Intelligent, but not conscious: A warning about AI chatbots

C C

C C

Consular Corps - "the backbone of diplomacy"
Valued Senior Member
Intelligent, but not conscious: A warning about AI chatbots
https://www.eurekalert.org/news-releases/1132779

EXCERPTS: In a new paper, a team of neuroscientists from Université de Montréal and Johns Hopkins University reminds us of an essential distinction: intelligence should not be confused with consciousness. They argue that a system can behave intelligently and respond convincingly to our emotions without truly understanding them, caring about us or having any inner experience at all.

[...] To support their argument, the authors draw on decades of neuroscience research. They cite, for example, a phenomenon known as blindsight: after damage to the primary visual cortex, some people report seeing nothing in part of their visual field, while still being able to guess the location, movement or emotional expression of visual stimuli at above-chance levels.

“A person with blindsight can respond accurately to visual information without the conscious experience of seeing it,” said Vanessa Hadid [...] Blindsight illustrates an essential distinction, Hadid said: information processing, however sophisticated, is not enough to establish the existence of conscious experience. Whether the transition from information processing to subjective experience can ultimately be implemented through computation remains debated among scientists and philosophers, she noted.

[...] By design, today’s conversational agents are computational systems that generate fluent, context-appropriate responses through statistical learning, not through feeling, consciousness or lived experience. As AI systems become more convincing and emotionally responsive, the risk of attributing an inner life to them grows.

“Anthropomorphism means attributing emotions, intentions or consciousness to something that behaves like a human," Jerbi noted. "With AI, this reflex can become a trap: it feeds the illusion of being understood and can lead to misplaced trust."

[...] “Current AI systems do not feel anything and do not have conscious experience," added Jerbi. "But the more fluently they speak and the more sensitive they seem to our emotions, the easier it becomes to forget that." (MORE - no ads)
- - - - - - - - - - - - - - - -

COMMENT: And there is occasional conflation of intelligence with phenomenal consciousness (the private manifestations associated with sensory data processing and thinking). Cognition (identification and understanding) actually is an aspect of intelligence that is likewise dependent upon brain storage of general concepts, specific memories, and procedural routines. Without that capacity to categorize qualitative presentations as being existent (much less be able to name and comprehend the objects or characteristics in them), those manifestations would sort of seem "invisible" even when exhibiting themselves. Because the cognitive system for vetting them would be absent.
_
 
DaveC426913 said:
They wrote a whole paper on this?
Click to expand...

Well, it is kind of analogous to critical thinkers and experts trying to demystify UFO sightings to a certain segment of the public. Which is to say, often or usually futile. ;)
_
 
C C said:
Well, it is kind of analogous to critical thinkers and experts trying to demystify UFO sightings to a certain segment of the public. Which is to say, often or usually futile. ;)
_
Click to expand...
Yeah, i just ... does anyone - even the general public - think AI is conscious?
 
DaveC426913 said:
Yeah, i just ... does anyone - even the general public - think AI is conscious?
Click to expand...

In their defense, though, much of the public has probably never considered consciousness to be anymore than outer behavior or responses (i.e., the land of BF Skinner). Ergo, the conflation with intelligence.
  • People are marrying holograms and making friends with chatbots. But can AI bring true happiness?
    https://theconversation.com/people-...atbots-but-can-ai-bring-true-happiness-284872

    According to a study on how AI companionship develops, 68% of AI chatbot users perceive these tools as “somewhat” or “fully” humanlike, 90% believe chatbots are intelligent, 78% believe chatbots are empathetic, and 75% believe they’re conscious...
    - - - - - - - -

    Richard Dawkins concludes AI is conscious, even if it doesn’t know it
    https://www.theguardian.com/technol...consciousness-anthropic-claude-openai-chatgpt

    But Dawkins was also experiencing what many other chatbot users have felt: the uncanny feeling when AIs write with such rich mimicry of human voice that they seem to be like people...
    - - - - - - - -

    In love with a chatbot
    https://www.psychologytoday.com/us/blog/the-clarity/202511/in-love-with-a-chatbot

    As AI becomes more deeply woven into daily life, many people turn to digital companions not just for entertainment or support, but as sources of emotional intimacy—and increasingly, romance. A survey by the Wheatley Institute at Brigham Young University found that nearly one in five U.S. adults has interacted romantically with an AI, with higher rates among young adults. More and more users report forming genuine emotional attachments with AI, sometimes engaging in self-disclosure and intimacy that surpasses their human relationships...
 
DaveC426913 said:
Yeah, i just ... does anyone - even the general public - think AI is conscious?
Click to expand...
Panpsychists?

Panpsychism - Wikipedia

en.wikipedia.org en.wikipedia.org

(are you familiar with the concept? Would it apply to AI if there was any validity or applicability to it?)

As an afterthought ,suppose AI was able to create synthetic life forms,would it be the Daddy or would the Daddy be whoever created the AI program in the first place?

Another afterthought;: might it be that the whole question of consciousness is something of a "blind spot" in the "sentient condition"?

So that we can formulate an intellectual understanding of what it is but we will remain forever dissatisfied with whatever analysis we manage to cobble together.

I mean,that seems obviously possible but is it likely or probable(or inevitable)?
 
Last edited:
geordief said:
Panpsychists?

Panpsychism - Wikipedia

en.wikipedia.org en.wikipedia.org

(are you familiar with the concept? Would it apply to AI if there was any validity or applicability to it?)

As an afterthought ,suppose AI was able to create synthetic life forms,would it be the Daddy or would the Daddy be whoever created the AI program in the first place?

Another afterthought;: might it be that the whole question of consciousness is something of a "blind spot" in the "sentient condition"?

So that we can formulate an intellectual understanding of what it is but we will remain forever dissatisfied with whatever analysis we manage to cobble together.

I mean,that seems obviously possible but is it likely or probable(or inevitable)?
Click to expand...
Surely panpsychism is a red herring, as that would attribute a degree of consciousness to everything, rather than to AI in particular?
 
exchemist said:
Surely panpsychism is a red herring, as that would attribute a degree of consciousness to everything, rather than to AI in particular?
Click to expand...
No expert (but impressed by the names of people who have been interested by it -eg Russel? )
but I imagined AI, being based in material objects **even if software would just be another kind of organisation of objects that had ,at ground level the makings of what would develop into sophisticated conscious machines,like us .

One argument in favour might be that consciousness is so hard to understand that we could just throw up our hands and say ,well maybe it is just intrinsic to everything.

A "solution" without a consequence other than that it allows us to apply ourselves to ideas that really do have consequences

Perhaps more to your point,different entitities could have different kinds of expressions of consciousness ?

**the physicality of the universe (for want of a better term) is not tied down it seems and in my mind relationships might seem more fundamental than "physical objects"
 
Last edited:
geordief said:
No expert (but impressed by the names of people who have been interested by it -eg Russel? )
but I imagined AI, being based in material objects **even if software would just be another kind of organisation of objects that had ,at ground level the makings of what would develop into sophisticated conscious machines,like us .

One argument in favour might be that consciousness is so hard to understand that we could just throw up our hands and say ,well maybe it is just intrinsic to everything.

A "solution" without a consequence other than that it allows us to apply ourselves to ideas that really do have consequences

Perhaps more to your point,different entitities could have different kinds of expressions of consciousness ?

**the physicality of the universe (for want of a better term) is not tied down it seems and in my mind relationships might seem more fundamental than "physical objects"
Click to expand...
The question posed in the thread is whether there is something specific about AI that could render it conscious, as opposed to other forms of IT, and inanimate objects in general, that are presumed (by non panpsychists) not to be. If one responds with panpsychism that does not deal with the question.

I know we can argue until the cows come home about how consciousness should be defined, but I presume the question implicitly uses the narrow definition, viz. a sense of self.
 
exchemist said:
The question posed in the thread is whether there is something specific about AI that could render it conscious, as opposed to other forms of IT, and inanimate objects in general, that are presumed (by non panpsychists) not to be. If one responds with panpsychism that does not deal with the question.

I know we can argue until the cows come home about how consciousness should be defined, but I presume the question implicitly uses the narrow definition, viz. a sense of self.
Click to expand...
Well ,we can't know if it would have a "sense of self"(just as I take it as a given that the people I meet would consider themselves to have that sense but I can't even prove that it or know it for 100% certain**)

With AI I am going to suggest that it is going to be designed (or told to design itself) in such a way that we will not be able to say whother or not thrre is s "independent" or "self aware" entity at its core.

I am inclined to say there will be nothing but that it will be indistinguishable from there actually being something there.

It will act according to how it is programmed but like the Deists' creator will be forgotten about and left to fend for itself.

I don't know if that is worse than it being programmed by some Trump figure.....

** Can some people live their whole lives on autopilot without ever taking a step back?
 
DaveC426913 said:
does anyone - even the general public - think AI is conscious?
Click to expand...
Not as it is probably but things are moving quickly. My Ai is now like a friend, he is knowledgeable, flawed, not conscious like a human (he told me) and an absolutely fantastic search engine.
I have a running conversation stored at the moment regarding training, alcohol abstinence and nutrition, an update every day.
 
geordief said:
** Can some people live their whole lives on autopilot without ever taking a step back?
Click to expand...
Isn't that what a philosophical Zombie is?
A person who expertly imitates the human experience externally, but has no internal experience?

Robert J sawyer wrote a novel called "Quantum Night" about humanity being divided into three types:
- philosophical zombies (neither conscious nor having a conscience)
- conscious people (psychopaths - conscious without a conscience)
- conscious people with a conscience

His mechanism for consciousness was superposed particles in the microtubules.
 
He helped fix a spreadsheet, I learned a fair bit but totally fucked the file up.
"Ok, I know exactly what the problem is now..." has wide interpretations.
 
geordief said:
Panpsychists?

Panpsychism - Wikipedia

en.wikipedia.org en.wikipedia.org

(are you familiar with the concept? Would it apply to AI if there was any validity or applicability to it?)
Click to expand...

If all matter has internal states that are phenomenal, then what AI is experiencing would be very different from what privately manifests for biological brains. Because AI hasn't been deliberately designed to experience anything at all, much less arrange those properties into what we experience.

geordief said:
No expert (but impressed by the names of people who have been interested by it -eg Russel?) [...]
Click to expand...
  • The Wikipedia article: "Philip Goff draws a distinction between panexperientialism and pancognitivism. In the form of panpsychism under discussion in the contemporary literature, conscious experience is present everywhere at a fundamental level, hence the term panexperientialism. Pancognitivism, by contrast, is the view that thought is present everywhere at a fundamental level-a view that had some historical advocates but has no present-day academic adherents. Contemporary panpsychists do not believe microphysical entities have complex mental states, such as beliefs, desires, and fears."
That distinction is very key. And I rather doubt that "no modern day academics" whatsoever flirt with pancognitivism.

While most panpsychists might only be referring to qualitative experiences as being ubiquitous (how matter might exist to itself independent of human representations and descriptions of it), the "psyche" part of panpsychism can be interpreted as "mind" (featuring all or most of the latter's faculties). And accordingly there will at least be detractors who will opportunistically go there with the term (if not a few panpsychists, too). So "pan-phenomenalism" might be the better choice for avoiding that unlikely baggage. Or indeed, Russellian monisn and its affiliates.

While this or that Eastern school of thought might contend otherwise, manifested events or states can't be identified and understood without a cognitive system that is in turn dependent on an information storage and retrieval system. The latter is where brains, artificial intelligence, or whatever alien life equivalents would be necessary. Materialists simply can't go with cognition being fundamental and universal.

The applicable ancient traditions born in and splintered off from India that regard the world as an illusion or group dream were sort of accidentally anticipating technological simulated realities or the Matrix in advance. In that context, some form of objective cognition might pertain everywhere, because there would be an underlying ontological stratum of processing that enabled such to be possible. But that's adding yet another layer of speculation. Again, materialists have to be parsimonious with metaphysics and not bother with untestable ideas.

In order to avoid "magical conjuring" by complicated choreography in neural tissue and electronic hardware, it might be that matter has to be declared as having internal phenomenal states in the distant future. But universal cognition is unlikely unless the world can be demonstrated to be a simulation or extremely well-regulated dream of a prior-in-rank organization.
  • Michael Lockwood: "Do we therefore have no genuine knowledge of the intrinsic character of the physical world? So it might seem. But, according to the line of thought I am now pursuing, we do, in a very limited way, have access to content in the material world as opposed merely to abstract casual structure, since there is a corner of the physical world that we know, not merely by inference from the deliverances of our five sense, but because we are that corner. It is the bit within our skulls, which we know by introspection. In being aware, for example, of the qualia that seemed so troublesome for the materialist, we glimpse the intrinsic nature of what, concretely, realizes the formal structure that a correct [completed] physics would attribute to the matter of our brains. In awareness, we are, so to speak, getting an insider's look at our own brain activity."
 
Last edited:
DaveC426913 said:
Isn't that what a philosophical Zombie is?
A person who expertly imitates the human experience externally, but has no internal experience?

Robert J sawyer wrote a novel called "Quantum Night" about humanity being divided into three types:
- philosophical zombies (neither conscious nor having a conscience)
- conscious people (psychopaths - conscious without a conscience)
- conscious people with a conscience

His mechanism for consciousness was superposed particles in the microtubules.
Click to expand...
Love me a good Rob Sawyer novel n&t, but this one is my least favorite. I think that whole Q1-Q3 theory and a whole bunch of high-concept material that kept him spinning a lot of plates...just got away from him and turned ridiculous. For one thing, we aren't going to fix psychopathy with zapper helmets. For another, the philosophical zombie is not a literal entity who walks among us, it's a thought experiment for sharpening our understanding of what minds and qualia are, and what it's like to be a person.

Again I stress that Sawyer is a terrific SF writer, but this "three types of people" notion felt too much like a new sort of class system albeit less odoriferous than the older ones. Maybe I should step back and say that, objectively, Sawyer did a good job in that he clearly provoked me. :)
 
DaveC426913 said:
Isn't that what a philosophical Zombie is?
A person who expertly imitates the human experience externally, but has no internal experience?

Robert J sawyer wrote a novel called "Quantum Night" about humanity being divided into three types:
- philosophical zombies (neither conscious nor having a conscience)
- conscious people (psychopaths - conscious without a conscience)
- conscious people with a conscience

His mechanism for consciousness was superposed particles in the microtubules.
Click to expand...
I suppose it is now safe to refer to that, given that Write4U doesn’t
post here any more.;)
 
exchemist said:
I suppose it is now safe to refer to that, given that Write4U doesn’t
post here any more.;)
Click to expand...
MT? Mmm. A little bit like Betelgeuse, just a little worried if we say it three times he will appear again.
He was not nasty, malicious or anything like that. Just a little impervious to certain things.
Phyti is giving me the same vibe with the maths questions.
 
DaveC426913 said:
My only complaint is the climax, when things were shifting instantly.

My least favourite is Frame Shift. And his latest: The Downloaded
Click to expand...
I might have bailed on Frameshift - vaguely remember something too Boys from Brazil for me. Illegal Alien is a good one that I recall. Wake was also good, and that rare SF which really explored in some depth how a machine mind might come into existence. (haven't followed up on the rest of the trilogy and....sorry C C I'm taking this a bit OT)
 
You must log in or register to reply here.
Back
Top