Viewing a single comment thread. View all comments

geneing t1_ivnp4o0 wrote

Why are we wasting time on this? Searle made a few subtle mistakes and played a few tricks.

  1. He never defines what "understand" means. Without a clear definition, he can play rhetorical tricks to support his argument.
  2. Is it really possible to translate from English to Chinese by just following a book of rules? Have you seen "old" machine translations that were basically following rules - it was trivial to tell machine translation from human translation.
13

red75prime t1_ivoiy3c wrote

  1. Sure. Take an ML translator algorithm and the weights and do all matrix multiplications and other operations by hand.
1

Merastius t1_ivp2wc2 wrote

The part of the thought experiment that is deceptive is that, the simpler you make the 'rule following' sound, the more unbelievable (intuitively) it'll be that the room has any 'understanding', whatever that means to you. If instead you say that the person inside the room has to follow billions or trillions of instructions that are dynamic and depend on the state of the system and what happened before etc (i.e. modelling the physics going on in our brain by hand, or modelling something like our modern LLMs by hand), then it's not as intuitively obvious that understanding isn't happening.

3

mjmikulski t1_ivp4c5g wrote

> it was trivial to tell machine translation from human translation.

Isn't it anymore?

1

Nameless1995 t1_ivp5o2x wrote

> He never defines what "understand" means. Without a clear definition, he can play rhetorical tricks to support his argument.

Right.

> Is it really possible to translate from English to Chinese by just following a book of rules? Have you seen "old" machine translations that were basically following rules - it was trivial to tell machine translation from human translation.

Newer one's are still following rules. It's still logic gates and bit manipulation underneath.

1

PrivateFrank t1_ivp7x8b wrote

>Newer one's are still following rules. It's still logic gates and bit manipulation underneath.

Yeah, but at the same time the translation "logic" is being continuously refined through learning.

The book of rules is static in the old example.

1

Nameless1995 t1_ivp9j9b wrote

> Yeah, but at the same time the translation "logic" is being continuously refined through learning.

Yes by following more rules (rules of updating other rules).

> The book of rules is static in the old example.

True. The thought experiments make some idealization assumptions. Current programs need to "update" partly because they don't have access to the "true rules" from the beginning (that's why the older models didn't work as well either). But in CRA, the book, represents the true rules all laid bare. One issue is that in real life, the rules of translations itself is dynamic and can change with change in languages. To address that CRA can focus on a specific instance of time (time t) and consider it as given that the true rules are available for time t, and consider the question about knowledge of Chinese at time t. (But yes, there may not be even true "determinate" rules -- because of statistical variations of how individuals use language. Instead there can be a distribution of rules each aligning with real life usages to varying degrees of fit. The book can be then treated as a set of coherent rules that belongs to a dense area in that distribution at time t.)

1

PrivateFrank t1_ivpaaxh wrote

>Yes by following more rules (rules of updating other rules).

But those rules are about improving performance of the translation according to some other benchmark from outside of the rule system.

Unless one of the Chinese symbols sent into the room means "well done that last choice was good, do it again, maybe" and is understood to mean something like that, no useful learning or adaptation can happen.

2

Nameless1995 t1_ivpc688 wrote

Right. The Book in CRA is meant to represent the true rules (or at least something "good enough" for a human with billingual capabilities) at a given time so the "need" for updating rules from feedback is removed (feedback is needed in practical settings, because we are not in a thought-experiment which stipulates some oracle access). The point is that the practical need of contemporary ML models for refination (given lack of magical access to data generating processes), doesn't entail the 'in principle' impossibility to write down serviceable rules of translation for a specific time instance in a book.

1

PrivateFrank t1_ivpfnm2 wrote

Then this instant-time version of the CRA doesn't need understanding.

But you have to compare that to a human for the analogy to mean anything, and an instant-time human being is as empty as the CRA of understanding and intentionality.

1

Nameless1995 t1_ivpjf1l wrote

This is bit of a "semantics". As I also said in the other comment, it all boils down to semantics about how we decide to define "understanding" (a nebulous notion) in the first place (even intentionality is a very unclear concept -- the more I read papers on it the more confused I become because some many diverse ways people go around this concept).

If we define that understanding such that it by definition "updating" information and such (which I think is a bit weird of a definition in terms of standard usage of understanding in community. Would an omniscient being with the optimal policy setup in theory be treated as incapable of understanding?), then yes the vanilla CRA wouldn't understand but would not make any interesting claim about capabilities of programs.

Either way, Searle's point of using CRA was more for the sake of illustration to point towards something broader (need of some meaningful instantiation of programs to realize understanding properly). The point mostly stand for Searle for any formal programs (with update rules or not). In principle CRA can be modified correspondingly (think of the person following rules of state transitions --now the CRA can be allowed to have update rules from examples and feedback signals as well). But yes, then it may not be as intuitive to people if CRA would count as "not understanding" at that point. Searle was most likely trying to point towards "phenomenality". And how arbitrary instantiation of "understanding-programs" would not necessarily realize some phenomenal consciousness of understanding. But I don't think it's really necessarily to even have phenomenal consciousness for understanding (although again, that's partly a semantic disagreement about how the carve out "understanding").

2

PrivateFrank t1_ivplvi0 wrote

Hey I'm not an ML guy, just someone with an interest in philosophy of mind.

Intentionality and understanding and first-person (phenomonologistic) concepts, and I think that's enough to have the discussion. We know what it is like to understand something or have intentionality. Intentionality in particular is a word made up to capture a flavour of first-person experience of having thoughts which are about something.

I think that to have "understanding" absolutely requires phenomenal consciousness. Or the "understanding" in an AI has could be the same as how much a piece of paper understands the words written upon it. At the same time, none of the ink on that page is about anything - it just is. There's no intentionality there.

It's important to acknowledge the context at the time that there were quite a few psychologists, philosophers and computer scientists who really were suggesting that the human mind/brain was just passively transforming information like the man in the Chinese room. It's important to not let current ML theorists make the same mistake (IMO).

The difference between the CRA and what we can objectively observe about organic consciousness is informative about where the explanatory gaps are.

3

Nameless1995 t1_ivpyc47 wrote

> Intentionality and understanding and first-person (phenomonologistic) concepts, and I think that's enough to have the discussion. We know what it is like to understand something or have intentionality. Intentionality in particular is a word made up to capture a flavour of first-person experience of having thoughts which are about something.

> I think that to have "understanding" absolutely requires phenomenal consciousness. Or the "understanding" in an AI has could be the same as how much a piece of paper understands the words written upon it. At the same time, none of the ink on that page is about anything - it just is. There's no intentionality there.

That's exactly Searle's bigger point with CRA. CRA is basically in the same line as Chinese Nation and Dneprov's game, paper simulations, and such arguments (not particularly novel). The larger point of Searle is that although machines may understand (according to Searle, we are such machines), programs don't understand by simply the characteristics of the programs. That is particular instantiation of programs (and we may be such instantiation) may understand depending on the underlying causal powers and connections instantiating the program but not any arbitrary instantiation may understanding (for example "write symbols in paper" or "arrange stone" instantiations).

The stress is exactly that in any arbitrary instantiation may not have relevant "intentionality" (the point becomes muddied because of the focus on the simplicity of the CRA.)

However, a couple of points:

(1) First I think at the beginning of disagreements, it's important to set up the semantics. Personally, I think a distributionalist attitude towards language. First, I am relatively externalist about meaning of words (meanings of words are not decided by solo people in the mind, but it's grounded in how the word is used in public). Second, like Quine, I don't think most words have clean determinate "meaning" and especially more so for words like "understanding". This is because there are divergences among how different individuals use words, and there may be no consistent unfying "necessary and sufficient" rules explaining the use of words (because some usages by different people may contradict each other). Such issues don't express that much in day to day discourse, but becomes more touchy when it comes to philosophy. So what do we do? I think the answer is something like "conceptual engineering". We can try to make suggestions about refinements and further specifications of nebulous concepts like "understanding" when it's necessary for some debate. Then it can be upto the community as a whole to accept and normalize those usages in necessary contexts or counter-suggests alternatives.

(2) With the background set up in (1), I believe "understanding"'s meaning is indeterminate. Now from a conceptual engineering perspective, I feel like we can go multiple different mutually exclusive branches in trying to refine the concept of "understanding". However, we need to have some constraints as well (we need to ideally keep the word's usage still roughly similar to how it is).

(3) One way to make the notion of "understanding" more determinate, is to simply stipulate the need of "intentionality" and "phenomenality" for understanding.

(4) I think making intentionality essential for understanding is fair (I will come back to this point), but not sure if "phenomenality" is as essential for understanding (not needed for the word to roughly correspond to how we use it).

(4.1) First note that it is widely accepted in Phil. of Mind, that intentionality doesn't require phenomenality. There is also a debate on whether intentionality ever has a phenomenal component at all. For example, Tye may argue in a phenomenal field, there is just sensations, sound, images, imageries etc. no "thoughts" representing things as about something. When thinking about "1+1=2", Tye may say the thought itself is not phenomenally present, instead you would have just some phenomenal sensory experiences associated with the thought (perhaps your a vague audio experience of 1+1=2, perhaps some visual experiences of 1,+1,=2 symbols in imaginations and so on). Functionalists can have a fully externalist account of intentionality. They may say say that some representation in mental state being "about" some object in the world is simply a matter of having the relevant causal connection (which can be acheived by any arbitrary instantiations of a program with proper interface to the io signals - "embedding it in the world") or relevant evolutionary history (eg. teleosemantics) behind the reasons for the selection of the representation-producer-cosumer mechanism. This leads to the "so-called" causal theories of intentionality. They would probably reject intentionality as being some "internal" flavor of 1st person experience, rather than it being grounded in the embodiment of the agent.

(4.2) Even noting Anscombe's work on intentional theories of perception that was kind of one of the starting points on intentional theories -- she was pretty neutral on the metaphysics of intentionality and she was trying to take a very anti-reificationist stance, even close to treating it more of a linguistic device. She also distinguished intentional content from being mental objects. For example, if someone's worshiping has the intentional object -Zeus (they worship Zeus), then it's wrong to say that the intentional content is "the idea of zeus" (because it's wrong to say that the subject is simply worshipping the idea of Zeus, because the subject's 'intention' is to worship the true Zeus which happens to not exist - but that's irrelevant). This raises the question - what kind of metaphysical states can even constitute or ground this "subject taking the intentional content of her worship as being Zeus" (the intentional content is not exactly the idea of Zeus, or Zeus-imageries that the subject may experience -- but then what does this "intentionality" correspond to?). After thinking I couldn't really think of any sensible answer besides again going back to functionalism: the intentional object of the subject is zeus because that's the best way to describe their behaviors and functional dispositions.

(4.3) That's not the only perspectives of intentionality of course. There are for example, works by Brentaro and approaches of intentionality from core phenomenological perspectives. But it's not entirely clear if there is some sort of differentiable "phenomenal intentionality" in phenomenality. I don't know if I distinctly experience "aboutness" rather than simply having a tendencing to use "object-oriented language" in my thinking (which itself isn't distinctly or obvious phenomenal in nature). Moreover, while me understanding certain concepts may "feel" phenomenally to be a certain way, it feels to be a very poor account of understanding. Understanding is not a "feeling" nor it is clear why having the "phenomenal sense" of "understanding xyz" is necessary. Instead upon reflection of me for example having the "understanding of arithmetic" constitutes, I don't find it to be necessarily associated with my qualitatively feeling a certain way with dispositions to say or think about numbers, +, - (they happen, and the "qualitative feeling" may serve as a "mark" signifying understanding -- "signifying" in a functional correlational sense), but it seems to be most meaningfully constituted by possession of "skills" (ability to think arithmetics, solve arithmetical problems etc.). This again leads to functionalism. If I try to think beyond that, I find nothing determinate in 1st person experience constituting "understanding".

2

Nameless1995 t1_ivpyckd wrote

(5) Given that I cannot really get a positive conception of understanding beyond possessing relevant skills, I don't have a problem in broadening the notion of understanding to abstract out phenomenality (which may play a contingent (not necessary) role in realizing understanding through "phenomenal powers"). So on that note, I have no problem with allowing understanding (or at least a little bit of it) to a "system of paper + dumb-rule-follower + rules as a whole" producing empty symbols in response to other empty symbols in a systematic manner such that the it is possible to map the input and output in a manner making it possible to interpret it as a function to do arithmatic. You may say that these symbols are "meaningless" and empty. However, I don't think "meaning" even exists beyond just functional interactions. "meaning" as we use the term simply serves as another symbol to simplify communication of complicated phenomena wherein the communication itself is just causal back-and-forth. Even qualia to me are just "empty symbols" gaining meaning not intrinsically but from their functional properties in grounding reactions to changes in world states.

(6) Note I said "little bit of it" regarding understanding of arithmetic the system "paper + dumb-rule-follower+ some book of rules" as whole. This is because I am taking understanding as a matter of degree. The degree increases with increase in relevant skills (for example, if the system can talk (or having functional characteristics mappable to) about advanced number theory, talk about complex philosophical topics about metaphysics of numbers, then I would count that as "deeper understanding of arithmatic")

(7) 5/6 can be counter-intuitive. But the challenge here is to find an interesting positive feature of understanding that the system lacks. We can probably decide on some functional characteristics or some need of low-level instantiation details (beyond high level simulation of computational formal relations) if we want (I personally don't care either way) to restict paper+dumb-rule-followers to simulate super-intelligence. But phenomenality doesn't seem too interesting to me, and even intentionality is a bit nebulous (and controversial; I also relate to intentional-talk being simply a stance that we take to talk about experiences and thoughts rather than taking it as something metaphysically intrinsic in phenomenal feels (https://ase.tufts.edu/cogstud/dennett/papers/intentionalsystems.pdf)). Some weaker notion of intentionality can definitely be allowed already in any system behavior (including paper-based TM simulation as long as that is connected to a world for input-output signals). Part of the counter-intuitive force may come from the fact is that our usage of words and "sense" of a word x applies in context y, can be a bit rooted in internal statistical models (the feeling of "intuition" that it doesn't feel right to say the system "understands" is the feeling of ill-fittingnes due to our internal statistical models). However, if I am correct that the words have no determinate meaning -- it may be too loose or even have contraidcting usages, in our effort to clean them up through conceptual engineering it may be inevitable that some intuition needs to be sacrificed (because our intuitions themselves can be internally inconsistent). Personally, considering both ways, I am happier to bite the bullet here and allow papers+dumb-writers to understanding things as whole when the individual parts don't: simply following from my minimalist definition of understanding revolving around highl level demonstrable skills. I feel like more idealized notion are hard to define and get into mysterian territories while also unnecessarily complicating the word.

2

waffles2go2 t1_ivqahzo wrote

Excellent and thoughtful :)

CRA pushes our thinking as a useful tool.

Semantics can get refined as we push our ideas - we are just starting to understand.

1

waffles2go2 t1_ivqb7pe wrote

Searle is pushing the ball forward. I love this topic but most in this thread don't seem to appreciate that this thinking is evolving, understands its weaknesses, and is trying to address them.

God and the "chemicals is math so the brain is math" logic is so "community college with a compass point on it"....

1

geneing t1_ivplj76 wrote

The new machine learning based translators don't really have a set of rules. They essentially learn probabilities of different word combinations. (e.g. https://ai.googleblog.com/2019/10/exploring-massively-multilingual.html), which we could argue should count as "understanding" (since Searle didn't define it clearly).

0

Nameless1995 t1_ivpnf6f wrote

> They essentially learn probabilities of different word combinations.

These isn't dichotomous with having a set of rules. The rules operate at a deeper (less interpretable level -- some may say "subsymbolic") compared to GOFAI. The whole setup of model+gradient descent correspond to having some update rules (based on partial differentations and such). In practice they aren't fully continuous either (though in theory they are) because of floating point approximations and underlying digitization.

2

waffles2go2 t1_ivq94uo wrote

We're "wasting our time" because you clearly don't understand the theory. We are trying to prove/disprove and what you offer is somewhat slow thinking...

1 - Captain obvious says "everyone knows this" - it seems that you somehow believe this is "done" and Searle (and others) are trying to push it forward and iterate.

Why do you hate the scientific process or did you invent something better?

2 - Derp - your own opinion, not something Searle offers, nor is it relevant if you understand to the problem (it is given that you cannot tell the difference between someone who understands Chinese and the output of the "searcher").

Overall, great evidence as to why we need more discussion on this topic.

Great job!

1

Flag_Red t1_iw1hqzr wrote

This comment is unnecessarily hostile.

1