Coherence arguments imply a force for goal-directed behavior

By Katja Grace, 25 March 2021

[Epistemic status: my current view, but I haven’t read all the stuff on this topic even in the LessWrong community, let alone more broadly.]

There’s a line of thought that claims that superior AI will are typically ‘goal-directed’—that’s, constantly doing no matter makes sure favored outcomes extra doubtless—and that that is to do with the ‘coherence arguments’. Rohin Shah, and possibly others, have argued in opposition to this. I need to argue in opposition to them.

The previous argument for coherence implying (worrisome) goal-directedness

I’d reconstruct the unique argument that Rohin is arguing in opposition to as one thing like this (making no declare about my very own beliefs right here):

1. ‘No matter belongings you care about, you might be finest off assigning constant numerical values to them and maximizing the anticipated sum of these values’
   ‘Coherence arguments

And because the level of all that is to argue that superior AI is perhaps arduous to take care of, be aware that we are able to get to that conclusion with:

5. ‘Extremely smart goal-directed brokers are harmful’
   If AI programs exist that very competently pursue objectives, they’ll doubtless be higher than us at attaining their objectives, and due to this fact to the extent there’s a threat of mismatch between their objectives and ours, we face a critical threat.

Rohin’s counterargument

Rohin’s counterargument begins with an commentary made by others earlier than: any conduct is in keeping with maximizing anticipated utility, given some utility operate. As an illustration, a creature simply twitching round on the bottom could have the utility operate that returns 1 if the agent does no matter it the truth is does in every state of affairs (the place ‘state of affairs’ means, ‘total historical past of the world to this point’), and 0 in any other case. It is a creature that simply needs to make the best twitch in every detailed, history-indexed state of affairs, with no regard for additional penalties. Alternately the twitching agent would possibly care about outcomes, however simply occur to need the actual holistic unfolding of the universe that’s occurring, together with this specific sequence of twitches. Or it may very well be detached between all outcomes.

The essential level is that rationality doesn’t say what ‘issues’ you may need. And specifically, it doesn’t say that it’s a must to care about specific atomic items that bigger conditions may be damaged down into. If I attempt to name you out for first spending cash to get to Paris, then spending cash to get again from Paris, there may be nothing to say you may’t simply have needed to go to Paris for a bit after which to return house. In reality, it is a frequent human state of affairs. ‘Aha, I cash pumped you!’ says the airline, however you aren’t nervous. The twitching agent would possibly at all times be like this—a creature of extra refined tastes, who cares about complete delicate histories and relationships, slightly than simply summing up modular momentarily-defined successes. And given this freedom, any conduct would possibly conceivably be what a creature needs. 

Then I might put the complete argument, as I perceive it, like this:

1. Any observable sequence of conduct is in keeping with the entity doing EU maximization (see commentary above)
2. Doing EU maximization doesn’t suggest something about what conduct we’d observe (from 1)
3. Particularly, figuring out {that a} creature is an EU maximizer doesn’t suggest that it’s going to behave in a ‘goal-directed’ approach, assuming that that idea doesn’t apply to all conduct. (from 2)

Is that this just a few disagreement in regards to the that means of the phrase ‘goal-directed’? No, as a result of we are able to get again to a significant distinction in bodily expectations by including:

4.  Not all conduct in a creature implicates dire threat to humanity, so any idea of goal-directedness that’s in keeping with any conduct—and so is perhaps implied by the coherence arguments—can’t suggest AI threat.

So the place the unique argument says that the coherence arguments plus another assumptions suggest hazard from AI, this counterargument says that they don’t. 

(There may be additionally not less than some selection within the that means of ‘goal-directed’. I’ll use goal-directed_Rohin to discuss with what I believe is Rohin’s most popular utilization: roughly, that which appears intuitively objective directed to us, e.g. behaving equally throughout conditions, and accruing assets, and never flopping round in attainable pursuit of some precise historical past of non-public floppage, or peaceably preferring to at all times take the choice labeled ‘A’.)

My counter-counterarguments

What’s incorrect with Rohin’s counterargument? It sounded tight. 

In short, I see two issues:

1. The entire argument is when it comes to logical implication. However what appears to matter is modifications in chance. Coherence doesn’t must rule out any conduct to matter, it simply has to vary the possibilities of behaviors. Understood when it comes to chance, argument 2 is a false inference: simply because any sequence of conduct is in keeping with EU maximization doesn’t imply that EU maximization says nothing about what conduct we’ll see, probabilistically. All it says is that the chance of a behavioral sequence is rarely lowered to zero by issues of coherence alone, which is hardly saying something.

You would possibly then assume {that a} probabilistic model nonetheless applies: since each entity seems to be in good standing with the coherence arguments, the arguments don’t exert any drive, probabilistically, on what entities we’d see. However:

2. An out of doors observer having the ability to rationalize a sequence of noticed conduct as coherent doesn’t imply that the conduct is definitely coherent. Coherence arguments constrain combos of exterior conduct and inner options—‘preferences’ and beliefs. So whether or not an actor is coherent is determined by what preferences and beliefs it really has. And if it isn’t coherent in gentle of those, then coherence pressures will apply, whether or not or not its conduct seems coherent. And in lots of circumstances, revision of preferences as a result of coherence pressures will find yourself affecting exterior conduct. So 2) isn’t solely not a sound inference from 1), however really a incorrect conclusion: if a system strikes towards EU maximization, that does suggest issues in regards to the conduct that we are going to observe (probabilistically). 

Maybe Rohin solely meant to argue about whether or not it’s logically attainable to be coherent and never goal-directed-seeming, for the aim of arguing that humanity can assemble creatures in that perhaps-unlikely-in-nature nook of mindspace, if we strive arduous. Through which case, I agree that it’s logically attainable. However I believe his argument is commonly taken to be related extra broadly, to questions of whether or not superior AI will are typically goal-directed, or to be goal-directed in locations the place they weren’t meant to be.

I take 1) to be pretty clear. I’ll lay out 2) in additional element.

My counter-counterarguments in additional element

How would possibly coherence arguments have an effect on creatures?

Allow us to step again.

How would coherence arguments have an effect on an AI system—or anybody—anyway? They’re not going to fly in from the platonic realm and reshape irrational creatures.

The primary routes, as I see it, are by way of implying:

1. incentives for the agent itself to reform incoherent preferences
2. incentives for the processes giving rise to the agent (specific design, or choice procedures directed at success) to make them extra coherent
3. some benefit for coherent brokers in competitors with incoherent brokers

To be clear, the agent, the makers, or the world are usually not essentially fascinated by the arguments right here—the arguments correspond to incentives on the planet, which these events are responding to. So I’ll typically discuss ‘incentives for coherence’ or ‘forces for coherence’ slightly than ‘coherence arguments’.

I’ll speak extra about 1 for simplicity, anticipating 2 and three to be comparable, although I haven’t thought them via.

Trying coherent isn’t sufficient: if you happen to aren’t coherent inside, coherence forces apply

If self-adjustment is the mechanism for the coherence, this doesn’t rely upon what a sequence of actions seems like from the surface, however from what it seems like from the within.

Think about the aforementioned creature simply twitching sporadically on the bottom. Let’s name it Alex.

As famous earlier, there’s a utility operate beneath which Alex is maximizing anticipated utility: the one which assigns utility 1 to nonetheless Alex the truth is acts in each particular historical past, and utility 0 to the rest.

However from the within, this creature you excuse as ‘perhaps simply wanting that sequence of twitches’ has—allow us to suppose—precise preferences and beliefs. And if its preferences don’t the truth is prioritize this elaborate sequence of twitching in an unconflicted approach, and it has the self-awareness and means to make corrections, then it would make corrections. And having completed so, its conduct will change. 

Thus excusable-as-coherent Alex continues to be moved by coherence arguments, even whereas the arguments haven’t any complaints about its conduct per se.

For a extra life like instance: suppose Assistant-Bot is noticed making this sequence of actions: 

• Provides to purchase gymnasium membership for $5/week 
• Consents to improve to gym-pro membership for $7/week, which is like gymnasium membership however with added morning courses
• Takes discounted ‘off-time’ deal, saving $1 per week for less than utilizing gymnasium in evenings

That is in keeping with coherence: Assistant-Bot would possibly desire that precise sequence of actions over all others, or would possibly desire incurring gymnasium prices with a bigger sum of prime components, or would possibly desire speaking to Fitness center-sales-bot over ending the dialog, or desire agreeing to issues.

However suppose that the truth is, when it comes to the construction of the inner motivations producing this conduct, Assistant-Bot simply prefers you to have a gymnasium membership, and prefers you to have a greater membership, and prefers you to have cash, however is treating these preferences with inconsistent ranges of power within the completely different comparisons. Then there seems to be a coherence-related drive for Assistant-Bot to vary. A technique that that would look is that since Assistant-Bot’s general behavioral coverage presently entails freely giving cash for nothing, and likewise Assistant-Bot prefers cash over nothing, that desire offers Assistant-Bot cause to change its present general coverage, to avert the continued trade of cash for nothing. And if its behavioral coverage is arising from one thing like preferences, then the pure strategy to alter it’s by way of altering these preferences, and specifically, altering them within the route of coherence.

One concern with this line of thought is that it’s not apparent in what sense there may be something inside a creature that corresponds to ‘preferences’. Typically when folks posit preferences, the preferences are outlined when it comes to conduct. Does it make sense to debate completely different attainable ‘inner’ preferences, distinct from conduct? I discover it useful to contemplate the conduct and ‘preferences’ of teams:

Suppose two automobiles are parked in driveways, every containing a pair. One couple are simply having fun with hanging out within the automobile. The opposite couple are coping with a battle: one needs to climb a mountain collectively, and the opposite needs to swim within the sea collectively, they usually aren’t shifting as a result of neither is keen to let the outing proceed as the opposite needs. ‘Behaviorally’, each automobiles are the identical: stopped. However their inner components (the companions) are importantly completely different. And in the long term, we count on completely different conduct: the automobile with the unconflicted couple will most likely keep the place it’s, and the conflicted automobile will (hopefully) finally resolve the battle and drive off.

I believe right here it is smart to speak about inner components, separate from conduct, and actual. And equally within the single agent case: there are bodily mechanisms producing the conduct, which may have completely different traits, and which specifically may be ‘in battle’—in a approach that motivates change—or not. I believe it is usually value observing that people discover their preferences ‘in battle’ and attempt to resolve them, which is means that they not less than are higher understood when it comes to each conduct and underlying preferences which can be separate from it. 

So we now have: even if you happen to can excuse any seizuring as in keeping with coherence, coherence incentives nonetheless exert a drive on creatures which can be the truth is incoherent, given their actual inner state (or could be incoherent if created). At the very least in the event that they or their creator have equipment for noticing their incoherence, caring about it, and making modifications.

Or put one other approach, coherence doesn’t exclude overt behaviors alone, however does exclude combos of preferences, and preferences beget behaviors. This modifications how particular creatures behave, even when it doesn’t fully rule out any conduct ever being right for some creature, someplace. 

That’s, the coherence theorems could change what conduct is doubtless to look amongst creatures with preferences. 

Reform for coherence most likely makes a factor extra goal-directed_Rohin

Okay, however shifting towards coherence would possibly sound completely innocuous, since, per Rohin’s argument, coherence contains all types of issues, comparable to completely any sequence of conduct. 

However the related query is once more whether or not a coherence-increasing reform course of is more likely to lead to some sorts of conduct over others, probabilistically.

That is partly a sensible query—what sort of reform course of is it? The place a creature finally ends up relies upon not simply on what it incoherently ‘prefers’, however on what sorts of issues its so-called ‘preferences’ are in any respect, and what mechanisms detect issues, and the way issues are resolved.

My guess is that there are additionally issues we are able to say typically. It’s is just too huge a subject to analyze correctly right here, however some initially believable hypotheses about a variety of coherence-reform processes:

1. Coherence-reformed entities will have a tendency to finish up trying much like their start line however much less conflicted
   As an illustration, if a creature begins out being detached to purchasing crimson balls once they value between ten and fifteen blue balls, it’s extra more likely to find yourself treating crimson balls as precisely 12x the worth of blue balls than it’s to finish up very a lot wanting the sequence the place it takes the blue ball possibility, then the crimson ball possibility, then blue, crimson, crimson, blue, crimson. Or wanting crimson squares. Or eager to experience a dolphin.

   (I agree that if a creature begins out valuing Tuesday-red balls at fifteen blue balls and but all different crimson balls at ten blue balls, then it faces no apparent stress from inside to change into ‘coherent’, since it isn’t incoherent.)

2. Extra coherent methods are systematically much less wasteful, and waste inhibits goal-direction_Rohin, which suggests extra coherent methods are extra forcefully goal-directed_Rohin on common
   Typically, if you’re typically a drive for A and typically a drive in opposition to A, then you aren’t shifting the world with respect to A as forcefully as you’ll be if you happen to picked one or the opposite. Two folks intermittently altering who’s within the driving seat, who need to go to completely different locations, won’t cowl distance in any route as successfully as both of them. An organization that cycles via three CEOs with completely different evaluations of all the pieces will—even when they don’t actively scheme to thwart each other—are inclined to waste quite a lot of effort bringing out and in completely different insurance policies and efforts (e.g. one week attempting to broaden into textiles, the subsequent week attempting to chop all the pieces not concerned within the central enterprise).

3. Combining factors 1 and a couple of above, as entities change into extra coherent, they typically change into extra goal-directed_Rohin. Versus, as an example, turning into extra goal-directed_Rohin on common, however particular person brokers being about as more likely to change into worse as higher as they’re reformed. Think about: a creature that values crimson balls at 12x blue balls is similar to one which values them inconsistently, besides rather less wasteful. So it’s most likely comparable however extra goal-directed_Rohin. Whereas it’s pretty unclear how goal-directed_Rohin a creature that desires to experience a dolphin is in comparison with one which needed crimson balls inconsistently a lot. In a world with a number of balls and no attainable entry to dolphins, it is perhaps a lot much less goal-directed_Rohin, regardless of its larger coherence. 

4. Coherence-increasing processes not often result in non-goal-directed_Rohin brokers—just like the one which twitches on the bottom
   Within the summary, few beginning factors and coherence-motivated reform processes will result in an agent with the objective of finishing up a selected convoluted moment-indexed coverage with out regard for consequence, like Rohin’s twitching agent, or to valuing the sequence of history-action pairs that can occur anyway, or to being detached to all the pieces. And these outcomes shall be even much less doubtless in apply, the place AI programs with something like preferences most likely begin out caring about far more regular issues, comparable to cash and factors and clicks, so will most likely land at a extra constant and shrewd model of that, if 1 is true. (Which isn’t to say that you simply couldn’t deliberately create such a creature.)

These hypotheses counsel to me that the modifications in conduct caused by coherence forces favor shifting towards goal-directedness_Rohin, and due to this fact not less than weakly towards threat.

Does this imply superior AI shall be goal-directed_Rohin?

Collectively, this doesn’t suggest that superior AI will are typically goal-directed_Rohin. We don’t understand how robust such forces are. Evidently not so robust that people, or our different artifacts, are whipped into coherence in mere tons of of 1000’s of years. If a creature doesn’t have something like preferences (past an inclination to behave sure methods), then coherence arguments don’t clearly even apply to it (although discrepancies between the creature’s conduct and its makers’ preferences most likely produce a similar drive and aggressive pressures most likely produce an identical drive for coherence in valuing assets instrumental to survival). Coherence arguments mark out a side of the motivation panorama, however to say that there’s an incentive for one thing, all issues equal, is to not say that it’s going to occur.

In sum

1) Though any conduct may very well be coherent in precept, if it isn’t coherent together with an entity’s inner state, then coherence arguments level to an actual drive for various (extra coherent) conduct.

2) My guess is that this drive for coherent conduct can also be a drive for goal-directed conduct. This isn’t clear, however appears doubtless, and likewise isn’t undermined by Rohin’s argument, as appears generally believed.

.

.

.