Rationality
Emma Borg
At a minimum, a system is rational when it utilises good reasoning principles to arrive at a conclusion on the basis of reasonable evidence. Beyond this, a rational agent might also be required to have appropriate goals, to optimise satisfaction of its preferences, and to pursue its goals efficiently.
Can an AI system make rational decisions? And why does this matter?
Key Points:
Rationality has often been taken to be a core human attribute, so when considering advances in AI it is important to ask whether the systems behave in a rational manner or not.
Judgements of rationality can attach to agents or systems as a whole, to the processes which systems embody, and to the actions which systems produce. Whether an AI system counts as rational, and the extent to which this matters, will depend on the aspect of rationality we are interested in.
Human rationality is often held to involve more than just correct logical reasoning, it also involves setting appropriate goals and acting in suitable ways to achieve those goals. So, arguably, an artificial system will not approximate human rationality unless it is able to set its own goals and work towards achieving them.
A classical view defines man as a rational animal. So, if we are interested in how closely AI capacities mirror human capacities we will want to know whether AI systems can also be rational. Furthermore, rationality is linked to a number of other things we care about, such as explicability and predictability: knowing somebody’s reasons, and understanding their reasoning, allows us to explain what they are doing and predict what they will do next. Similarly, if an AI system arrives at its outputs rationally, that would allow us to better understand and predict AI behaviour. Accounts differ on what is required for a person to be rational, so extending the notion to AI is going to be complicated, but there are a number of distinctions which can help in thinking about this issue:
1. Narrow vs wide conceptions of rationality
A common view is that rationality involves a kind of coherence between one’s goals, one’s evidence, and one’s actions or verdicts. This is to adopt a narrow (internalist) perspective on rationality, capturing the sense in which a person counts as rational if what they believe and do is based on good reasoning using the evidence available to them. From this perspective, it is rational to believe that spiders have six legs, even though that claim is false, if one derives the belief correctly from other things one believes (e.g. that spiders are insects and insects have six legs). From a narrow, coherence perspective, an AI system can count as rational even if its internal states do not accurately represent the world. A chatbot which outputs “the world is flat” because this sentence was prevalent in its training data counts as rational from a coherence perspective even though the claim is false. Advocates of narrow rationality will, however, need to specify which transitions between internal states are rationally acceptable (see 3).
This narrow conception of rationality can be extended in various ways. For instance, rationality might be taken to incorporate an external dimension (in addition to internal constraints), requiring rational processing to aim at achieving accuracy or truth, representing the world as it actually is. This externalist perspective captures the sense in which, say, someone with Capgras delusion (who believes their loved ones have been replaced by aliens) counts as irrational: although they may have a fairly coherent worldview, we take what they believe to diverge too seriously and too significantly from reality for the person to count as rational. A second way to extend the notion of rationality is to require a system to set its own goals and to pursue those goals efficiently. In economics, rational behaviour is often taken to involve ranking preferences and acting in such a way as to maximise preference satisfaction. From these wider perspectives, to count as rational an AI system will need to be sensitive to its environment (in order to aim at the truth) and be capable of setting its own goals and devising plans to achieve them.
2. Theoretical vs practical rationality
Theoretical rationality concerns information processing/belief formation: rational agents must process information in accordance with rules (such as those embedded in classical logic or probability theory) and in line with the need for proper evidential support. Practical rationality concerns what to do: rational agents must adopt appropriate means to achieving outcomes and (perhaps) aim at outcomes which are themselves valuable in a specified respect. When considering AI rationality, ask what rules the system is following when processing information, what outcomes it is aiming at, and how those outcomes are achieved. Also ask whether the system is generating its own goals or whether all goals have been set by programmers.
3. Ideal vs Bounded Rationality
Ideal Rationality (often assumed in economics) requires a system to (i) aim at maximising the benefits of outcomes (known as ‘utility’), (ii) consider all evidence, and (iii) utilise classical logic in deterministic environments and probability theory in uncertain/vague situations. Bounded rationality, on the contrary, allows a system to (i) yield good-enough outcomes, (ii) consider limited evidence, and (iii) utilise any appropriate problem-solving rule or approach. Bounded rationality is associated with heuristics: non-logical, approximating rules (e.g. ‘objects which appear blurry tend to be further away’). Bounded rationality is often taken to provide the proper standard for human thought. When asking about AI rationality, consider whether it should be judged by Ideal or Bounded standards, and whether human and artificial rationality need be judged by the same standards.
4. Rationality as context-sensitive
What evidence is taken to be relevant to a decision, and how that evidence should be weighed, are context-sensitive matters. It may be rational for me to believe that my train leaves at 5pm today on the grounds that it left at 5pm yesterday, but if getting the time of the train right really matters, or if I’m ignoring easily available evidence to the contrary, then continuing to believe that the train leaves at 5pm today may no longer count as rational. This context-sensitivity creates problems for assessments of AI rationality. Imagine that an AI system is asked to deliver ways to combat human-created climate change and it suggests exterminating all humans (this example is an urban myth). Such an output might count as rational given the task set, and information provided, by the programmers. However, given additional information (such as the value placed on human life) the answer may no longer seem rational. Determining what kind and how much evidence a rational AI system (or indeed a person) needs to be sensitive to when making a decision is far from straightforward.
5. Rationality as all-or-nothing vs a matter of degree
Sometimes rationality is understood as a binary notion: an individual, belief, or action either is or is not rational. Once the bar for rationality is passed, however, rationality is treated as a matter of degree: one individual, belief, or action can be more or less rational than another. This is reflected in the fact that aspects which matter for rationality - how much evidence is looked at, how good one’s reasons are, and perhaps what kind of goals one is pursuing - are also graded (more or less evidence can be looked at, reasons and goals can be better or worse). An AI system might count as rational per se if it arrives at a verdict or conclusion on the basis of the evidence it has, whilst pursuing goals set by its programmer, but such a system might still count as more rational if it considers more evidence, sets its own goals, pursues those goals more efficiently, etc.
6. Common-sense reasoning
Human rationality connects to other intellectual and emotional virtues such as intelligence, problem-solving, and empathy. Designing an artificial system which can deploy non-logical, heuristic, or common-sense rules, which can decide what evidence should be taken to be relevant to a problem, and can display sensitivity to human values in goal-setting and reasoning, is a big ask. However, using neural network architectures able to learn from vast data sets has resulted in systems which seem able to reason in more human-like ways. However, findings here have been disputed and some initially impressive results turn out to involve mere parroting of information in the system’s training data. Human-like reasoning is often seen as a benchmark for intelligence, so when thinking about AI systems it is important to ask whether they succeed in common-sense reasoning, not just logical reasoning tasks.
The need to understand and predict the behaviour of AI systems is likely to become more pressing in the coming years. AI systems would be significantly more interpretable if we could assume that they are rational systems, responding to evidence and acting on reasons in the way that we assume typically holds true of people. However, while coherence between internal states is something we can expect good AI systems to deliver on now, becoming fully ‘reasons-responsive’ seems a long way down the line and will require richer capacities (such as AI systems which aim at believing what is true, can determine how much evidence to look to, and can set their own appropriate goals and work efficiently to achieve them).