The Last Border Is the Mind

The Algorithm of the Heart “The deepest algorithm is not in any machine. It is in the human heart, and it has been training itself for a hundred thousand years.” There is a question that has, in the last several years, been raised more often than answered. It is the question that animates the deeper layers of the AI discussion, beneath the policy frameworks and the safety protocols and the regulatory debates. The question, simply stated, is this: Can a machine be moral? I have been asked this question, in various forms, by students in universities on three continents, by engineers at AI laboratories, by policy makers in capital cities, by religious leaders, by ordinary citizens trying to make sense of the technologies that increasingly mediate their lives. The question is asked with a tone of genuine uncertainty. The questioner is not, usually, looking for a glib answer. They are looking for a way to orient themselves in a moment in which their inherited categories no longer fit cleanly onto the phenomena in front of them. I want to offer, in this chapter, the answer I have arrived at after some years of reflection. It is not the final answer. It is, however, the answer that I have come to find most useful for ordering the rest of the conversation. The answer is this: The machine cannot, in any meaningful sense, be moral. The machine can, however, reflect or distort the morality of the humans who build it, deploy it, and inhabit the world it shapes. The right question, therefore, is not whether the machine is moral, but whether the humans involved are. This is not a deflection. It is, I think, the deepest available statement of the position. Morality, in any of the senses in which that word has been seriously used across the philosophical traditions of the species, is a property of beings capable of suffering, of preferring, of caring, of being in relation. The contemporary AI systems, however impressive their behavioural outputs, are not such beings — at least not yet, and possibly not ever. They are sophisticated statistical artifacts. They produce outputs that look like moral judgments because they have been trained on the outputs of beings that do, in fact, make moral judgments. The appearance is impressive. The substance is, currently, absent. This does not, however, render the moral question moot. Quite the opposite. It concentrates the moral question on the humans who participate in the system: the designers, the deployers, the regulators, the users, the publics whose participation legitimates or de-legitimates the entire enterprise. The moral burden does not disappear because we have built machines. The moral burden intensifies, because the machines amplify whatever moral disposition the humans have brought to them. A society of generous, dignity-respecting humans, building AI, will produce AI that amplifies generosity and dignity. A society of narrowly self-interested, attention-mining humans, building AI, will produce AI that amplifies narrow self-interest and attention-mining. The technology is, in this sense, a mirror at scale. It magnifies the moral substrate of its makers. What, then, is the moral substrate of our current makers? This is the question that this chapter is, in the deepest sense, about. The Algorithm Already Running Long before we built the first computer, the human species was running an algorithm. It is the algorithm of care — the slow, evolved, cross-generational pattern of human beings choosing, over and over, to invest in the survival, the flourishing, and the dignity of beings other than themselves. The algorithm of care is older than language. It is older than civilization. It is, in some respects, the substrate on which language and civilization were both built. Without it, the cooperative achievements of the species would not have been possible. The algorithm of care is not a metaphor. It is a real, biological, cultural phenomenon, and it has been studied — under different names — by every serious anthropological, evolutionary, and psychological tradition. The mother who nurses her child, the elder who teaches the youth, the stranger who offers water to a thirsty traveller, the surgeon who stays an extra hour to finish an operation, the teacher who learns the name of every student — all of these are expressions of the algorithm of care. They are not exceptional behaviours. They are the default behaviours of human beings under conditions where the default has not been disrupted. Much of the history of modern industrial civilization, by the analysis of this book, has consisted of disruptions of the algorithm of care. The privatization of belonging, the financialization of time, the optimization of attention, the unbundling of meaning — each of these has, in measurable ways, suppressed the conditions under which the algorithm of care can run at its evolved level of intensity. The kingdom of ice is the kingdom in which the algorithm of care has been most disrupted. The boats of ghosts are launched from places where the algorithm of care has been most disrupted in a different way — through deprivation of the material conditions that allow the algorithm to express itself. The renaissance project of this book is, at the deepest philosophical level, the restoration of the conditions under which the algorithm of care can run at its full evolved capacity. The Index of Dignity is one such condition. The Moral GDP is one such condition. The Vocational Renaissance Centers are one such condition. The Peace Economy is one such condition. Tech-Yes-Logy is one such condition. Each of these, in its own way, is an attempt to remove the disruptions that have, over the past century, attenuated the algorithm. The question of whether the machine can be moral is therefore the wrong question. The right question is whether the machine can be placed in service of the algorithm of care. The answer to this question is unambiguously yes. The machine can be placed in service of the algorithm of care. The machine can also be placed in service of the algorithm’s disruption. The choice is ours. The choice has, until now, been made primarily by people who were not even aware that a choice was being made. The choice must, going forward, be made deliberately. What the Algorithm of the Heart Asks of Our Tools If we accept that the right design question is not “can the machine be moral” but “can the machine be placed in service of the algorithm of care,” then a series of specific design implications follow. The algorithm of care prefers thickness over thinness. It prefers extended human relationships over transactional ones. Therefore, any technology whose net effect is the substitution of thin transactions for thick relationships should be viewed, in the framework of this chapter, with suspicion. Therapy applications that replace human therapists, social platforms that replace embodied gatherings, dating applications that replace community-mediated introductions — each of these may be net positive in specific contexts, but at population scale they require continuous moral audit, because their tendency is to substitute thin for thick. The algorithm of care prefers presence over performance. It prefers being seen for oneself over being evaluated for one’s metrics. Therefore, any technology whose net effect is the intensification of performance pressure on its users should be viewed with suspicion. The contemporary social media environment has, by the empirical evidence, produced significant population-level intensification of performance pressure — particularly among adolescents. This is a care-algorithm-hostile design pattern, and it is reformable through better defaults, better incentive structures, and better governance. The algorithm of care prefers slow over fast. It prefers the patient cultivation of long relationships over the rapid accumulation of short ones. Therefore, any technology whose business model depends on the maximization of throughput at the expense of depth should be viewed with suspicion. This is a hard constraint for the contemporary internet economy, much of which is built on the optimization of throughput. It is, nonetheless, a constraint that the renaissance project asks the technology sector to take seriously. The algorithm of care prefers attribution over anonymity. Not in the sense that all communication must be identified — there are important contexts in which anonymity protects vulnerable speakers — but in the sense that the human beings affected by a technology should know, to a reasonable degree, who is responsible for the technology’s design and deployment. Diffuse responsibility, as we discussed in Chapter Three, is the most dangerous moral condition. Technologies that concentrate responsibility, that make designers visible, that make deployment decisions traceable, are technologies that operate in service of the algorithm of care. Technologies that diffuse responsibility behind layers of corporate structure, jurisdictional shopping, and algorithmic black-boxing are technologies that operate against it. The algorithm of care prefers repair over replacement. The contemporary consumer technology economy has, for several decades, been organized around the rapid obsolescence and replacement of devices, applications, and services. This pattern has significant ecological and economic costs that we have begun, in this book, to discuss under the Moral GDP framework. The algorithm of care prefers, where possible, the repair of what exists over the replacement of what exists with something new. Right-to-repair legislation, durable design, software longevity guarantees, ecosystem stability — each of these is a technology policy that reflects the algorithm of care. These five preferences — thickness, presence, slow, attribution, repair — are the operational core of what I am calling, in this chapter, the algorithm of the heart. They are not the only preferences. They are the ones I have found most consistently relevant when evaluating contemporary technologies. They can be supplemented, refined, and contested. The point is that there is a thing to align our tools with, and the thing is not invented by this book or by the technology sector. The thing is the evolved cooperative behaviour of the species itself, refined over thousands of generations, and currently being disrupted by technologies whose makers did not intend the disruption but produced it as a side effect of optimizing for other things. To re-align the tools with the algorithm of the heart is the work of the technology renaissance. It is the most consequential alignment problem of our century — far more consequential, in my view, than the more publicized alignment problems of advanced AI safety, important though those are. The technology that already exists, at the scale at which it has already been deployed, is reshaping the algorithm of care of every population it touches. The reshaping is happening now. It will continue, in whatever direction it is currently set, until the direction is deliberately changed. I close this chapter with one observation about what kind of person will, in the end, be required to do the changing. The engineers who are currently building the technologies of this decade are, in many cases, brilliant. They are, in many cases, thoughtful. They are, in many cases, well-intentioned. They are, in many cases, working inside institutional structures that constrain the moral discretion they are permitted to exercise. The problem is not, on the whole, that the engineers are bad people. The problem is that the institutional surround in which the engineers work has not yet been redesigned to ask of them the kind of moral questions the renaissance requires. The redesign of the institutional surround is a governance problem, not a moral education problem. The engineers do not need to be retrained. The institutions in which they work need to be reformed such that the moral questions are in the institutional flow, not at its margins. This is the same observation we made about civilization itself in Chapter Twelve: the question is one of organized refusal, not of individual heroism. The same applies to the technology sector. The renaissance does not depend on the rise of a generation of saintly engineers. It depends on the construction of an institutional surround in which ordinary, competent, professional engineers — operating with the discretion and the accountability the institutions provide them — produce, as a matter of routine professional practice, technologies that serve the algorithm of the heart. That institutional surround is buildable. Pieces of it already exist. Other pieces will need to be built, and built quickly. The remainder of this book — particularly Part VI, on the Divine Presence of Peace — will sketch the broader institutional architecture in which the technological alignment is one essential component among several.