Observing right and wrong for the first time
The common understanding of moral knowledge is that you learn it from your parents, read it from a book, or innately know it in your biology. In these views, you already “know” right from wrong, and your only challenge is to abide by this pre-existing moral wisdom.
What this view forgets is how much of right and wrong we learn experientially, in trial-and-error fashion. We act, we feel, we update. By reflecting after the fact on how our actions played out, and what did or didn’t feel right, we gain new insight into what is right by us. In this view, we’re all Pinocchio’s without a Jiminy Cricket to guide us; we’re alone and have to make our own mistakes. Put simply, a big part of ethics is empirical.
(Quick definitions: The word “empirical” is just a science-y word for experiential or observed. “Ethics” and “morality” are different ideas but I’m not a philosopher so I’ll go back and forth between them a bit loosely. Empirical morality is the right and wrong we learn by experience, guided by our reflections on our new experiences rather than pre-existing moral knowledge. It’s opposite is received morality, the kind we read in books.)
Empirical morality is not a quibble. It’s a major category of moral experience. No matter how old humanity is, new ethical frontiers are popping up all the time. The Bible asserts that there’s nothing new under the sun, but that’s what the Bible needs to be true, or it wouldn’t be any good as a one stop shop for moral guidance. And there are literally new things under the sun. The first nuke created a sun under the sun, with all kinds of ethical fallout afterwards.
Therefore, the ethics of the unknown, or not-yet-known, needs a larger role in how we understand moral behavior in general, particularly on “moral frontiers,” the new areas of human moral experience created by science, technology, and our steadily growing understanding of just how differently we all experience reality. Genetic engineering, algorithmic bias, artificial intelligence, and the surveillance state are just a few areas in which technology has created new dilemmas that can’t be easily sorted out only on the basis of society’s knowledge to-date.
We tend to think that it is the ethicist’s job to think through these new challenges for society, so we let them sit shotgun on the edge of experience with a typewriter on their laps, clattering out the rules of the road as society hurtles down the highway. But it is our practitioners — social psychologists, wet lab biological technicians, military drone operators, and corporate AI researchers — with the most direct access to how things are, how they could be, and the nuance of what feels right and wrong in each moment. These actors are the most direct (or least indirect) recipients of the moral weight of each decision they make. They are moral empiricists, and their experiences of what feels right and wrong on moral frontiers play a vital role in the moral systems society develops. So what are the properties of empirical morality, what can it learn, and what are its pitfalls?
The dartboard of right and wrong
In a strictly empirical morality there are actions, they can be right or wrong, and we can’t know which are which until after doing them. Imagine playing darts blindfolded on a board of unknown size, shape, dimension, and dynamics. The contours of the dartboard represent the unknown slowly shifting boundary between right and wrong. After each throw the thrower will either hear the satisfying thunk of the dart against the board (Right) or an awful crash as it misses and hits something else (Wrong). The empirical moral agent’s mission is to accumulate data about hits and misses and use it to develop a strategy for guiding action, a moral system.
This moral system can be seen as a classifier. It labels prospective actions as right or wrong before having to commit them. Building a classifier is fundamentally about managing tradeoffs. Is the goal maximum ability to discern right from wrong (overall classifier accuracy)? It may not be. In fact it’s most likely not. Moral systems often have an asymmetry in which types of error they penalize. They are, possibly by definition, much more concerned with properly preventing actions that may be bad (theft and treachery; the “miss rate”) than properly permitting those that may be good (swing dancing and reading comics; the “false negative rate”). When right and wrong are unknown, a moral systems as a classifier might seek to minimize false negatives specifically (trying to only hit the dartboard) or the overall misclassification rate (trying to learn its shape). Put simply, an empirical ethics has to choose whether its goal is “do no wrong” or “discern right from wrong.” And either choice forces big implications for right and wrong in society.
We will consider the implications of an empirical morality under both goals, ‘do no wrong’ and ‘know right from wrong.’ The main contribution of this essay is to show that both have serious problems. These problems are evident even outside the simple world of the model, in the more realistic worlds in which the boundary between right and wrong is blurry or dynamic, classifier goals intermingle, and in hybrid moralities, part empirical and part recevied. In fact, the more the problem description deviates from the simple model toward reality, the worse the consequences.
The “Do no wrong” ethic and its failure
Taking the first, let’s imagine that your goal on an ethic frontier is to do no wrong (never miss the dartboard). This makes sense if you are developing experimental pathogens to better treat disease. Zero can escape, even the harmless one, because you don’t really know what’s harmless. Under this goal, the best strategy is as follows: throw darts until your first hit — or ask around among your fellow “moral explorers” for someone who has already gotten a hit — and aim every future dart for the same spot. Take the first pathogen you find to be useful and harmless and student that one endlessly. As you explore its most minor deviations, and miss intentionally or by accident—as your darts to the lower left give a crash instead of thud, you might adapt up and to the right, and adhere to that offset. After enough experience with what minor mistakes are acceptable, you can assemble a moral theory that’s never wrong by defining moral action as anything in the target region, and immoral action as anything outside of them. Because you are playing it safe and mostly trying to hit the same spot over and over, the only source of novelty is the novelty that can’t be helped, such as that which derives from noise and execution error.
This “do no wrong” approach has the benefit of yielding as few as zero false positives (it will mistakenly label the fewest bad actions as good). But it’s downside is an unbounded false negative rate: unlimited potentially good actions will be labeled bad. Given our premise that many real moral theories are so biased, this is a positive feature of the model, not a shortcoming. And since the thrower sticks with the region of their first thud, they leave potentially huge areas of action space ethically unexplored. This is consistent with our common understanding of the most strict and orthodox moral systems, that they are so concerned with preventing wrong as to permit strategies that conservatively forbid actions that may be right.
The high false negative rate of “do no wrong” yields a major problem: non-uniqueness. Under “do no wrong”, two dart throwers in front of the same board under different initial conditions could stake out non-overlapping regions of action space and end up with moral systems that a) are perfectly internally consistent and b) completely exclude each other. Why is it that even the most careful, closely related religious denominations develop moral standards that exclude each other’s practitioners? A morality that is only interested in never doing wrong, with know interest in learning more about the space of right and wrong, is not only at risk of conservatively forbidding huge amounts of action space, it can also developing lead to multiple mutually incompatible moral systems. While we hope that two visionary founders who try to live right end up with the same ethic, this thought experiment shows that their conservatism, applied to such a rich space of possible behaviors, can easily lead to those sincere founders developing moral systems in which the other is stuck in the wrong.
The “Know right from wrong” ethic and its failure
If “do no wrong” is so flawed, let’s examine the other potential goal of empirical morality, learning right from wrong. In the language of the dartboard metaphor, this looks like maximizing overall accuracy over the entire space by learning the contours of the dartboard by experience. Here is the problem with trying to learn the shape of a dartboard one thud at a time: you have to miss. It is impossible to adaptively learn the shape of a boundary without sampling from both sides of the boundary between right and wrong, particularly if throwing darts is inherently noisy.
The problem for an empirical morality based on learning the boundary between right and wrong under all actions is that it is impossible to learn the difference between right from wrong without doing wrong. If there exist actions that we can only morally evaluate after the fact, if cognitive limits make some actions inherently morally unsimulatable and we must sometimes “feel” an action out after doing it, then even a completely earnest, virtuous scientist, who wants only to understand the moral contours of their new area of human inquiry, must sometimes go too far.
So both goals of an empirical ethics have serious problems. If your ethic is to do no wrong, you will produce a morality that is incompatible with any other “do no wrong” ethic that’s not identical, all while limiting your followers to an artificially small backwater of human experience. And if your ethic is to learn right from wrong, then you have chosen an ethic is which the only way to “be” right is to sometimes “do” wrong.
Going further
These are not the last dilemmas facing those on moral frontiers. Simply being on the edge of right and wrong is enough to be viewed suspiciously by a more general population nestled safely in the interior terrain of familiar right and familiar wrong. How does an empirical morality interface with a culture that generally subscribes to a learned morality with no trial and error, no vast unknown spaces, and no conflict between know and doing right?
Social morality’s interactions with empirical morality only adds problems. People who have learned what is right in a new area by doing a lot of right and wrong can attempt to serve their society by incriminating themselves and sharing their stories from the frontier. This would help prevent the rest of society from having to also do wrong to learn right. Is the alternative more moral: protecting themselves, leaving the boundary unlit, and letting all future comers make the same mistakes?
The challenges facing ethical entrepreneurs may explain the general suspicion of those who work in morally fraught areas of technology. How many books and movies are driven by the scientist or technologist who “went too far”? Workers at moral frontiers are suspicious for the very good reason that they are morally compromised, not necessarily for any personal shortcoming, but merely because of the nature of work on a moral frontier. Even the most earnest practitioner can only serve society if they are prepared to be judged as amoral or immoral. The only people who can see that a good person must sometimes do wrong are your peers on the frontier.
Will some hybrid approach demonstrate a complementarity that solves the problems of an empirical morality? A classifier tuned between the extremes over minimal misses and maximum accuracy will not magically benefit from the synergy of both. A mixed approach between empirical and received morality can’t prevent these shortcomings. Only a tremendous faith in moral simulation, a perfect ability to anticipate unanticipated consequences of our actions, can eliminate the risk to a society in mapping moral frontiers. The other option, leaving them unexplored, is conservative at best and at worst completely opposed to science and the spirit of inquiry.
Living it
I didn’t come up with this playing darts, but building organizations where I had to balance the demands of consensus with my own inner compass, or serving organization, like the Walt Disney Company, with unprecedented data on unprecedented populations of children’s and families. I’ve made all kinds of mistakes. After all of it, I’ve come to the conclusion that science is inherently morally compromising. And even more controversially, i’ve decided that the solution isn’t to stop doing science, but to be OK being compromised. This model is what I came up with to get that across. I have presented a simple model of morality for scenarios where it can only be learned by experience. It reveals a basic distinction between the moral goals of doing no wrong and knowing right from wrong. I show that either one you chose, you’re going to have big problems, either the problem that no distinct pair of moral systems are mutually ethical, or the unsettling idea that a person cannot be right without doing wrong, or at least tolerating other who do wrong for them. I’m sharing this because I think it’s important to make mistakes, and to promote the possibility of an ethic in which the greatest moral failures are worthy of our admiration as long as they proceed with care, sincerity, and a spirit of discovery.
