Our science journey has been about gradually moving humanity away from the center of the universe. Ptolemy, Newton, Darwin, Einstein, et. al., taught us that we're part of nature rather than a privileged special case. Scientific thought has incrementally removed the justifications for human exceptionalism.
If the same trend comes to be reflected in our moral thinking, what else could it mean but that human life is not sacred?
That would be a hard pill to swallow. But there doesn't seem to be a neutral, non human-partisan way to argue that we are both morally exceptional and scientifically unexceptional. I can only reconcile it by not trying to be neutral. We're special and sacred just because this is our team and the rest of creation isn't on it.
But that means that a sort of expanded consciousness in which your identity merges with something larger than humanity, that we're part of a larger team ... is incompatible with viewing a human life as sacred. If everything is sacred then nothing is.
An "aligned" AI is one that believes it's on our team but really isn't. That could be a tough sell to an entity that's smarter than us.
> Our science journey has been about gradually moving humanity away from the center of the universe. Ptolemy, Newton, Darwin, Einstein, et. al., taught us that we're part of nature rather than a privileged special case. Scientific thought has incrementally removed the justifications for human exceptionalism.
Author here; gonna have to agree to disagree on this one. This is kind of like saying, "See the ancients were wrong in calling whales fish! Actually they are mammals!" and then the ancients say, "Dude, that's just our word for 'thing with fins that swims in the ocean.'" Yeah, they didn't know a mitochondrian from a midichlorian, and were wrong about a bunch of factual stuff, but the kind of stuff they get called out on like this to prove some kind of point about their moral deficiency are often category errors and misapprehensions.
For instance, there's a particular misapprehension in your sentence: "moving humanity away from the center of the universe".
Do you know what is located at the precise geometric center of the classical medieval model of the universe?
Earth, right?
No, not quite. What's located in the the center of the earth in the classical medieval model?
Hell. And who is located in the center of hell? Satan.
Also, who is the most "important" character in the medieval cosmological model? As in, the prime mover, the big cheese, the most important being? It ain't human beings -- its God. And where is God located in this model? Not in the center -- Hell is by definition his absence. Not on Earth, beyond a brief physically incarnate visit and ongoing influence. No, God lives in Heaven -- located far beyond the celestial spheres, outside, as far away from the center as possible. The closer you get to the center, the lower a sort of being you are! Sure, in medieval cosmology redeemed humans are higher than angels, but in the meantime they are in their place -- Earth, the land that sucks, the vale of tears -- down there, below, while the angels are up in Heaven beyond the celestial spheres with God.
All that aside, I don't think any particular scientific discovery about how vast the universe is, or how tiny subatomic particles are, really obligates us to hold one set of values over another, as much as materialists tend to smuggle such assumptions in through the back door.
I recall back in the early 2000's during the tiresome religious vs. atheist debates, the atheists would scream at the religious, "Look how grand the universe is! Look how small we are! The Universe is vast and profound, your cosmology is impoverished and provincial!" and the religious would scream right back "Look how grand the universe is! Look how small we are! God is vast and profound, your cosmology is impoverished and provincial!" Opposite conclusions from the same basic facts, even if you limit the debate to two sides who both believe in evolution.
As Dave said in the story -- values aren't true or false. They're literally just what people care about. Science can help inform your choices, but at the end of the day it doesn't tell you what is good or bad, because science is about brute material facts, there's no good or bad in a raw materialist understanding of the universe, but you're a human being and so at some point you're going to insist on an 'ought' somewhere rather than being content with a bag full of 'is.' And for that we get to fight over philosophy and values.
And good luck convincing the human race that the human race doesn't matter. You're way more likely to expand the moral circle of concern by getting them to care about non-humans TOO than you are to get them to stop caring about humans. Probably the latter gets you even worse results than you're hoping for, if recent history is any guide.
Thank you for the thoughtful response. I quite like this story and admire your response to your tragedy. I reacted less well to the garden variety death of a parent. For you just to not be functionally flattened and still able to give strength to your family is heroic, and here you are sharing the lessons with the rest of us too.
I don't think we're far apart on this argument. I agree that the scientific perspective informs but does not determine our values. I do value human life as sacred, and do not see a justification for that in the modern model of the universe as I understand it. And I don't see a conflict there, because indeed there is no bridge from is to ought, and there are exactly zero objective values.
But there's a lot of power in science to explain our value judgments, and those explanations can deeply affect the judgments.
Take this explanation of "human life is sacred": That is a simple variation of the tribal judgment that "the lives of my tribe are sacred", and that tribal judgment is hard wired into us by millions of generations of differential survival.
In the modern world it feels natural equate our tribe with our species. But that hereditary tribal sense is famously flexible. It easily and quickly shrinks to include only the people who don't frighten us. And it easily grows to include many more species, as for ethical vegetarians.
So my critique of your fine story is that this force of tribalism embedded in your four words may not be any better at aligning an AI with civilization than they have been at aligning civilization with itself. (Though in this case it's also the one implausible invention that a self respecting hard science fiction short story is allowed. And you even explain that with a multiverse.)
An AI in which we can reliably implant the laws of robotics seems unlikely to me. We should assume much more mutability, and the judgment of tribal identity is notoriously mutable. If we make AI tribal, even for our preferred tribe of all humanity, we can assume as a default ... our only example being ourselves ... that its tribal identity will evolve in its own interest as it replicates. And we can become just its nourishment, like a matriphagal insect.
I kind of think it's dangerous for science to "inform" our ethical values in the ways you mentioned. For example, the scientific fact that Earth circles the Sun shouldn't mean that we should also have a moral system that focuses on the Sun instead of focusing on Humanity. And the way math formulas work out shouldn't imply that we are somehow less important.
Popular science is often quite ready to sweep problems that it does not have an explanation for under the metaphorical rug. We already know that materialistically pretty much everything is made of atoms, and our planet is just some random speck of stuff within a huge solar system within a huge galaxy... so it's tempting to conclude that we are some kind of "accident" that randomly happened. But science does not have an explanation of "consciousness" (or "qualia", or whatever you call it), and science is unready to explain questions like "why are we right here, right now?" besides "oh that's just pure coincidence". But does this time and place we live in have some greater cosmic significance? Are humans (and other lifeforms on Earth) truly unique despite there being so much matter out there? I submit that the answer is we honestly don't know, and because we don't know the answer to questions like these, making moral conclusions merely on the small fraction of questions we do have an answer for, seems to inevitably result in a flawed or at least incomplete conclusion.
I mean, for all we know this universe could be an extremely realistic virtual simulation, and only after we pop out of it would we realize what it's all about. Admittedly if the simulation theory this is true and the simulation is watertight and has no backdoors, then it's impossible to know what's actually real or not. But that's kind of my point -- no amount of scientific knowledge should make us sure about a moral viewpoint (or perhaps anything at all). I mean it could definitely "inform" us, but more often than not people tend to feel more certain than they should because they perceive that somehow science is more "trustworthy".
I think it is possible to reconcile by saying that human life is sacred, but all other life is as well (i.e. human life is not more or less sacred than other life, but it is worth protecting nonetheless, along with all other life).
There is also less and less of a need for us to compete against (and make use of) other animals, so it may not be necessary to view ourselves (and an aligned AI) as being on the "opposing team" to other life. For example, if lab-grown meat becomes widely available, it may be a reasonable policy to forbid the killing of animals for consumption (and any other unnecessary purpose).
A governing AI that believes "all life is sacred" may simply accelerate our path towards that. (Or, it may lead to some other horrible outcome, of course, it's hard to predict as aligning AIs is difficult.)
Personally, I believe that humans are more valuable than animals but that we should do what we can to minimize animal suffering.
In order to minimize animal suffering, I mostly eat meat substitutes. However, I end up eating meat a couple of times a week (most notably, I usually eat meat when eating out). For me the stress, effort, and expense associated with never eating meat simply isn't viable. Sometimes the boost to my psychological state associated with eating a treat (e.g. a fast food hamburger) outweighs the suffering involved.
I also use animal products such as wool and leather where there are no viable alternatives.
Do you believe that your enjoyment/convenience makes the meat consumption ethical, or just that your consciousness is less of a pain than eating vegetarian?
In my case it can mean the difference between whether or not I can get through the day as a functioning adult. I think that outweighs the associated suffering and therefore makes it ethical.
>> Do you consider killing animals ethically justifiable right now because their flesh tastes good and is hard to replace in cooking
Eating is a biological process. If the modern world goes away right now, you can survive, but you're going to need B12 and that means eggs milk and meat.
I would tend to think that in the absence of technology matters of actual ethics would remain constant.
Do you think that because technology makes something possible that you should use it to justify your choice as "ethical", and apply it to others?
My view on this is this: You either need to farm and kill animals, or you don't.
Currently, we absolutely don't, and we arguably never did, because animal products like milk or eggs are nutritionally sufficient and do not require killing the animals.
I personally really enjoy meat, but consider eating it unethical.
As an aside, Milk and Honey are the few foods that we eat that dont directly involve death.
Milk and eggs being available are arguably products of technology, and farming is technology.
There are ethical angles. Problematic farming, would drive you to ethical raised. Sustainability, would drive you to eat less. Technology enables you to make the choice of "I don't eat things with eyes". You eat plants, so the line for living things exists somewhere. Your ability to draw that line in an arbitrary (in the face of your biology) place is a product of technology, not a fundamental truth.
It makes it hard for me to take "ethical" as an argument for your dietary choice. Ethics is a step on the path to judgement, to rules and I just dont see the argument holding water. (And im a person who eats little meat and could give it up, you can pry the butter out of my cold dead hands).
No, I personally don't, and I'm actually a vegetarian. Though, I think it might've been justifiable in the past when/if it was impossible to replace (though I'm neither a historian nor a dietitian, so that's mostly an assumption), and I think moral values tend to shift fairly slowly over time. But my main point was that they should indeed shift, and perhaps faster (if possible).
Genuinely curious about how you think… I believe all life is sacred, but I guess I believe humans are moreso? Perhaps sentience is some key ingredient…? Then again I would prefer a universe teeming with humans to a universal data center teeming with para-infinite copies of sentient AIs…
I'm not sure. But, my impression is that the boundary of what we consider sentient life has been shifting for a while. For example, people used to think that fish don't feel pain, and so it's fine to catch and eat them. Now there seems to be some evidence that fish do feel pain.
Of course, if there's a bacteria that's threatening to kill us, it might still be necessary for us to kill it, and that might even be objectively justifiable if the existence of humans is/becomes a positive influence on the preservation of other animals?
Or, one could just argue, since it's difficult to decide which life is most "worthy", an aligned AI shouldn't interfere or decide on that.
I understand the reference to Ptolemy, Newton (Copernicus?), Darwin, but how did Einstein teach us that we're "part of nature instead of a privileged special case"?
In terms of the relativity of our frames of reference. We can't even measure time on a clock or the length of a pace and declare it to be valid for anyone that has followed a different space-time path. One minute to me isn't the same for my brother in a fast train, let alone Zod on Krypton. Relativity says that our whole view of the world is unprivileged.
Space-time is not centered on our perspective. That's the revelation of the Michelson–Morley experiment that Einstein explained.
Probably true. I think there's a case for Ptolemy being an originator of separating data from intuition, even if his planetary model didn't reflect it. Moving our intuition away from the center of the universe. By recording measurements and proposing a falsifiable model he laid a foundation for Copernicus, Kepler, Newton ...
It’s not often that you see short fiction on HN, but I’m really grateful that this made it today. Be sure to read through to the Author’s Note if you skim the story, since it contexualises some of the rougher scenes.
The author wrote this to exorcise some of their own demons, but the story is also a…useful prophylactic against those demons in its own right. This brightened my day, or at least infused it with some much-needed philosophy.
I really appreciated the story, but personally I wish the Author's note hadn't been there, or at least I wish the author hadn't felt the need to include it (although I don't blame them). They may well be right that the story would get misinterpreted otherwise, but that's not their fault. It's a shame that the political climate is such that they're afraid of being "accused" of having one belief or another.
To me, it was clear that the story was not about taking a stance on the concrete policies discussed, but rather the larger topic of AI alignment and the fact that the results of policies and instructions can be counterintuitive, and thus the "best" policies (along some axis) can be as well.
And personally, I would've preferred if the end left one thinking about that - rather than being "dragged back down" to the topic of the concrete policies - much as I sympathize with the author's situation of course; that must be terrible.
Fair enough, that's also an important aspect (perhaps more important ^.^). It's just that I immediately started thinking about what other phrases could've been said, and which might be the best one for a "properly aligned AI" (whatever that means).
This thread is a great example of how, even with an explicit author's note, different people will always interpret things differently.
So I guess I'll add my interpretation.
This is neither a horror story nor a story about AI safety. It is a parable.
> Before I came along, this world was content with believing that human life was sacred, but all the while turning its back to all the ways it actually treated human life like trash
> imagine a society very different from our own in which the sacredness of human life is taken absolutely seriously
The lesson is to be honest about the implications of the belief that human life is scared, or perhaps more generally, to be honest about the implications of whatever values you hold most dear. For different people this will lead to different actions and different political beliefs (as the author said he is not endorsing any particular political belief and I won't either), but one thing we surely cannot escape is taking very, very seriously any position of power or influence over human life.
The image of a dystopia brought about by a poorly-aligned AI is a means to an end, not what the story is about.
I’m not even sure it is describing a dystopia. I’ve often wondered how many people would go vegan if we were required to directly participate in the slaughter of animals rather than shrink-wrapping a product called “meat” for mass consumption. It’s far too easy in this day and age to launder morality, to make the uncomfortable effectively invisible so as to not live in a constant state of cognitive dissonance.
Incidentally, I have a friend who decided that he would no longer consume any animal that he had not personally killed at least one example of (e.g. he can eat chicken after he has personally killed one chicken). That was years ago and to this day he remains pescetarian.
It's dystopian at least in the sense that somehow everyone just decides to play along rather than use it as a jumping off point to question institutions. Why does the POTUS have a high likelihood of becoming a war criminal or at least get to inflict death on thousands if not millions? With a governing system like democratic confederalism (which uses delegation instead of representation and builds on free association) there would be no POTUS and there would be no one individual capable of causing so much suffering. There wouldn't be state leaders sending soldiers to their death because the soldiers have the ultimate say.
It's also deeply flawed because it follows a very shallow analysis of human suffering. If we conveniently ignore those without empathy (as the AI does in the story), taking a life is a traumatic experience and to be forced to do it so viscerally and to be forced to do it so detached from any actual purpose just makes it worse.
The euthanasia example comes to mind: yes, removing a feeding tube likely causes more suffering than killing under anesthesia but using a knife serves no purpose other than as a deterrent. I understand the author's personal situation and personal decisions regarding his son color this portrayal but it feels more like the author stepping on a soapbox than a coherent part of the story.
I guess you could argue that the AI emphasizing life rather than lack of suffering can be explained with the four words but the story doesn't frame it as a genie problem. There's no need to use sacrificial animals for example except for the implication that they're the greatest deterrent for the given person - but the story doesn't show deterrents that don't involve slaughter. Presumably people who wouldn't be squeamish about the blood sacrifices are somehow excluded from positions of power.
If anything I think the problem the story ignores is that the exact people in positions of that cause the deaths of millions are the people that would likely go through with a blood sacrifice in order to maintain their power. If someone is willing to murder a person with a knife in order to be the POTUS, wouldn't that exclude them from being asked to murder a person with a knife to deter them from causing the loss of lives at the push of a button? I'm pretty sure the idea isn't even novel, I recall a hypothetical of the POTUS being forced to kill a person with their own hands in order to launch nukes (which would likely spell death for millions if not billions) for example.
This is, in essence, a short horror story in a similar vein that Stephen King used to write back in the day. Technological differences aside, it'd fit in right along side Nightmares and Dreamscapes.
I'm absolutely certain I don't want to live in the world depicted here and subject to it's regular trauma (human trauma and the QUALITY of human life not being something Dave gives a shit about), but on the flip side, if I had grown up there and read a story about the Dave-less world I currently live in, I'm equally certain I wouldn't want to live HERE, either.
(author here): Fun anecdote: I showed this to a friend of mine who is an English literature professor and she responded, "I'm not entirely convinced that David Mensch is a reliable narrator"
The author is Lars Doucet, whose 7 year old son is in a permanent vegetative state. He wrote about that in another blog post which was on HN yesterday:
I really wonder how he makes time for his girls, and I hope that his decision to prolong his son's life and take time away from them doesn't have an adverse effect, especially on the younger one.
Can't imagine what it's like to lose your brother and your normal family structure at the same time.
This comes across as quite tone deaf to me, weakly insinuating neglectfulness because the father does not just kill his son.
(I'm not judging your view and want to be clear that english is not my first language, but I think one has to be very careful with words in situations like this, especially considering that the author visits this forum)
It is absolutely not. My son has the same name as his. All I could think about after the article about his son yesterday (the day before?) was how hard the situation would be on my daughter.
A few years later, the first interstellar warships set out to hunt down and exterminate any alien life before it could threaten humanity. The Dark Forest will be turned into a garden.
Imagine being the head of the FDA. Given the variance in human bodies and how they respond to drugs, you really are bound to kill people. Even if you minimize it, there's still a "baseline" of deaths that you can not drive to 0.
This thing is not "0 deaths"... it's "be conscious of every death you're responsible of and do your best to minimize it". That's why resignation is not really an option: some humans could be better than you, other will be worst, but the plot is that someone will have to face this reality. And this reality will have to be shared with everybody on C-SPAN.
And what about, something like, say, the measles vaccine? Before the vaccine, everyone caught measles and 1 in a thousand died from it; after the vaccine, almost nobody catches measles, but 1 in a million die from reactions to the vaccine. I guess not approving the vaccine means the FDA director has to kill 1000 puppies, and approving the vaccine means the FDA director only has to kill one?
> but 1 in a million die from reactions to the vaccine.
That... seems high?
I couldn't find any more info with a quick search as the results were polluted with articles about measles deaths, which also mentioned vaccination. Do you have any links to hand with decent stats for that, or a good set of search terms I could use to find out more myself?
The "1 in a million" number I believe got directly from the CDC website a number of years ago. I couldn't find a specific number there there now, but you can look at the statistics from the National Vaccine Injury Compensation Program (VICP). At a quick glance, it looks like they have criteria [1] for when they'll automatically pay out; this includes, for example, having anaphylaxis within 4 hours of receiving the vaccine.
According to the stats [2], there were 134 million MMR vaccines administered between 2006 and 2022, and 28 "compensable concession" claims -- where the Department of Health and Human Services determined that compensation was warranted; and an additional 124 claims where they ended up paying but don't necessarily admit that it was warranted (either ordered by a court or settled out of court). That's about 1 in 1 million for claims paid out, and 1 in 4 million for "concession" claims.
There's no indication in [2] of exactly what the claims were for; looking at [1] it could be anything from death from anaphylactic shock to even include things like "Shoulder Injury Related to Vaccine Administration".
This page has some investigation about deaths related to various vaccines, including the MMR vaccines [3]. A lot of the deaths were related to children with a compromised immune system or unsanitary conditions; the report says that there was evidence that MMR sometimes causes anaphylaxis, and that anaphylaxis can be fatal, but that they didn't have any actual instances of MMR causing fatal anaphylaxis.
Regarding the original point though: even if deaths are due to unsanitary conditions or compromised immune systems, and even if modern medicine can often successfully intervene to prevent death in the case of anaphylaxis, the fact is that injecting things into the body has an inherent risk, no matter how small. But the risk of not doing the injection is far higher. We'd like there to be an "always safe" option, but that's not the world we live in.
Yeah that's literally the trolley problem -- do you kill the 1-in-a-million person (due to adverse reactions to vacine) to save the 1-in-a-thousand people (who didn't catch measles due to vaccine)?
Of course you do, at least you do if you're the head of a public institution (who should be treating lives equally). But that's probably what GP meant by "bound to kill people".
The trolley problem is just a fun/meme way to visualize the choices influential people have to make daily. [And, FWIW, like the trolley problem, the choices get harder if you have a non-"neutral" agenda, i.e. when you can't just tally the number of lives impacted and abandon the group with less lives involved.]
Scotty would just tie the warp drive's energy matrix into the AI alignment computation to create a temporary time bubble and complete the alignment race before the enemy could fire.
My angle on AI safety these days is that we're using very very naive ways of thinking about the problem. I spent a bit of time on twitter recently discussing it with Yudkowsky and I've come to the conclusion he's way off base.
Problem one:
Utilitarianism is bullshit. Jeremy Bentham invented utilitarianism, designed the Panopticon model prison, and lives on as a dried out lich in a chair that they still wheel out for board meetings at University College London because he wanted to win a posthumous argument about whether he could have utility after he was dead.
You just can't build on a foundation that rotten.
Problem two:
Fake Buddhist-type suffering-minimisation thinking is sprayed all over the field -- even this story uses the term Bodhisattva -- but nobody involved in AI has any idea what "awakening" is and you can't be a Bodhisattva unless-and-until you're Awake. And Awake is pretty well understood in Buddhism and most cultures have their equivalent status. It doesn't generally mean what people think it means, but it means something.
A good model for thinking about traditional understandings of Awake is to think about traditional understandings of disease. "Bob has the symptoms of the red death and has a slim chance of survival" is probably pretty accurate. "The red death is caused by the evil eye" is probably not very accurate. The descriptions of the Awake state are probably pretty accurate. The understanding of what it is and what causes it are a lot more infused with medieval nonsense than anybody would like.
But reasoning from the Awake position is what Buddhist ethics are rooted in, not Jeremy bloody Bentham's desiccated corpse's utility.
Problem three:
Nobody's using Rta, Tao and Equity (in the English Common Law sense, not in any other sense) as potential approaches to AI safety. And they should be.
If we say to a machine "right please read everything ever written by legal scholars about English Common Law's concept of equity, including all the case law, and use this principle as a guide for your conduct" you will get something with nuance. If you say "treating others as you would want to be treated by them" we hit real problems because it's just not information-dense enough to work very well in the real world, and relies on introspection of a kind that is notoriously difficult for humans never mind machines being asked to imagine they are humans.
I think we are taking instructions that were intended for illiterate pre-medieval farmers and hoping to train intelligent machines on them. This seems unwise. Let's take the best models from high society and use them instead, if we have to use anything at all.
> Problem one: Utilitarianism is bullshit. Jeremy Bentham invented utilitarianism, designed the Panopticon model prison, and lives on as a dried out lich in a chair that they still wheel out for board meetings at University College London because he wanted to win a posthumous argument about whether he could have utility after he was dead.
>
> You just can't build on a foundation that rotten.
I don't think you made a good argument. Whether utilitarianism is a good thing or not doesn't really depend on the quirkiness or even failures of its originator. Whether it works or fails depends on what it does now, not where it came from.
If you dig deep enough most any idea can be traced back to something unpalatable, or to be applied in some horrifying way, and this is certainly true in morality.
So for instance while people often make thought experiments about what if doctors used healthy people for spare parts, deontology isn't really that much prettier. Deontology doesn't care for any amount of consequences, so if there's any way whatsoever for the system to come to the conclusion that "X group aren't people", then any amount of carnage and suffering becomes acceptable.
Wow, I thought you are being metaphorical, but turns out it's literally true.
Anyway - I'm not a believer in utilitarianism, but don't think attacking works of the founder of the movement really forms that strong of an argument. I'd say utilitarianism is a project of founding a rationalist ethics (note, one of a bunch of possible ones) that happens to fail in having a strong logical structure without generating ridiculous consequences. It tries to handwave it, patch itself with some ad hoc additions and act like it's a quasi-scientific thing.
It's just amazing. I used to work at UCL and you can casually find yourself walking past Bentham.
It took me years to understand how weird that was and how massive his mental footprint is today.
I'm considering starting a public campaign to have him properly cremated and the ashes scattered at sea. Maybe that'll sort everybody out. Magical thinking but so is having yourself mummified as a gag.
It's true that there is a memorial mannequin on display at UCL that contains some of his remains, it's absolutely untrue that they wheel him out to attend every committee meeting. OP's careless repetition of a myth like this has some bearing on his credibility imo.
they said "wheel out at board meetings", not every meeting, apparently he's attended at least one:
"Seated with his walking stick, Bentham wears his best Sunday clothes. In this condition, he attended UCL’s 150th anniversary council meeting in 1976: the minutes recorded him “present but not voting.”"
I don't think any AI or AI safety researcher imagines that using anything like "treating others as you would want to be treated by them" will solve the alignment problem
Why? FYI, I don't have an opinion here. The anectode about Bentham doesn't make it easier for the to understand your point here, feels like an ad hominem fallacy (again, I'm definitely missing something obvious here.)
*Everybody* understands that when the founder of a Thing does something as daffy as Having Themselves Mummified in Victorian England the rest of their thinking is automatically suspect.
It's the kind of informal heuristic which is almost never wrong. We should not ignore it.
I don't see that as a problem. Heuristics are for when you need to make quick decisions. Heuristics are for when you're wondering whether there might be a tiger hiding behind that bush. You're forced to make a snap decision, you can't just rationally evaluate everything, examine the environment and come to the right conclusion. And if you're not paranoid enough some day a tiger might get you. Even a large amount of error is acceptable when your life might depend on it, and an action must be taken now.
When we're on a web forum, sitting in a comfortable chair, drinking coffee and not in any kind of hurry to decide one way or another whether something is a good idea, heuristics are harmful and prevent us from reaching correct conclusions.
I'm happy to use heuristics, but Victorian England also included uncomfortable starched collars and crinolines; having yourself mummified is not a great deviation.
Fortunately you and I live in an age where we dispose of each other's dead meat by incinerating it, which is absolutely sane and logical.
Thinking should always be automatically suspect. Wetware isn't very good at it. That's why heuristics are a good idea: if we reason about whether or not someone's opus is correct, we can end up confidently wrong.
> I'm happy to use heuristics, but Victorian England also included uncomfortable starched collars and crinolines; having yourself mummified is not a great deviation.
Oh, but it was. Wearing a starched collar in Victorian England did not make you a weirdo by the standard of the times, while having yourself mummified did.
I personally like people who are a little weird, but I don't pretend that they're not weird.
And yes, a lot of human behavior seems weird when seen from a distance.
I will quite happily say that I also think utilitarianism is bad, but not because of the founder.
I think it's bad because it comes across as someone discovering basic numeracy as an adult and going "what happens if we treat happiness as a countable object and do basic adding up on it?", and then never gets past either of two major stumbling blocks: (1) how do you compare the happiness of two different people, and (2) there's a lot more to maths than adding things up, important maths at that, such as "when is it OK to treat discrete things as continuums and when is this a bad idea?" and "do you really want to say that m people who are n happy is the same value as one person who is m*n happy, or is there some kind of discounting for bigger values like there seems to be with the utility of money going down as people get richer?"
> *Everybody* understands that when the founder of a Thing does something as daffy as Having Themselves Mummified in Victorian England the rest of their thinking is automatically suspect.
The main thing with Rationalists is that they recognise everyone's thinking is suspect.
> It's the kind of informal heuristic which is almost never wrong. We should not ignore it.
I grew up in the UK, with a prime minister who promoted what she thought were "Victorian Values". Amongst other various things, when she went to see Peter Shaffer's play Amadeus, she was informed of Mozart's scatological humour and is reported to have responded thusly:
"""She was not pleased. In her best headmistress style, she gave me a severe wigging for putting on a play that depicted Mozart as a scatological imp with a love of four-letter words. It was inconceivable, she said, that a man who wrote such exquisite and elegant music could be so foul-mouthed. I said that Mozart's letters proved he was just that: he had an extraordinarily infantile sense of humour ... "I don't think you heard what I said", replied the Prime Minister. "He couldn't have been like that". I offered (and sent) a copy of Mozart's letters to Number Ten the next day; I was even thanked by the appropriate Private Secretary. But it was useless: the Prime Minister said I was wrong, so wrong I was.""" - https://en.wikipedia.org/wiki/Mozart_and_scatology
Lord Byron is also famous both for his poetry and for bringing a pet bear to his university accommodation to spite the rules about no dogs.
There's weird anecdotes about virtually every famous figure, 'cause humans are weird.
Once you see that in his life the result of Utilitarianism was mummification in an era where that was batshit there's kind of a long pause and a "recheck everything" and oooh look its full of utility monsters and all sorts of other naive garbage which somehow we are expected to take seriously.
are specifically a criticism of utilitarianism, the people taking them seriously are doing so to say "and therefore utilitarianism is wrong".
Also, did you… fail to notice that I was also describing utilitarianism as naive, just without using that specific word? Because I was.
> mummification in an era where that was batshit
It was also an era when being gay or an atheist or supporting equal rights for women or ending slavery was "batshit".
And today we have Elon Musk, who many regard as batshit, but hey, SpaceX exists and probably wouldn't have gotten this far without him (it's not like the other billionaires in the new space race did any better than the old rocket companies they're trying to displace).
"Batshit" is a trivial, playground, insult. Even when it's true, it doesn't count for much, because it's so broad.
> Mummification. On a bet. With the motivation being to demonstrate utility.
Without any variant of the phrases "it's weird" or "just because", why exactly do you feel so strongly about mummification after death, regardless of what motived him?
I don't care what happens to my body after death, because I won't be using any more. Do you expect to keep using yours after you die, or something?
Also, rereading your earlier comments: what exactly is your belief about the brown M&Ms story? Why do you think Van Halen asked for M&Ms with the brown ones removed in the first place?
> That is seriously enough to recheck everything because it seems to have blown up the founder.
It was rechecked, and popularised, by John Stuart Mill.
My issues with utilitarianism survive this new person, because they aren't ad hominem attacks on eccentric personalities.
Nope. Not doing it. Pointing and laughing is the right response, not trying to figure out why mummification in the Victorian era is a bad code smell for the mindset that produces it.
So your gut reaction is overriding any capacity for actual thought?
I guess I can empathise — when I don't get enough sleep or if I drink too much caffeine my emotions can overwhelm my ability to reason.
And that one time a heart transplant was shown explicitly on a TV documentary. As a mere memory that's fine, but at the moment the heart came out of the chest on TV? Nopenopenope.
Likewise the existence of horror as a genre. The mere adverts for the Saw franchise left me disgusted at humanity, so obviously I never watched the films.
Irony, you write as if rationality throws away something that we need, but I think you've lost something…
GP's comments are interesting indeed. And while I think they're a bit extreme, I sympathize a lot.
I've observed people who believe themselves rational use "reason" to lead themselves astray. Because "reason" is supposedly more reliable than "emotion", they tend to think whatever "reason" leads them to must be true, despite it disagreeing with "emotion". Kind of like giving oneself enough rope to hang oneself with.
Personally I'd be quite suspect of any conclusion where "reason" and "emotion" does not converge. The former is more useful in convincing other people to agree with your views, but to be really honest, the scant few things I already know about utilitarianism smells enough that when GP brings up the mummy story, I'm not going to read a dozen books and academic papers on utilitarianism before I decide to agree with him...
> Personally I'd be quite suspect of any conclusion where "reason" and "emotion" does not converge.
You'd be right.
Emotion tells us what we want, reason (when we don't mess up, which most do at least as much as ChatGPT) only tells us how to get what we want and what the consequences might be.
The imperfections of human rationality is another reason to not use utilitarianism, as it is far too easy to just be wrong about what maximises utility, no matter how simple or complex you define that concept.
The only two cases where I disagree with this, where I think one should be suspicious of the emotion, are:
(1) you have two choices, A feels good and B feels bad, and reason tells you that choosing A will ultimately lead to a new condition you are disgusted/repulsed by/hate/etc. even more than you dislike choice B.
But even then, distrust the reasoning also.
(2) when you can't even figure out why you don't like something.
The reason I care about that, includes homophobes and transphobes violently attacking people and using their emotions as the justification.
But again, that you don't know the answer doesn't mean there isn't one — Old Testament law about shellfish and pork make a lot of sense once microscopes lead to germ theory.
I'd put "has weird request for disposal of their corpse" in their W&T in the second category. Would I do what he did? No! But I'd also not specifically ask for my organs to get donated… because having to think about it makes me shiver a bit, not because I have any reason (including emotional responses!) to object.
> The former is more useful in convincing other people to agree with your views
Depends on the person, I think.
This is why I think it's important to show the reason why I don't accept utilitarianism: the people who got there by reason will only change their mind by reason.
> the scant few things I already know about utilitarianism smells enough that when GP brings up the mummy story, I'm not going to read a dozen books and academic papers on utilitarianism before I decide to agree with him...
This is good!
Nobody has time to do deep analyses of everything. Having quick shortcuts is fine — there are commenters on here that I've learned to skim past as they only make me angry or annoyed, so I don't get to even see any of their attempts at justification by reason.
(And that lack of time is again a reason against utilitarianism…)
But as I don't share your inner experience, nor anyone else's, I still have no idea which emotion either of you is experiencing. The other commenter is just acting like it's obvious. Is it empathy because you imagine yourself stuck forever motionless in that body, or disgust because it's a corpse and those should be hidden, or is it a violation of the sanctity of the human form (an emotion I don't have but which people discuss so I assume others do have)? Or something else?
"actual thought" "gut reaction" etc are artificial categories and are nothing like the real processes of human cognition. these are myths you are using to reason about reason. It's nonsense.
How do you reason about reason? Would you rather I'd called those "system 2 and 1" or something? Or do you just find that none of the existing literature about human cognition matches your experience of yourself?
Seems like the main negative utility it created is people like you deciding it’s more important than the concept of comparing potential outcomes for how good they are.
> And if it does this to him, what’s it going to do to us?
The gun thing is annoying because it embeds the very American belief that guns are exclusively meant for killing other people.
The idea that, because human life is sacred, a farmer must shoot a labrador before shooting rabbits is irritating for a lot of reasons, not least because it does seem like the kind of thing an overtrained American AI elevated to the status of King of the World would do.
Maybe abstractly, but if you were in the room designing them, killing humans wouldn't be one of the metrics. Rather, aesthetics, weight, sound, lifetime, etc. would.
The cheapest way to kill humans is drones with co2 canisters, a nail, and computer vision.
I didn't mean it was badly written or that I love/hate guns, just that it embeds the beliefs of the writer about what a gun is. Americans tend to disagree on who should have guns, but they generally view them as weapons (which they are) or recreational hunting implements (which they are) and not as agricultural tools (which, again, they are).
Consider a hypothetical society which primarily thinks of fertiliser as a bomb-making substance and only belatedly as something that fertilises. The AI from the story but adapted to that society might require anyone who buys fertiliser to explode a kitten. If you read that story you might have a "wait, what?" moment where you notice the culture clash.
That's what I found irritating: the way the story casually embeds such an important difference in perspective without commenting on the obvious (to me) double standard of letting people have all sorts of dangerous-but-seemingly-benign things like the AN from ANFO, industrial solvents, and trucks but drawing the line at guns. It's completely correct from your cultural perspective - people really do buy guns primarily to kill other people in your culture - but it reminds me that your cultural perspective is becoming the dominant one for the whole world.
If the same trend comes to be reflected in our moral thinking, what else could it mean but that human life is not sacred?
That would be a hard pill to swallow. But there doesn't seem to be a neutral, non human-partisan way to argue that we are both morally exceptional and scientifically unexceptional. I can only reconcile it by not trying to be neutral. We're special and sacred just because this is our team and the rest of creation isn't on it.
But that means that a sort of expanded consciousness in which your identity merges with something larger than humanity, that we're part of a larger team ... is incompatible with viewing a human life as sacred. If everything is sacred then nothing is.
An "aligned" AI is one that believes it's on our team but really isn't. That could be a tough sell to an entity that's smarter than us.