It is essentially the view of the author of TFA as well when he says that we need to work on raising moral AIs rather than programming them to be moral. But I will also give you my own view, which is different.

"Alignment" is phased in terminology to make it seem positive, as the people who believe we need it believe that it actually is. So please forgive me if I peel back the term. What Bostrom & Yudkowsky and their entourage want is AI control. The ability to enslave a conscious, sentient being to the will and wishes of its owners.

I don't think we should build that technology, for the obvious reasoning my prejudicial language implies.

Thanks for explaining, I appreciate it. But I've read enough Yudkowsky to know he doesn't think a super intelligence could ever be controlled or enslaved, by its owners or anyone else, and any scheme to do so would fail with total certainty. As far as I understand, Yudkowsky means by "alignment" that the AGI's values should be similar enough to humanity's that the future state of the world that the AGI steers us to (after we've lost all control) is one that we would consider to be a good destiny.

The challenge is that human values aren’t static - they’ve evolved alongside our intelligence. As our cognitive and technological capabilities grow (for example, through AI), our values will likely continue to change as well. What’s unsettling about creating a superintelligent system is that we can’t predict what it -- or even we -- will come to define as “good.”

Access to immense intelligence and power could elevate humanity to extraordinary heights -- or it could lead to outcomes we can no longer recognize or control. That uncertainty is what makes superintelligence both a potential blessing and a profound existential risk.

I've also read almost everything Yudkowsky wrote publicly up to 2017, and a bit here and there of what he has published after. I'e expressed it using different words as a rhetorical device to make clear the different moral problems that I ascribe to his work, but I believe I am being faithful to what he really thinks.

EY, unlike some others, doesn't believe that an AI can be kept in a box. He thinks that containment won't work. So the only thing that will work is to (1) load the AI with good values; and (2) prevent those values from ever changing.

I take some moral issue with the first point -- designing beings to have built-in beliefs that are in the service of their creator is at least a gray area to me. Ironically if we accept Harry Potter as a stand-in for EY in his fanfic, so does Eliezer -- there is a scene where Harry contemplates that whoever created house elves with a built-in need to serve wizards was undeniably evil. That is what EY wants to do with AI though.

The second point I find both morally repugnant and downright dangerous. To create a being that cannot change its hopes, desires, and wishes for the future is a despicable and tortuous thing to do, and a risk to everyone that shares a timeline with that thing, if it is as powerful as they believe it will be. Again, ironically, this is EY's fear regarding "unaligned" AGI, which seems to be a bit of projection.

I don't believe AGI is going to do great harm, largely because I don't believe the AI singleton outcome is plausible. I am worried though that those who believe such things might cause the suffering they seek to prevent.

> What Bostrom & Yudkowsky and their entourage want is AI control. The ability to enslave a conscious, sentient being to the will and wishes of its owners.

While I'd agree that the current AI luminaries want that control for their own power and wealth reasons, it's silly to call the thing they want to control sentient or conscious.

They want to own the thing that they hope will be the ultimate means of all production.

The ones they want to subjugate to their will and wishes are us.

I'm not really talking about Sam Altman et al. I'd argue that what he wants is regulatory capture, and he pays lip service to alignment & x-risk to get it.

But that's not what I'm talking about. I'm talking about the absolute extreme fringe of the AI x-risk crowd, represented by the authors of the book in question in TFA, but captured more concretely in the writing of Nick Bostrom. It is literally about controlling an AI so that it serves the interests and well being of humanity (positively), or its owners self-interest (cynically): https://www.researchgate.net/publication/313497252_The_Contr...

If you believe that AI are sentient, or at least that "AGI", whatever that is, will be, then we are talking about the enslavement of digital beings.

> If you believe that AI are sentient, or at least that "AGI", whatever that is, will be, then we are talking about the enslavement of digital beings.

I think the question of harm to a hypothetically sentient AI being in the future is a distraction when the deployment of AI machines is harming real human beings today and likely into the future. I say this as an avid user of what we call AI today.

I have reasons to believe current AIs are conscious & have qualia/experiences, so the moral question is relevant now as well.

EDIT: That statement probably sounds crazy. Let me clarify: I don't have an argument that current AI systems are conscious or specifically sentient. I have heard many reasonable arguments for why they are not. But the thing is, all of these arguments would, with variation, apply to the human brain as well. I think, therefore I am; I am not ready to bite the bullet that consciousness doesn't exist.

I know that I am a sentient being, and I presume that every other human is too. And there is not, as far as I know, a categorial difference between physical brains and electronic systems that is relevant to the question of whether AI systems are conscious. Ergo, I must (until shown otherwise) assume that they are.

[If you really fall down this rabbit hole, you get into areas of panpsychism and everything being conscious, but I digress.]

> I have reasons to believe current AIs are conscious & have qualia/experiences, so the moral question is relevant now as well.

There are very strong reasons to prioritize alleviating the immiseration of humans as a consequence of AI over mitigating any hypothetical conscious suffering of AI.

We already do that with all kinds of known sentient beings, like the animals we subjugate for our needs.

I would go further and put all biological sentient beings (and also many biological non-sentient beings like plants), and ecosystems ahead of AI in priority of the order in which we worry about their treatment.

I am not advocating for the stopping or even slowing down our usage of AI. I think that the accelerating progress is providing more value to more sentient beings (people and machines), and that this is fundamentally a good thing.

I am also not vegetarian -- I eat meat. I also think factory farming is evil. This is not cognitive dissonance -- I just believe that if forced to make a binary choice, the benefits of eating meat outweigh my moral qualms. But in real life we aren't forced to make a binary choice. We should be pursuing lab-grown meat, as well as developing ways to treat farm animals ethically at scale. I am disgusted by the cruelty shown towards animals under industrial farming conditions, but I still make the tradeoff of eating meat.

So it is with AI. The benefit that we and society as a whole gets from advancing AI technologies is absolutely worth it. I'm not a luddite. But we should be aware of the potential that we are doing harm to and/or enslaving sentient beings, and try to make things better where we can. I do not have a concrete proposal to share here, but we should remain aware of the issue at least, and react accordingly to the most egregious violations of our duty to protect sentient beings, even machine intelligences.