Being able to say the word "should" is something I don't want to give up. You can say it's just a problem of semantics and I should admit to being a nihilist, but I think the use of language is important, because it's how people get things done. I say "should" because I want to be able to judge an agent on how good it is on its own terms. For example, if someone tells me their dream for the future, I want to be able to tell them they "should" work out a reasonable plan, assuming they haven't done so already.
I haven't read the sequences (or "Rationality: From AI to Zombies" I think the old version was called). I didn't get my definition from LessWrong, I think it's partially from Bruno de Finetti and partially from old AI literature, but I read it from the standard recommended books.
> I say "should" because I want to be able to judge an agent on how good it is on its own terms.
Again, not standard use of language, but unobjectionable once it's explained. (That's also how I like to use the word, but it's not often understood by those around me.)
> I haven't read the sequences
Oh, then pretend I referenced later Wittgenstein instead. (He made the point better.)
I haven't read the sequences (or "Rationality: From AI to Zombies" I think the old version was called). I didn't get my definition from LessWrong, I think it's partially from Bruno de Finetti and partially from old AI literature, but I read it from the standard recommended books.