Why are you asking a token generator to explain its prior output?
You are proceeding from a false premise. You are not getting an explanation of its prior output. You are getting a series of tokens that forms a response to your query, same as it did for the initial answer. Now you've asked it why it's wrong, so the text conforms to that request, but that doesn't change the fundamental nature of the software you're interacting with.
This is your mistake right here. It doesn't think. It's a text generator. It can no more think about what year it is than Swiftkey on your phone "thinks" what year it is when you type
I'm as bearish as anyone on the current AI hype, but this particular ship has sailed. Research is revealing these humongous neural networks of weights for next token prediction to exhibit underlying structures that seem to map in some way to a form of knowledge about the world that is, however imperfectly, extracted from all the text they're trained on.
Arguing that this is meaningfully different from what happens in our own brains is not something I would personally be comfortable with.
> Research is revealing these humongous neural networks of weights for next token prediction to exhibit underlying structures that seem to map in some way to a form of knowledge about the world that is
[[citation needed]]
I am sorry but I need exceptionally strong proof of that statement. I think it is totally untrue.
> Why are you asking a token generator to explain its prior output?
I swear I'm not. I'm trying to get it to fix the bug. I know it's a stateless slop generator, but I need it to be an obedient stateless slop generator.
The "magic words" I'm trying to come up with are whatever will prompt it to see the bug at all. I've tried standing instructions demanding that it simply not ever question me about whether a bug I've mentioned exists, because I'd rather it "fix" a bug that doesn't exist (so it can fail fast and I can realize I'm the dumb one) than fall into this loop of trying to argue it into doing what I say.
edit: that tactic does not work, even with much repetition, all caps, and many exclamation points. Eventually the instructions read like I'm having a mental breakdown.
You still seem to be expecting some degree of thought and understanding from these tools.
They generate tokens. The output has a probabilistic relationship to the established context and prompts, plus whatever prompting is happening as you interact with the model.
There is no understanding of "don't do [thing]". Sometimes, you can get something closer to what you wanted by putting stuff like that in the prompt. But it's still probabilistic token generation. It's not interpreting that as a literal command to not do the thing. It has that command in its context now, and maybe that changes the output. Maybe it changes in a useful direction, maybe it doesn't. But it's not going to be treated as a literal command because the model does not have the capability to do so. Phrasing it differently doesn't change the fundamentals.
You are proceeding from a false premise. You are not getting an explanation of its prior output. You are getting a series of tokens that forms a response to your query, same as it did for the initial answer. Now you've asked it why it's wrong, so the text conforms to that request, but that doesn't change the fundamental nature of the software you're interacting with.