That could be post-hoc reasoning, though. It would be interesting to pre-register your hypotheses, or see whether you could tell bandit outcomes from random ones.
This is literally the logical fallacy. You could get lucky. Maybe you have obvious gains to chase. But bad logical arguments are bad because they never work forever. They are corrupted heuristics that can get you in trouble without critical thinking.
Edit: added in forever. Phone dropped some wording I originally had. I think.
Incremental wins can still lead to dead ends. My phrasing was off in my post. I meant to say that the fallacies aren't that the tactics never work. Just that they can stop working without you really realizing it. A heuristics that can lead you down a dead end.
By all means, keep doing it if it is working for you. But don't confuse it as good advice. And stay vigilant.
Products exist in human reality not some science paper. There are no absolute truths, everything dead-ends eventually. It’s like trying to prove that one set of genes is better than another for future survival - an impossible task.
This belies a belief that science doesn't reflect the real world. It absolutely does.
Again, it may be working in your case. Argument to authority can go a long way. Even ad hom attacks often exist due to a "smell" of the person speaking. It is not, however, logically sound and can easily lead to unsupportable positions.
So, take care. And realize that a lot of the damage of poor practices may be tangential. For example, a belief that the real world can not be described by science.
Isn't this problem also an issue when people talk about transferring what they learn from one test to another test? That is frequently cited as a benefit of A/B testing.
