I didn’t see it linked, but a paper by my coworker at Netflix, Michael Lindon, seems relevant: Anytime-Valid Inference for Multinomial Count Data. https://arxiv.org/abs/2011.03567

Thanks for sharing - will add to my reading list!

If you’re worried about having too little data then A/B tests are the wrong tool for the job.

It’s better to just use ML scorers and rankers.

Can you elaborate on this?

How would one convert a classic conversion-rate optimization task to use an ML scorer? Would that mean converting it into a classification problem?

It becomes a regression problem. You use ML to estimate the conversion rate for each item. You still have to do feature engineering, but fast regressors like XGBoost make this much easier. https://improve.ai is one such scorer/ranker built on XGBoost.