Embed Notice

@cthos@mastodon.cthos.dev @xgranade@wandering.shop This is an issue that came up a lot during simulations: you know, you can calculate the confidence intervals of your distribution that you're sampling, but it's a measure about the statistical estimator you used on your distribution, not necessarily your sampling process.

If you knew the ground truth answer, that could tell you whether your sampling is good. If your statistical estimate is far away from the ground truth, your measures of confidence can give you information about whether it's random noise, or whether you have a genuine problem with something but without that ground truth it's not sufficient to tell you whether your sampling process is good or accurate (like, it might just say, "your estimates are very consistent", or in this case, "our estimates of how many clickthroughs resulted in sales are very consistent" or inconsistent or whatever. And that number might seem high, but again, it's only a measure of the sampling process). And it's an industry that has a vested interest in not answering that question.

Anyway... sorry, that was a bit of a ramble. I had to deal with this a lot in grad school (basically I would end up with answers with a super high degree of confidence that didn't agree with what I should have been getting and it sucked) since simulations are a huge sampling problem.

also I know I am preaching to the choir here. Just sort of writing as I'm thinking about this since it's one I spent a lot of time on in the past... much to my chagrin.