Was reading back through the old Neural Net Lottery Picker thread and it got locked before the most interesting part really got going. The MCMC/Markov discussion in there was the best thread I've found on here, so starting a fresh one to keep it alive.
The point that stuck with me: someone noted they found no real advantage running Markov on draw history because the independence of each draw keeps failing to reject "it's just a discrete uniform distribution." That matches what I keep hitting - the models will happily find structure, but it doesn't survive once you hold back a test set. Classic overfit.
Which lands on the thing I think actually matters more than the model: the backtest. Before trusting any of this, I've started running it backwards over full draw history to see how often the real winning line would've actually survived the filter or ranked well under the model. More often than not, the fancy approach does no better than flat random - which is its own kind of useful to know.
So for the people in here actually building this stuff:
- When you test an ML or MCMC approach, do you hold back an out-of-sample set, or judge it on the full history?
- Has anyone gotten any method to beat uniform-random out-of-sample, even slightly?
- The IQR hot/cold/neutral classification idea someone raised — anyone actually backtested whether the "hot" bucket outperforms?
Not looking to relitigate "it's all random" — I know where that lands. More interested in how you separate a method that genuinely does something from one that just looks impressive on the data it was built on.
