cross validation

This is what happens to people have have made a successful CV (ICR - Identifying Age-Related Conditions):

“I woke up today with my teammate congrats and I got intrigued as to what place we got? I burst out laughing when I saw it was 2nd place, and the funniest thing is my solution is just CV - no probing, no nothing”

https://www.kaggle.com/competitions/g-research-crypto-forecasting/discussion/323098 - An easy way to see if your CV scores have too much variance is to look at a plot of CV score vs. a parameter you’re tuning. A good plot will usually be smooth, with a knee and a plateau, or maybe a peak or a valley. If the plot looks too noisy to clearly see those things, then try it again, with different seeds, and see if you get different results. If you do, then your CV results are too noisy. You can try using more folds, or running CV several times with different random seeds and averaging.)
I also think that from my experience:
- If you do something that SHOULD improve the CV, but it worsens, your CV isn’t strong enough
  - e.g. I cleaned my data, but my CV worsened

Note: Doing the right thing could mean getting a worse CV score (e.g. remove easy examples)

forward chaining cross validation is pretty popular
hold-out cross validation also works where you only validate on the last bit of data
- although I’m afraid that you could be overfitting to the validation set (even if you run the validation multiple times with diff model seeds)
  - because you aren’t looking for trends in other time periods

🏖️ Kaggle Solutions