"Stopped loss" (1/13) Investment AI development Struggle Diary
This is also something I posted about on X, and I will hold a retrospective meeting regarding this matter.
https://x.com/firstpips_peace/status/1878652789309764010
What were we doing?
To determine the architecture of a time-series learning model, we were performing a grid search using CPCV.
The training data consists of daily data from 2011 onward, so it isn’t a very large dataset.
Therefore, we estimated it would be completed within a few hours.
The next morning, it was still running, so I thought it might take a little longer and
added the day’s tasks before going home.
Is it still running? Why
The next day, is it still running?
When I calculated it, the number of grid patterns was 4,096, and with CPCV having 3 folds, it was 4,096*3 = 12,288 training runs.
Additionally, the training had a callback set for 100 iterations or early stopping after 10 iterations.
Even if one training run takes 1 minute, that’s 12,000 minutes = 200 hours.
It would take almost 10 days, right?
So we decided to stop.
Amen to that.