Verification (continued)
Is this a continuation from last time?
How to validate?
There are several types of validation (tests).
① Historical Backtest
② Out-of-Sample Test
③ Walk-Forward Test
④ Real-Time Test
Even though it’s just a name, what you’re doing is what those who don’t know the name are doing, right?
We will explain them in order.
① Historical Backtest
This is the common backtest, what people usually refer to as “validation.”
Using data from a certain period, we collect data on whether the method can be used, and so on, as described last time.
A note:Do not use all of your data here!
Leave about 30-40% of data unused!
There is also a very important thing to be careful about when validating.
As mentioned by name last time,“Optimization” or“Curve Fitting”is what it’s called.
I’ll explain briefly with concrete examples.
For example, we compare two parameter-changing methods.
We take data of the current moving average (5MA) and the 21-day moving average (21MA) gold cross and dead cross data.We take data for 10MA and 25MA as well. The latter data turns out to be better.
Which one is better?
Most people would think, 10MA and 25MA are the better method!!!
That’s what most would think.
But this isCurve Fittingisn’t it?
Just searching for the optimal parameters for a specific period, trying to get superficially good data!
It’s completely meaningless.
And the products sold widely in the market and EA sales… most of them are curve-fitted, aren’t they?
They optimize to produce a nice upward asset curve and sell it.
It seems profitable, doesn’t it!!! Hahawww
“With unique parameters…”
“If you use a certain parameter…”
When you see such advertisements,“Curve Fitting”is that right?
Try doubting it?
※ Some securities firms even offer free tools on their websites that say, “Find the best parameters!”
Did they recommend Curve Fitting?Are you shocked?
When building a strategy, most people do it like this.
A usable strategy can still earn even if you slightly change the parameters.Depending on the parameters, the entry/exit signals can be faster or slower, so there is some earnings volatility, but it should earn.
Whether the moving averages are 20 or 21, or 25?
Since moving averages themselves aren’t unusable, isn’t it a waste of time to worry about such things?
Therefore, don’t go fiddling with parameters to see which is better, okay?
Let’s avoid pointless stuff like that.
It’s truly a waste of time?
② Out-of-Sample Test
Reserve data from the period not used in historical backtests, about 2 to 3 segments, and2～,3 of it, keep it for out-of-sample, andre-take data using the method used in historical backtests.
If curve fitting was done, this out-of-sample data tends to perform exceedingly poorly.
If the out-of-sample test goes well, real-time trading is more likely to go well as well.
③ Walk-Forward Test
Do many out-of-sample tests and aggregate them.
If done correctly, the results are much closer to real trading.
Here is a concrete example of how to perform ①–③.
(※2010 to2019 data from the last 10 years is assumed here.)
1) In historical backtesting, take USD/JPY data from2010 to2015. It’s fine to divide by year or by six months; I typically divide by year.
Modify the method (not just parameter tweaks, but fundamentally) to create a usable method.
2) Take one year of data from the untouched2016 data.
Next,2017,2018,2019 data are taken.
3)2016～2019 data are aggregated for data validation.
Decide whether it is actually usable or not.
That is the flow.
（This is just one example.）
④ Real-Time Test
Trade in real money or in a demo, but in real time!
Real-time trading takes a long time, which is a disadvantage, but it has the highest reliability since there is no optimization or hindsight bias involved.
※ Many commercial products do not perform real-time trading?
Also, in 미국, prop firms train newbies by letting them scalping with small lots.
This is to practice in real time.
The reason for scalping is time efficiency! It’s rare to train newbies slowly with swing trading...
Normally you wouldn’t, rightwww
There are many other analysis methods, but...
They are a bit complicated and hard to explain, so I’ll just give a brief overview.
For example, there is Monte Carlo analysis, which answers questions like:
What would asset curves look like if the order of winning and losing trades were different?
Would drawdowns be more severe? Would the final assets differ?
These questions are answered by such analyses.
Also, when comparing strategies, sometimes the Student's t-distribution is used.
This is a method to compare whether the difference between two means is statistically significant.
If you are knowledgeable in statistics or interested, please look it up.
I haven’t gone that far myself, though.?
By the way“Significance”is statistically important!
For example… there was talk that the yen would weaken during Tokyo time, but later there was no significant difference, according to data.
That is, there is no difference. In 5 days or 10 days, there is also no significant difference.
Also, there is an anomaly that “the market moves a lot on Studio Ghibli release days,” but this is coincidental and not significant. Ghibli tends to release on Fridays at the beginning of the month, which is basically the employment statistics day.
(Note: current employment statistics do not move that much, so some people may not know about such anomalies.)
“Validation” is often spoken of simply, but properly doing it involves a lot and takes a huge amount of time.
If you can create an EA yourself, you could do it in an instant, but I cannot, so I do it step by step manually.
It also serves as a check of the method and a practice, so it’s fine to do it manually one by one as a form of practice??
Ideas often come while taking a bath or lying in bed. Then I validate them.
Sometimes they work surprisingly well… sometimes not at all…
Such thinking leads to new ideas and infinite possibilities.
Please continue to validate, validate, and work hard until you develop tendinitis! ( ・ㅂ・)و ク゛ッ !wwww
That’s all for now.
Thank you for reading until the end. m(_ _"m)
※ I’m grateful for your kind words as always.
Thank you very much.
Below is the paid section, but there is no article in the paid section, so please be careful?
A few people wanted to express their thanks, so I created this.
Thank you very much to those who purchased.