Uncover the EA's edge that cannot be seen in backtests, through out-of-sample tests and walk-forward tests
------------------------------------------------
Empathy level 99.9%!
What is the FX manga that’s a hot topic now??
↓
------------------------------------------------
◆Introduction◆
Most trading EAs come with a backtest.
But this backtest can be quite tricky.
Backtests feel likestatistical proof of an edgein reality, that alone cannot measure an EA’s edge.
For consumer EAs, backtesting being good is obviously expected…
andwe’ve written similar stories in another articleand I’d like to write a sequel now ♪
Keep in mind that easily creating an upward-sloping backtest curve is simple
“Backtests being good for consumer EAs is natural.”
This is something I can say again and again.
”An EA that doesn’t even include a backtest”
is, of course, out of the question.
However,
having an excellent backtest does not prove the EA’s performanceeither.
Why is that??
Because a backtest that seems superb may beover-optimizedand thus not robust.
With over-optimization, you can easily make a backtest curve rise steeply to the right.
Well, that’s somewhat a developer’s concern, but I want buyers to know this too.
Anyway,
“Anyone can make an upward-sloping backtest curve.”
Please keep that in mind.
“Then what is the point of backtesting…?”
This is a reality we may need to accept (^^;)
How to uncover the true edge of an EA
So, what is the method to know the true edge of an EA?
Backtesting alone is uselessas you’ve already understood, right?
In fact, many EAs show superb backtest curves but have no forward-test reproducibility, and many people have fallen for such baseless claims.
That is the strongest evidence.
However, there is a way to reveal the true edge of an EA.
It is a strict reproducibility test,
- Out-of-sample testing
- Walk-forward testing
to be conducted.
However, this is a test that cannot be done if the logic is a black box. right? (lol)
“Huh—If the logic is a black box, how can you reveal the EA’s edge?”
Most people probably think so.
・
・
・
・
Yes, that’s correct (;・∀・)
Until the logic is known, the only trustworthy edge data is forward performance.
Forward-running EAs that perform well can be considered to have an edge.That’s right.
Forward testing is essentially the same as out-of-sample testing.
Let’s look at what that means in detail.
What is Out-of-Sample Testing??
In strategy creation,Out-of-Sample Testingmeanstesting whether parameters optimized during a certain period still apply outside that optimization periodas a method.
It tests strategy robustness and reproducibility.
For example, you optimize parameters over a 10-year period from 2005 to 2015, then backtest over a different period outside that optimization window.
As shown above, the optimization period is called the “in-sample period,” and the period outside it is called the “out-of-sample period.”
Of course, it’s natural to tune to profit in the in-sample period.Right.
The problem is whether the strategy determined in the in-sample period can also work outside the in-sample period.
That’s the essence.
Forward running is exactly out-of-sample testing.
The backtest period shown for selling is the in-sample period, and actual forward trading tests whether it holds after that period.
If the EA’s forward performance does not reproduce the backtest results, it’s likely to perform poorly in out-of-sample testing as well.
What is Walk-Forward Testing??
Walk-forward testing is a series of consecutive out-of-sample tests to check reproducibility.
Out-of-sample testing selects an in-sample and out-of-sample period arbitrarily, but
“Which period to choose?”
is a dilemma.
So we shift through all periods a bit at a time, performing multiple out-of-sample tests.
As shown above,by repeatedly performing strict out-of-sample tests, the strategy’s robustness and reproducibility become clearer.
We perform walk-forward testing across the entire backtest period, joining the out-of-sample periods to view the equity curve.
Only a very few EAs pass this higher bar.
Backtesting is the Optimal Solution!
Backtest curves are naturally upward-sloping because they’re derived from optimal parameters within the backtest period.
There is a catch, though.
Even strategies that only work with a very limited set of parameters will show an upward backtest curve if you derive the optimal solution.
However, such a strategy will not work in walk-forward testing.
No matter how high the backtest PF is, or how well it withstands high spreads, or the drawdown is small, or the curve is neatly rising,
these are optimal solutions and are likely not applicable in the future.
- Out-of-sample testing
- Walk-forward testing
andEA that has not undergone strict reproducibility tests is dangerousas well!
Forward testing for about half a year isn’t trustworthy
Often you’ll see
“It’s profitable in forward testing too!”
and some EAs advertise forward periods of three months to six months.
The results of forward testing are the same as out-of-sample testing, so they are data worth trusting, buta six-month or so upward trend can be a fluke in many cases(^^;)
So, when assessing forward periods, you should ideally haveat least one year of operationor more.
Three years or more would be safer.
By that point, screening becomes quite rigorous though.
There aren’t many long-running EAs that stay profitable for 3+ years.
What you can do as someone choosing an EA
When selecting an EA, you want to pick a highly competitive one with universal edge, right?
So,
“Develop your ability to evaluate backtests to choose good EAs!”
This is not very helpful and likely not meaningful, as you’ve understood.
Backtesting mainly reveals characteristics of the EA.
Still, knowing its characteristics is important, so backtesting remains useful, but…
So, what can you do as someone who buys via a logic black box EA?
What works is
“Ask the EA author a question.”
Specifically,
“Have you verified the logic’s reproducibility with walk-forward testing during the development process?”
Such a question will yield responses, though many authors may not respond (laugh).
“Annoying guy asked questions. Ignore!”
Is that what you’ll get? (^^;)
“We perform walk-forward testing to confirm the logic’s robustness and reproducibility.”
If that’s true and not a lie, then that EA has value.
But people can say they perform walk-forward testing, and even if they do not, they can argue and justify it in various ways.
In short,you should judge how trustworthy the author is.If the author has a platform (blog or newsletter), read it to assess whether they have a proper mindset.
EA authors who keep their logic as a black box generally hide the logic-related parameters.
Therefore, out-of-sample and walk-forward tests are often impossible, and it’s hard for buyers to rigorously assess robustness.
Thus, to gauge the edge of a consumer EA, you must look at forward performance and the author’s intent.
Ideally, you’d use an EA whose logic you understand.
But if you must use a commercially sold EA, there are cases where you cannot, so you may end up concluding that it’s faster to develop your own logic (^^;).
Be careful of people who are only skilled as programmers
EAs are built on code.
Therefore, from programmers’ backgrounds, some people enter the algo-trading world with the belief that
“EA can be profitable.”
However, programming and strategy development ideas are entirely different things.
“A programmer who makes a perfectly coded EA but doesn’t understand market reality”
releases EA products confidently.
Being a capable programmer is not enough if you don’t understand the market.
BBP is a strategy that passed strict reproducibility tests
You’ve probably understood the importance of strict reproducibility tests such as out-of-sample testing and walk-forward testing by now.
Then the question becomes
“Does the BBP released by FX Ryushou conduct proper reproducibility tests like walk-forward testing?”
This is a question you can’t avoid.
Of course BBP has properly conducted strict reproducibility tests such as out-of-sample testing and walk-forward testing, so please rest assured＾＾
BBP has been a long-run EA since its release in 2015, continuing to generate profits despite various twists and turns, so by that point it can be seen as having passed out-of-sample and walk-forward tests.
However, when I first developed BBP, I didn’t know about out-of-sample testing or walk-forward testing (laugh).
Still, intuitively, BBP felt strong.
Three years after release, with forward results continuing, performing out-of-sample and walk-forward tests, the result was that it remained excellent as well (laugh).
BBP, from its development, didn’t set a basic trading rule and then iterate parameters; it produced an upward-sloping equity curve in one shot.
This isthe secret to making a good strategy.
Optimization of an EA should not be done many times, and walk-forward testing should not be done many times either.
If you can get a good backtest curve in one try but fail in walk-forward testing, discard that logic.BBP is a logic that has succeeded in a one-shot manner, so its reproducibility is guaranteed.
Of course there are ups and downs, but I believe it has long-term viability as a strategy.
I’ll discuss BBP’s walk-forward testing in more detail in another article.
I’d like to prove BBP’s edge from a more scientific perspective!
There are many FX information products that claim superiority without any scientific basisand that’s insulting to consumers.
BBP’s reproducibility, which stands apart from such products that lack any scientific grounding, will continue to be monitored by the developer＾＾
