The “best” among 89 methods disappeared by 80% during the verification period

Backtests rise steadily, but in forwards they do not follow the same pattern. This is a phenomenon often encountered when validating EAs.

This isn’t that the forwards are abnormal; in many casesthe backtest numbers are structurally higherthan what actually happens. It makes more sense to interpret it this way. I measured the magnitude in a nearby validation example.

Experiment

Using a moving average crossover strategy, I followed a simple procedure. USD/JPY, daily, 2020–2023.

1. Prepare combinations of short-term and long-termin 89 ways.
2. Run all of them on the early data (IS: in-sample period) and select the best performing setting
3. Apply that setting to the later data (OOS: out-of-sample period) as is

OOS is “data after the choice,” so it corresponds to a pseudo- forward test.

Results

Over 80% of performance disappeared.

We did not change the logic. The market did not suddenly shift.The only thing that created this difference is the operation of “choosing the most suitable one from 89 possibilities”.

Why does selecting cause a drop

If you try many configurations, you will inevitably find a setting that perfectly matches random quirks in the past data. That setting is not showcasing reproducible edge; it’s a coincidence limited to that period.

As the number of trials increases, the probability of hitting a “coincidentally good-looking” setting rises. In real EA development, the number of trials can reach thousands or tens of thousands, and the deviation can be even larger than in this example.

What’s important here is not that backtest numbers are lying. The calculations are correct. What’s lacking is information about how many different configurations were considered, making it hard to distinguish skill from luck.

Other factors causing divergence

Beyond overfitting, there are other factors that work in the same direction.

• Differences in cost settings— If spreads and slippage are more favorable in reality than in the testing environment, the backtest appears more advantageous
• Market environment changes— If the validation period is trend-focused, the premise collapses as it enters a range
• Look-ahead— A coding issue that uses information that was not yet known. Backtests become unusually good

All tend to bias the forward-looking results downward toward the backtest side.

4 points to confirm in sales materials

With the above in mind, when assessing EA validation results, pay attention to the following:

1. Is there forward performance?— If only backtests exist, read by factoring in the above divergence
2. Forward period and number of trades— 1–2 months and a few dozen trades are not yet beyond the realm of coincidence
3. Difference between backtest and forward— The smaller the gap, the weaker the chance of overfitting
4. How many parameter variations were tried— The larger the trial count, the more the “best” result can be explained by chance

If forwards are published with sufficient duration and number of trades, and the forward-backtest gap is small, such a performance becomes much easier to interpret than an unknown-high number with no context.

About the author

I have compiled the above validation steps into a free PDF, presented as a fill-in checklist and a 6-question template for sellers, and distributed it on GogoJungle.

• EA sales page 5-minute checklist [Free](0 yen, A5, 8 pages)

The method to statistically adjust “which among the many variations performs best” (Deflated Sharpe Ratio) and the 89-variation validation procedure described here are covered in more detail in the paid version.

• Is a Sharpe Ratio of 4.51 high?

All of these are PDFs created by the author. They do not constitute the purchase advice for any particular EA or product, and the validation examples in the text are designed as fictional strategies for instructional materials. Please make final investment decisions at your own risk.