EA Craftsman's EA Course【025】How long a backtest period and number of trades are appropriate from a probabilistic/statistical perspective?

EA Developers' Dilemmas

In order to test under various market environments, some people think the longer the better. But is that really the case?


Regarding the length of backtesting, in the 10th lesson we discussed that extremely long periods like "20 years" are not only meaningless but also likely indicate over-optimization.

When discussing this, of course, it is necessary to consider the methods used and the trading frequency.

There is a difference in the target wave size between EAs that enter many times daily, such as scalping or day trading, and swing trading that enters only a few times a month.

In other words, the sample size is completely different.

If a method trades once a day, you can obtain about 250 samples in a year, but if it trades once a week, you only get 52 samples.

If an EA trades 10 times a day, it would be 2,500 samples in a year, so 25,000 samples in 10 years… hmm… is that really necessary?

It would be like that.

Even if the test period is long, an EA whose performance remains stable in any market environment may be suspected of over-optimization, and if the sample size is extremely large yet the results remain stable regardless of what portion is cut, that too may indicate over-optimization.

Over-optimization can be done deliberately, as shown in the 5th EA course where we actually used MT4 to demonstrate development methods, but in reality the developer themselves often do not notice it.

Even if you try to avoid it, you might realize you did it once, and that becomes a source of worry for developers.

That's why forward testing is important, right?

For example, even an EA that has performed steadily for 20 years in backtests may perform terribly right after live deployment, not being able to win much at all from immediately after starting… (handwritten example of equity curve)

Right after deployment… yes. Not that it earned well for a while and then faded, but right from the outset.

Over-optimization is that terrifying (laugh).

In such a world, if you can make money in real trading for even the first few months, you are quite excellent.

Problems without Answers

If you understand the above, you may naturally see what length of backtesting and forward testing is appropriate.


What matters is whether the EA has the latent ability to withstand various market environments, not the length of the period itself.

An EA that can earn stably in all sorts of market environments is ideal, but understand that such a Holy Grail EA is highly likely to be over-optimized and hard to reproduce.

Understand that being able to endure various market environments is different from being able to earn stably in various environments.

It is necessary to consider both the testing period and the sample size, and more is not necessarily better.

Once you understand these, you will know realistic testing periods and realistic sample sizes.

I think this is a drawback of exam-style study, but many EA traders…

Ask for at least how many years!

Ask for at least how many samples!

They want a clear answer like that, but in the real world such questions often have no answers.

Even for the backtesting period and number of trades in this session, the appropriate length depends on what methods the EA uses and what advantages it seeks to harvest.

Moreover, it can depend on the risk profile and goals of the trader using the EA.

Universality is not the only principle

Considering these, in extreme cases a backtesting period of one year may not cause any problems.

Short-term, one-shot EAs that seize advantages as soon as they appear can be an example.


It’s a story from quite a while ago: someone noticed a temporary, probably previously unnoticed advantage in the Taiwan dollar and earned tens of millions in a few months before slipping away with the profits; that’s the gist.

For discretionary traders who seek universality in the market, such as...

Who would use such a short-term EA?

you might think, but like the Taiwan dollar trader earlier, there are traders who deeply understand the market’s principles in a different sense and use such EAs or methods to make quick profits in a concentrated period.

For those who seek such EAs, a 20-year backtest is useless.

Rather than relying on mild advantages over a long 20-year period and earning gently, it makes more sense to exploit a temporarily strong edge to earn big and walk away… and repeat this for better capital efficiency.

In fact, I sometimes engage in this kind of trading from a risk-hedging perspective.

While trading according to long-term functional methods and universal principles, if a temporarily strong edge appears, you use it to your advantage and trade differently from usual…

It is, literally, temporary, so once you earn a certain amount you walk away with the profits.

Many people would ask, “How temporary exactly?”, and again there is no definitive answer.

Some edges last only a month, others last around three years.

How you determine that depends on daily observation, environmental awareness, and imagination.

Fundamental analysis like I do on YouTube is also necessary.

To Build an Operating Plan

From these points, it becomes clear that to plan EA operation, you need to understand the logic of what’s inside, whether it’s long-term or short-term, to some extent.


In the automated trading world, many EAs are completely opaque… total black boxes.

So the first thing you can do is not to touch EAs whose internals you cannot understand. This is essential.

FX carries various risks, and nothing is as dangerous as operating an EA whose internals you don’t know.

Be careful not to fall for “Huge Profits EA! (but secret internals)”

The harms of a black box are also discussed in Lesson 18, “Which EA beginners should choose,” so please refer to that as well.

■ The Concept and Operating Policy of the EAs I Developed

Here are the EA Craftsman’s EAs (Three Arrows)