How far can you trust backtests for EA? Let's organize how to read the results

“In backtesting, it rose so neatly to the right, but when I actually ran it, the results were completely different.”
When you’re involved with EAs, you’ve probably heard such stories, right? Actually, I’ve had that kind of experience too.
Backtesting is a reliable tool when creating or selecting an EA. But if you use it incorrectly, it can become a trap where you “feel relieved just by looking at the results.”
This time, including my own mistakes, I’ll carefully整理 the question of how to read backtest results, including how to interpret figures like PF (Profit Factor) and other numbers.
It’s a slightly longer column, but please join me.

1. What backtests can teach you and what they cannot

Backtesting uses historical price data to test “if these rules were followed, what would have happened.”
It’s very useful for comparing EA behavior and settings, and for checking that results aren’t excessively artificial.
However, this is still a calculation based on past conditions. Please keep separate from actual live performance.
And there are things backtesting cannot tell you: what the market will do in the future, whether orders will pass in the account you actually use, and so on. These cannot be understood by just looking at the test screen.

Points

• Grasp the trading behavior and trend of movement
• See how results change when you alter settings
• Look for markets that are difficult or periods when equity drops sharply
• Don’t conclude “the future will be the same” just from the results
• Don’t treat test results and real trading performance as the same

2. First numbers you notice: How to read PF (Profit Factor)

When you open the test results screen, the first thing that catches your eye is likely PF (Profit Factor).
PF is calculated as “Total Profit ÷ Total Loss.” It shows how many times more profit you had than loss.

Calculation illustration

• Total profit 200, total loss 100 → PF 2.0
• Total profit and total loss are equal → PF 1.0 (break-even)
• If total losses are larger → PF below 1.0

This is just an illustration to understand the calculation and not actual performance.
In rough terms, PF above 1.0 means profits exceeded losses during the test period.
However, don’t be reassured just by PF alone, please.

Notes when looking at PF

1. With few trades, PF can swing greatly by chance
2. Be wary if PF looks high due to a few large profits
3. If PF is very high, consider whether settings were over-fitted to past data
4. If you change trading costs, how much PF drops (closer to 1.0 means more sensitive to costs)
5. PF alone doesn’t tell you how much capital was drawn down (drawdown)

PF is a very intuitive number. But deciding “this EA looks good” from PF alone can cause you to miss other important parts.

3. Other numbers you should look at besides PF

The test results screen shows many numbers. You don’t need to understand all of them perfectly. First, focus on the following to see results in three dimensions.
(Display names may vary slightly depending on tools and environments)

Numbers to look at

• Maximum drawdown: how far equity fell from its peak. Useful for considering whether you can tolerate this drop. I recommend looking at both amount and percentage. Note this number isn’t a guarantee of a bad EA (more on this later)
• Expectancy (expected value): average profit or loss per trade. If this is small, small changes in costs can push you toward a loss
• Total number of trades: small sample size increases susceptibility to luck
• Win rate and the balance between profits and losses: high win rate but large single losses can still lead to overall loss. Don’t judge by win rate alone
• Longest losing streak: how long losses might continue. Helps prepare mentally
• Recovery Factor: how many times the max drawdown the profits recover. An indicator of how much you can recover
• Shape of the profit/loss graph: whether it grows smoothly or has sharp movements in certain periods. Graphs can reveal patterns not obvious in numbers

The important thing is not to judge by a single number, but to look at several numbers together.
For example, even if PF is high, if drawdown is large or the number of trades is low, the impression changes a lot.

When looking at drawdown, be mindful of compounding as well.
With an account that compounds (increases trading size as funds grow), the amount of drawdown grows in dollar terms as the balance increases. In other words, as funds increase, drawdown amount tends to rise accordingly.
Therefore, a large drawdown does not necessarily mean a bad EA.
Also, looking only at the “maximum drawdown amount” in test results can mislead you toward the large amounts seen later. Check as a percentage as well, and consider whether you can withstand the drop when funds grow.
Furthermore, fixed-amount trading tests and compounding tests show drawdown differently. It’s helpful to check under which conditions the results were produced.

How to read drawdown

1. Look at it in both amount and percentage
2. Check whether the test used fixed lot size or compounding (increasing lots with funds)
3. Separate periods when funds were low from periods after growth
4. Compare with the same period’s profits (Recovery Factor is also a reference)
5. Consider whether your funds and psychology can endure that drop

4. Confirm price data and test conditions

Even the same EA can yield different results if you use different price data or test conditions.
For example, how price movements are reproduced at small scales, and whether trade costs are set close to real conditions, are important to check.
If you don’t know the test conditions and only look at results, it’s hard to judge what scenario is being replicated.

What to verify concretely

1. Which currency pair or instrument and time frame were used in the test
2. What period of price data was used
3. How tick data (very fine price movements) is reproduced
4. Whether spreads and other costs are included
5. Whether there are gaps or anomalies in the data or settings used

5. My failure story: differences between testing environment and real-use environment

Here I’ll share my failure story.
I was backtesting in an environment with very low spreads from an FX broker.
When I actually started live trading with a slightly different FX broker, the results were completely different from the test.
Even though the EA logic was the same.

The per-trade spread difference might seem small, but as trades accumulate, the difference stacks up. In particular, EAs that trade often or make small per-trade profits are more susceptible to this effect.
Test results are only the results within the tested environment. This experience underscored that for me.

Besides spreads, differences in commissions, slippage (orders executed at prices different from requested), and broker price differences can change results.

What to confirm before testing

• Spread on the actual account you’ll use (typical range and time of day when spreads widen)
• Whether there are commissions
• What the spread settings were in the test
• Order conditions during volatile events like earnings announcements
• If the test broker and actual broker differ, whether prices and times align
• Whether testing with spreads wider than actual still yields robust results

6. Look at whether you’ve over-tuned the settings

If you repeatedly adjust settings to make past results look better, the results may fit past movements perfectly but fail in other periods.
That is, overfitting to past data.
PF being unusually high is a red flag. Not only the numerical performance but also how results respond when you slightly alter settings should be checked to judge calmly.

Concrete checks

1. Can you explain why you chose that setting?
2. Is there a short period that disproportionately boosts the overall result?
3. Do results worsen suddenly when you tweak the settings slightly?
4. Are you also watching the periods of losses and capital drawdowns, not just profits?
5. After trying many settings, are you selecting only the best outcome?

7. Verify gradually even outside testing

Don’t stop at backtesting. Check data from periods you didn’t test to see whether you’re overfitting to past data.
Also, running in a demo account helps confirm order and stop behavior.
Still, future performance is unknown, and live trading can incur losses due to market conditions and trading environment differences.

Points

• Separate the period used to decide settings from the period used for confirmation
• Check whether the same trends persist in the confirmation period
• In a demo environment, verify orders and executions behave as expected
• When using a real account, prepare funds considering potential losses
• Record the testing conditions and confirmations so you can review later

Summary: Backtesting is one of several clues

Backtesting is a clue for investigating an EA’s past movements.
Don’t decide based solely on clear numbers like PF; also consider drawdown (both amount and percentage) and the number of trades in combination.
And check for differences in data and costs, and differences between testing and real environments, and avoid over-tuning. Also evaluate in periods beyond the test and in a demo environment step by step.
Even if results look favorable, they do not guarantee future profits or safety. Whether to use an EA should be your own decision, considering the possibility of losses.

Disclaimer

This article is for information purposes and not investment advice. Operating an EA carries the risk of capital loss. Past results do not guarantee future performance. Please make decisions at your own judgment and responsibility.