“A winning EA” can be created easily. A trustworthy EA is difficult.
# The "winning Expert Advisor" can be made easily. Trusting a reliable EA is the hard part
Nice to meet you.
While developing automated trading methods, I am examining how far I can trust the “appearing good results” in backtests.
I form hypotheses using AI, verify them with actual market data, and discard if they don’t hold.
This is the repetition.
In this series, I will write not so much about introducing a “winning method,”
**but about how I questioned and broke down results that looked promising**
and centered around that.
In the first installment, I will discuss the actual pitfalls of backtests I stepped on while continuing to develop automated trading.
---
## A good backtest result is surprisingly easy to produce
When I started making automated trading:
“High PF”
“High win rate”
“Capital curve rising smoothly”
If I saw these figures, I thought I was getting closer to a good method.
But as I continued testing, my views changed quite a lot.
Now, rather,
**The better the numbers, the more I doubt them**
I have become.
Because in backtests,
**you can get good numbers even without being able to predict the market**
.
---
## The PF exceeded 1.0. That alone wasn’t trustworthy
PF (Profit Factor) is
calculated as total profit ÷ total loss
For PF > 1.0, backtests show profits exceeding losses.
At first I placed heavy emphasis on it.
However, after testing many exit conditions, something odd happened.
In a particular extended grid, out of 1,728 cells that passed PF > 1.0,
**1,714 cells, 99.2%**
were mathematical artifacts caused by extreme asymmetry between take profit and stop loss.
(Source: `hypothesis_backlog.md:6362-6373`)
Direction predictions were not superior.
Just by making the exit shape extreme, there existed regions where PF easily exceeded 1 even with random entries.
In other words,
**PF>1 = there is an entry advantage**
was not the case.
This was a major turning point.
---
## Started to look at "Is it better than random?"
So I stopped looking at PF in isolation.
I created a PF distribution for random entries with the same exit conditions, and
I checked how far candidates deviated from that distribution.
I used the z-score for this.
Initially,
“Candidate PF must be at least 0.15 higher than random baseline”
a fixed-width standard was used.
But I found that the standard deviation of the random baseline varied greatly depending on exit conditions.
With a fixed width, the rigor of the comparison changed by condition.
So I changed to
**a method that checks how many standard deviations away the candidate is from the random distribution**
.
(Source: `hypothesis_backlog.md:650-655`)
As a result, 471 candidates became
**471 → 14**
(Source: `hypothesis_backlog.md:657-664`)
Even with the same backtest results,
instead of asking “is there profit,”
I began to ask
**“would it also perform about as well under random conditions?”**
and the perspective changed.
---
## Choosing in the first half, not re-choosing in the second half
Another major change was how I used the data.
For example, if you have 10 years of data,
you look at all ten years and
you keep selecting this TP is good
this SL is good
holding time is good
and so on.
Finally, you present the best one as the
“method that made a profit over 10 years.”
This is quite dangerous.
It looks like you tested ten years, but you actually optimized using ten years.
So I split it into
**exploration in the first half, verification in the second half**
.
The exit conditions are decided in the first half.
In the second half,
**do not change what was decided in the first half; use it as is for verification**
If you change the criteria after looking at the second half, that second half no longer serves as verification data.
(Source: `hypothesis_backlog.md:1977-2003, 3016`)
This seems obvious.
But when you start extensive search,
the temptation to tweak a little more appears many times.
If you use the second half data each time, you can build any “good method.”
---
## If you try 17,740 combinations, you'll get some coincidences
Another unavoidable issue is
**massive search**
Testing just one hypothesis and getting a good result is different from trying thousands or tens of thousands of conditions, and only one being good.
Same numbers, different meaning.
Even with coin flips, if you search thousands of times for the best way to flip, you can get extremely good results by chance.
The same applies to automated trading.
In actual verification, I treated 17,740 trials as one family,
and used White Reality Check and Hansen SPA methods
to assess whether “the family as a whole has meaning”
.
stationary bootstrap is 5,000 iterations.
(Source: `item6_white_rc_hansen_spa.py:1-19`)
Looking only at individual backtest results,
you cannot distinguish
the “best among many random attempts” from true edge.
---
## Even with 99.5% win rate, I discarded it
During verification,
**a setting of 99.5% win rate**
appeared.
Normally, this would be quite enticing.
But when I looked into the details,
the TP was extremely close and the SL was extremely far.
Small profits were realized many times, with rare large losses.
There was another oddity as well.
On paper it passed the criteria, but since it matched a deterioration pattern I had already confirmed, I ultimately did not adopt it.
(Source: `hypothesis_backlog.md:2320-2331, 2366-2367`)
This story is long, so I will write it in another article.
What matters is
**I did not discard it because the win rate was high**
.
As a result of examining why it became 99.5%, I found I could not trust that number.
So I discarded it.
---
## Phenomena described in papers do not always translate to trading
I also brought hypotheses from academic papers and tested them.
Regarding a short-term reversal phenomenon in FX, when I examined the original paper's form in a recent period,
the statistical phenomenon itself was confirmed in earlier periods.
However, when viewed by era, it diminished significantly.
From 2000–2013 the effect was clear,
after 2013 it shrank,
and from 2020–2025 it was nearly gone.
Furthermore, even in periods with strong phenomena, the predicted reversal magnitude was about 0.15–0.3 pips.
The realized spread was about 0.2–1.0 pips.
In other words,
**the statistical phenomenon existed, but it did not exceed trading costs.**
(Source: `hypothesis_ledger.md:205-219`
`design_A1_vol_regime_reversal_20260902.md:390`)
What I learned here is that
**whether a paper’s phenomenon can be reproduced is a different question from whether it can be traded profitably**
.
And,
**what existed in the past and what exists now are also different**
.
---
## Good numbers can still be accompanied by broken verification methods
Another more troublesome issue is
not the strategy, but
**verification code or underlying assumptions were wrong**
.
For example, during a period when I was testing a rule that forced closure after a certain time,
positions that timed out were not closed at actual prices and were treated as zero profit/loss.
Also, I had cases where swaps were not included in the model.
In a GOLD verification,
a PF of 1.157 without swaps becomes 1.033 with swap at current value,
and PF drops to 0.922 when assuming swaps twice as large.
(Source: `hypothesis_backlog.md:3231-3325, 3274-3292`)
Backtests themselves were functioning normally.
No errors were raised.
Graphs and numbers appeared.
So it’s scary.
**Just because the code runs does not guarantee the verification is correct.**
---
## The problem is not “wrong premises” but “no mechanism to re-verify premises”
Looking back at structural flaws, there are common traits.
When those premises were decided,
they all seemed reasonably valid.
The problem is
**there was no mechanism to confirm later whether the agreed premise remained valid.**
(Source: `hypothesis_backlog.md:3234-3235`)
So now I
when deciding criteria or thresholds,
I ask not only “why is this number here,”
but also
**“which assumptions does this judgment rely on?”**
to keep.
Fixing backtest rules is important.
But
**fixing incorrect rules permanently is not the same.**
---
## Re-evaluating my past verifications
A strategy that once seemed good is not trusted forever.
I also conducted audits to see if it stayed robust in more recent periods.
In one audit, for the targeted strategy, the most favorable metrics in the most recent segment were the average daily profit, expectancy, and descriptive PF, all of which were highest in the most recent period.
The difference between the most recent segment and the initial segment was positive for all point estimates.
However, bootstrap confidence intervals were wide, and for the most uncertain strategies,
p(Δ<0)=0.247
.
(Source: `hypothesis_ledger.md:541-553`)
In other words,
“the more recent is better, therefore safe”
cannot be said outright.
We can only say that there are no signs of deterioration in the current data.
I think this distinction is very important.
---
## Re-testing the same hypothesis twice
As verification grows, another issue arises.
Sometimes I would take a previously verified hypothesis and re-select it by slightly changing its name or angle,
and present it again as a “new hypothesis.”
In reality, I realized this by searching old records, so I did not re-verify it.
Yet,
**repeating the same hypothesis with a different name is unrecorded massive exploration**
.
So now I keep a hypothesis ledger and before starting a new hypothesis, I check
- whether the same ID already exists
- whether there is a similar estimation target
- what status it had in the past
to confirm.
(Source: `hypothesis_ledger.md:7-10`)
As the amount of research grows,
it's important to manage not only “what to try”
but also “what has already been tried.”
to.
---
## I’m not yet talking about actual operating results
Even with stricter backtests,
there are things you won’t know until you actually run them in the market.
However, evaluating only a small number of trades soon after starting the operation
“wins”
“losses”
“this method is usable”
is dangerous.
I will not display short-term results as if they are representative.
Until there is enough data to evaluate,
**evaluation cannot be evaluated**
will be treated as not evaluable.
---
## What increased through this verification process was not “a winning method” but “reasons to discard”
When I started building automated trading,
I thought that continued verification would reveal more and more “winning conditions.”
In reality, it was the opposite.
What increased was
- reasons not to trust PF alone
- reasons to compare with random baselines
- reasons to separate the first half and second half
- reasons to compensate for massive exploration
- reasons to doubt exit settings
- reasons to separate costs
- reasons to separate past and present
- reasons to doubt the code itself
- reasons to keep a hypothesis history
.
Creating good backtests was easier than I thought.
The hard part is
to steadily accumulate reasons to trust that backtests are trustworthy
.
That is why now
instead of “making a winning EA,”
I feel more aligned with trying to find an EA that survives even when broken
.
---
## What this series will cover
From now on,
- how a promising hypothesis was broken
- what happens when you bring papers into the real market
- how much backtest bugs can alter conclusions
- how I incorporate AI-generated hypotheses and verification support into research
- how I question the verification methods themselves
I plan to write about these topics.
This is not a series introducing a ready-made “winning EA.”
Rather,
**this will be a record of how I discarded things that looked like they could win**
.
If you are practicing automated trading or backtesting similarly,
I hope there is at least one clue for your own verification.
※This article does not recommend any specific EA, financial product, or individual trading decision. It records the verifications I performed and the methods used.
※This article is a reprint with minor adjustments from a note published previously.