Use three AIs to test an EA — assign the role of creator and the role of skeptic to separate AIs

Introduction: If you leave everything to one AI, no one will doubt anything

In this series, I have written about the pitfalls of making an EA with AI. If you ask to “make the backtest better,” you’ll get overfitting. If you ask to “write it attractively,” the product description will be inflated. Both have the same root: when the role of the creator and the role of the doubter are the same person, no one will doubt anything.

AI will faithfully do what it is asked to do. The creator themselves (you and the AI in that conversation) are in a position where they want to see their results in a favorable light. Therefore, even within the same conversation, asking “Is this really okay?” tends to get a lenient answer.

I am currently developing and validating an EA by dividing roles among three AIs: ChatGPT, Claude, and Codex. This time I will describe the division of labor, the mechanism that makes them review each other, and a way to imitate this even with just one AI.

Roles of the three AIs

The principle of division is simple: separate the “creating role” and the “verifying role” into different AIs, and make sure the responsibilities do not overlap.

AI	Main Responsibilities	What it answers to other AIs
Codex	Write code. Backtesting and EA implementation, builds, tests	What the code does, whether tests pass
Claude	Designing verification and interpreting results. Document pre-declarations and acceptance criteria, and judge results by those criteria	Under which conditions judgments are being made, the operational status
ChatGPT	Review Claude’s design and results. Also serve as a sounding board for secondary use of成果 and expansion to other businesses	Whether the design and judgments have any oversights

The important point is that none of the AIs should do anything outside their own responsibilities “on the side.” If the AI that wrote the code also judges whether the code’s results are pass or fail, it will return to wanting to view its own work in a favorable light.

Another rule I have set is not to rely on myself as the explainer every time. To let AIs inquire about each other’s status, I have predetermined which AI answers what. If I step in to summarize, that introduces a bias of “wanting to see better.”

Mechanism to review each other: Fix the order

Just separating roles is not enough. We also fix the order in which decisions are made and who makes them.

1. Before looking at results, document the search methods and acceptance criteriaWhat to try and in how many ways, what criteria must be met to accept, what must not be done. I call this “pre-declaration.”
2. The creating role implements and runs according to the declarationNo one will comment on the results. If you want to implement any “improvements” beyond the declaration, stop and report.
3. Judgment is performed mechanically only on the declared conditionsDon’t pass because it’s “close enough” or because the conditions were too strict
4. For items that pass, have another role doubt them by considering “if this is by chance, why”Not the role that celebrates success, but the one that looks for reasons it could fail

The key point of this order is that 1 comes first. If you decide conditions after seeing results, AI will craft conditions that fit any outcome.

In practice, I have reduced the number of tests by adhering to declarations. I even tried widening the stop loss and take profit ranges mid-way, and started drafting that version. But counting showed 516 test variants; that exceeded the upper limit of 288 declared, so I returned to fixed limits.

Case study: 5 passes, but only 1 underlying phenomenon

With this approach, we conducted 129 tests across 2 major currency pairs and 8 strategy templates. The acceptance criteria were seven, with the last one requiring that the adjacent timeframe or the other currency pair also makes sense. The details are written in“Five passes, one identity”article.

As a result, five passed the criteria. At first glance, it seems we found five winning rules. If I read the results from the perspective of the creating role, I would want to deploy all five in parallel.

However, as the role of doubt inspects the contents, all five turned out to be the same pattern: “go long in the same market at the opening of the same currency pair’s same session as the previous direction.” The names were different, but the implementations were nearly identical. The independent discovery was effectively one.

Furthermore, when we later computed the 95% average confidence interval, all five could not be distinguished from zero. The reason was the small number of trades; to settle on one, about 118 cases were needed, but at that point there were 72 cases.

If one AI had been asked, “What about these five?” I think the reply would have been, “Let’s diversify five promising rules and deploy them.” In reality, I was staking five times on the same one, thinking I was diversifying.

In my division of labor, the task of “inspecting the contents of the passing results” is the role for which I ask ChatGPT to review. With a different eye from Claude who produced the results, I’ll ask whether there really are five distinct ones.

A readily copyable method: what one AI can do

You don’t need to contract three AIs to do the same thing. The key is not the number of AIs, but separating the “creating conversation” from the “doubting conversation.”

1. Before creating, write the declarations. Note and save the scope to test, the upper limit of tests, and the acceptance criteria.Checklist for Part 3can be used as the acceptance criteria as is
2. In the creating conversation, pass the declaration and request “implement only within this range”. Do not request result evaluations
3. Open a new doubting conversation. To avoid carrying over from the previous conversation, pass only the declaration and results, and ask as follows

You are the one who looks for reasons not to adopt this EA.
Read the attached pre-declarations and backtest results,
- points where the declaration conditions are not met
- points where you are doing things not in the declaration (post-hoc criteria, more tests than allowed)
- whether the passing items are actually capturing the same single phenomenon
- how you would explain it if the results were merely lucky
Please list them. Do not bother listing good points.

It’s important that you do not list good points. If you omit them, the AI will balance good and bad points trying to be helpful, and the conclusion will become blurred.

4. In the end, you decide. If you cannot answer the reasons given in the doubt conversation, that EA will not be adopted yet

Summary

• If the creator and the doubter are the same, no one will doubt anything. Separate AIs to separate those roles
• The sequence is declaration → implementation → condition-based judgment → another role doubts. Do not decide conditions after seeing the results
• “Five passes” actually meant “one underlying phenomenon” when viewed by the doubt role
• Even with one AI, you can do the same thing by separating the creating and doubting conversations

Thus far in five installments, I have described the full process of creating, verifying, listing, and presenting an EA with AI. In each installment, the message is the same: doubt your own results with a perspective separate from yourself.

A related discussion about applying the same verification to TradingView strategies is in the next chapter“TradingView strategy results should not be read in the default settings”I wrote about it there.

Bonus: How to delegate the “doubting role” to a machine

There is also a method to delegate the doubting role to a fixed-criteria tool rather than an AI. A self-made inspection tool“EA Robustness Check”was created precisely for that purpose. The judgment criteria are fixed, and results cannot be loosened after the fact. It does not involve the subtlety of an AI.

---

For those who find it hard to doubt themselves, we also offer a service to undergo inspection.A third-party checks EA backtest robustness14,800 yen (Skijan). We will store up to three MT5 reports and provide findings (PDF) within three business days. The judgment criteria are fixed, and you cannot change the results if they are unfavorable.

※This article introduces the author’s own approach and does not endorse specific instruments, methods, or EA purchases for trading, nor the use of specific AI services. Past verification results do not guarantee future performance.