The calculation result matched. However, it was the side that did not match that turned out to be correct.

The calculation results matched. However, the correct one was the one that did not match.

In automated trading research, you must question not only the trading method but also the method of verification itself.

In the first article, I wrote about the possibility that code can run correctly and verification can be incorrect as separate things.

This time, this is a continuation.

In current research, a mechanism to redo calculations and compare results is being incorporated.

And actually, problems were found through that cross-check.

Moreover, the stocks whose calculations did not match were more correct, and the ones that did match had the problem.

Decide the rules before looking at the results

In current research, before starting verification, we decide what to examine and under what conditions to judge.

This is the idea of pre-registration.

While conducting research, there are times when you want to change the conditions after seeing the results.

If the results aren’t as expected, you want to try other conditions.

If good results appear, you want to adopt those conditions.

However, if you change the judgment criteria while looking at the results, it becomes difficult to tell whether you were testing that hypothesis from the start or selecting conditions to fit the good results.

Therefore, we decide the rules before looking at the results.

I believe it’s necessary to prevent yourself from changing your judgments after favorable results.

Of course, pre-registration does not guarantee correct results.

This is a mechanism to deliberately reduce the freedom of judgment.

Opening the latter half of the data is allowed only once in the entire exploration

In the first installment, I introduced the idea of dividing the data into the first half and the second half, and not reselecting the conditions chosen in the first half for the second half.

Now, we are making the rules even stricter.

Opening the second half of the data is allowed only once in the entire exploration.

Before that, we narrow down the candidates and finalize the verification conditions.

And even if the first chosen candidate fails in the second-half verification, we will not promote another candidate to retry with the same second-half data.

Once a result is seen, that information cannot be forgotten.

The candidate that took time to find might be rejected at the final check.

Even so, if you replace candidates one after another after seeing the results, the purpose of leaving the second-half data would be diluted.

This is a part I want to guard very strictly as we advance the research.

Only the Nikkei 225 had mismatched results

This time, the problem actually occurred.

When recalculating and cross-checking, only the Nikkei 225 did not match.

The other four stocks matched.

Normally, you would want to doubt the calculation for the Nikkei 225 that did not match.

However, when I investigated, I found something surprising.

The issue was not the Nikkei 225 but the remaining four stocks that were matching.

In automated trading, an ATR indicator is used to determine the width of take-profit and stop-loss.

ATR is a measure of price movement magnitude.

There are several ways to calculate the average of ATR.

The originally decided rule used was to use the simple average (SMA).

The Nikkei 225 was calculated according to that rule. It also matches the calculation method used in actual operations.

However, for the other four stocks, a different calculation method called Wilder's method was used partway through.

Moreover, that change was not recorded.

The verification code we used was also Wilder's method, so the results for the four stocks calculated by the same method matched.

On the other hand, only the Nikkei 225, which followed the original rule, did not match.

In other words, a match of results does not guarantee correctness.

If you make the same mistake, the calculation results will match.

The issue this time is not about which method—Simple Average or Wilder's—is superior.

The issue is that the pre-decided calculation method was altered without being recorded.

The four stocks were recalculated with the predetermined method. Several days of calculations had to be redone, but it is far better than piling up results on the basis of an erroneous premise.

Subsequent calculations were redone as well.

Previous calculation results were not deleted but left as "invalidated."

If I had only looked at verification results and adjusted the Nikkei 225 accordingly, I would have made the correct calculation fit the incorrect one.

From this experience, I again realized that not only whether results match, but also what criteria are used to judge correctness, are important.

About 45 hours of calculations disappeared due to a restart

Not only the calculation method but also the saving of calculation results failed.

About 45 hours of calculated results were wiped out by restarting the Mac.

The cause was the save location.

The results were saved in a temporary folder that disappears after a restart.

The calculations themselves were not wrong.

However, if the results do not remain, you cannot verify those calculations later or use them for the next stage of the research.

In response to this failure, we now save interim results roughly every 20 minutes, and have a system that can resume from where it stopped.

Even if you stop processing midway multiple times, you can resume from where you left off and the final results will fully match the results of uninterrupted calculation.

Not flashy improvements.

However, for long-running verifications, I feel such mechanisms are indispensable.

Build a mechanism to detect mistakes before producing correct results

In the past, when a promising backtest result appeared, I would doubt whether that trading method was really effective.

That mindset hasn't changed.

But the things to doubt have increased.

Whether the calculation method has changed partway.

Whether the judgment criteria were changed after seeing the results.

Whether the calculations can be reproduced under the same conditions.

Whether long-running calculation results can be reliably preserved.

And when there is a mistake, whether there is a mechanism to detect and fix it.

The ATR issue this time could have gone unnoticed if we hadn’t recalculated and cross-checked.

What matters is not that we never make mistakes, but that we create a state where mistakes can be found and corrected.

We are continuing to validate new trading methods.

Results have not yet been obtained.

They might be good results, or the hypothesis might be refuted again.

Still, at least compared to before, there are more mechanisms to judge whether results are trustworthy.

As we continue research to find profitable methods, we are rebuilding the verification environment that underpins it.

I want to have a reason to trust those results even before they turn out well.

That is how I feel now.