[Technical Report] Forward Test Deviation Comparison between Overfitting Logic and WFA Passing Logic — Survival Strategy Through Process Design
In the operation of automated trading systems (EAs) and quantitative strategies, the most serious and universal challenge is the discrepancy between backtesting and forward testing. Why does a system that showed a beautifully rising asset curve in past validation continue to update its drawdowns as soon as it moves to live operation?
We at Semura Lab. design and diagnose the structure of the system itself as process design engineers in order to eradicate this “discrepancy” in system trading. In this report, we technically explain the difference between the main cause of discrepancy, “overfitting (curve fitting),” and the structurally eliminating method, “WFA (Walk-Forward Analysis),” using a comparison table.
1. Conceptual Comparison: Overfitting Models vs. WFA Models
First, we organize the differences in how past data is handled in system development. The usual overfitting approach that many developers fall into and the WFA approach, which our lab regards as the absolute standard, have fundamentally different process designs.
| Comparison Item | ❌ Overfitting (Curve Fitting) Model | ? WFA (Walk-Forward Analysis) Model |
| Data Split | Optimizes using the entire period as a single block | Strictly splits into a “learning period (IS)” and a “testing period (OOS)” |
| Parameters | A single fixed value that yields the best performance over the entire period | Learned from past data and dynamically evaluated through regular sliding validation |
| Adaptation to Unknowns | Untested on unknown data (OOS) | Always evaluates performance only on unknown data |
| Discrepancy Risk | Weak to regime changes, extremely high | Low discrepancy risk due to a structure adapting from past trends to the future |
| Design Focus | Maximizing past profits | Ensuring survival and robustness in an unknown future |
Overfitting models are, in a sense, “rote memorization of past answer sheets.” In contrast, WFA models require the system to have the ability to “discern laws from past trends and solve problems it has not seen before (unknown markets).”
2. Structure of Forward Test Discrepancy
Next, we illustrate the structural differences that produce performance differences when both approaches transition to actual trading (forward testing).
| Indicator / Phenomenon | ❌ End result of Overfitting Logic | ? Behavior of WFA Passing Logic |
| Backtest PF | Produces unrealistic and overly optimistic values such as 3.0–5.0 | Converges to realistic and modest values around 1.2–1.5 |
| Immediately After Forward Transition | High probability of sharp performance drop, turning downward | Maintains a gradual trajectory close to the backtest expectations |
| Drawdown | Loses the assumed maximum DD and risks account ruin | Controlled within the DD budget (cage) set during design |
| Resilience to Market Changes | Dies immediately when volatility or trend characteristics change | Shows structural adaptability (persistence) to regime changes |
Overfitted logic fits even to rare past noise perfectly, so even minor real-market deviations are fatal. Meanwhile, logic that has passed through WFA is a specimen that has survived durability tests against unknown data (out-of-sample), so forward discrepancy is minimized to the extreme.
3. Perspective of a Process Design Engineer: Thorough “Complete Causality”
As a fundamental prerequisite for WFA to function correctly, the system must adhere to the physical law of “complete causality.”
When the system makes a judgment, if even one bit of future information (look-ahead bias), such as an endpoint price that is not yet determined, leaks into the process, the test results of WFA become illusions. In all stages—entry, settlement, and adaptive sizing—a strict process design that computes only with data that is completely determined at that time (past data up to T-1) is indispensable.
In Conclusion: Abandon Illusions, Embrace Robustness
Dear readers of Investment Navigator, when selecting a system, please discard the criterion of “how much profit it had in the past.” What you should look at is, “how was the system evaluated against unknown data (presence or absence of WFA)?”and“Have constraints (causality and capital management) been designed to prevent self-destruction of the system?”
Semura Lab. never trusts an unnaturally upward-sloping curve. As process design engineers, we will continue to communicate only the structure and verification facts of truly robust systems that survive in real markets, even if it takes more effort.