Here is the uncomfortable truth: most trading strategies fail. Not because the traders who built them are not smart, not because they did not work hard, not because the markets are unfair. They fail because the strategies were never truly validated in the first place.
The title of this episode is not designed to be provocative. It is a direct statement of probability. If you build a strategy using standard optimization techniques – even if you use walk-forward analysis – there is a strong likelihood that strategy will not perform as expected in live trading. Bob Pardo, the inventor of walk-forward analysis, said as much himself: even a walk-forward result can be profitable purely by chance.
That statement, from that person, changes how you have to think about validation.
How Most Traders Start – And Why It Goes Wrong
Tomas’s own beginning in systematic trading is instructive. When he first decided to move from discretionary trading to automated systems, he did what many beginners do: he bought strategies online. Five or six of them, at $300 – $400 each. He paper-traded them for about six months without putting real money in.
Every single one failed. Not just underperformed – failed in the specific way that purchased strategies tend to fail. Each one had equity curves that peaked right around the time he started watching them, then declined immediately. He had effectively bought strategies at their equity highs, right before they broke down.
This is not coincidence. It is the natural lifecycle of an over-optimized strategy: it performs well on the historical data it was fitted to, looks spectacular at the moment of sale, then stops working as soon as market conditions shift even slightly from the period used to build it.
A friend in Las Vegas attended a trading expo around the same time. A vendor approached him with the pitch: “Buy my strategy and you will make a million.” The friend asked, straight-faced: “And if I buy two strategies, will I make two millions?” The vendor nodded, in total seriousness. Yes.
That is the environment most retail traders learn in. Not an environment that encourages rigorous validation thinking.
The Consumer Mentality vs. the Trader Mentality
The real problem is not that bad strategies exist. It is that most traders evaluate strategies with a consumer mentality rather than a trader mentality.
Consumer validation: Does the website look professional? Are there testimonials? Is there a disclaimer and a refund policy?
Trader validation: Does the strategy have a genuine, identifiable edge? Has it been tested out-of-sample? Is the robustness of the results distinguishable from curve-fitting? What does the logic look like across multiple markets and time periods?
Tomas’s insight is that building your own validation process is the central intellectual challenge of systematic trading. Not strategy building – anyone with access to good software and a few tutorials can build a strategy. The hard part is building the interrogation machine that tells you whether what you have built is real or not.
The Evolution of a Validation Framework: Versions 1.0, 2.0, and 3.0
Tomas has been developing his validation framework for roughly nine years. What started as a version 1.0 that was slightly better than a coin flip has evolved through two major iterations into the framework he now uses in his hedge fund.
Framework 1.0: Slow Hardware, Basic Ideas
This first version was built when TradeStation ran on a single CPU core and 8GB of RAM was considered luxurious. It incorporated some non-standard ideas – things that were not widely published in trading literature – and produced results that were marginally better than 50/50. Not meaningfully actionable, but it was a start, and it planted the seeds of the right questions.
Framework 2.0: The Czech Republic Era
Two to three years after the initial version, Tomas had collected enough research and tested enough validation techniques to build something substantially more powerful. Framework 2.0 went beyond the standard in-sample/out-of-sample split and beyond basic walk-forward analysis. It incorporated multi-market validation and combinations of techniques that were not commonly used together.
He shared this version with students in Czech Republic. Multiple students built successful track records using it. Two of them used it to launch their own hedge funds.
Framework 3.0: Hedge Fund Grade
When Tomas began building his own hedge fund – staffed by former Czech students who had become skilled enough traders that he wanted to work with them at a deeper level – the goal was to take everything from 2.0 and push it further. The team worked together to add completely new research, new elements, new tests. The result is framework 3.0.
The performance numbers speak for themselves: over nearly two years of monitoring strategies in the hedge fund database, the strategy survival rate is approximately 93%. Only about 7% of strategies validated through the framework have genuinely failed. Some additional strategies have broken even. But 93% of strategies validated by this process have continued performing as expected in live trading.
That is not an achievable number with basic walk-forward analysis. It requires something significantly more comprehensive.
Why Walk-Forward Analysis Is Not Enough
Walk-forward analysis is a genuine advance over simple in-sample/out-of-sample splits. It is better. But “better than the alternative” is not the same as “sufficient.”
Three specific limitations matter:
- Statistical luck. Bob Pardo’s observation stands: a profitable walk-forward run can happen by chance, without reflecting any genuine robustness. If you are running thousands of parameter combinations, some of them will look good in walk-forward purely by statistical noise.
- Fitness function gaming. The choice of what metric to optimize during walk-forward – the fitness function – significantly affects results. Using combinations of fitness functions (weighting net profit and drawdown together, for example) introduces instability that tends to produce unreliable out-of-sample behavior. The hedge fund approach is to use simple, individual fitness functions rather than composite ones.
- Window sizing is an art, not a science. There is no universal rule for how much data to use in the in-sample window versus the out-of-sample window. An 80/20 split is a reasonable starting point, but the optimal ratio depends on the strategy’s trading frequency, the data history available, and properties specific to the market being traded. Getting the window sizing wrong systematically biases your results in ways that only become apparent in live trading.
Parameter Ranges: Where Science Meets Judgment
One of the most frequently asked questions in strategy validation is: how do you choose the parameter ranges to test during optimization?
Tomas’s honest answer: it is more art than science. This is uncomfortable for technically-minded traders who want a precise algorithm for every decision. But the discomfort is necessary to sit with.
The analogy he draws is to musical performance. Andrew played trombone professionally for years. Learning the technical mechanics of the instrument – the positions, the breathing, the theory – is the science part. But producing music that actually sounds like music, rather than like a student practicing scales, requires a developed sense that cannot be fully codified. You build it through repetition.
The practical guidance: start with a simple general framework – three or four rules about parameter ranges that you apply consistently across all strategies. Do not try to optimize the optimization. Stick with your general rules even when they feel imperfect, and focus on building your first hundred strategies. After a few months of practice, you will start developing a feel for when ranges are too tight or too wide. Your walk-forward results will also give you feedback – a strategy that looks very sensitive to small parameter changes is probably over-fitted, regardless of what the walk-forward shows.
What Robust Validation Actually Requires
The honest summary of what validation framework 3.0 involves is that it is layered, complex, and not possible to fully reverse-engineer from a description. Even Tomas sometimes needs to consult the documentation his hedge fund team maintains to recall specific details. That complexity is not a feature – it is the inevitable result of nine years of systematic refinement.
What you can take away practically:
| Validation Layer | What It Tests | Common Mistake |
|---|---|---|
| In-sample / Out-of-sample split | Basic generalization to unseen data | Too little out-of-sample data (fewer than 30 trades) |
| Walk-forward analysis | Performance stability across rolling windows | Treating a passing WFA as sufficient validation |
| Multi-market testing | Whether the edge is market-specific or structural | Only testing on the market where you found the strategy |
| Fitness function selection | What you optimize for affects what you find | Using composite fitness functions that introduce gaming |
| Parameter sensitivity analysis | Whether results are robust across parameter variations | Accepting a narrow performance peak as a valid result |
The strategy validation guide covers the core principles in detail, and the final test every strategy must pass addresses the robustness layer specifically. If you are starting from scratch with strategy validation concepts, the algo trading introduction provides useful foundational context.
Want to build strategies you can actually trust before going live? Check out Breakout Trading PRO. It’s a structured, visual strategy builder inside BreakoutOS, powered by the proven Mr. Breakouts Formula used by thousands of traders.