Anatomy of a false edge: auditing our five-model ensemble

The first serious system we built for XAUUSD was, on paper, exactly what this industry sells as sophistication: an ensemble of five models — gradient-boosted trees (XGBoost), an LSTM sequence model, a vision CNN reading rendered chart images, a PPO reinforcement-learning agent, and a meta-fusion layer that weighted their votes into one signal. It traded 5-minute gold bars on a demo account, retrained itself on a schedule, logged every decision, and drew equity curves that went up.

It looked like the future. It had a dashboard. It had online learning. If we had been a signal channel, this is the point where we would have opened a Telegram group and started selling access.

Instead, before risking real money, we applied the rule that later became the first line of our methodology: a positive result triggers more skepticism, not celebration. A backtest that looks profitable is not evidence; it is a hypothesis that has not been attacked hard enough. So we attacked our own system — line by line, config by config — the way we would attack a stranger's claim.

The audit found four independent, fatal defects. Read them in order and notice the pattern: not one of them ever threw an error.

The feature pipeline normalized inputs with a min-max scaler. Standard practice — except ours fit its min and max across the nine features of a single bar, instead of across time per feature. One of those nine features was price, sitting around 4,300. Another was RSI, sitting around 50.

Scale those together and the arithmetic is merciless: price defines the entire range, and every other feature is crushed into a sliver a fraction of a percent wide. RSI, momentum, volatility — all of them arrived at the models as near-constant noise, at training time and at serving time. Five sophisticated architectures were staring at one number and a haze. The models were not learning the market; they could not even see it.

The backtest's cost model contained two numbers, and both were wrong in the system's favor by enormous factors:

• Point value: 100× wrong. One lot of gold is 100 oz — a $1 price move is $100 per lot. The config had it as $1. Every simulated win and loss was scaled down two orders of magnitude, which quietly rescaled risk, drawdown, and position sizing into fiction.
• Spread: ~34× wrong. The config charged a raw “20 points” ≈ $12 per trade. The real spread, measured later directly from live order-book data, is about $0.35. Two errors, opposite directions, no relationship to any market that exists.

A P&L built on these numbers is not an approximation of trading gold. It is a simulation of an imaginary instrument.

The deepest flaw was structural. Three separate stages of the system were fit on the same full history: the five models were trained on it, the fusion layer's weights were tuned on it, and the final performance was evaluated on it. Nothing in the pipeline had ever been asked a question it hadn't already seen the answer to.

This is the quiet failure mode of most retail “AI trading” results: memorization graded as skill. The fix is boring and non-negotiable — chronological splits, where models are trained on the past and judged only on a future they never touched. We rebuilt everything around that rule.

The reinforcement-learning component — the part that sounded most impressive — turned out to have been trained on Binance crypto order-flow data and then deployed, unchanged, to trade gold on 5-minute bars against a completely different market, microstructure, and session clock. There was no transfer learning story, no fine-tuning, no evaluation of whether the domains had anything in common. A bitcoin-shaped policy was casting votes on gold.

None of these bugs crashed anything. All four inflated the results. That is not a coincidence — it is a selection effect. When a bug makes performance look worse, you investigate until you find it. When a bug makes performance look better, you find reasons to believe it. The graveyard of retail quant projects is full of systems whose authors stopped looking exactly one bug too early.

The only defense we know of is procedural: treat every positive result as the start of the investigation, audit the pipeline before admiring the equity curve, and price every simulated trade at real, measured costs.

We then rebuilt the entire evaluation from scratch as a clean walk-forward harness: real economics ($100 per $1 move per lot, measured $0.35 spread, $7 round-turn commission — a ~$4.30 wall per 0.1-lot round trip), per-feature scalers fit on training data only, triple-barrier labels, chronological folds, and decision thresholds chosen on validation and applied untouched to out-of-sample test windows.

With every bug removed, the same class of models on the same market produced predicted probabilities whose 5th-to-95th percentile range was [0.498, 0.503] — statistically indistinguishable from a coin flip. Forcing trades anyway lost between $604 and $6,803 per fold. The edge had never existed. The ensemble's performance was manufactured, in equal parts, by a scaler bug, imaginary costs, leakage, and borrowed conviction from another market.

Verdict: killed by honest labels and real costs. What survived is the discipline — one shared feature implementation across train, verify, backtest, and live; costs on every simulated trade; and the standing assumption that our own results are lying to us until proven otherwise. Every number now published on this site passes through that gauntlet first. That is the entire reason this Lab exists.