The selection problem

Why one impressive Sharpe ratio is not evidence.

Search N trading rules on one price history and record the best in-sample Sharpe ratio. Even when every rule is worthless, that maximum grows with N. The quantity of interest is therefore not the winner's Sharpe but how far it sits above what pure search would produce anyway.

E[max Sharpe | no skill] ≈ σ_SR · [ (1 − γ)·Z⁻¹(1 − 1/N) + γ·Z⁻¹(1 − 1/(N·e)) ]

σ_SR is the cross-sectional standard deviation of trial Sharpe ratios, γ ≈ 0.5772 is the Euler–Mascheroni constant, Z⁻¹ is the inverse standard normal CDF, and N is the number of trials. This is the Gumbel approximation of the maximum of N independent normal draws.

Probabilistic Sharpe Ratio

Probability the true Sharpe exceeds a benchmark, given track length and non-normality.

PSR(SR*) = Φ( ( (SR − SR*) · √(n − 1) ) / √( 1 − skew·SR + ((kurt − 1)/4)·SR² ) )
  • SR and SR* are expressed per observation (annualized Sharpe ÷ √252) so the n-scaling is correct.
  • n is the number of daily observations in the window being judged.
  • skew is the sample skewness, kurt the raw kurtosis (normal = 3) of the strategy's daily returns.
  • Negative skew and fat tails reduce PSR at the same Sharpe ratio — the estimate is less trustworthy.
  • The denominator is checked at run time; if it is non-positive the metric is reported as unavailable rather than guessed.

Deflated Sharpe Ratio

PSR measured against the expected maximum, not against zero.

DSR = PSR( SR* = E[max Sharpe | no skill] )

The winner must beat the level that search alone would deliver. A DSR of 0.95 or more is treated here as passing. Assumptions made visible: trial Sharpe ratios are treated as independent and normally distributed around zero, and σ_SR is estimated from the realized dispersion of this run's trials. Overlapping parameter grids make trials correlated, so the effective number of independent trials is lower than N and this correction is conservative in that direction.

Probability of backtest overfitting

A lightweight combinatorial cross-validation score.

  1. Split the post-warm-up history into S equal chronological folds (S is configurable).
  2. Take every combination of S/2 folds as a training set, with the remaining folds as the test set (capped at 140 combinations for responsiveness).
  3. Pool the daily returns of the chosen folds and pick the strategy with the highest pooled Sharpe ratio in the training set.
  4. Rank that strategy's Sharpe ratio among all strategies on the complementary test folds.
  5. PBO is the share of combinations where the training winner landed at or below the test-set median.
PBO = (1/C) · Σ_c  1{ rank_c ≤ 0.5 }      rank_c = #{ SR_test ≤ SR_test(winner_c) } / (M + 1)

A PBO near 0.5 means in-sample ranking is no better than a coin flip out of sample. Folds are chronological and not purged or embargoed, so serial dependence at fold boundaries is not removed — a documented simplification relative to full combinatorially purged cross-validation.

Backtest mechanics

Identical for every trial, so comparisons are fair.

  • Returns are simple daily close-to-close returns: r_t = P_t / P_(t−1) − 1.
  • The position decided from information up to close t−1 earns r_t, so there is no lookahead.
  • Costs: each unit of absolute position change is charged the configured bps, which also penalizes high turnover.
  • Annualization uses 252 trading days; volatility uses the sample standard deviation.
  • An indicator warm-up window (up to 260 days) is excluded from both train and test so long-window rules are not handicapped.
  • Edge cases: zero-volatility tracks report Sharpe 0, degenerate variants are discarded and counted, and runs are rejected when the test window would be under 120 observations or the history under 400 closes.

Strategy families

Parameter ranges sampled by the seeded generator.

Moving average crossover

Long when a short moving average sits above a long moving average.

short window 5–60, long window max(40, short+10)–250

Momentum breakout

Long when the trailing return over a lookback window exceeds a threshold.

lookback 20–250 days, threshold −2% to +6% trailing return

Mean reversion

Buys stretched-down z-scores and exits as price reverts to its rolling mean.

z-score lookback 10–120, entry threshold 0.5–2.5 sigma

Volatility-filtered trend

A trend rule that only trades when realized volatility sits in a chosen percentile regime.

trend windows as above, vol window 10–60, regime percentile 20–80, above or below

Pass thresholds

Fixed in advance, applied to every run.

  • Deflated Sharpe Ratio ≥ 0.95
  • Probability of backtest overfitting ≤ 0.50
  • Out-of-sample Sharpe positive and at least half the in-sample Sharpe
  • Out-of-sample Probabilistic Sharpe Ratio ≥ 0.90

Three or more failures is reported as likely overfit, one or two as inconclusive, none as more credible.

Disclaimer

Scope of this tool.

This is an educational research tool built to demonstrate selection bias in backtesting. Demo datasets are synthetic, public price data is unadjusted and best-effort, and no result here is investment advice or a recommendation to trade any security.