Why one pair is not proof: single-market backtests and selection bias

The coin-flipping contest

Put a thousand people in a room and have them each flip a coin ten times. Around one of them will flip ten heads. Interview that person and they will tell you about their technique. Their wrist action. Their focus.

Now test one strategy on twenty pairs and pick the pair where it shines. You just ran the contest. The winning pair is the ten-heads flipper. The performance is real, in the sense that it happened. The skill is not, in the sense that it will not repeat. This is selection bias, and it does not feel like cheating while you do it. It feels like research.

The arithmetic is blunt. Test enough pairs and one of them must look great by chance alone. The better the best result, the more you should suspect the process that selected it, not admire the result. See overfitting for the full family of this mistake.

Why a single pair misleads even without shopping

Suppose you only ever tested one pair. No shopping, no selection. The single-pair backtest still carries two problems.

One history. The pair lived through one specific sequence of regimes. Maybe your test window caught LINK in two long trends. A trend strategy looks brilliant there, and the brilliance belongs to the window, not the rules. Run the same rules on a pair that spent those years chopping and the verdict reverses. You did not test the strategy. You tested the strategy multiplied by one particular past. See market regimes.

Few effective samples. Trades on one pair in one period are correlated with each other. The same trend produced many of the wins. Statistically, your 400 trades carry far less independent evidence than 400 coin flips. The sample is smaller than it looks, and it already looked small.

What cross-asset validation actually tells you

Run the same rules, same parameters, on several related pairs. Three outcomes are possible, and each one is informative.

It works broadly. The edge shows up, weaker or stronger, on most of the majors. This is the good outcome. A pattern that exists across BTC, ETH, and SOL is probably a property of how these markets move, not a property of one chart’s history. The trader’s equation has a reason to stay positive.

It works on one pair only. The default explanation is luck plus selection. The honest response is suspicion, not celebration. Demand a reason.

It works nowhere, including the original pair, after costs. Also useful. The strategy died on paper, which is the cheapest place to die.

The honest exception

Some edges are legitimately pair-specific or venue-specific, and pretending otherwise would be its own mistake. A funding-carry strategy lives on the funding behavior of one contract on one venue. A microstructure edge lives in one book’s depth and fee schedule. A thin altcoin can have inefficiencies that BTC arbitraged away years ago.

The test is whether you can state the mechanism. “This works on this pair because of this measurable, structural feature, and here is that feature in the data.” That is a reason. “It works on this pair because that is where the backtest was green” is not a reason. It is the ten-heads flipper describing wrist action.

A real mechanism also gives you a sunset signal for free. If the edge lives on high funding, you can watch funding. When the feature fades, you retire the strategy before the equity curve makes the announcement. See strategy decay.

What to require before believing a backtest

• The same rules tested on at least a handful of related pairs, not the best one.
• Either broad performance across them, or a written mechanism explaining the exception.
• Out-of-sample confirmation on every pair you intend to trade. Cross-asset breadth and time-based holdout answer different questions, and a real strategy should pass both. See out-of-sample testing.
• Full costs everywhere. A pair can pass on signal and fail on spread.

This is more work than testing one chart. It is much less work than funding one chart’s lucky streak.

How Edgecraft handles this

Edgecraft makes the multi-pair version of the test the easy version. The same strategy runs across assets with identical rules, results are shown side by side, and optimization studies can score parameter sets across several pairs at once, so the winner has to be a winner in more than one history. When a strategy only performs on one pair, the comparison makes that visible immediately, and the question “why this pair?” gets asked before deployment instead of after.