Optimization: what it is worth and when it hurts

What the machine does

You define a strategy with knobs. An RSI period. A stop distance. A lookback. The optimizer turns the knobs, runs a backtest for each combination, scores each one, and hands you the highest score.

It is a search algorithm. It is excellent at searching. It has no opinion about whether the point it found is a market edge or a statistical accident, because having opinions was never its job. Telling edge from accident is your job. The rest of this article is that job.

What the machine is not

• It is not a strategy factory. A strategy that loses on sensible defaults does not get fixed by optimization. It gets fit harder to the noise, and the illusion gets more convincing. You cannot tune a broken hypothesis into a true one.
• It is not magic. Sharpe 0.5 becoming Sharpe 3.0 through knob-turning is the signature of overfitting. It is not hidden potential. Nothing was hiding.
• It is not permanent. Markets move. Today's optimum drifts stale. See strategy decay.

The data-mining trap

Test 1,000 parameter combinations and a handful will look spectacular by luck. They must. That is what testing 1,000 things produces. The best result of a big search is always impressive, and the impressiveness is exactly as informative as the best flipper in a coin-flipping contest.

The honest rule of thumb: optimizing a strategy that already works yields an improvement on the order of 10 to 30 percent. An optimization that turns mediocre into stellar found noise. And note the direction: more knobs and more trials make the trap worse, not better. Every degree of freedom is another place luck can dress up as skill. See overfitting.

What honest optimization looks like

Every honest practice either constrains the search or audits the output:

• Constrained ranges. Search realistic values. A long moving-average window does not need testing from 1 to 10,000. A space that wide is an invitation to nonsense, and the invitation will be accepted.
• Multiple objectives. Score return and drawdown and trade count together. One metric is easy to game. A panel in tension is not. See Pareto frontiers.
• Out-of-sample, always. Every candidate gets judged on data the search never touched. Not optional. See out-of-sample testing.
• Stability checks. A real optimum is a region. Its neighbors also work. If a 1 percent nudge collapses the result, you found a spike of noise wearing a crown.
• Cross-asset, cross-period. An optimum that exists on one pair in one year is a coincidence with a parameter set.

When to optimize and when to stop

Optimize when the strategy is already positive on defaults, the data is deep enough for a real holdout, the knobs number five or fewer, and you intend to deploy a balanced pick rather than the single best score.

Do not optimize when the strategy loses on defaults, when there is under two years of relevant data, when the knobs number ten or more, or when the plan is to ship the top score untested. In each of those cases the optimizer will work perfectly and the output will be worthless. The machine does not know the difference. You have to.

The mental model

Optimization is fine-tuning a working engine. It is not building an engine. Sharpe 0.8 becoming 1.1, confirmed out-of-sample, is a real win. That is what winning looks like: small, confirmed, and durable. Sharpe 0.0 becoming 2.5 is a curve fit in a suit. Throw it away and be glad it died on paper, because paper is the only place it dies for free.

How Edgecraft handles this

Edgecraft studies are capable of multi-objective by default, scoring return, drawdown, and robustness together or any other goal the user might have, so one metric cannot quietly buy its score with the others. Every parameter set carries an out-of-sample score next to its in-sample score, and both must pass before deployment. Fragile optima get flagged when small perturbations break them. Redundant trials get pruned so the search budget buys exploration instead of repetition. The machine still has no opinions. It just refuses to hide the evidence.