Strategy Methodology

Probability Calibration

A method for adjusting raw model outputs so that predicted probabilities match observed frequencies — a 70% prediction is correct 70% of the time.

Most models produce scores, not probabilities. A machine learning model that outputs 0.8 for a trade signal is not saying there is an 80% chance of success — unless it has been calibrated. Calibration maps raw scores to reliable probability estimates by comparing predictions against actual outcomes over a historical dataset.

The botwir3 calibration module applies Platt scaling or isotonic regression to transform raw signal scores into calibrated probabilities. These calibrated outputs feed into position sizing: a signal with 65% calibrated probability gets sized differently than one with 85%. The gate uses these probabilities when evaluating whether a proposed position stays within the configured tolerance band.

Calibration is only as good as the data it was trained on. If the market regime changes, calibration accuracy degrades until the model is recalibrated. The user is responsible for monitoring calibration quality and recalibrating when performance drifts.

Sources

Dawid, A. P. (1982). “The Well-Calibrated Bayesian.” Journal of the American Statistical Association, 77(379), 605–610.Formalized the definition of calibration for probabilistic forecasters.
Platt, J. C. (1999). “Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods.” Advances in Large Margin Classifiers, MIT Press, 61–74.Introduced Platt scaling — the most widely used post-hoc calibration method.

Used in

◎Probability Calibration module — builder

See this in action

→ polymarket→ kalshi→ manifold→ betfair

Related

→ Monte Carlo Simulation→ Edge→ Overfitting→ Backtesting
Build your bot →