rungfolio

Fitting the past

What it is

Run enough backtests over one date range, keep the settings that did best, and you'll find settings that did very well there. Part of that is the settings, and part is luck: the gap between rungs that happened to match that year's dips, or the target that happened to sell just before a fall. The past can't change, so a result tuned to it keeps every bit of that luck. The next stretch of prices has different dips, and the luck doesn't carry over.

This is called fitting the past, or overfitting. The more settings you try on one window, the better the best one looks, and the more of its result is likely to be luck.

Signs of it

  • A sharp best: the best settings beat their near neighbors by a lot. Nudge the gap or the target a little and the result drops away.
  • A result that hangs on a few trades: most of the gain came from a handful of cycles, or from one rally the ladder happened to be fully bought into.
  • Lots open at the end that carry the result: the return looks good because the range ends on a high close. Ended a few weeks earlier, it would look worse. Reading a backtest explains why lots open at the end matter.
  • Many settings changed at once to reach the result, each of them tuned.

How to check a result

Treat a backtest you've tuned as one good guess, then try to break it:

  • Another window. Run the same settings over a range you didn't tune on, such as the year before or the year after. Settings that only work in the window they were chosen in won't hold up.
  • Another ticker. Try them on a similar fund. A rule that's sound on one leveraged fund should do something sensible on another, even if the numbers differ.
  • Buy-and-hold, every time. Compare each run with buy-and-hold over the same days. A run that beat it in one window and trailed it in the next two hasn't shown much.
  • The risks, not just the return. Look at maximum drawdown, peak capital in use, days paused and lots open at the end in each window. A drawdown that's shallow in one window and deep in another shows how much the ladder's risk depends on the prices it meets.
  • Small changes. Move each setting a step either way. If the result holds roughly steady, the settings are less likely to be fitted to one stretch of prices.
  • Side by side. Save each run, then compare two to four of them at once: their returns on one chart, and their settings and numbers in one table, with the rows they differ in marked (reading a backtest shows how).
  • Keep a window back. Choose settings on one stretch, say 2016 to 2021, and only then look at a later one, say 2022 on. The later one is the closest a backtest gets to a fresh test.

Settings that do reasonably well everywhere you try them are usually a better bet than the ones that did best in one window.

What a backtest can't tell you

Even a result that holds up across windows and tickers is still about the past. Leveraged funds can behave in ways no stored year shows, and a backtest leaves out what the simulator doesn't model. Practice mode is the next step: it runs the settings on prices no one has seen yet.

Educational software, not investment advice.