A key step in investment product construction is determining how a product might perform in the real world. Before investing real dollars, it often makes sense to “kick the tires” and track hypothetical performance. However, to gain a significant undertanding of portfolio behavior, a product would have to be monitored in realtime for many years.
Instead, many firms employ a form of testing called “backtesting.” At its core, backtesting is applying the rules developed in portfolio implementation to historical data to reconstruct hypothetical portfolio performance. This allows the immediate and rapid testing of a portfolio over a large time-period without having to wait several years.
The goal of backtesting is to determine how a portfolio or model would have performed in the past. While backtesting is a useful tool in product construction, it is ultimately limited because it is done over a single, realized path of returns and is not determinative of future performance. Maximizing historical performance over this path is dangerous and leads to brittle portfolio designs; nevertheless, many firms design their portfolio process in just this way, optimizing their performance over the historical data they have.
Furthermore, many firms commingle rule development and testing. This is dangerous because excessive portfolio development after backtesting often leads to data-snooping. Backtesting also has considerable risks due to inaccurate, or altogether missing, data. Even worse is restated data, which gives an unrealistic picture of behavior and performance because the backtester is able to use information from the future. The more data dependent the strategy, the greater these risks.
However, when used appropriately, its limitations realized, and the results looked through with a lens of scrutiny, backtesting can be an extremely useful tool.
At Newfound Research, we use backtesting as a means to explore hypothetical portfolio behavior, and we think that is a key distinction in what makes our portfolio development process unique. We use backtesting not as a means to explore, or optimize, historical portfolio performance, but rather as an opportunity to explore the robustness of the rules we developed and the portfolio assumptions we are making.
When we perform a backtest, we look at the number of times each rule is applied to the portfolio and the different environments it is applied in. If a single rule dramatically changes the behavior or performance of the portfolio and does not occur a statistically significant number of times in a varying number of market environments, we are skeptical about portfolio behavior going forward. An ideal backtest for us is one in which all portfolio rules are applied multiple times over varying market cycles.
We also look to see how rules co-exist within the portfolio in an attempt to find hidden assumptions we have yet to identify. If two rules are constantly clashing with one another, it may be because we have a lurking or confounding assumption.
It takes a defined process to ensure that the rules developed during the implementation phase do not become over-optimized, at the cost of robustness, to achieve excess historical performance.
What we consider unique about our process is that we will actively look for periods of underperformance in market cycles as evidence of a well-behaved portfolio, so long as our assumptions warrant it. For example, if we build a portfolio on the assumptions of moderate inflation, we would expect underperformance in a deflationary environment. While many firms would look at a backtest that also outperforms in a deflationary environment as evidence of an “exceptional” product, we look at it as one we have yet to fully understand.
At Newfound Research, backtesting is a tool we use to explore the statistical significance and robustness of portfolio rules, both over multiple market cycles and in how they co-exist. In exploring historical behavior, we attempt to find and identify confounding or lurking assumptions. In our process, we consistently stress that optimization to historical performance is a way to guarantee a broken portfolio process in the future and that backtesting should only ever be used to accept or reject portfolio construction assumptions, never to develop them.