Crunching numbers in a monastery
During the first half of the 1600s a French monk, Marin Mersenne, had many acquaintances in the scientific world. Mersenne studied (and taught) theology, philosophy, mathematics and music. He communicated extensively with other scholars like Descartes, Pascal, Huygens and Galilei.
In spite of being a theologian and philosopher primarily, Mersenne’s name is associated with prime numbers that compound to Mn = 2n – 1. Such numbers are called Mersenne primes. The first four Mersenne primes are 3, 7, 31 and 127 and significantly a Mersenne prime (219937−1) is elementary for the most commonly used version of the Mersenne Twister.
The Mersenne Twister is a fast generator of high-quality pseudorandom integers. Recently AmiBroker’s already extensive feature set was expanded with a Mersenne Twister based Monte Carlo simulator which is capable of rendering 30+ million trades per second (!). More specific, the Monte Carlo simulator runs series of trade sequences based on backtest output and uses the high-quality Mersenne Twister for randomizing the order of the trades.
And so we finally arrive down the stairs of the famous "Casino de Monte-Carlo" in mondain Monaco ;-)
Why stress test strategies with Monte Carlo Analysis?
Before we start familiarizing ourselves with Monte Carlo Analysis let’s first pick a sample strategy for illustration purposes: SeekingAlpha's contributor Varan's Simple GMR. Each month all available trading capital is re-allocated to the top performing ETF out of a basket with IJJ, EFA, IEV, EPP, QQQ, EEM and TLT. See Varan's post for details. For establishing points of reference and collecting the trade data required for a Monte Carlo Analysis, a backtest is run starting at year-end 2003 and ending August 2015 using high-quality monthly total return data as provided by Norgate Premium Data (Alpha-tester program).
The equity curve as well as the distribution of the yearly returns obtained from the backtest look reasonable, even considering the 2008 drawdown when compared to the market in general. Volatility is not too high. Actually, the ratios for Sharpe, Sortino and Calmar are quite nice. The complete chart suite is available in the Google drive folder connected to this post (zooming required!).
 |
| Portfolio performance over 2004 - 2015 |
 |
| Yearly returns |
 |
| Profit contribution |
 |
| Monthly trade results |
Looking at the charts and the key performance metrics, the strategy appears to be profitable. But how confident can a trader really be that “live” application of the strategy will yield approximately the same results? After all, the backtest rendered just one single equity curve and nobody knows what the future will look like. What the trader needs to do is examine the likelihood of the reported backtest metrics to reflect “live” trading results with a suitable degree of confidence. Welcome to the “Casino de Monte-Carlo”!
What is Monte Carlo Analysis?
For judging the probability of a specific outcome, Monte Carlo Analysis is applied. A Monte Carlo simulation is performed to address problems too complex for equations. It relies on repeated random sampling from a set of trades to estimate the mean and distribution of performance metrics by applying statistics. As Howard Bandy once put it:
"Monte Carlo Analysis is used to rearrange the sequence of trades many times. Typically many sequences are used -- 1000 or more -- each of many trades -- 100 or more. After all 1000 runs, all of the equity curves are draw on a single chart and statistics computed that will allow you to estimate the final equity and probability of both going broke and retiring wealthy. The plot looks like a straw broom with the straws angled upward to the right."
The above “straw broom” chart paints 10 individual random sequences of the change in equity (in gray) that were performed in the backtest with Simple GMR universe as well as the average of all simulated sequences as an approximation of the expected change in equity (in blue). Note that these 10 example sequences are equally likely, there are many more possibilities (140140 to be precise), only one sequence will materialize and nobody can predict which sequence that is going to be.
In his book "Modeling Trading System Performance" Howard Bandy states:
The result of a single simulation ( such as a single backtest run ) is a qualified statement. For example, "The system appears to be profitable".Put differently, Monte Carlo Analysis is all about gaining confidence based on statistical probability. In trading nearly any outcome is possible, but the trader wants to know the probability of his expectations. By imposing a reasonably conservative confidence level of 95% to himself, the trader can assess the probability of a specific outcome. However, even with 95% confidence, still a 5% risk lies in wait for embarrassing the trader’s statistically founded expectations and, more important, to wreak havoc to his account.
The result of a probabilistic Monte Carlo simulation is a quantified probability. For example, "There is an 80% probability that the system will have a compound annual rate of return of 12% or more."
Croupier chorus (joyful chanting): “Il n'y pas de certitude!” (encore)
Monte Carlo stress test results
By backtesting IJJ, EFA, IEV, EPP, QQQ, EEM and TLT over 2004 - 2015 with monthly reforms into the top performing fund, the backtest spans 11 years and 8 months or 140 trades. The results of each of these 140 trades are the sample set used by the Mersenne Twister to randomize the sequence. In their twisted sequences trades may occur in a different order, trades may be omitted or trades may be duplicated. See AmiBroker's website for an explanation of the general methodology. Due to the blazing speed of AmiBroker’s Monte Carlo engine 1,000,000 simulations were applied for generating the following metrics and graphs (total i7-CPU time: 6s (!)).
The above table is very helpful by interpreting the results from the graphs about to follow, so there is no need to "guess" these readings each time from the graphs.
The CAR level as reported by the backtester (see first chart in this post) is 24.71%. This is a near match to the 50 percentile score on the above numerical table. In other words, the table informs the trader that there is (only) a 50% probability that CAR will be around this level or perhaps higher. Of course, sitting at the 50 percentile level, the likelihood of a lower CAR is equally high, so 50% too.
Given the earlier adopted reasonably conservative 95% confidence level, probability states that CAR will be 14.62% or higher as is shown on the following Monte Carlo CAR graph.
Regarding drawdown, the below Monte Carlo graph informs the trader his maximum drawdown percentage will probably be 33.42% or better. So the registered 14.40% maximum drawdown from the backtest isn’t really informative at all. Frankly, in view of this 95 percentile score, one could maintain that the reported -14.40% drawdown is rather deceptive. The likelihood that the incurred maximum drawdown will be near the reported -14.40% or better is only about 10%.
Lastly the trader may be quite confident that a wipe-out of his account is very unlikely to happen: given an initial capital of $100,000, there is only a 1% probability that the trader will lose more than $26,000 from his starting capital.
Wrapping up, Monte Carlo Analysis takes away any false expectations entertained by a trader from a (single) backtest and it enables him to become realistic about the prospects of a strategy and his financial future.
A simplified rule of thumb for Monte Carlo based probability derived from a backtest might be: split CAR in half and double MaxDD.
Final remarks
AmiBroker's developer, Tomasz Janeczko, informed me that using Monte Carlo Analysis is not feasible for models that invest in multiple ETFs at the same time. Compounding by itself is not a problem as long as there are no overlapping trades. I have yet to fully understand why, but please take notice.
For for the graphs in this post the following AFL driven code was used:
The full code for Simple GMR including the Monte Carlo routine is available on the Google drive folder connected to this post. Run the code on monthly periodicity. On the Google drive also the full trades list is shared in html format allowing import into Excel for further analysis or even Monte Carlo DIY.
Further reading
Howard Bandy provides in-depth coverage about Monte Carlo Analysis as well as probability based position sizing in his aforementioned book "Modeling Trading System Performance", including instructions for applying Monte Carlo Analysis with Excel.