Guy Carpenter’s extensive back-testing of reserve distributions created using the popular paid chain ladder bootstrap method has shown that these distributions materially under-estimate reserve risk.
With the U.S. industry annual statement data (1) and thousands of hours of research, Guy Carpenter in collaboration with Risk Lighthouse have bootstrapped thousands of loss triangles in an attempt to answer the question: Does the bootstrap model work?
How we did it:
Back-testing as at December 2000:
1. We can go back in time 11 years and create a reserve distribution using the paid chain ladder bootstrap method as at December 2000, using Schedule P data for a particular company A’s homeowners book of business. (Figure 1) (2)
2. Now it turns out that, with hindsight, it actually cost USD45 million to settle these claims. We call this the “actual” reserve - what the reserve should have been, with perfect hindsight. The USD45 million falls above the 90th percentile of the original distribution. This is just one result, so we can’t tell if the model is “good” or “bad”.
3. We can repeat steps 1 and 2 for another 50 companies. The percentiles for these companies are listed in figure 2. It is good to see that not all of the percentiles are above the 90th percentile.
In fact, what we want to see when we plot a histogram of these percentiles is a uniform distribution, as shown in figure 3, where we show 5 buckets each with 20 percent of the companies.
This is because that is the definition of a percentile - for example, the 80th percentile is the value you expect to be above 20 percent of the time, so, if the model “works” then 20 percent of the companies will fall in the 80th to 100th percentile bucket. There should also be a 20 percent chance of falling in the 60-80th percentile buckets, and so on.
When we plot the percentiles in figure 2, what we actually see is shown in figure 4:
That is, 26 out of the 51 companies had “actual” reserves that fell above the 80th percentile of the original distribution. This distribution is biased toward adverse developments. This is not a good result, but there is more to the story.
Back-testing as at December 1996:
4. We can try this at another time period - instead of December 2000, we can try December 1996. That is, we repeat steps 1 to 5, but this time, we are creating reserve distributions for 51 companies as at December 1996. When we do this, below is the histogram that we see:
Figure 5 shows a distribution biased toward favorable developments, which is the opposite of figure 4!
Back-testing multiple periods:
5. Our next attempt is to try the test across many time periods as well as many companies, and plot all the percentiles into one histogram, which might result in a uniform distribution of percentiles. In this test, reserve distributions were created as at December 1989, December 1990, … up to December 2002, across almost 100 companies. We bootstrapped a 10 accident year by 10 development year triangle, but only considered the reserve for the most recent accident year. This gives us just over 1,000 percentiles that we can plot in one histogram, creating figure 6.
Figure 6 shows that, around 20 percent of the time, the actual reserve is above the 90th percentile of the bootstrapped distribution, and 30 percent of the time the actual reserve is below the 10th percentile of the distribution.
When you tell management the 90th percentile of your reserves, this is a number they expect to be above 10 percent of the time. In reality, we find that companies have exceeded this number 20 percent of the time. The bootstrap model is under-estimating the probability of extreme reserve movements, by a factor that is clearly material for the purposes of capital modelling and therefore Solvency II.
Want to know more?
Guy Carpenter in collaboration with Risk Lighthouse have back-tested thousands of companies on all lines of business represented in the U.S. annual statements, with similar results. In the next article in the series, we will ask the questions: Why are we under-estimating reserve risk and what can we do about it?
An academic paper on our back-testing research is in the works.
Statements concerning tax, accounting, legal or regulatory matters should be understood to be general observations based solely on our experience as reinsurance brokers and risk consultants, and may not be relied upon as tax, accounting, legal or regulatory advice, which we are not authorised to provide. All such matters should be reviewed with your own qualified advisors in these areas.
1. Every year, insurers in the United States are required to submit an annual statement, which includes Schedule P. This schedule contains paid, incurred and booked ultimate loss triangles, of 10 accident years by 10 development years, net of reinsurance, by line of business. Importantly, we can track the losses and booked reserve estimates for the same book of business for 10 years. Guy Carpenter and Risk Lighthouse have developed a research database of these filings going back to December 1989.
2. It is important to note that company A is a real company, the loss triangle used in the bootstrap model is their real loss triangle as at December 2000 and the “actual” reserve is how much it really cost to fulfil the claims that were reserved for as at December 2000 (we have pain-stakingly tracked this). There is no simulated data in this back-test.
- “Actual” reserve = what the reserve should have been, with 9 years of hindsight. For example, for a reserve set as at December 2000, by December 2009 we know what the reserve should have been, with hindsight.
- “Actual” reserve as at December 2000 = Ultimate loss as at December 2009 less Paid to date as at December 2000