Is an enormous issue, and one of the biggest NO-NOs if you are a quant.
If you are not familiar with survivor bias it is when a backtest, or reported results, exclude companies that have been delisted due to mergers, acquistions, bankruptcies, etc. If you are a long time World Beta reader you would recall the fantastic Blackstar study that shows just how big of a deal this is. Roughly 20% of all public company stocks go to zero. And that is the Russell 3000, not even small and micro penny stocks where it would be much worse.
Anyways, I was reading an article that reported some backtests based on Wealth Lab (Fidelity bought them) and I seem to recall that their tests suffer from survivor bias?
A somewhat befuddled Fidelity rep confirmed that indeed this was the case. Any Wealth Lab Pro users on here?
wealthlab pro seems like a cool product, only problem is the underlying database. Reuters has one of the best point in time databases. they even keep track of earnings re-statements etc. so true walkforward testing is possible using fundamental data.
FYI, the survivorship bias impact is larger for small caps studies as they have a higher blow out rate.
Wish the world of free data sources started including the deletions. Getting bias-free data seems to add 10x to the cost.
Is there any way to get access to the point-in-time as a small trader without paying up? Do universities/students commonly get a break on access?
Indeed, WealthLab is no better than the database it reads from. When using the Fidelity version, it uses Fidelity data that do have survivorship bias.
In its “free” form (available to non-US residents) it can read several other databases, like TC2000, but all suffer from survivorship bias.
[...] Suvivor Bias – Via World Beta – Is an enormous issue, and one of the biggest NO-NOs if you are a quant. If you are not familiar with survivor bias it is when a backtest, or reported results, exclude companies that have been delisted due to mergers, acquistions, bankruptcies, etc. If you are a long time World Beta reader you would recall the fantastic Blackstar study that shows just how big of a deal this is. Roughly 20% of all public company stocks go to zero. And that is the Russell 3000, not even small and micro penny stocks where it would be much worse. [...]
Survivorship bias (SB) is just one of several issues with respect to “backtesting” stock trading systems. Nick points out that SB has less impact on “large cap”, say S&P 500 member stocks. That's true, but there is another problem to contend with. I'll use an example. XTO Energy (symbol: XTO) is the best performing S&P 500 stock over the last 10 years. Problem is…it wasn't in the S&P 500 while most of its gains were being enjoyed. Getting into the S&P 500 is the natural result of a long and profitable run for a company. If you are BACKtesting on TODAY'S S&P 500 stocks you are guaranteed to have the best performing stocks in your unrealistic sample. But you couldn't have known BACK THEN which stocks would go on to become S&P 500 members. I call this postdictive selection bias. An analogy would be keeping a record of all the legendary race horses, and then going back in time and only betting on them.
Another problem has to do with non-representative prices due to stock splits. I can’t count the number of times I’ve seen people build systems that focus on buying $2 stocks in anticipation of a rise to $100. They often cite stocks like Microsoft or Cisco as having been under $2 in the early 1990’s as evidence that epic winners originate from the ranks of penny stocks. Problem is…neither stock was trading below $2 back then. It just looks that way on a typical chart from a typical database because of stock splits. Microsoft was actually trading above $100 per share while Cisco was in the $40’s.
Another problem has to do with dividends. Nearly half the market’s total return over time has come from dividends. Most databases and virtually all charting packages fail to account for dividends. This biases the backtest away from dividend yielding value stocks and towards growth stocks.
Furthermore, even if the typical backtesting software packages could get the data issues right…they still don’t aggregate portfolio-level results correctly. They simply blend individual equity curves together at the end of a simulation run or rebalance period. That’s not how real life works.
Products like PowerST, Mechanica, and TradingBlox allow one to do things right…but doing it right is expensive and incredibly time consuming.
SB is often not a big issue (at least for small investors like me) as you are likely to trade trends and therefore not likely to keep your stocks until they reach 0.Furthermore few traders do buy and hold.
Only the international version of Wealth-lab is interesting, and it is one of the best back-test software out there.SB is not caused by WL, you put in the data you wish (mines are clean of SB).
You obviously didn't read anything I just wrote; but then again, that's okay…carry on.
Eric I did read your comments. One of the nice features of WL Pro and the Fidelity
database is that it does include dividends.
It's SURVIVOR, not SUVIVOR.
Man, you need to get some sleep. Or a proofreader.
Funny how the title means something completely different when you don't include all letters.
Norgate is a great source for stock data.
Uh, I'm not a longtime reader — where's the Blackstar study? The link leads to a paper on “The Capitalism Distribution”, which in turn had a link to “Does trend following work on stocks?” (an excellent read), but no study specifically on SB.
Would like to verify that “Roughly 20% of all public company stocks go to zero.”
I don't know that the information “Roughly 20% of all public company stocks go to zero.” is valid or useful to everybody.
I assume that any stock which goes to zero is probably trading under a moving average during its descent …
Wealth Lab backtesting is great for computer expert. For monthly or quarterly balancing, 2008 and 2009 data are enough (great bear and bull in two years) for manual testing. I use etf selector from http://www.etfquest.com and do manual paper/pencil back testing.
So it's still spelled incorrectly in the title, and Fidelity is misspelled in the body.
It is almost like it is intentional (Suvivor, not Fidelity) . . .
I'd love to hear from other readers, but I know of no generalized backtesting tool that deals with survivorship bias well – I'm not sure that even TradingBlox deals with it – would love to be corrected on it. The problem, however, is both the tools and the databases. First, no one yet offers a tool where with a dynamic portfolio that can have components come in and out at different times based on database data. They all offer the ability to have, say, a filter on a watchlist that selects stocks based on, say, volume. But no tool that I know of allows you to select, say, the S&P 500 and then the tool deals with all the events that bring stocks in and out of the index. The second problem is the database which others have mentioned.
It is amazing to me that no one has dealt with this so far – Norgate has said that they are working on the issue and will have a solution for Amibroker which I am excited about.
Portfolio123.com has addressed survivor issues
Great point Carl – I had forgotten about P123.
Thanks for sharing
Diana
Thanks for sharing
Diana