Smart(er) Investing: How Academic Insights Propel the Savvy Investor

Part IHow to Read and Evaluate Academic Research

© The Author(s) 2019

E. Basilico, T. JohnsenSmart(er) Investinghttps://doi.org/10.1007/978-3-030-26692-9_1

1. What Constitutes Good Investment Research?

Elisabetta Basilico¹   and Tommi Johnsen²  

(1)

Applied Quantitative Analysis LLC, Denver, CO, USA

(2)

Reiman School of Finance, University of Denver, Denver, CO, USA

Elisabetta Basilico (Corresponding author)

Tommi Johnsen

Email: tjohnsen@du.edu

In this and the following chapter, we provide the reader with a tutorial on how to read and evaluate a scholarly research article in the field of finance. We will explain how information is typically organized in an article, what to look for in each section of the article and most importantly, how to evaluate the quality and strength of the research. Journal editors accept manuscripts for review and possible publication primarily if they believe the article will make a contribution to the field. Editors of finance journals are especially alert to the many pitfalls in designing and conducting statistical tests using financial and economic data. There are commonly known methodological errors that are at the crux of failures in our ability to predict investment returns in the real world. It is primarily these failures of prediction that give rise to memorable phrases about the use of statistics and data we are all familiar with:

Facts are stubborn, but statistics are more pliable—Mark Twain

Contrary to expectations, many researchers and practitioners of finance do not comprehend the degree to which faulty application of traditional econometric methods can compromise published investment results. The most frequently committed methodological errors, biases and out-and-out mistakes found in finance articles include:

Ignoring biases that result from the survivorship problem

Ignoring the effects of look-ahead bias due to the time lag structure in financial reporting and the impact of restatement effects in financial reports

Failure to account for transactions costs and liquidity when trading

Failure to make the appropriate risk adjustments to return performance

And the most egregious: datamining, data snooping and p-hacking

We begin our discussion with the Datamining topic.

1.

Datamining, Data Snooping and P-Hacking

Cliff Asness (June 2, 2015) defines datamining as discovering historical patterns that are driven by random, not real, relationships and assuming they’ll repeat…a huge concern in many fields. In finance, datamining is especially relevant when investigators are attempting to explain or identify patterns in stock returns. Often, they are attempting to establish a relationship between characteristics of firms with returns, using only US firms in the dataset. For example, a regression is conducted that relates, say, the market value of equity, growth rates or the like, to their respective stock returns. It is important to note that the crux of the datamining issue is that a specific sample of firms observed at a specific time produce the observed results from the regression. The question then arises as to whether or not the results and implications are specific to that period of time only and/or that specific sample of firms only. It is difficult to ensure that the results are not one-time wonders within such an in-sample-only design.

In his Presidential Address for the American Finance Association in 2017, Campbell Harvey takes the issue further into the intentional misuse of statistics. He defined intentional p-hacking as the practice of reporting only significant results when the investigator has conducted any number of correlations on finance variables; or has used a variety of statistical methods such as ordinary regression versus Cluster Analysis versus linear or nonlinear probability approaches; or has manipulated data via transformations or excluded data by eliminating outliers from the data set. There are likely others, but all have the same underlying motivating factor: the desire to be published when finance journals, to a large extent, only publish research with significant results.

The practices of p-hacking and datamining are at high risk to turn up significant results that are really just random phenomena. By definition, random events don’t repeat themselves in a predictable fashion. Snooping the data in this manner goes a long way toward explaining why predictions about investment strategies fail on a going forward basis. Even worse, if they are accompanied by a lack of theory that proposes direct hypotheses about investment behavior, the failure to generate alpha in the real world is often a monumental disappointment. In finance, and specifically in the investments area, we therefore describe datamining as the statistical analysis of financial and economic data without a guiding, a priori hypothesis (i.e. no theory). This is an important distinction in that if a sound theoretical basis can be articulated, then the negative aspects of data mining may be mitigated and prospects for successful investing will improve.

What is a sound theoretical basis? Essentially, sound theory is a story about the investment philosophy that you can believe in. There are likely numerous studies and backtests that have great results that you cannot really trust or believe in. You are unable to elicit any confidence in the investment strategy because it makes no sense. The studies and backtests with results that you can believe in are likely those whose strategies have worked over long periods of time, across a number of various asset classes, across countries, on an out-of-sample basis and have a reasonable story.

The root of the problem with financial data is that there is essentially one set of data and one set of variables all replicated by numerous vendors or available on the internet that can be used. This circumstance effectively eliminates the possibility of benefiting from independent replications of the research. Although always considered poor practice by statisticians and econometricians, datamining has become increasingly problematic for investors due to the improved availability of large sets of data that are easily accessible and easily analyzed. Nowadays, enormous amounts of quantitative data are available. Computers, spreadsheet and data subscriptions too numerous to list here are commonplace. Every conceivable combination of factors can be and likely has been tested and found to be spectacularly successful using in-sample empirical designs. However, the same strategies have no predictive power when implemented on an out-of-sample basis. Despite these very negative connotations, datamining is not only part of the deal in data driven investing, it requires a commitment to proper use of scientific and statistical methods.

What is the ANTIDOTE to Datamining?Develop and present a theory regarding the underlying mechanism of interest and what hypotheses can be derived from such a theory. Do this before conducting any data analysis. Define the methodology including the period of analysis, how the data will be handled or transformed and what statistical approach will be used. Use a t-statistic criterion that is greater than 3 to avoid p-hacking. Be sure to include out-of-sample testing of some sort. For example, out-of-sample testing conditions can include time periods surrounding the actual period, different asset classes , as well as non-US markets, sectors and countries with varying governance norms, and varying tax rates and trading costs. Confirming results not only within the context of the question being addressed but also across the conditions just mentioned provides evidence that the results are robust. And remember, even though the temptation to datamine is great in terms of asset-gathering… not all significant investment successes are the results of data mining.

2.

Survivorship Bias: Successful Companies Outlive and Outperform Unsuccessful Companies

The contents of a data sample may be inadvertently stacked in favor of positive results if the firms included are exclusively survivors. Although it is intuitively obvious, defining the sample of companies that are alive or operating today and examining their history incorporates a likely positive bias in the performance metrics, returns and other financial data. Excluding companies that have merged, gone bankrupt, gone private or otherwise have become inactive in the past is likely to cause a favorable bias in the sampling process. If we define a sample of firms, let’s say, from the S&P500 at a specific date and trace their membership historically, we see that their numbers decline dramatically. Often the firms that are removed from the index do so for poor performance. Companies fail and disappear, some merge and some are taken over or taken private. If we define the universe sample to be the S&P500 as of the end of the year 2001, and follow just those firms forward, we will also see that the sample decreases for the same reasons. Poor performers disappear leaving the survivors. Those performance characteristics of firms that survive are generally superior to those that dropped out of the index and will bias the results.

What is the ANTIDOTE for Survivorship Bias?The correct sample universe is the set of companies that are in business at any Point in Time that data is being collected and analyzed. There are numerous vendors (including S&P Capital IQ and Thomson Reuters Financial—now Refinitive) that provide this type of historical data. We advise always to ask about survivorship bias in any analysis you are presented with. If it is not addressed, then beware.

3.

Look-ahead Bias: There Is a Lag Structure Imposed by Financial Reporting Delays in Quarterly and Annual Data

Due to the delay in releasing financial information to the public relative to a company’s fiscal year end, tests of historical performance may be subject to look-ahead bias. This occurs because there is a time difference between the date that the reporting period ends and the publication date of the data in the financial report. For example, at the end of the fiscal quarter, the delay in availability of the company’s quarterly report can be delayed as much as two months, and three to four months for annual reports. For the researcher, the error occurs when the assumption is made implicitly or explicitly that the lags do not exist. For example, using annual rebalancing if stock selection occurs on January 1, 2017, the appropriate data to utilize in the study is from the financial statements as of December 31, 2015. If the data from the 2016 annual report is used, it would be essentially looking ahead into the future as of the January 1 date because the annual reports for 2016 will not be available until March or April of 2017.

The results of a five-year simulation conducted by the authors, where stocks were selected on the basis of low price-to-cashflow and rebalanced annually are presented in Table 1.1. Note the size of the upward bias of 34.10% over the period. When appropriate data lags for financial reporting are used the total return is 261.18% compared to 295.28% when unlagged data is used. In every metric presented, the unlagged backtest exhibits superior performance.

Table 1.1

An example of an upwardly biased backtest when unlagged data is utilized

What is the ANTIDOTE for Look-ahead Bias?Use a Point-in-Time (PIT) data source that matches data available with actual calendar dates, or ensure appropriate lags that match the reporting period are implemented in the empirical design. PIT databases offer the researcher accurate information about what information is publicly known at the point in time it is known. Data items in a PIT database are stamped with the real-time company filing, while non-PIT data is stamped with fiscal period end dates. Even worse, non-PIT data is generally overwritten with data that may have been changed by a firm as a result of restatement due to error or accounting changes, for example. Preliminary results, a source of considerable value, are not available. On the other hand, PIT data includes both initial and revised data with a real-time data stamp.

4.

Originally Reported Data Versus Restated Data

The three most likely reasons for restating financial statements include correcting simple errors, changing required accounting methods (GAAP) and changing company ownership or structure. In the case of material errors, the financial statements are required to be reissued. Material errors are defined as those that cause inaccurate conclusions when accounting data is utilized to analyze the financial statements. Currently GAAP rules require a restatement if the new accounting methods would change the previous statements if applied retroactively. The same rule is applied if there is a change in the ownership or structure of the company. In order to maintain comparability, the prior year’s statements must be restated, such that the statements only reflect fundamental or real changes in operations.

Models of stock selection have been shown to vary significantly depending on the restated versus originally reported status of the data reported. Researchers have reported upward performance biases as large as 60% when ranking stocks on factors, for example when restated data is used.

What is the ANTIDOTE for Restatements?The ultimate fix from the researcher’s perspective is again to use a Point-in-Time database, where the original as is data is stored along with the associated revised data. With reasonable justification, the researcher may also infer that the restatement effect is randomly distributed across the data set used and therefore unrelated to the results.

5.

Transaction Costs, Liquidity and the Cost of Shorting

The growing popularity of factor investing and smart beta products has taken center stage for equity investing. The incentives to develop these products are highly attractive, both in academia (more publications) and in the industry (increases in $AUM). Since these strategies are by and large quantitative, they raise the concern not only over data snooping, but over the real costs of trading strategies centered on the most important factors or anomalies. There is quite a bit of disagreement in the literature over the impact of trading costs on excess returns for factor strategies.

An article published in 2016 in The Review of Financial Studies, by Robert Novy-Marx and Mihail Velikov, examines the trading costs of the most important anomalies. They find transaction costs typically reduce value-weighted long/short strategies by 1% of the monthly one-sided turnover. In other words, for a strategy that turns over 20% per month, the spread will be at least 20 bps lower per month. Many of the strategies based on the anomalies studied (at least those with turnover less than 50%) remain profitable but in all cases transaction costs significantly reduce their profitability and statistical significance.

Other studies have documented the substantial impact of costs on the profitability of trading strategies. However, there is an argument to be made that the costs used in conducting research studies are a poor proxy for realized trading costs. The costs of trading typically used in academic studies in the literature are the trading costs for the average investor, which are vastly different (higher) than those incurred by large institutional asset managers. Frazzini et al. (2015) utilized live trading data in a study designed to examine trading costs and found quite a large discrepancy when compared to the average.

Setting the financial crisis of 2008 aside, most researchers would agree that 90% of US stocks are very liquid and relatively cheap and easy to trade and borrow. This raises the question as to whether or not the remaining 10% matter at all if they are excluded from a research sample. An examination of the content of poorly ranked stocks on most factors reveals that they are typically the most expensive to short. Under the limited-arbitrage framework, this type of shorting difficulty would prevent arbitrageurs from exploiting the anomaly, leaving those stocks underpriced in the market. Hence, strategies that depend on those stocks representing the shorting universe will exhibit an upwardly biased rate of return in the statistical analysis. Interestingly, and in spite of the reality that some stocks are either unavailable