Changes of sentiment don’t cause markets to ebb and flow; but their short-term fluctuations, which are largely random, affect sentiment.
“So far, so good!” Ken Fisher recently exulted (“To forecast financial markets, get sentimental,” The Australian, 16 April). “My 2024 outlook (issued in January) called for strong gains down under and worldwide – but with Aussie stocks lagging early before soaring late.” What underpins Fisher’s ebullience? “Sentiment. Stocks always move most on the gap between expectation and subsequent results, so gauging the former is crucial. There are hard ways and easy ways to do it … Whether you choose easy or hard methods, tracking sentiment is key. Today it all signals more (market) gains ahead.”
Unlike Fisher, in this article I explicitly distinguish and analyse subjective (based upon investors’ unobservable, hard-to-measure but undoubtedly existing feelings) from objective (based upon directly-observable and easily quantifiable market prices) measures of investor sentiment. Further, and unlike his principal source of evidence, I analyse publically-available data. Finally, decades-long series substantiate my conclusions; Fisher’s, in sharp contrast, rely upon a handful of recent anecdotes.
I conclude that in three key senses Fisher is incorrect. First, he greatly overstates the importance of investor sentiment. In the short term, markets’ fluctuations are largely random; their systematic variation is thus a relatively small portion of their overall variation. Secondly, Fisher mistakes the nature of sentiment’s limited influence: it’s a “contrarian” correlate (but not cause) of tomorrow’s returns, such that higher sentiment today typically means lower returns tomorrow.
Finally, and bearing in mind markets’ largely but not completely random short-term fluctuations, both objective and subjective measures of sentiment presently signal the opposite of what Fisher expects: either mediocre gains or outright losses.
Except on the very infrequent occasions when it reaches extremes, Leithner & Company ignores investor sentiment: our actions stem from our own rigorous analyses rather than others’ transitory moods; moreover, our actions acknowledge that returns always transmit information – but recognise that often they don’t emit sensible signals.
As conservative-contrarians, and like Benjamin Graham, we derive no pleasure from acting with or against the crowd, and take no comfort because prominent, vocal or great numbers of people agree – or, more likely, disagree – with us. Adherence to passing fashion is no substitute for timeless principles, reasonable premises, valid reasoning and hard evidence.
What Is Investor Sentiment?
Investor sentiment, also known as market sentiment, refers to the mood of investors toward the stock market (as measured by an index such as the S&P 500) and its short-term prospects – that is, their belief that prices will rise or fall.
The many speculators who follow sentiment seldom express their key assumption explicitly: the more positive is today’s sentiment, the greater is the likelihood that stocks’ prices will rise tomorrow and in the near future. If so, speculators exert positive momentum. Conversely, the more negative is today’s sentiment, the more likely it is that stocks’ prices will fall. That’s negative momentum.
Several difficulties – and thus risks – accompany the incorporation of sentiment into investment decisions.
First, its measurement is hardly an exact science. To cite but one example: drawing representative samples of investors’ sentiments is always difficult and often prohibitively expensive. Second, what’s influencing what? Does today’s sentiment influence tomorrow’s returns? Or do yesterday’s returns influence today’s sentiment?
In other words, is the sentiment of today’s market participants merely a by-product of their recent returns, such that strong returns generate high sentiment and poor returns produce weak sentiment? For this and other reasons, it’s easy – as, I’ll show, Fisher does – to misstate and overestimate sentiment’s ability to anticipate changes of major market indexes.
Finally, and adding to the complications, assuming that you possess a valid and reliable measure of sentiment and have clarified what’s doing the influencing and what’s being influenced, what’s the direction of the relationship?
Specifically, is sentiment a “momentum” indicator, such that positive and rising sentiment today begets increasing returns tomorrow? Or is it a “contrarian” indicator, such that positive and rising sentiment presages decreasing and even negative returns, and negative sentiment signals stronger results?
Fisher’s Assessment of Investor Sentiment – and My Criticism
Ken Fisher asks: “how do you assess sentiment? One tough way my firm does you probably can’t replicate … plotting professional sentiment bell curves.” Not unreasonably, he assumes that the sentiments of “Wall Street professional” investors at a given point in time follow, at least roughly, a “normal” (bell-shaped) distribution.
The distribution’s (bell’s) top quantifies its central tendency – that is, where most professionals’ sentiments cluster. Sentiments are distributed symmetrically around the mean (some professionals sentiments are above-average, others’ are below-average), creating downward-sloping curves on each side of the peak. The wider is the distribution of sentiments, the larger is its standard deviation; in particular, the distribution’s “tails” (the bell’s bottom) represent the relatively few professionals whose sentiments lie far above and far below the mean.
Fisher doesn’t say so explicitly, but (again not unreasonably) assumes that the distribution of the accuracy of pros’ sentiments – that is, their mood today vis-à-vis their subsequent returns – isn’t normally distributed; in particular, its tails are “fatter” than those of a normal distribution.
If so, then the actual likelihood of extreme events – subsequent results that are much better or much worse than today’s expectations – is greater than a normal distribution implies. In other words, the unexpected is more likely than the consensus anticipates. That’s a crucial insight.
Fisher elaborates: “an easier sentiment-tracking tool is to watch how economic (and financial) data compare to (consensus forecasts) … Are subsequent results worse than expected? If so, sentiment is likely too optimistic. Are they beating (expectations)? Too dour. Low expectations mean weak results don’t doom stocks.”
Yet he omits to ask: do high expectations tend to beget disappointment – and consequent low returns? He also ignores what I recently detailed: a pillar of the mainstream’s obsession about “consensus forward earnings” – namely that markets boost the shares of companies whose earnings “beat the consensus estimate” – is generally false.
Specifically, investors don’t consistently punish companies whose earnings “miss” expectations. Moreover, companies which provide earnings guidance typically aren’t valued more highly than those which don’t; nor does their “forward guidance” tend to tamp the volatility of their shares’ prices (see Everything the mainstream says about earnings is wrong, 13 March 2024).
But let’s not let mere disconfirming evidence spoil a good story! In support of his claim, Fisher cites two examples (I’m tempted to say “anecdotes”):
- In early-2018, professionals expected that during the year to come (he doesn’t say, but I assume American) stocks would rise 5.3% (excluding dividends). The lower tail of the distribution captured the actual result: a decrease of 6.2%.
- In 2019, the median professional prophesied a rise of 15.8%, yet the actual result (28.9%) was in the upper tail.
At the beginning of this year, Fisher says, the median professional forecast that stocks would rise 1.8%. As he notes, that’s not very optimistic. Of the 54 professionals he tracked, the expectations of 40 clustered in the range 2.9-9.0%. Another nine envisaged market declines of 3.0% or worse, and one foresaw losses exceeding 17%.
Expecting the unexpected, Fisher therefore predicts a good 2024: “the relative void above 10% (at the beginning of this year) suggested above-average gains were quite possible, even probable. That is happening now.”
For two reasons, I doubt it. First, he looked at the wrong investors – or, at least, an overly restricted stratum of investors. Indeed, in this respect Fisher’s previous research – which to my knowledge he hasn’t recanted and others haven’t repudiated – undermines his current glib assertions! In “Investor Sentiment and Stock Returns,” Financial Analysts Journal, (vol. 56, no. 2, 2000), Fisher and Meir Statman “show that the sentiment of Wall Street strategists is unrelated to the sentiment of individual investors or that of newsletter writers, although the sentiment of the last two groups is closely related.”
Fisher’s current expectations of market gains rest exclusively upon professional investors. What about semi-professionals and non-professionals? Does he ignore their sentiments because he believes that professionals are superior forecasters? If so, he’s clearly mistaken (see, for example, Experts can’t predict yet investors must plan: What, then, to do? 23 November 2020).
Secondly, Fisher’s contention that professionals’ sentiment will extend the past year’s strong returns is, in light of his previous research, highly questionable. He and Statman “found a negative relationship between the sentiment of each of these three groups and future stock returns, and the relationship is statistically significant for Wall Street strategists and individual investors.”
In other words, if past is prologue and contra Fisher’s assertion, the recent sharp rise of bullish sentiment among non-professionals (see Figure 1, Figure 4 and Figure 6 below) will likely beget modestly positive and perhaps even negative returns tomorrow. In his and Statman’s words: “the sentiments of both small and large investors are reliable contrary indicators for future S&P 500 Index returns.”
Two Measures of Sentiment
In this article I’ll analyse two indicators of market sentiment. The first, the American Association of Individual Investor’s weekly survey of its members, is subjective: it gauges respondents’ opinions or feelings about the S&P 500’s prospects. The second, the CBOE Volatility Index (“VIX”), is objective: based upon today’s prices in options markets, it measures expected price fluctuations of S&P 500 Index.
AAII’s Investor Sentiment Survey
Each week since 24 July 1987 – that’s more than 1,900 weeks! – the American Association of Individual Investors (AAII) has asked its members: “Do you feel the direction of the stock market over the next six months will be up, no change or down?”
Various organisations and media outlets, including Barron’s and Bloomberg, follow and publicise its results. Birinyi Associates (“We are unique in that we do not analyze the economy, have little interest in corporate developments and fundamentals, and have little use for traditional, technical, quantitative or other market indicators. Our approach is to understand the psychology and history of the market, and most importantly the actions of investors”) has been following AAII’s survey for years,
According to AAII, the higher is the percentage of respondents that say the market will rise over the next six months, the more confident (“bullish”) is sentiment. If so, then high levels of confidence likely signal overconfidence.
Conversely, the larger is the percentage of respondents that say the market will fall over the next six months, the more fearful (“bearish”) is sentiment. Finally, the higher is the percentage of neutral responses, the more uncertain sentiment becomes.
The AAII’s average member is male, aged in his late 50s and holds a tertiary degree, and over half of its members own an investment portfolio of $500,000 or more. Its survey thus represents a small group of active and very well-to-do investors. Clearly, the views of respondents to its survey may well differ from those of investors as a whole.
The Wall Street Journal (“Investor Survey Says: Bet Oppositely,” 9 December 2010) goes much further: “what … few on Wall Street know is that the (AAII’s) survey’s sample size is typically so small, and its methodology fraught with holes, as to render it statistically worthless.”
WSJ elaborates: “Just 200 to 300 investors respond each week … From a strict, statistical perspective, the survey is ‘pretty much useless,’ said David Madigan, professor and head of the Department of Statistics at Columbia University, who is particularly troubled by survey’s reliance on voluntary self-reporting. ‘The thing you worry about is the bias of the people who volunteer … But maybe the opinions of the 200 who are motivated enough to respond is predictive of what the markets are going to do.’”
Madigan’s criticisms are valid. Yet they apply just as much or even more to Fisher’s sample of ca. 60 Wall Street professionals, The Wall Street Journal’s long-running survey of ca. 80-100 economists, etc.
The AAII’s sample is small; accordingly, its margin of error is wide (aggregating weekly samples into monthly ones mitigates this defect). Moreover, the survey’s sample is likely unrepresentative of the general population of investors.
Why, then, consider the AAII’s survey? Although it doesn’t pass professional statistical muster, it’s nonetheless very useful. As WSJ acknowledged, “despite its formidable statistical limitations, … over the past two decades it has proved a compelling contrarian indicator: if (its) reading is overly bearish, for instance, it is often a sign the market will rally.”
CBOE’s Volatility (“VIX”) Index
The Chicago Board Options Exchange (CBOE) is America’s and the world’s largest. VIX, which CBOE created and CBOE Global Markets maintains, measures the implied volatility, based upon the bid and ask quotes of a range of near-term call and put options traded on CBOE, of a hypothetical option on the S&P500 Index with 30 days to expiry. (Until September 2003, VIX’s formula was very different. According to Macroption.com, “obviously, the two methods produce different index values, although the differences are relatively small …”)
Volatility, says Investopedia, “is often seen as a way to gauge market sentiment, and in particular the degree of fear among market participants.” Hence VIX is widely known as “the fear index.” The higher VIX rises, the greater, by implication, is investors’ fear. The lower it falls, in contrast, the lower is their fear and the greater is their confidence – and at extremely low levels, overconfidence and even greed.
Investopedia concludes: VIX “is an important index in the world of trading and investment because it provides a quantifiable measure of market risk and investors’ sentiments.”
Results: AAII’s Investor Sentiment Survey
Figure 1 plots the percentage of “bullish” respondents to AAII’s survey; that is, those who believe that stocks will rise during the next six months. In order to tamp a considerable amount of random week-to-week fluctuation, I’ve expressed these percentages as 26-week moving averages (MAs). From 24 July 1987 to 21 January 1988, for example, an average of 39.4% of respondents reckoned that stocks would rise, and so on until the average of 44.3% from 12 October 2023 to 11 April 2024. The overall mean of these 26-week MAs is 37.5%.
Figure 1: Percentage of “Bullish” Responses, Six-Month Moving Average of Weekly Data, January 1988-April 2024
The MAs are only approximately normally distributed. If their distribution were perfectly normal, then 95% of the observations would lie within two standard deviations of their overall mean – and thus 5% of observations would lie more than 2 SDs from the mean. In reality, only at the height of the Dot Com Bubble at the turn of the century was the percentage of bulls exceeded two standard deviations above the overall mean.
Similarly, only rarely and very briefly has the MA fallen more than two standard deviations below its overall mean. Most recently, between December 2021 and July 2022 it repeatedly crossed this threshold. Finally, the data’s trend is parabolic: before the GFC, bullishness mostly waxed; since then, with some brief exceptions it’s mostly waned.
Indeed, the sharply rising bullishness over the past two years, from a very low to an above average level (19% in April 2022 to as high as 49% in December 2023, and 45% in April 2024), ranks among the strongest and most rapid on record.
Figure 2 plots the percentage of bearish respondents to AAII’s survey; that is, those who believe that stocks will fall during the next six months. This percentage has trended weakly upwards, and the 26-week MA has never been more than two standard deviations (2 × 6.5% = 13.0%) below its overall mean (31.0%). On four occasions, in contrast, the MA has risen more than two standard deviations above the overall mean.
Figure 2: Percentage of “Bearish” Responses, Six-Month Moving Average, January 1988-April 2024
The first commenced in November 1990 and concluded in March 1991, i.e., coincided with the recession of the early-1990s; the second occurred from January 2008 to August 2009, i.e., encompassed the Global Financial Crisis; the third occurred from July to October 2020, i.e., in the wake of (rather than during) the COVID-19 panic; and (perhaps in anticipation of a recession which, at least according to its semi-official arbiter, the National Bureau of Economic Research, hasn’t materialised) the fourth and most recent from June 2022 to March 2023. Since this latter month, bearishness has plunged.
Figure 3, which plots the “bullish” and “bearish” series as 52-week MAs, reveals four key points that Figures 1 and 2 obscured:
- from July 1994 to February 2008, bulls outnumbered (by an average of 14.3 percentage points) bears;
- from the nadir of the GFC to 2016, the percentages of both bulls and bears fell;
- since 2016, these percentages have fluctuated sharply but erratically;
- presently, investors are among the most bullish (12-month MA on 11 April was 44.4%) they’ve been since before the GFC.
Figure 3: Percentage of Bulls and Bears, 52-Week Moving Average, July 1988-April 2024
Figure 4 plots the “bull-bear spread,” that is, the percentage of bulls net of the percentage of bears, as a 52-week MA. A spread greater than 0% indicates that the percentage of bulls exceeds the percentage of bears, and a spread less than 0% indicates that bears outnumber bulls. The mean of the spread’s 12-month MA is 6.4%; since 1988, bulls have outnumbered bears by an average of 6.4 percentage points.
Figure 4: “Bull-Bear Spread,” 52-Week Moving Average, July 1988-April 2024
On just two occasions – the height of the Dot Com Bubble (September-December 2000) and before the GFC (February-September 2004) – did the spread rise two standard deviations above its mean (25.6%). Similarly, just twice (January-February 1991 and November 2008-July 2009) has it sunk two standard deviations below its mean (-12.8%). But only once – from September 2022 to June 2023 – has it breached this threshold.
Does the spread’s subsequent sharp rise (to 11% in April 2024) actually indicate a genuine increase of bullishness? Or does it reflect the rebound of a heavily mean-regressing series from its previous extreme – indeed, unprecedented – low?
Finally, for the sake of completeness Figure 5 plots the percentage of AAII’s respondents which believes that stocks will neither rise nor fall during the next six months. The mean of the 26-week MAs is 31.4% and their standard deviation is 8.2%. From 1988 until the GFC, the moving average fell by more than half to 20%; it then rose steadily and doubled to more than 40% in 2016; since then, it’s gradually sagged to 30%.
Figure 5: Percentage of “Neutral” Responses, 26-Week Moving Average, January 1988-April 2024
What Explains Short-Term Changes of Bullishness and Bearishness?
Do responses to AAII’s survey during a given week reflect the market’s direction and degree of volatility during the previous week? In order to answer this question, for each week since 24 July 1987, I computed the percentage change of the S&P 500’s closing level from the previous Friday (or the nearest business day in the case of holidays). I also computed
- the S&P’s volatility during the week (defined as its weekly maximum minus its minimum divided by the minimum);
- the weekly change (in percentage points) of the survey’s bullish percentage;
- the weekly change (in percentage points) of the survey’s bearish percentage;
- the change (in percentage points) of the survey’s “bull-bear spread” (that is, the percentage of bulls net of the percentage of bears vis-à-vis the previous week).
Finally, I sorted the data by the Index’s one-week percentage change, divided these sorted data into quintiles (five groups of equal size net of rounding) and computed relevant ranges and means for each quintile. Table 1 summarises the results.
Table 1: Effect of S&P 500 Index’s Change and Volatility upon Investors’ Bullishness and Bearishness, Weekly Data, July 1987-April 2024