Can AI Really Predict Stock Returns?

Can AI Really Predict Stock Returns? An original review of *Empirical Asset Pricing via Machine Learning* by Shihao Gu, Bryan Kelly, and Dacheng Xiu. This episode is educational commentary, not investment advice. It does not reproduce the paper's prose, tables, charts, or figures. Exact source and license links appear on the episode page. ## 1. The question Can artificial intelligence predict stock returns? That question sounds like an invitation to hype. It can summon images of a machine seeing tomorrow's prices, beating every human trader, and turning uncertainty into a smooth upward line. The paper in today's episode asks a narrower and much better question. If researchers take decades of U.S. stock data, give several machine-learning models the information that investors could have observed at the time, and then test the models on later years, do the flexible models forecast returns better than conventional statistical methods? The paper is *Empirical Asset Pricing via Machine Learning*, by Shihao Gu, Bryan Kelly, and Dacheng Xiu. It appeared in *The Review of Financial Studies* in 2020. It has become influential because it brings modern prediction methods into one of finance's most difficult settings: estimating the expected return of an individual stock. The answer is interesting because it is neither “AI solves the market” nor “machine learning makes no difference.” The authors report a real historical improvement. But the statistical improvement is small, the portfolio implications are much larger, and the journey from those two facts to a commercial investment product is full of friction. So our real question is not simply whether the model predicts. It is: what exactly was predicted, how honestly was it tested, and what survives after we ask the questions a portfolio manager, risk officer, regulator, or paying customer would ask? ## 2. Why it matters now The commercial language of AI in finance often runs ahead of the evidence. Investment products are described as intelligent, adaptive, or powered by machine learning. A label can be attached to an old strategy without changing much beneath it. This paper gives us a more useful standard. First, define the target. Second, separate training from validation and testing. Third, compare the new model with serious baselines. Fourth, report not only a statistical score but also what happens in portfolios. Finally, expose the uncomfortable details: turnover, drawdowns, small-stock dependence, and sensitivity to research choices. This matters because stock returns are extraordinarily noisy. A company can execute well and still fall because interest rates change. A weak company can rally because expectations were worse. News, liquidity, positioning, taxes, and risk appetite collide in the same monthly return. In that setting, an out-of-sample R-squared of less than one percent can sound trivial. It means the forecast removes only a sliver of squared prediction error compared with a simple benchmark. Yet when thousands of small forecasts are ranked and combined, a sliver may change portfolio construction. That creates a dangerous communication gap. A researcher may correctly say, “The model improved prediction by four-tenths of one percent.” A marketer may turn the associated portfolio result into “AI finds alpha.” Both sentences can refer to the same experiment while giving a listener radically different expectations. For education, the paper is valuable because it teaches us to keep three layers separate: predictive accuracy, historical portfolio performance, and a live commercial product. They are connected, but they are not interchangeable. ## 3. What came before Traditional empirical asset pricing often begins with a regression. Researchers choose a small set of characteristics—perhaps company size, valuation, or recent price momentum—and estimate how those variables relate to future returns. Linear models are attractive because they are transparent and disciplined. But they impose a shape on the world. They may assume that a one-unit change has a similar effect across companies, or that two variables matter separately when their interaction is the real signal. Machine learning offers a larger toolbox. Penalized regressions can shrink noisy coefficients. Principal-components and partial-least-squares methods can compress many variables. Random forests and boosted trees can capture thresholds and interactions. Neural networks can build nonlinear combinations through layers of learned features. Flexibility is both the promise and the hazard. A model with enough freedom can fit patterns that happened…

Continue with the full plain-text transcript.