Can AI Really Predict Stock Returns?

Can AI Really Predict Stock Returns? An original review of *Empirical Asset Pricing via Machine Learning* by Shihao Gu, Bryan Kelly, and Dacheng Xiu. This episode is educational commentary, not investment advice. It does not reproduce the paper's prose, tables, charts, or figures, and exact source and license links appear on the episode page. That framing holds throughout, so I will state it once here rather than repeat it. ## 1. The question and the bottom line Can artificial intelligence predict stock returns? Here is the honest answer before any detail, so you have it even if you stop after this minute. In this landmark study, machine-learning models did predict individual U.S. stock returns better than conventional methods — but the statistical improvement was tiny. The best model explained about four-tenths of one percent of next month's variation in a stock's return. That sliver, ranked across thousands of stocks and formed into portfolios, produced striking historical returns on paper. The biggest caveat, in the same breath: those portfolio numbers are gross historical backtest results, not a live track record. They come with punishing turnover, deep drawdowns, and a dependence on hard-to-trade small stocks. And the surrounding literature warns that this kind of edge tends to shrink after it is published and swings wildly with research-design choices. Who should care? Anyone who hears "AI beats the market" and wants to know what that claim can and cannot mean. The useful reading is neither "AI solves markets" nor "the gains are meaningless." It is: a small, real, honestly tested predictive edge, and a long, cost-laden road from that edge to any actual product. So the real question is not simply whether the model predicts. It is what exactly was predicted, how honestly it was tested, and what survives the questions a portfolio manager, risk officer, or paying customer would ask. ## 2. Why it matters now The commercial language of AI in finance runs ahead of the evidence. Products are called intelligent, adaptive, machine-learning-powered; a label can be attached to an old strategy without changing much beneath it. This paper offers a more useful standard: define the target, separate training from validation and testing, compare against serious baselines, report not just a statistical score but what happens in portfolios, and expose the uncomfortable details — turnover, drawdowns, small-stock dependence, and sensitivity to research choices. This matters because stock returns are extraordinarily noisy. A strong company can fall when interest rates move; a weak one can rally because expectations were worse. In that setting a monthly out-of-sample R-squared below one percent sounds trivial, and for a single stock it nearly is. Yet ranked and combined across thousands of names, a sliver can shift portfolio construction. That creates a communication gap: a researcher can honestly say "the model improved prediction by four-tenths of one percent," and a marketer can turn the associated portfolio result into "AI finds alpha." Both describe the same experiment. Keeping three layers separate — predictive accuracy, historical portfolio performance, and a live product — is the whole skill. ## 3. The field so far To judge whether this paper is a breakthrough or a careful step, you have to know the field it entered — and that field carries a warning label. Start with the problem the paper is really answering. In 2016 Harvey, Liu, and Zhu described a "factor zoo": hundreds of published variables claim to predict returns, and after so much data mining, most would not clear a proper statistical hurdle — they argued a new factor should show a t-statistic above about three, far stricter than usual. That is the backdrop. When someone reports a new return predictor, the base rate for false discoveries is high, which is exactly why this paper's discipline about testing on later, held-out years is its real contribution. Two more results frame the findings, and they pull in a cautionary direction. First, an independent machine-learning study in the same 2020 journal issue, by Freyberger, Neuhierl, and Weber, found that nonlinearities genuinely matter — agreeing with today's paper — but also that many previously published predictors add no incremental information once you select carefully. A smaller set of characteristics may carry most…

Continue with the full plain-text transcript.