Recently, Large Language Models (LLMs) have gained significant attention for their applications in various fields. However, in the field of finance, the use of LLMs to predict stock market returns is still relatively new development. On one hand, some may consider the value of LLMs in this regard to be limited, as many of these models have not been trained on recent market data. On the other hand, others expect LLMs to be a powerful tool for processing textual information and predicting stock returns due to their ability to understand natural languages more accurately than previous language models. Therefore, the potential of LLMs in predicting financial market dynamics is still worth examination and discussion.
This research can aid regulatory authorities and policymakers in understanding the potential benefits and drawbacks that may arise from the use of LLMs in the field of finance. It also provides inspiration for asset managers and institutional investors for new portfolio construction. Incorporating LLMs into investment strategies could optimize traditional quantitative investment models such as multi-factor model. In addition to this, by taking this research as a starting point, we can promote the exploration of the potential and limitation of LLMs in the financial field, providing useful experience and references for the development of more complex LLMs tailored to the specific needs of the financial industry. However, it is important to note that the promotion and application of LLMs is essentially an expansion of AI technology in real world applications. Compared to traditional Natural Language Processing (NLP), LLMs have more training parameters, enabling them to perform better in understanding and generating natural language. Simultaneously, LLMs also demonstrate a considerable degree of logical reasoning.
Currently, the most successful LLM in the market is ChatGPT, developed by AI research company OpenAI and launched on 30 November 2022. Once launched it achieved explosive growth, exceeding 100 million users in January 2023, just 2 months after its launch, with over 13 million active users daily. ChatGPT has brought unprecedented potential for application to various industries, shaping emerging business scenarios. Because of this, it is worth examining whether ChatGPT can outperform traditional models, producing better results in predicting stock price movements.
Data for Predicting Stock Yields with ChatGPT
The research paper [1] by Alejandro Lopez-Lira and Yuehua Tang is one of the earliest papers evaluating ChatGPT’s performance in predicting stock market returns. Using the model’s Sentiment Analysis Capabilities 1, the authors evaluated ChatGPT’s performance using news headline data, comparing it to existing Sentiment Analysis methods from other leading data vendors.
In this paper, the authors utilised two data sets: stock yield data from the Center for Research in Security Prices (CRSP), and news headline data from a leading data provider. The sample data was selected from October 2021 to December 2022 2 , as training data used by ChatGPT was only available until September 2021, with this timing set to avoid data leakage. In order to generate a corresponding sentiment score based on each news headline in the dataset 3, the authors used a pre-trained version of GPT 3.5, a LLM developed by OpenAI 4.
1 ChatGPT utilises deep learning and is able to understand the context of text and accurately identify emotions.
2 Data leakage refers to the accidental leakage of information and test data used during the model training process, thereby having an impact on the model prediction results. In this case, the model may give a very optimistic result, but the prediction performance on the actual new data may be very bad. Since the training data of ChatGPT is as of September 2021, using testing data from October 2021 can avoid the risk of data leakage. As such, after this point in time, when testing new data, there is no need to worry about the negative impact of data leakage on model performance.
3 The specific calculation is explored in more detail in the explanation of Sentiment Analysis below.
4 GPT-3.5 was chosen due to its ability to perform stably and well in various natural language processing tasks, including Sentiment Analysis.
How ChatGPT Predicts Stock Returns
The following section describes the Sentiment Analysis process employed by ChatGPT, and the corresponding empirical results. The authors first make use of prompts to start the interactive conversations with ChatGPT, asking it to conduct Sentiment Analysis of each news article from the perspective of a financial analyst and convert it into a ‘ChatGPT score’. For example, the user would enter news headline relating to XXX company into ChatGPT and ask, “Is this good or bad news for XXX company?”. If ChatGPT’s answer is “Yes”, then the corresponding output would be 1 (which indicates good news for the company). The remaining corresponding outputs, for “UNKNOWN” and “No”, are 0 (indicating the emotional reaction to the news is unclear) and -1 (indicating negative news for the company), respectively. If the company has multiple news pieces on the same day, the average of the corresponding scores is used as the overall ChatGPT score. The authors then used a ChatGPT Sentiment score, lagged by one day 5, to predict a company’s stock return for that day. Specifically, the authors make use of linear regression to examine the predictive accuracy of ChatGPT’s Sentiment scores on daily stock returns, while accounting for year-fixed effects and firm-level fixed effects 6. Simultaneously, the authors also compare the results of the prediction generated by ChatGPT with predictions based on traditional Sentiment Analysis methods provided by other data vendors.
| Daily Return (%) | Headline Length | ChatGPT Response Length | GPT Score | Event Sentiment Score | |
| Daily Return (%) | 1 | – | – | – | – |
| Headline Length | 0.00 | 1 | – | – | – |
| ChatGPT Response Length | 0.00 | 0.26 | 1 | – | – |
| GPT Score | 0.02 | 0.08 | 0.44 | 1 | |
| Event Sentiment Spore | 0.00 | -0.08 | 0.10 | 0.27 | 1 |
5 This is to ensure that inappropriate, future data is not used in the regression. Regression is a statistical analysis method used to explore a linear correlation between a dependent variable (the daily return of a stock in this case) and one or more parameters (the ChatGPT Sentiment score in this case).
6 By introducing dummy variables or fixed effects for a year, the model can capture implicit effects for the year, thereby controlling the interference of these year-related hidden factors on the outcome variables. Specifically, the model introduces a dummy variable for each year. Likewise, the introduction of dummy variables or fixed effects allows us to control the unobserved effects of these companies on the outcome variable. This means that the model treats each company as an independent class and introduces a dummy variable for each company.
ChatGPT’s Performance in Predictions
The authors first find the strongest correlation between ChatGPT’s sentiment score and the subsequent daily returns of the sample stocks (see Table 1). Therefore, it makes sense to use Sentiment score to predict future stock returns.
Figure 1 intuitively presents the net value changes of different investment portfolios under backtesting of trading strategies, assuming an initial capital of only $1 in October 2021 without considering transaction costs. The black line corresponds to the net worth of the equal weighted portfolio of all companies with news on the previous day, and their portfolio returns are the lowest at the end of the backtesting period. The green line corresponds to the historical net worth of buying an equal weighted portfolio of companies with good news based on ChatGPT’s Sentiment scores. The red line corresponds to the historical net value of an equal weight portfolio that short sells companies with bad news based on ChatGPT scores. The blue line represents an equal-weighted zero-cost portfolio constructed from ChatGPT scores, meaning that companies with good news are bought and companies with bad news are shorted at the same time. Among all the portfolio construction methods shown above, the long-short combination method based on ChatGPT’s Sentiment score achieved the best performance, obtaining a result of about 4 time the initial capital during the sample period 7. This clearly indicates that statistically the ChatGPT sentiment score has a significant ability to predict daily stock market movements. By leveraging news headlines and GPT generated scores, a strong correlation was found between the ChatGPT’s scores and subsequent daily returns. This result serves to further highlight the potential of ChatGPT using Sentiment Analysis as a valuable tool for predicting the direction of the stock market.
To further enhance the stability of the results, the authors compared the performance of ChatGPT with traditional sentiment analysis methods provided by a major data vendor. The results demonstrated that after controlling for the Sentiment scores generated by ChatGPT, the predictive power of other Sentiment scores on daily stock returns dropped to zero. This indicates that the Sentiment score generated by ChatGPT is superior to other existing Sentiment Analysis methods in predicting stock market returns. In other words, ChatGPT is able to capture additional information that other models are unable to.
7 The left axis represents the value of the investment, with all of them beginning at $1. Having rose from $1 to $5, this practice has undergone a 400% increase in size.
ChatGPT’s Future in Stock Returns Prediction
The development of LLMs specifically designed for the financial industry is of high importance, and future research should focus on understanding the predictive capabilities of these models. Moreover, with the emerging popularity of LLMs in the financial industry, their potential impact on market dynamics, price formation, dissemination of information and market stability should be examined. Further research can investigate the possibility of combining LLMs with other machine learning techniques and quantitative models to create intelligent investment systems, further improving the predictive capabilities of AI-driven models in the field of economics and finance.
References
[1] Lopez-Lira, A., & Tang, Y. (2023). Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models. arXiv preprint arXiv:2304.07619.
The content of this webpage is not an investment advice and does not constitute any offer or solicitation to offer or recommendation of any investment product. It is for general purposes only and does not take into account your individual needs, investment objectives and specific financial circumstances. Investment involves risk.
The work described in this article was supported by InnoHK initiative, The Government of the HKSAR, and Laboratory for AI-Powered Financial Technologies.
(AIFT strives but cannot guarantee the accuracy and reliability of the content, and will not be responsible for any loss or damage caused by any inaccuracy or omission.)
