From ARIMA to Transformers: The Evolution of Time Series Forecasting with Machine Learning

Abstract

This study examines the evolution of time series forecasting in the context of crude oil price prediction, tracing the methodological shift from classical statistical models, such as ARIMA, to advanced machine learning architectures, including LSTM networks and Transformer models. Using a dataset of daily crude oil prices from 1995 to 2016, the analysis evaluates model performance across multiple horizons and error metrics. Results indicate that ARIMA, while interpretable, shows limitations in handling volatility and nonlinearities (RMSE = 2.10). LSTM improves accuracy by capturing long-term dependencies (RMSE = 1.55), while Transformer achieves the highest performance (RMSE = 1.32, R² = 0.87). These findings highlight how attention-based models outperform traditional econometric approaches by addressing volatility clustering and long-range dependencies. The study contributes to the forecasting literature by demonstrating the paradigm shift from ARIMA to Transformers and offering a framework for selecting models based on accuracy, interpretability, and forecasting horizon.