Can We Predict a Breakthrough?

In 2023, a widely debated Nature paper argued that science is becoming less disruptive, using a metric called the CD index to measure whether later research builds on a paper or moves past it. For the third assignment of the Applied Machine Learning course in my BSc, I set out to predict a paper's five-year disruption score (CD5) using only information available at publication, without looking at the future papers that cite it. Starting from a SQLite citation database of 149,078 papers, I joined five tables and kept the 48,422 papers with a CD5 score that were published by 2017, so that a full five-year citation window existed. After dropping papers with no subject data, 44,553 remained. I engineered five features: number of authors, number of references, mean age of references, publication year and number of subject areas. Reference years were often missing or malformed, so I recovered them by sharing known years across every appearance of the same DOI. I also explored TF-IDF and K-means clustering of research fields, but excluded it from the final models to avoid leaking future information. I then trained two models: an XGBoost gradient-boosted regressor, which reached an R² of 0.96 and an MSE of 0.01 on a 20% held-out test set, and a Keras neural network (two 64-unit ReLU layers, early stopping), which reached a test mean absolute error of 0.021. Both models predicted the dominant CD values of 0 and 1 well but struggled with the rarer values in between. Number of references was by far the most important feature.

Some research papers open entirely new fields; most build on what came before. The CD index measures the difference: if later papers cite a study but ignore the work it cited, that study broke new ground. The catch is that you only know a paper's score years after it's published, once the citations come in. The question I tackled was whether a paper's future impact could be predicted from what's known on day one: its authors, its references and its subject areas.

An end-to-end ML pipeline in Python (pandas, scikit-learn, XGBoost, TensorFlow/Keras):

• Data pipeline: extracted and joined five tables from a 149,078-paper SQLite database, then filtered to the 44,553 papers with a valid five-year CD score and complete subject data.
• Feature engineering: five publication-day features. I cleaned more than 850,000 reference records, repairing malformed and missing years by matching DOIs across the dataset, and imputed the remaining gaps with the median.
• Exploratory clustering: grouped research fields with TF-IDF and K-means (18 clusters), then deliberately left this out of the models because it risked using future data.
• Model 1, XGBoost: 1,000 boosted trees on an 80/20 train/test split, with feature-importance analysis.
• Model 2, neural network: a Keras Sequential model with a normalisation layer, two 64-unit ReLU layers and MAE loss. I diagnosed a noisy validation curve, enlarged the validation split and added early stopping, which ended training at 65 epochs.

• XGBoost: R² = 0.96, MSE = 0.01 on unseen test data.
• Neural network: test MAE = 0.021, with training and validation curves converging smoothly.
• What drives the predictions: number of references was the most important feature by a wide margin, followed by publication year and reference age.
• Where the models struggle: both predict papers at CD = 1 and near 0 well, but are much weaker on the rare values in between.

Looking back: a paper with no recorded references scores CD = 1 by definition, and at least a quarter of the training set had none. That goes a long way to explaining both the high R² and why reference count dominates. If I did it again, I'd benchmark against a simple "no references → 1" rule, evaluate papers with references separately, and use a time-based train/test split.