Why gradient boosting beats LLMs for student retention prediction

If you have sat through a vendor pitch in the last twelve months, you have probably heard some version of "we use AI to predict student outcomes," with "AI" doing a lot of unspecified work in that sentence. Sometimes it means a large language model wrapped around a dashboard. Sometimes it means a classical machine-learning model that does the actual prediction, with an LLM bolted on for the summary screen. The two are not equivalent, and the difference matters when the output is going to shape advising caseloads or program decisions.

This post is the short version of the argument for IR leadership: on structured institutional data, gradient-boosted trees (XGBoost, specifically) outperform LLMs by a clear margin, the benchmark literature is unambiguous about it, and the reasons are practical, not ideological. None of this means LLMs have no role. It means they have a different one.

The pull toward LLMs for retention prediction is understandable. They are general-purpose, they produce fluent output, and they require less obvious infrastructure than a trained model. Hand the chatbot a CSV of student records, ask "who is most likely to drop out next term," and get an answer in seconds. It feels efficient.

The output is also confident, which is part of the problem. An LLM will return a graduation rate for nine students with no warning that nine students cannot support a graduation rate. It will assign a "high risk" label without telling you how it got there. It will produce a different answer the second time you ask, and a third one the third time. In a procurement demo, this looks like sophistication. In production, it is the worst possible foundation for decisions that affect students.

The retention question is, at its core, a numbers problem over 18 to 20 structured features. It is not a language problem. Asking a language model to solve it is using the wrong tool for the right reason.

There are exactly three structural limitations that disqualify a general-purpose LLM as the predictor in this kind of system. Each one is fatal on its own.

None of this is hypothetical. Each limitation has been demonstrated repeatedly in peer-reviewed benchmarks on exactly this kind of structured prediction problem.

The argument that LLMs win on structured tabular prediction has been made in print. Reading the actual studies, including the ones that argue the LLM case, produces a consistent pattern: classical machine learning wins on structured data with adequate volume, and LLMs only catch up in the tiny-data regime (roughly 32 to 64 rows) where there is essentially nothing else to fit.

Three results are worth knowing.

The pro-LLM papers (UniPredict, TP-BERTa, GTL, the Amazon survey) reinforce the same boundary condition: LLM gains appear only in low-data, heavily categorical, or untuned-baseline settings. On numeric institutional data with enough rows, XGBoost wins or matches every time. An institution's six years of student-term records is not the LLM-favorable regime. It is the XGBoost-favorable regime by a wide margin.

Saying "use gradient-boosted trees" still leaves a choice. The tree-based family includes Random Forest, the original GBM, AdaBoost, XGBoost, LightGBM, CatBoost, and NGBoost. They are not interchangeable, and the right pick depends on the dataset's size, feature mix, and operational constraints.

For a dataset on the order of 240,000 student-level rows expanding to roughly 1.84M student-term rows (six years at 40,000 students), with 19 to 20 features and a mix of numeric, binary, and categorical inputs plus class imbalance and an explainability requirement, the operating ranges look like this.

The selection logic at this scale:

Logistic Regression is the right baseline but learns essentially one additive equation. With this much data, it cannot capture the higher-order interactions (e.g. attendance only helps if grades are adequate) that actually drive the outcome.

Random Forest handles interactions but is memory-heavy and harder to balance for the rare not-retained class on a wide course matrix.

CatBoost handles categoricals natively but trains slower at this scale than XGBoost with explicit encoding.

LightGBM is the planned successor once the dataset grows past roughly 6 to 7 million rows, where its leaf-wise growth becomes the speed advantage. For now, it is overkill.

XGBoost is the chosen model: strong on tabular data, robust regularization, native handling of missing values and class imbalance, mature tooling, and excellent SHAP support. The safest high-accuracy choice at this scale.

A trained model returns a probability between 0 and 1 for each student. A 0.85 means an 85% modeled probability of the outcome, based on patterns from similar students whose outcomes are known. The most common piece of confusion when a risk score reaches leadership is what "85%" actually claims.

"Precise" does not mean "certain." It means the uncertainty has been quantified and is attributable to specific factors, rather than guessed. That distinction is the entire point.

When a model is reported as having about 86.6% accuracy, ~91% recall, and ~0.91 AUC, here is what those numbers actually say.

Two things to note. First, accuracy alone is misleading when 87% of students are retained. A model that predicts "retained" for everyone would score 87% accuracy and catch zero real dropouts. That is why recall, not accuracy, is the operating metric for this problem. The retention deep-dive post covers why and what to do about class imbalance.

Second, AUC of 0.91 is genuinely strong on this kind of data. The reason this is achievable is not magic. It is that there is enough signal in six years of attendance, grades, prior-term performance, and course difficulty for the model to rank students reliably.

A bare 67.3% graduation probability is not useful to an advisor. The advisor needs to know what is dragging the number down or pushing it up, because that is where the conversation with the student starts.

SHAP (Shapley Additive Explanations) decomposes each prediction into per-feature contributions. The score is read as a base rate plus and minus the effect of each factor that the model considered.

The same machinery, run across the whole population, surfaces which factors dominate across the board. On a well-built retention model, academic-progress features (cumulative credits earned, cumulative percent credit completion, prior-term GPA) dominate. Demographic features (gender, race/ethnicity) typically sit near zero, which is what you want: the model is relying on performance and progress, not on demographic shortcuts.

SHAP is what makes the prediction defensible in a committee meeting. Without it, the score is a black box. With it, every prediction is a conversation an advisor or dean can have.

None of this excludes LLMs from the system. It defines what they should and should not do.

LLMs are excellent at two things in this stack. First, enriching course features. Pulling course descriptions and catalog metadata to tag courses by workload, prerequisite difficulty, and topic cluster is a language problem, and that is where LLMs shine. Those tags then become inputs to the gradient-boosted model.

Second, translating SHAP output into committee-ready language. Turning "feature X contributed -8.4% to student Y's prediction" into a paragraph an academic affairs committee can read and act on is exactly the kind of summarization LLMs are good at.

The division of labor is straightforward: classical machine learning does the prediction and the confidence estimation. The LLM does the language work around it. Mixing those up is how predictions stop being trustworthy.

Clema's predictive models follow exactly this pattern. Gradient-boosted trees do the prediction, SHAP produces the per-student explanation, and we use LLMs only for the supporting tasks they actually do well (feature enrichment and natural-language summaries of the model output).

For the broader strategic case (catalog of predictions, economic argument, where to start), the reactive-to-proactive playbook is the companion to this post. For the cost question (open-source stack vs $30K to $200K incumbent platforms), the cost and vendor reality post covers the math.

XGBoost documentation, Distributed (Deep) Machine Learning Community. SHAP documentation, A unified approach to explaining model output. arXiv 2411.06469, LLMs vs ML on clinical prediction. JAMIA, comparing LLM and ML for early discharge prediction (2025). arXiv 2406.12031, TABULA-8B tabular transfer learning. scikit-learn, machine learning in Python.