Introduction
At the 2025 AIR Forum in Orlando, a well-attended session walked through the University of Arizona's deployment of a retrieval-augmented generation system: organizing unstructured campus data, building the vector database, and letting people ask research questions in plain English. A year later, AI sessions filled the 2026 Forum program in Washington, D.C. The interest is not mysterious. Nearly every AI tool pitched at IR offices this year, including ours, has RAG somewhere under the hood, and almost none of the marketing explains what that means or where it fails.
You should understand it at the working level, the same way you understand sampling or cohort adjustment. Not to build one, but because you cannot evaluate an AI tool, write a governance policy for it, or debug its wrong answers without knowing what the architecture actually does. This post explains RAG in plain English, shows it working on real IPEDS data, and names the specific ways it breaks on higher ed questions.
If you have read why higher ed needs its own AI assistant, this is the next level down: the mechanics behind "it reads your data instead of guessing."
What RAG is in plain English
Retrieve
When a question arrives, the system first searches your actual data for relevant pieces: rows, documents, definitions, prior reports. The search is usually a mix of keyword matching and semantic similarity, which is why the data gets converted into embeddings, numeric representations of meaning, stored in a vector index. The search returns a handful of candidate snippets it thinks are relevant to the question.
Augment
The retrieved snippets are inserted into the prompt, ahead of the question. The model is no longer asked to answer from memory; it is handed the relevant material and told to use it. This step is the entire difference between a chatbot that guesses and a system that quotes.
Generate
The language model writes the answer with the retrieved material in front of it. If the retrieval worked, the answer traces to real rows. If it did not, the model either says it cannot find the answer or, worse, papers over the gap. Generation is the strongest and least trustworthy step; the quality ceiling is set by retrieval, not by the model.
RAG vs the alternatives
| Approach | Answers come from | When the data is wrong | Fit for IR questions |
|---|---|---|---|
| General chatbot | Training data and the public web | You cannot tell; it sounds the same either way | Poor: no access to your data, no source trail |
| Fine-tuned model | Patterns learned during additional training | Baked in until you retrain; expensive to find | Poor: teaches style, not facts; numbers still invented |
| RAG | Your data, retrieved at question time | Traceable to the exact retrieved rows | Strong: grounded, refreshable, auditable |
The comparison that matters most is with fine-tuning, because vendors blur the two. Fine-tuning changes the model itself; RAG changes what the model sees at question time. For factual institutional data that updates annually, that distinction is decisive. You cannot fine-tune a model on numbers that change every release cycle, and even if you could, a fine-tuned model still cannot show you the row behind its answer. RAG can, because the row is sitting right there in the prompt.
Why RAG fits IR work specifically
- Traceability is a job requirement. Accreditors, cabinets, and federal reporters need the table, the year, and the calculation. RAG keeps the provenance because the answer literally came from retrieved rows.
- Higher ed data refreshes on an annual cadence. When the new IPEDS or Scorecard release lands, you re-index and every answer updates. No retraining, no new model version.
- Definitions are half the question. A good higher ed retrieval layer indexes the data dictionary too, so "retention" pulls the definition as well as the number. This is where general RAG stacks fail and domain-aware ones earn their keep.
- Access control survives. Retrieval can respect row-level permissions: the same question returns different scopes for a dean and an analyst. A chatbot cannot promise that because it has no rows to gate.
- It answers the long tail. The questions IR gets are too varied to pre-build into dashboards. Retrieval over the whole warehouse covers the 80 percent of asks nobody built a view for.
A worked example on real IPEDS data
Here is RAG working on a real question with real data, using the same IPEDS Fall Enrollment residence file behind our enrollment cliff analysis.
Ask a general chatbot: "How many first-time students enrolled at US institutions in fall 2024?" You get a confident number, often in the right zip code, sometimes sourced to a press release from a different year. There is no way to check it, because there is no row behind it.
Ask the same question of a RAG system over IPEDS. Retrieval searches the fall enrollment tables, finds the EF2024C residence file, and pulls the total row: 3,109,185 first-time, degree/certificate-seeking students. The prompt now contains that row plus the column definitions. Generation produces: "3,109,185 first-time, degree/certificate-seeking students, per the IPEDS Fall Enrollment residence component, fall 2024 collection." The number, the exact definition, the component, and the collection year, because all four were in the retrieved material.
Now the part vendors skip. Ask the follow-up: "How about 2025?" Retrieval searches, finds the 2025 partition exists but holds no data yet, and the correct answer is "not published in this warehouse." A good system says exactly that. A bad one retrieves the 2024 row anyway and answers "in fall 2025, 3,109,185 students enrolled," which is how a missing-data artifact becomes a number in someone's board deck. Same architecture, different retrieval discipline, opposite outcomes. The plain-English querying guide shows what this looks like end to end when the discipline holds.
The five ways RAG still breaks
Retrieval misses the right vintage
The warehouse holds multiple collection years. Retrieval that grabs the 2016 row for a 2024 question produces a grounded, well-cited, three-years-stale answer. On higher ed data this is the most common failure, because "the enrollment number" is ambiguous until a year is pinned.
Definition drift
"Retention," "first-time," even "enrolled" carry different definitions across IPEDS, state, and internal reporting. If the retrieval layer does not resolve which definition applies, the model averages them into an answer that matches none. This is the higher ed-specific failure that generic RAG stacks hit first.
Hallucination over retrieved context
Generation can still embellish. The retrieval hands the model three rows and it computes a fourth, or narrates a trend the rows do not show. Grounding reduces fabrication; it does not eliminate it. Every output still needs the source trail checked, which is why the citation has to be a feature and not a marketing claim.
The stale index
Federal data refreshes on a schedule. If nobody re-indexes after the new IPEDS release, the system keeps answering confidently from the old vintage. A higher ed RAG deployment without a refresh calendar tied to the federal release calendar is a time bomb of confident old numbers.
Permissions leakage
If retrieval ignores row-level access, a question from someone without FERPA-scoped rights can surface protected rows in the prompt. The model never "decides" to leak; the leak happens at retrieval, before generation runs. This is the first thing your security review should ask about, and the reason SSO and access design are not afterthoughts.
Build vs buy
Building it yourself
The parts list is knowable: a vector database, an embedding pipeline, connectors to your SIS and warehouse, and a domain-aware retrieval layer that understands your definitions and vintages. What kills internal builds is rarely the first demo; it is the maintenance: refresh calendars, permission changes, definition governance, and the evaluation harness that tells you answers are still right a year later. Budget for that before the pilot, not after.
Buying it
A vendor should arrive with the federal datasets pre-indexed, the higher ed definitions modeled, and the audit trail built. Your evaluation job shifts to verification: does it actually cite rows, does it respect your access rules, does it survive a data refresh. The five questions to ask before buying were written for exactly this, and work on any vendor, including us.
The honest middle
Plenty of offices will end up with both: a bought system for federal data and routine asks, and a small internal pilot for genuinely proprietary data. That is fine, as long as each has an owner, a refresh plan, and a policy answer for what it cannot do yet. The failure mode is the ungoverned middle: a pilot someone built in a spare month that quietly became load-bearing.
Questions to ask any vendor
- "Show me the row behind that answer." If the demo cannot surface the source record on demand, the grounding is decorative.
- "Where does processing happen, and what data leaves our boundary?" This is the data governance committee's first question; make it yours too.
- "How are definitions handled when a term means two things?" A vendor without a concrete answer here has not thought about higher ed.
- "What happens the day after the new IPEDS release?" You want a refresh process with dates, not "the model stays current."
- "How does retrieval respect role-based access?" Ask for a live demonstration with two users, not an assurance.
- "What does the system do when the data does not exist?" The correct answer is a refusal; anything else is a future correction in a board deck.
See RAG done right on data you know
Bring the questions you already know the answers to. Watch retrieval pull the rows, definitions, and vintages, and see the source trail behind every number.
Book a Demo