Every one of FreeFellow's 45,000+ practice questions starts as an AI draft. Before it reaches you it has to pass automated checks and a set of statistical checks that run like software tests. Separate AI review passes and candidate reports catch more of what those checks miss. Once it's live, it gets recalibrated against real candidate answers when there are enough of them. This post walks through each step and the problems each one was built to catch.
I've written separately about the seven tells of an unaudited AI question bank, and most of those came from FreeFellow's own early banks. This post covers what the process looks like now.
Before a question goes live
An AI model drafts each question for a specific learning objective, topic and difficulty target, following a long list of format rules that came from past mistakes. Choices give the answer without arguing for it. Wrong numeric answers come from real mistakes, not numbers spaced evenly around the right one. Choice lengths stay balanced. Each exam's mix of calculation and concept questions is respected. Drafting this way costs a small fraction of an expert-written question, which is why the whole bank can be free.
Next, automated validators check the mechanical things: structure and formatting, math rendering, signs that the answer key and the solution disagree, missing pieces in the solution, and dozens of format rules. None of this proves the math is right.
Then come the statistical checks. A question can pass every mechanical check and still be gameable, so a set of detectors runs on every content change. Most of them block a release that adds a new problem, the same way a failing test blocks a software deploy. The near-duplicate scan is an audit rather than a gate. They check:
- Answer-position balance, across the whole bank, by difficulty level and by topic, with hard limits. The by-topic check exists because a candidate noticed a letter pattern that the bank-wide numbers didn't show.
- Giveaway choices: choices that share a leading number and add "because" reasoning, choices that argue for themselves, and sets where one choice is so much longer than the others that its length gives it away.
- Duplicates and near-duplicates across the whole bank. Exact duplicates are gated, and the near-duplicate scan finds reworded twins so they can be rewritten to test the objective a different way.
- Each exam's mix of calculation and concept questions, kept within a target range based on the exam body's outline.
Last, AI review passes look at what the statistics can't: whether the question tests the learning objective or some side detail, whether a wrong choice is clearly wrong or could be argued as right, and whether the solution teaches or just states the answer. They run as scoped passes over an exam or a topic, so not every question has had one. Since fall 2026, every CFA question has also been checked against the curriculum, and I wrote up what that found in how accurate AI-written practice questions are.
After it goes live
When a question is published, FreeFellow saves a fingerprint of its content. Any later change, like a corrected solution, a rebalanced set of choices or a clearer question, saves the old version instead of overwriting it. Questions with problems that can't be fixed get retired, not deleted, so your past practice records stay the same.
Difficulty keeps getting checked. FreeFellow had recorded more than 870,000 candidate answers as of September 7, 2026. A reviewed calibration pass compares each question's difficulty label with how often candidates got it right on the first try and changes the label where the evidence is strong. One pass moved nearly 1,000 labels. The readiness score uses each question's current label, measured where there's enough data and the written-for level where there isn't.
Candidates can report a suspected error on any question. Reports go through an AI verification step against the source material. I decide the cases the process flags for me, like a rewrite that would change what a question tests. Confirmed errors get fixed in the live bank with the old version saved, and the reporter usually hears back by email with what changed, or with the reasoning if the question holds up. Some of the most useful checks in the process started as candidate reports, and there are probably more blind spots someone will find.
What this means for you
Guessing a letter won't help, since answer positions are checked for balance at every level. When you pick a wrong choice, its note tells you which mistake it came from. The checks for near-duplicates and giveaway choices keep finding them so they can be fixed, which makes your practice score a better read on what you know. And once enough candidates have answered a question, the difficulty label your readiness score uses has been checked against how they did.
Anyone can try the sample questions without signing up, and the full bank needs a free account. You can run the checks from this post on FreeFellow yourself.