Can ChatGPT Write Good Practice Questions? The 7 Tells
Yes, a frontier model can draft a plausible multiple-choice question in seconds. And no, you should not study from an unaudited AI question bank: in FreeFellow's own audits of AI-drafted content, one early bank had 47% of its correct answers sitting on a single letter, and a corpus sweep found more than 2,000 answer choices that argued for their own correctness, a shape real exams never use. Every defect in this post is one FreeFellow has found, measured, and gated in its own pipeline, which is exactly why I can describe them specifically.
I am Jeffrey Ting, FSA, CFA, the founder of FreeFellow. FreeFellow's first drafts are AI-generated too. The thesis of this post is not that AI content is bad; it is that generation and verification are different jobs, and an AI question bank is only as good as the audit layer behind it. Here are the seven tells of a bank that skipped that layer, each one testable yourself.
Tell 1: The Correct Answer Clusters on One Letter
A generation pipeline typically produces the correct answer first and the distractors after, so unless a separate rebalancing step shuffles positions, correct answers pile onto A. This is not hypothetical. One of FreeFellow's banks was perfectly uniform for months, then two expansion batches were added without the rebalancing step, and the bank drifted to 47% A. A candidate who always guessed A would have scored 47% without reading a single question.
The subtle version is worse. A candidate once wrote in that "most of the right answers are A" in one of our banks. The bank as a whole was a clean 25% per letter, but her instinct was half right: one topic within the bank leaned 33% A, invisible in the aggregate.
The fix: FreeFellow runs automated answer-distribution checks with hard thresholds (no letter above roughly 1.7 times its uniform share, none below 0.4 times) on every content change, the way software teams run regression tests. After that candidate's report, the check was extended to run per difficulty tier and per topic, so a skewed batch can no longer hide inside a balanced aggregate.
Test it yourself: tally the correct letter over 40 questions from any bank. On a 4-choice exam, anything near 40% on one letter is a defect, not chance.
Tell 2: On Math Questions, the Middle Number Wins
Ask a model for a calculation question with five numeric choices and it does something very natural: it computes the right answer, then builds distractors symmetrically around it (a little lower, a little higher). Sort the choices in ascending order, as actuarial and finance exams do, and the correct answer lands in the middle. One of FreeFellow's early actuarial banks had the correct answer in the middle position 47% of the time.
Real exams do not look like this. On released SOA and CAS papers, the correct answer is the smallest or largest of the five choices roughly a third of the time, because real distractors are not a symmetric ring: they are the specific wrong numbers produced by common mistakes (forgetting to annualize, dropping a sign, using the wrong table), and those mistakes land on one side of the truth.
The fix: FreeFellow regenerated its numeric distractor sets to be mistake-derived rather than symmetric, and the distribution gate now checks where correct answers sort, not just which letter they carry. The side benefit is pedagogical: every distractor now encodes an error a real candidate actually makes, and the per-choice note can tell you which one.
Test it yourself: across 30 calculation questions, count how often the correct answer is the smallest or largest choice. Near zero means machine-made distractors.
Tell 3: Choices That Argue Their Own Case
Real exam choices state what, never why. An unaudited generator produces choices like "527,000, because the translation loss is excluded from net income" next to "505,000, because unrealized gains reduce comprehensive income." The reasoning belongs in your head during the exam and in the solution afterward, never in the choice.
This shape is worse than a style problem. When a generator cannot make numerically distinct distractors, it hangs the discrimination on the appended prose, and the prose often papers over broken math: in one FreeFellow audit of 39 such questions, 16 had at least one "because" derivation that did not actually arithmetic to its own stated value.
The fix: FreeFellow runs three detectors for this family (shared-value choices with reasoning tails, distinct numeric choices with reasoning or method tails, and the conceptual variant like "Descriptive analytics, because it summarizes what happened"). The numeric variants sit at zero tolerance: one cleanup pass stripped reasoning tails from about 1,900 choices, and any new occurrence fails the build. Questions where stripping would collapse two choices into one get rewritten instead.
Test it yourself: search any sample set for ", because" inside the answer choices. On a real exam paper you will not find it.
Tell 4: The Longest Answer Wins
Test-writers have known this tell for a century: the correct option ends up longest because the author lavishes detail on it. Generators reproduce the bias enthusiastically. If the most detailed, most qualified choice keeps being right, the bank is coachable without knowledge.
The fix: FreeFellow's choice-quality detector flags lopsided length spreads within a question's choice set as a rewrite trigger, alongside the reasoning-tail checks above.
Tell 5: The Same Question in Different Clothes
Generate 1,000 questions on one syllabus and the model will circle back to its favorite setups: the same bond, the same fact pattern, lightly reworded. Near-duplicates inflate both the bank's advertised size and your practice score, since you are re-answering a question you have already seen.
The fix: FreeFellow runs similarity detection across the corpus and regenerates twins as genuinely new angles on the learning objective; one campaign replaced 121 near-duplicate pairs across 23 exams, and the detector now gates new content.
Tell 6: Numbers From the Wrong Year
A model's training data spans years, so it blends them. Ask a chatbot for a tax question and it may quietly use last year's thresholds, a superseded lease standard, or the prior syllabus's topic weights, stated with total confidence. For tax-anchored exams (EA, CPA REG and TCP, CFP), a single stale constant flips the answer key.
The fix: FreeFellow anchors tax-year figures to the current law and the relevant testing window, and its tax-anchored banks are deliberately walled off from bulk find-and-replace updates: when a figure changes, affected questions are regenerated with full question context so dependent arithmetic moves together. A watch process also tracks the credentialing bodies' own pages for syllabus and policy changes.
Test it yourself: ask any AI-generated question source which tax year its figures assume, and check one number against the IRS or the exam body. You will learn a lot from whether anyone can answer.
Tell 7: The Wrong Kind of Question Entirely
Each exam has a signature mix of computation and judgment, and a generator defaults to whatever is easiest to produce, usually calculation. FreeFellow learned this in public: its FRM Part 2 bank shipped at about 48% computational when the real Part 2 is famously qualitative (roughly 30 to 35% computational), and a candidate called it out. He was right.
The fix: every FreeFellow exam now carries a target band for its calculation-versus-conceptual mix, sourced from the exam body's outline and candidate consensus, enforced by the same always-on checks. The FRM Part 2 bank was rebalanced with per-topic caps to match how the real exam actually tests.
The Part No Generator Can Do: Difficulty
Even a perfect generator cannot tell you how hard a question is, because difficulty is not a property of the text. It is an empirical fact about candidates: what fraction of prepared people answer it correctly. A chatbot labeling its own question "hard" is guessing.
FreeFellow measures it. Millions of real candidate answers feed a monthly calibration pass that compares each question's labeled difficulty against actual first-attempt accuracy and re-labels where the evidence is strong; one round moved nearly 1,000 labels. That measured difficulty is what feeds the readiness score, which is why the score can mean something. No static book, and no chat session, has this loop at all.
The Ten-Minute Audit, Summarized
Before trusting any question bank, AI-built or not:
- Tally the correct answer letter across 40 questions. Flag anything near 40% on one letter of four.
- On calculation questions, count how often the correct answer is the smallest or largest sorted choice. Flag near-zero.
- Search the choices for ", because". Flag any hits.
- Check whether the longest choice keeps winning.
- Ask when difficulty labels were last changed, and what data changed them. Flag silence.
FreeFellow publishes its entire bank free, browsable without signup, so you can run every one of these checks on it before creating an account. That is deliberate.
What Chatbots Are Still Great At
None of this is an argument against studying with AI. A chatbot is the best explanation engine ever built: use it to rephrase a concept until it clicks, to interrogate a solution you do not follow, or to quiz you conversationally on a topic you name. FreeFellow's free tier even ships a copy-to-AI prompt builder for exactly that workflow on written-answer practice. Let the chatbot be your tutor. Just make sure your examiner, the fixed bank you measure yourself against, has been audited by someone.