Fake News Detection Using TF-IDF, Word Embeddings and BERT: A Comparative Study
We teach computers to spot fake news in three different ways, compare them fairly on three public datasets, and propose HyFuse, a model that combines them and beats each one on its own.
❓The problem
Fake news spreads faster than people can check it. A computer cannot know the truth about the world, but it can learn how fake stories tend to be written: their words, tone and style. The question we answer: which way of turning text into numbers works best for this, and can we do better by combining them?
📚The data: three kinds of news
Every dataset is split into 70% for training, 15% for tuning and 15% for the final test. The test part is used only once, at the end.
🛠️How a check works, step by step
🔬The three approaches we compare
1. TF-IDF: counting important words
Idea: give each word a score: high if it appears often in this article but rarely in others. Words like "shocking" or "according to" become strong clues.
Like: a teacher who spots copied homework from repeated unusual phrases.
Models: Logistic Regression, Linear SVM, Naive Bayes, plus a version that also looks at letter patterns (2 to 5 characters).
Fast and easy to explain · cannot tell that "car" and "vehicle" mean the same thing
2. Word embeddings: understanding word meaning
Idea: place every word as a point in a 300-dimensional "meaning map", so words used in similar ways sit close together (king near queen). An article becomes the average of its words' points.
Like: knowing synonyms, so a new word with a familiar meaning is still understood.
Models: word vectors learned from our own news text, then Logistic Regression and a neural network (MLP).
Understands similar words · loses word order
3. BERT: reading in context
Idea: a deep transformer network, pre-trained on billions of words, reads every word together with all the others. "Bank" in river bank and in bank loan gets a different meaning.
Like: a careful human reader who understands the whole sentence, not just single words.
How we trained it: fine-tuned bert-base-uncased (110 million parameters) on a GPU for every dataset. For long articles it keeps both the start and the end of the text.
Most accurate single model on headlines and articles · slower and needs a GPU to train
Extra: writing-style checker
20 simple, human-readable signals such as ALL-CAPS words, exclamation marks, sensational words, vague wording, quoting sources and emotional tone, fed to a gradient-boosting classifier. It explains how a text is written.
🏆Our proposed model: HyFuse
Each approach makes different mistakes. HyFuse is a "judge" model that listens to all of them and has learned whom to trust, and when.
- Why it works: when BERT and TF-IDF disagree, HyFuse uses the context (for example a very short text, or lots of capitals) to decide which one is usually right in that situation.
- Fair tuning: HyFuse is trained only on the tuning split and picks its own settings with 5-fold cross-validation, so the test score is a fair one.
- Which parts matter (ablation study): we removed one ingredient at a time and re-measured. Removing BERT causes the biggest drop on every dataset, and removing TF-IDF also lowers the score on all three, so the combination is stronger than either one alone.
🤝Live fact check: the models lead, the evidence assists
Writing style cannot tell whether a fact is true, so Verity answers two questions and combines them:
What our models do inside the fact check
- Choose what to verify: the facts the models find most suspicious are checked first.
- Screen the sources: every news or web source is scored by the models before anything reads it; clickbait-looking sources are skipped and the rest are ranked by credibility.
- Weigh the evidence: less credible sources count less.
- Shape the verdict: true facts written in a manipulative style are reported as "⚠️ TRUE, but written like fake news", something a fact check alone cannot see.
Role of the AI evidence reader
A large language model is used only to read the sources: it fixes spelling mistakes ("sentinent" → "Sentinel"), splits a text into separate facts, and labels each source as confirming, contradicting, only rumouring, or unrelated, quoting the exact sentence. It never judges the writing. Every quote is then checked by our own rules: it must really appear in the source, our NLI fact-checking model (RoBERTa trained on natural language inference) must not disagree with it, and the claim's key details (names, numbers, dates) must be in it.
Only strong models are allowed to read the evidence (small models mislabel sources). Verity uses the first one that is available, in this order:
Verdict rules: only three answers
- ✅ TRUE when a trusted source (official website, encyclopedia, fact-checker or reputable news outlet) states the claim with all its key details, and no trusted source contradicts it.
- ❌ FALSE when a professional fact-check rates the same claim false (this beats any number of supporting posts), or a trusted source states the opposite, or the sources show that a key detail of the claim is wrong. Verity then shows "What the sources actually say".
- ❔ NOT VERIFIED only when no trusted source settles it (nothing found, only rumours, or the sources disagree). Our trained models then say whether it is likely fake or real from the writing alone.
- Fairness checks: a newer report from a trusted source replaces an older one; an announcement or schedule ("will be held") does not prove that something already happened; an article that only says a video or post claims something went viral is not proof; social media and video sites are never used as proof.
Possible verdicts
How the evidence framework is tested for the paper
For the IEEE evaluation one fixed judge is used for every claim: GPT-OSS-120B on Cerebras (no switching to another model), so every reported number comes from the same judge. 300 test claims: 100 from LIAR, 100 from FakeNewsNet and 100 current news items (50 real headlines, 50 claims rated false by fact-checkers). Sources that published a dataset's own labels (PolitiFact for LIAR, PolitiFact and GossipCop for FakeNewsNet) are excluded, so the judge cannot simply look up the answer.
🔍Explainable AI (XAI): seeing why the AI decided
An AI verdict is only useful if people can see the reason. Verity explains every decision in three ways:
Method: leave-one-out (occlusion) explanation, the same principle as LIME and SHAP, computed on the final HyFuse score. The page also shows plain-language clues (capital letters, clickbait words, missing sources) and colours each sentence by risk.
📏How we measure "better"
⚠️Limits
- The trained models judge how a text is written, not whether it is true, so a well-written false claim can pass them. This is why the live evidence check exists.
- Short one-sentence claims carry few writing clues; the best model is right about 64% of the time on them.
- The live check needs internet, and can only confirm what trusted sources have reported; brand-new rumours often have no evidence yet.
- The AI reader can misread a source, which is why its quotes are verified and always shown with a link.
- Claims that something does not exist ("X offers Y") are hard to prove false: if no source says so directly, Verity answers NOT VERIFIED instead of guessing.
- The internet AI readers have free daily limits; when all are used up the offline model is slower and less accurate.