Does it work?
Two claims, measured against baselines, with the failures shown.
Results generated Nov 7, 2026, 5:05 PM CT · loaded from the committed eval/results.json
Dataset: synthetic stand-in, template-generated (NOT T-Mobile data). These numbers show the pipeline works; they are not a result on T-Mobile's data.
Claim A: detection
Beats the baseline by 13.9 F1 ptsScam vs. legit on a held-out split (seed 42, stratified 80/20 (83 train / 21 test); only the held-out test ids are scored). Agent mode: basic (rules-only 'basic' verdicts; the Nemotron agent was NOT used). Positive class = scam.
| Method | n | Precision | Recall | F1 |
|---|---|---|---|---|
| Bluff (unsure counted as scam) | 21 | 84.6% | 100.0% | 91.7% |
| Bluff (unsure counted as legit) | 21 | 84.6% | 100.0% | 91.7% |
| Stand-in baseline: contains a link → scam (not T-Mobile's baseline) | 21 | 100.0% | 63.6% | 77.8% |
Pipeline check only. The messages are template-generated by the same team that wrote the detector's heuristics, the test set is tiny (n=21), and the rules-only path (not the Nemotron agent) produced these verdicts. Do not read this as real detection accuracy; the real T-Mobile set and a live agent run are still pending.
Category accuracy against the taxonomy: 0.0%
Where Bluff is still wrong (Claim A)
Your verification code is 710001. Don't share it with anyone.
Expected legit · Got scam
verdict likely_scam (31/100, 85% sure); evidence: Asks for a PIN, password, SSN or card number; Limited check: the page content was not analyzed
Your verification code is 639573. Don't share it with anyone.
Expected legit · Got scam
verdict likely_scam (31/100, 85% sure); evidence: Asks for a PIN, password, SSN or card number; Limited check: the page content was not analyzed
Claim B: exposure-to-charge linking
Beats the baseline by 4.0 F1 ptsDid the linker find the true source site? held-out test split (20 charges: 13 with a true exposure, 7 with none, 5 hard). Dev split (10) used only to check the threshold.. Method: SQL linker: normalized descriptor + trigram/abbreviation/token name score + time window (agent confirmation not applied in this eval).
| Method | n | Precision | Recall | F1 |
|---|---|---|---|---|
| Bluff linkerthreshold 0.55 | 20 | 100.0% | 100.0% | 100.0% |
| Naive substring match | 20 | 100.0% | 92.3% | 96.0% |
Optimistic by construction: these synthetic charges were written by the same team as the linker and every positive was checked to be solvable by it (data integrity test). Treat F1 near 1.0 as "the pipeline works", not as expected real-world accuracy. The naive baseline mainly loses on abbreviated descriptors.
Where the linker is still wrong (Claim B)
No failures in this run.
Where the naive baseline fails (Claim B)
SQ *SNGLSSOTLT
Expected sunglass-outlet · Got no link
missed the true exposure (prefix + abbreviation)
How to read this
- Splits are fixed and committed before any tuning. Thresholds were tuned on the dev split only; every number above is on the held-out split.
- Small test sets mean wide error bars. Treat differences of a few points as noise.
- Failure cases are listed on purpose: they show what Bluff still gets wrong.