Teaching a machine to hear a mistake

Why ordinary speech recognition gets it wrong

Speech-to-text is designed to be forgiving. When you mumble, it writes down the most likely correct sentence. That is exactly what a recitation checker must not do: if someone skips a word or says the wrong vowel, a forgiving model quietly “hears” the correct verse and the mistake disappears.

We turned the problem around. When someone recites, we already know which verse they are reciting, so we don’t need the model to guess words. We need it to write down the sounds that were actually said, and then compare them with the sounds that were expected. The checker looks for four kinds of mistake:

• a skipped word,
• a wrong or missing letter, including a dropped shadda (a doubled letter),
• a wrong short vowel,
• a dropped ending where the reader did not stop.

Just as important is what it must not flag. A reciter may stop at any word out of necessity, such as running out of breath, and stopping changes how the word ends: the last vowel falls silent, and ة is read as a soft h. Some long vowels may validly be held for different lengths; a long vowel before a stop, for example, may be held for two, four or six counts. Going back to repeat a word and correct it is normal and is not a recitation error. Speech before the verse, such as the isti’adha (“a’udhu billahi mina sh-shaytani r-rajim”), is not part of the check.

Tajweed (the rules for giving each letter its correct articulation and qualities) is a separate discipline and is out of scope for this version. The checker is a practice tool that supports a qualified teacher; it detects likely mistakes in recitation, it does not replace the teacher.

What CTC is, and why we chose it

Most modern speech-to-text is built differently, as an encoder and a decoder. One network listens, and a second writes the text a token at a time, predicting each token from the audio and from the text it has already written. The decoder therefore learns an internal language model (Variani et al., 2020; Meng et al., 2021), which is why these systems are so good at dictation. For checking recitation, it works against you:

• It favours fluent, likely text over what was said. A verse the model has seen many times tends to come out correct, even when it was misread.
• It can repeat, drop or invent text, a known failure mode of encoder-decoder models on long audio with pauses, and pauses are normal in recitation.
• It gives no reliable timing for each sound. These models do not natively report when each sound was said, so you cannot measure how long a vowel or a shadda was held.
• It scores whole outputs, not single sounds. To ask “does this audio fit the correct word or the mistake better?” you need a score for any sequence of sounds you choose.

A CTC model has none of these problems by design. There is no decoder, so nothing writes text and nothing auto-corrects. The output follows the audio in order. And for any sequence of sounds we give it, the forward algorithm computes exactly how likely the model finds that sequence given the audio. That last property is what the rest of this article is built on: we can score the correct reading and a mistaken reading against the same audio and compare them.

We use a pretrained self-supervised speech encoder (a model that first learned general speech representations from unlabelled audio; see Mohamed et al., 2022) with a CTC head over a phonetic alphabet for recitation. The small model that runs on the phone is built the same way: an encoder with a CTC head, no decoder.

CTC is also what makes tajweed reachable later. The alphabet we transcribe into makes tajweed visible: a shadda is a doubled letter, a long vowel is written once for each count it is held, and nasalisation (ghunna) and the echo of qalqala have their own symbols. With a probability for every slice of audio, each of these becomes something we can measure rather than guess. One caution from the literature: CTC outputs are “peaky”, mostly blank with a sharp spike for each sound (Zeyer et al., 2021), so timing read directly from them is approximate. For a shadda we therefore measure how much probability the doubled sound receives and over how many slices, rather than trusting a single spike. Our hearing model already reports letter qualities such as heavy and light, and the judge uses them today. Tajweed checking is the next step, and we will release each rule only once it has been validated on its own.

Most false alarms were ours, not the model’s

Our first version used a pretrained speech model as it was. On correct recitation by professional reciters, it marked 12% of words as wrong. That sounds like a bad model. It wasn’t.

The problem was the comparison. We had compared each verse against a single expected reading. Once we accepted every valid way a phrase can be said (stopping or continuing at each word, and every allowed length of each long vowel in Hafs), false alarms fell to 2.4% after handling stops, and to 1.1% after handling vowel lengths, without training anything. Before you blame a model, make sure you are asking it the right question.

The phone problem

Professional recitations played into a phone were marked wrong on 46.7% of words. We measured why. In the phone recordings, only 1.5% to 5% of the sound energy was above 1 kHz, against about 21% in studio recordings.

That matters because several Arabic letter distinctions depend on cues phone audio weakens. The heavy (emphatic) letters ط ص ض are marked mainly by a lowered second formant, roughly 1 to 2.5 kHz, in the vowel that follows (Jongman et al., 2011). ص and س are both hissing sounds whose noise sits mostly above 3 to 4 kHz. ع differs from ء mainly through changes to the vowel’s lower formants and voice quality. When the channel suppresses those bands, some of the information a checker needs is simply not in the recording.

Three changes brought false alarms down to 15.8% before any retraining:

• Ask the phone for its speech-recognition microphone source. Android requires this source to have noise reduction and automatic gain control off by default (Android Compatibility Definition, §5.4.2). A flat response is only recommended, and only from 100 Hz to 4 kHz, so the channel still varies by phone.
• Measure how “bright” each recording is, as the share of energy above 1 kHz.
• Treat what the channel cannot show as unclear, not wrong. On muffled audio, the kinds of difference that even professional reciters produce are reported as unclear. A system should know when it cannot tell.

Scoring confidence, not just differences

Two sound sequences can differ for many harmless reasons. So for each difference we ask how much better the audio fits the mistake than the correct reading, using the CTC likelihood of each:

score = log P(mistaken sounds | audio) - log P(expected sounds | audio)

A positive score means the audio supports the mistake; a large one means it supports it strongly. This is the CTC form of a classic idea in pronunciation assessment, the goodness of pronunciation score (Witt & Young, 2000), and recent work builds the same comparison by substituting or deleting sounds in the expected sequence (Cao et al.; Parikh et al., 2025). It is exact under the model, but it is not a calibrated probability that a mistake happened, so it needs testing.

To test it we used planted mistakes: keep a reciter’s correct audio but change the expected sounds, for example ع to ء. The audio no longer matches the expectation at that spot, so the checker must flag it. That gives thousands of labelled mistakes for free. With confidence scoring, false alarms fell below 1% (0.85%) while 93% to 99% of planted mistakes, depending on the kind, were still caught.

Planted mistakes are useful, but they flatter you. Today we catch 97.4% of planted mistakes and 80.9% of real mistakes in held-out phone recordings. Real mistakes are what people make, so that is the number we work on.

Fine-tuning the “ears”

Next we fine-tuned the speech model for recitation. We track the phoneme error rate (PER): the share of expected sounds that come out substituted, deleted or inserted. Three lessons came out of it.

• Check your plumbing. Our first fine-tune started from a 36% PER where we expected about 2%. The cause was a timing assumption: the model produces its outputs half as often as we had assumed, so the sound timings we used to cut training audio were wrong. We found it by feeding in ten seconds of silence and counting the outputs.
• Then it worked. With the timing fixed, PER on recordings held back from training fell from 3.2% to 0.75% for professional reciters and from 17.0% to 7.3% for learners.
• Be careful with correct-only data. A later run added correct recitations by learners, most of them of one short surah. Its PER on correct learner speech fell from 8.6% to 0.31%, which looked excellent. But on that surah it caught 9.7% of vowel mistakes instead of 96.8%. The model had learned what the surah should sound like and heard that, whatever was said. It never shipped. Where we train on mistakes, the label is what was actually said, not what should have been said.

Separate hearing from judging

That run taught us that a lower error rate is not the same as catching more mistakes. So we split the system in two: a model that hears, and a small, separate model, gradient-boosted decision trees, that judges each word from about 25 measurements (the differences found, the confidence scores, timing, how bright the recording is, the word’s position, and more).

One rule made the judge trustworthy: a word may only turn red if a real error was found, or a targeted check (below) favours a mistake. Before that rule, 90% of the red words on correct learner recordings had no actual error behind them. The judge had learned a shortcut that did not hold for real voices.

Because judging is separate, strictness becomes a setting rather than a retraining job. Easy, Normal and Strict modes let a teacher choose. When we introduced them, on phone recordings held out from training, Easy caught 52% of mistakes with 1.0% of correct words red, Normal 68% with 2.6%, and Strict 76% with 3.9%.

Listening for what the model “fixes”

Even a fine-tuned model leans towards what it expects. So for each word we also run targeted checks: we build its likely mistakes (another short vowel, a dropped ending, a dropped shadda, a commonly confused letter, a long vowel cut short) and score each against the audio with the same likelihood ratio. That catches errors the model’s own first transcription hides.

Not every idea survived. Counting letter swaps on muffled phone audio, and a hard rule for long vowels cut short, both turned professional reciters red: with them, false alarms on professional recitations played into a phone more than doubled (4.35% to 9.78%). Both were dropped. A phone can turn a heavy letter into its light partner on a correct reading, so on muffled audio those swaps are not tried.

Some measurements did earn their place. Measuring how strongly and over how many slices each shadda is heard, and how heavy or light each letter sounds, and giving those to the judge, raised missed shaddas caught from 5% to 26% on held-out phone recordings, while correct words turned red less often (2.7% to 2.1%). A plain rule on the same shadda evidence did the opposite: it more than doubled false alarms on correct learner words (7.4% to 17.7%), because learners hold a shadda short and inconsistently. The judge can weigh evidence; a rule cannot.

Finally, we added real phone recordings made by our team (each with one deliberate mistake or none) to the hearing model’s training. On held-out recordings from the same team, mistakes caught rose from 69% to 82%, and correct words marked red fell from 2.4% to 1.6%. At the same 1.5% false-alarm rate, the new version caught 82% against 68%. Because the held-out set comes from the same people and phones, part of that gain may be adaptation to them; new readers will show how much carries over.

Making it fast enough to feel instant

A checker is only useful if it keeps up with the reader. We tried the standard speed-ups and kept a strict rule: the live path must give bit-identical answers, or pass the full scorecard again.

• A smaller distilled student model ran 1.37 times faster but caught fewer real mistakes on held-out phone recordings (75.0% against 80.9%).
• 8-bit integer and 8-bit floating-point versions ran slower than 16-bit on our hardware.
• A graph compiler was 1.2 to 1.3 times faster but changed the sounds heard in 3 of 231 test recordings; another compiler could not convert the model.
• Batching recordings together changed a small share of output slices and flipped a few verdicts on learner recordings. Several independent copies of the model, each handling one recording at a time, keep every answer identical instead.

What worked was splitting the work by urgency. CTC can stream, but a model trained on whole recordings loses accuracy when it only sees short chunks: in our tests, its PER on held-out phone recordings rose from 3.4% to as much as 6.7% (see Zhang et al., 2020, on why chunk-aware training is needed). So the cheap streaming pass is used only to follow the reader and show a word in amber (“may be wrong”), a median of about three seconds after the reader reaches it. When the reader pauses, a full pass hears the whole phrase again and gives the final verdict, so a red word never disappears. Model time fell from 0.29 to 0.08 seconds per second of reading, 3.6 times less, with identical word results on saved readings. Checking one verse takes under a tenth of a second.

On the phone itself, a small model follows the page and catches skipped words, even offline. Our first approach judged skips in fixed short windows and found only 41% of the real skips in our test readings. The same method with the large server model found 46%, so the method was the limit, not the model. Keeping a running transcription lined up with the page raised it to 90%, with about 1% of read words falsely marked. A check that almost nothing was heard between a word’s neighbours then cut false marks on held-out readings from 12 to 7.

How we measure

We test on reciters and verses the system never saw in training, with four kinds of check:

• correct recitation by professional reciters, from the studio and through a simulated phone microphone (how often it wrongly flags a word);
• planted mistakes (many cheap labels);
• phone recordings by our team, each with one deliberate mistake or none, held out from training;
• real mistakes made by learners (the number that matters most).

Two rules keep us honest. We compare versions at the same false-alarm rate, the standard way to compare detectors (Martin et al., 1997), because a checker that marks more words red will always “catch” more. And nothing ships unless the full scorecard beats the version already live, with a rollback ready. Several “improvements” in this article failed that test and never shipped.

Where it stands

On held-out tests today:

Real learner mistakes are harder than planted ones, and that is where we are focused now: closing the gap between planted and real mistakes, heavy and light letters on phone microphones, and missed shaddas. We will publish the learner numbers when they meet our own target.

Pronunciation assessment with CTC models is an active research field, including recent public benchmarks for Quranic recitation. Our work builds on that research; our focus has been making it work live, on ordinary phones, with hearing and judging kept apart.

Ten lessons that apply beyond the Quran

1. Model the valid variation before blaming the model. Our false alarms fell from 12% to 1.1% without training.
2. Train on what was said, not on what should have been said.
3. Measure mistakes caught, not only error rate. A better error rate can mean fewer catches.
4. Planted mistakes are cheap and useful, but they overstate real performance.
5. Measure the recording channel. Some cues are physically missing from phone audio.
6. Keep hearing and judging separate, so strictness is a setting, not a retrain.
7. Compare options at equal false-alarm rates, or you are comparing nothing.
8. Test a method’s ceiling with your best model before blaming a smaller one.
9. Spend the expensive computation only where the verdict is made.
10. Gate every release on a full scorecard, with rollback ready.

What’s next

We are turning the checker into an API for Quran apps and teaching platforms, so they can add word-by-word recitation checking without building the speech stack themselves. If you run a Quran learning app, a tahfiz programme or an Islamic school and want early access, get in touch.

References

1. Graves, A., Fernández, S., Gomez, F. and Schmidhuber, J. (2006). Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. ICML 2006. https://www.cs.toronto.edu/~graves/icml_2006.pdf
2. Zeyer, A., Schlüter, R. and Ney, H. (2021). Why does CTC result in peaky behavior? https://arxiv.org/abs/2105.14849
3. Variani, E., Rybach, D., Allauzen, C. and Riley, M. (2020). Hybrid autoregressive transducer (HAT). ICASSP 2020. https://arxiv.org/abs/2003.07705
4. Meng, Z. et al. (2021). Internal language model estimation for domain-adaptive end-to-end speech recognition. IEEE SLT 2021. https://arxiv.org/abs/2011.01991
5. Mohamed, A. et al. (2022). Self-supervised speech representation learning: a review. IEEE Journal of Selected Topics in Signal Processing 16(6). https://arxiv.org/abs/2205.10643
6. Witt, S. M. and Young, S. J. (2000). Phone-level pronunciation scoring and assessment for interactive language learning. Speech Communication 30(2-3).
7. Cao, X., Fan, Z., Svendsen, T. and Salvi, G. Segmentation-free goodness of pronunciation (preprint). https://arxiv.org/abs/2507.16838
8. Parikh, A., Tejedor-García, C., Cucchiarini, C. and Strik, H. (2025). Enhancing GOP in CTC-based mispronunciation detection with phonological knowledge. Interspeech 2025. https://arxiv.org/abs/2506.02080
9. Jongman, A., Herd, W., Al-Masri, M., Sereno, J. and Combest, S. (2011). Acoustics and perception of emphasis in Urban Jordanian Arabic. Journal of Phonetics 39(1), 85-95.
10. Android Compatibility Definition Document, Android 16, §5.4.2 Capture for voice recognition. https://source.android.com/docs/compatibility/16/android-16-cdd
11. Zhang, B. et al. (2020). Unified streaming and non-streaming two-pass end-to-end model for speech recognition. https://arxiv.org/abs/2012.05481
12. Martin, A. et al. (1997). The DET curve in assessment of detection task performance. Eurospeech 1997.