The hardest thing about building voice search for India is that a large share of the queries are not in any single language.
Someone says “mujhe red colour ka kurta chahiye under 500”. That sentence is Hindi grammar, English nouns, an English number, and a product category that is a Hindi word most speakers would not think to spell. There is no language model for it, because it is not a language. It is what people actually say.
Flipkart’s engineering blog put it plainly back in 2020: “People often speak in more than one language in the same conversation, and frequently even within the same sentence.” Everything else in this post follows from that sentence.
The scale that actually matters
The number I can point at is public, from the ACL 2023 industry paper the speech group published after I left: Flipkart Voice Search “handles substantial traffic of approximately 6 million queries per day.”
Six million queries a day is a useful number precisely because it is smaller than the ones people reach for. It means roughly seventy queries a second averaged, several times that at peak, and it means that a one-percent regression in recognition accuracy is sixty thousand broken searches before anyone finishes reading the alert.
It also means the interesting constraints are not about throughput. They are about what happens in the four hundred milliseconds while somebody is still speaking.
Why WER is the wrong metric for this
Word error rate is the metric every speech paper optimises. For voice search it is close to useless, and this is the first thing that surprised me.
If someone says “bluetooth headphones” and the recogniser returns “blue tooth headphones”, WER records an error. The search results are identical. If someone says “boAt” and it returns “boat”, WER records exactly the same single-word error, and the results are completely different, because one is a brand with thousands of SKUs and the other is a watercraft.
Those two errors are identical to the metric and unrelated in the product. So the thing worth measuring is not whether the transcript matches the words, it is whether the transcript retrieves the same results. Amazon reached for the same idea independently and calls it content WER. Once you look at errors that way, a large fraction of them stop mattering, and the ones that remain are almost all proper nouns.
Voice search is the worst case for end-to-end models
End-to-end speech models are known to be weakest on proper nouns and rare words, because those are exactly what a model trained on general speech has least evidence for.
In a voice assistant that is survivable. “Play Bohemian Rhapsody by Queen” is eight words of context around two entities, and the context does a lot of the work. In e-commerce voice search the query is two or three words and the entity is the entire query. There is no context to recover from. Get the brand wrong and there is nothing left that was right.
Then add that Indian e-commerce brand names are frequently neither English nor Hindi (they are coined words, or English words used as brands, or Hindi words transliterated inconsistently), and that the same brand is pronounced several ways depending on which language the speaker is thinking in.
This is why the general-purpose benchmark numbers you read for ASR are not the numbers you will get. Everyone benchmarks on read speech with sentence context. Voice search is short, entity-dense, and adversarial by construction.
The 1,300 millisecond decision
The single clearest trade in the published work is about endpointing: deciding when the user has stopped talking.
That is the whole job in one row. A second and a third of latency is the difference between a feature people use and one they abandon. One percentage point of WER and one in forty-five users being interrupted is a real cost. Neither number tells you what to do; you have to decide which you would rather have, and that is a product decision wearing a modelling costume.
My general experience is that latency wins more often than accuracy people expect it to, and that the accuracy loss is recoverable elsewhere (in ranking, in query understanding, in a spell-correction layer) while the latency is not.
The architecture, and why not the obvious one
The system described in that paper is a three-level hierarchical CTC model (LSTM with windowed multi-head self-attention, predicting characters, then 300-token subwords, then 5,000-token subwords), trained on roughly 22,500 hours and 14 million audio-text pairs, with a 390 millisecond forward lookahead in streaming mode.
The interesting part is what it is not. RNN-T is the natural streaming architecture and the paper is direct about why it was not used:
RNNT provides a natural alternative in streaming scenarios but has high training complexity and inference latency rendering it difficult to use in a real-world setting.
Same for the streaming Conformer, which it describes as having “much higher training complexity and inference latency.”
I find that a healthy thing to see in print. The literature-optimal architecture and the one you can afford to train, serve and debug are frequently not the same object, and a paper that says so plainly is more useful than one that reports a better number on a system nobody could operate.
One script for two languages
A detail from the earlier 2021 paper that I think is the single most elegant decision in the whole system:
The English text was transliterated to Hindi using our inhouse transliteration APIs. Hence, the output text is in the Devanagari script for both Hindi and English utterances.
Rather than build a model that decides which language it is hearing and routes accordingly, put everything in one script and let the model stop caring. Code-switching stops being a special case and becomes ordinary text. You pay for it downstream (the search index does not speak Devanagari), but you have moved the problem to a layer that can be fixed with a lookup table instead of a model.
That is a pattern worth stealing. When a model keeps failing on a distinction, check whether you can delete the distinction.
A benchmarking trap that was invisible at the time
Worth knowing if you evaluate Indian-language ASR against published numbers: text normalisation can dominate the result.
Whisper’s text normaliser mishandles Indic scripts, deleting matras: the vowel marks that make Devanagari and related scripts work. Reported WERs for some Indian languages fall by tens of points once that is fixed, and the languages that appear to “improve most” from normalisation are the ones whose text was being destroyed worst.
This was established formally in the literature after my time on this, so I cannot claim we knew it. But if you were comparing your Hindi WER against a published Whisper baseline in 2022, some part of what you were measuring was a Unicode handling bug rather than a model.
What I would do now
Whisper and its successors have changed the starting point substantially, and I would not build an acoustic model from scratch for this today. What has not changed:
- Entity recognition is still the problem. A general model will not know your catalogue. Contextual biasing towards your own brand and product vocabulary is where the wins are, and it is unglamorous work that no foundation model does for you.
- Code-switching is still under-served. Benchmarks are largely monolingual. Your traffic is not.
- Endpointing is still a product decision. No model choice removes it.
- WER is still the wrong metric. Measure whether the results change.
The pattern underneath all four: the model was the part that had a paper written about it, and the part that took the least time.
Sources
- Goyal & Garera, Building Accurate Low Latency ASR for Streaming Voice Search in E-commerce, ACL 2023 Industry Track
- Joshi & Kannan, Attention based end to end Speech Recognition for Voice Search in Hindi and English, FIRE 2021
- Goyal, Singh & Garera, End-to-End Speech to Intent Prediction, EMNLP 2022 Industry Track
- The future of voice-powered shopping in the land of language, Flipkart Engineering, 2020