Meet Unitra Engine 2

Unitra Engine 2 is the new engine behind everything Unitra AI does with sound, from hearing what is said to putting a translated subtitle on screen. It halves the error rate on real gameplay audio, handles mixed Chinese and English better than anything else we measured, covers 27 languages, and runs entirely on your own GPU. This post covers how well it hears.

Why we rebuilt it

Engine 1 was tuned for clean speech in one language at a time, which is what most benchmarks contain. It is not what our users do. They speak in short bursts mid-match, half-finish words, and switch languages inside a sentence. Engine 2 was built for that, and everything below is a measurement of the difference.

Accuracy on real gameplay audio

Public benchmarks are people reading prepared sentences in quiet rooms. To find out how the engines behave on the audio the product really receives, we recorded 237 clips of ourselves playing, transcribed them by hand, and scored both engines on those.

Error rate, lower is better:

English drops from 45.45 to 19.58, and mixed Chinese and English from 45.69 to 26.09. Both engines score much worse here than on any clean benchmark.

The averages leave out something users notice more than accuracy. When Engine 1 meets speech it was not trained for, it stops early, and the subtitle simply ends partway through the sentence.

Utterances losing 30% or more of what was said:

Neither engine registers on Chinese because that bucket is made of short utterances, a median of 11 characters, and the worst case on either engine is a single character missing out of five. The mixed clips are twice as long and Engine 1 loses up to 65% of one. On the larger Chinese set in the next section the same measure is non-zero for both.

Mixed-language speech

Switching between Chinese and English mid-sentence is ordinary in the games Unitra AI serves, and it is the hardest case on the list. We wanted this claim to be checkable, so we measured it on ASCEND, a public conversational code-switching corpus released under CC BY-SA 4.0, and ran Whisper large-v3-turbo alongside as an outside reference. Whisper is OpenAI's speech recognition model, released under the MIT licence, so we can benchmark it and publish the result. Anyone can download both and re-run what follows.

The two monolingual rows are the control. They come from the same speakers, the same recordings and the same segmentation, one language at a time. Engine 1 copes with either language on its own and falls apart when they are mixed. Engine 2 is the most accurate system on all three.

On mixed speech Engine 2 is 3.7 times more accurate than Engine 1 (14.20 against 52.98) and ahead of Whisper large-v3-turbo as well (18.69). The truncation pattern is even starker than the error rate: Engine 1 loses a third or more of the sentence on 56% of mixed clips, against 7.8% for Engine 2.

On mixed speech, which language you configure a system for changes its score a lot. Engine 1 scores 52.98 set to Chinese and 85.94 set to English; Whisper scores 18.69 detecting the language itself and 24.05 given a Chinese hint. Both are reported above at their better setting.

This also holds regardless of who is speaking. Grouped by the speaker's first language, Engine 2 improves on Engine 1 by 1.6 to 2.0 times for Chinese-native, Japanese-native and English-native speakers alike.

Speech under gunfire

Games are loud. We pulled 393 real gunshot transients out of game audio, mixed them into real microphone speech at nine controlled levels, and measured both engines across all of it.

Word error rate as the gunfire gets louder, lower is better. The column heading is the signal-to-noise ratio in dB, so gunfire is loudest on the right:

Engine 2 keeps roughly a 2x lead at every level, and is still 1.85x ahead with the gunfire 5 dB louder than the speech itself.

A subtitle that appears when nobody is speaking is distracting. On 1414 clips of pure game audio with no speech in them at all, Engine 2 stays silent on 46%. Engine 1 produces something every single time and has to have it filtered back out.

Speed and resource use

How much of your machine the engine takes matters as much as how quickly a sentence finishes. Real-time factor measures that directly: seconds of computation per second of audio. Lower is better.

Engine 2 uses between 3 and 11 times less compute for the same audio, and its cost stays about the same whichever language you are in. Each figure is the mean of three complete runs over the same clips.

Both engines read the same clips, one job at a time, each on the fastest setup we could get it to run on.

27 languages

Engine 2 covers 27 of the 35 languages Unitra AI supports, against Engine 1's 19. It adds Cantonese, Indonesian, Thai, Turkish, Hindi, Malay, Swedish, Danish, Finnish, Czech, Tagalog, Greek, Hungarian, Macedonian and Romanian. Eight languages are not covered yet, so Engine 1 stays available in Settings and remains the lighter choice on a small GPU.

How it compares to open weights

The same outside comparison, this time on clean read speech and against OpenAI's full-size Whisper large-v3. Same audio, same scorer, on FLEURS, which anyone can download.

Engine 2 leads every system on English and Chinese, the two languages Unitra AI is used in most. Whisper large-v3, a model two and a half times the size and far slower to run, leads on Japanese, where Engine 1 also stays available.

How we measured

Scoring uses five cumulative normalization stages covering surface form, punctuation, numerals, units and reading, so a difference in how something is written is never counted as a mishearing. Every system reads one identical copy of the audio, with each runner's own preprocessing switched off.

FLEURS and ASCEND are public, so those tables can be reproduced. The gameplay and gunfire results use internal recordings and licensed game audio, which we cannot redistribute, so for those the method is published rather than the material.

Every reference transcript was written or corrected by a person listening to the audio, never taken from another model's output. On the internal clips the human changed 84 of the 204 drafts they were shown, which is roughly what a benchmark scored against a machine's guess would have gotten wrong.

Available now

Engine 2 is on by default in Unitra AI 0.11. If you already have Unitra AI, update and you are already using it.

ASCEND is used under CC BY-SA 4.0: Lovenia et al., ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation, LREC 2022. FLEURS and LibriSpeech are publicly available. Whisper is MIT licensed.