Language Lab: Invented Languages and Machine Voices for Games

What it does

A game wants its characters to speak a language nobody knows. The same word should always sound the same, names should come out as written, and the player should be able to learn the language by listening. Language Lab does that offline, in four stages:

1. Meaning. An English line becomes a typed semantic representation: who did what to whom, tense, negation, question or command, radio procedure words.
2. Language. That representation is realised in an invented language with its own grammar, not word-for-word substitution.
3. Speech. A neural TTS model speaks the invented words from explicit phonemes, so it never guesses an English reading of an invented spelling.
4. Voice. A resynthesis engine replaces the voice with a designed one, from a light tint to a fully alien one.

The output is what a game needs: the audio, every word’s start and end time, its English gloss and a learning key, so the game can show words as they are said and teach them over time.

One meaning, two grammars (v5)

Versions up to 4 replaced English words one by one with stable invented ones. That keeps English word order, so it reads as a cipher, not a language. Version 5 adds a typed meaning layer: a line is a list of clauses, events and states with roles (agent, patient, theme), tense, aspect, polarity and force. A rule-based analyser builds it from English. Telegraphic radio text such as “Walker not now, guns on us.” becomes fragments with lowered confidence rather than a forced parse.

Two languages realise the same representation with different grammars:

	tala (transparent)	dovek (distant)
syllables	CV / CVC, 10 consonants	clusters (kr, dr, st), richer codas
clause order	SVO	SOV; OSV when the object is already known
tense, negation	particles before the verb	a suffix chain on the verb
agreement	none	subject person and number on the verb; subject pronouns dropped
objects	article la	accusative suffix, only on definite objects
harmony	none	front/back vowel harmony through every suffix

English	tala	dovek
I opened the door.	ha ta nepan la sumu .	dobartunu rudatum .
Do you see the little robot?	we noku la yemi lontun ki ?	buyur trurnonsumunu largurunumu ?
We did not find the cold water.	se ta ma ningka la kansi haken .	kepel manggurnu gumkormatuk .

Every word keeps a learning key: the concept (robot), however it is inflected or moved, or a grammatical tag such as #past. A hearing counts for the concept, so a player who hears yemi and yemi mi is learning one word. On the 132-line corpus, tala keeps English word order in 52 % of lines and dovek in 27 %.

Hearing it the way the game will

To judge the voices the way a player meets them, I wrote a small Godot window: a speaker on a low-resolution, scanlined monitor, and the line appearing word by word as it is said. The word being spoken is lit and underlined, and English glosses can sit above it.

The hard part is the clock. A word should light when it is heard, not when the engine mixes it, and the line can be sped up or slowed down with the voice keeping its pitch:

# main.gd: seconds into the line's audio
_t_last = maxf(_t_last, maxf(0.0, player.get_playback_position()
        + (AudioServer.get_time_since_last_mix()
           - AudioServer.get_output_latency() - _shift_delay()) * speed))
return _t_last

The time since the last mix and the output latency pass speed times as fast in the audio, and the pitch-shift effect that undoes the speed-up holds back about 35 ms of its own. The value is never allowed to step back within a line: between mixes the estimate can dip a few milliseconds, which made a word vanish for one frame. A check plays a line with the master bus muted and logs each word against the clock: the worst lag is 20–28 ms at 0.8×, 1.0× and 1.3×, about one frame.

A toggle plays the same take over the radio. It is a chain, not static laid on top. The voice is band-limited to 350 Hz – 2.8 kHz, compressed like a radio’s automatic gain control and pushed through a small speaker’s overdrive. The signal fades and briefly drops out, a hiss rises only as it weakens, and a key-up burst and a squelch tail frame each line. Recorded from the window’s output, energy below 300 Hz falls from −3 to −23 dB of the total and the voice stays within about 4 dB of its dry level.

Delivery and casting: tuning by ear (in progress)

In v5 every line is delivered the same way. “Mayday, mayday, mayday” was three identical takes of 0.853 s with fixed 0.22 s gaps, as calm as “the kettle is on”. Version 6 picks a delivery (urgent, wary or calm) from data the line already has: a distress marker in the meaning layer, or the context it is spoken in. Urgent lines speed up from phrase to phrase, use short uneven gaps and sometimes an inhale. Each repeat is its own take, so no two maydays are alike, and all of it is seeded, so a line renders the same every time.

The voices need the same treatment. The male voice sounded alien and the female one sounded like stock TTS. The difference turned out to be the processing, not the gender: the male preset pins the pitch and lowers the formants, while the female voice ran a light tint over a well-known stock speaker. Every candidate is now a blend of less familiar speakers plus a full voice design. Each gets an alien score: the mean distance between the cepstral envelopes of its final audio and of its own dry TTS take, and the liked male voice sets the bar at 0.38. A score can show that a voice has moved away from stock, not that it sounds good. Of the first six female candidates, the two busy, swarming designs were rejected by ear, one of them with the highest score of all.

Voices that speak differently, not only higher or lower

Listening through the window above, I found the voices too similar: male and female differed mostly in pitch. Both kept voices ran the same excitation, a pulse bank with the pitch pinned, and only the anchor moved (90 against 170 Hz). So round six changes what the voice is made of and how it talks:

• Carriers. Seven families over the same controls: a thin melodic reed, a wide hollow pulse, a breathy noisy one, a metallic inharmonic one, one that keeps nearly all of the original melody, a quantised rough one, and the iron one. The swarm and detune controls stay off, since the swarm designs were the rejected ones.
• Habits. Per voice: syllable rate, gap length, chance of an inhale, trimmed phrase edges, a pitch glide at the end of each phrase, soft attacks.
• Bases. Single speakers from the Japanese, Chinese, Hindi, Spanish, Portuguese, Italian and French groups of the model, or a blend weighted 80/20 towards one.

The owner’s 1.25× speed (set with the window’s slider; it keeps the pitch) is now baked in: rates times 1.25, every gap and inhale divided by it. A calm line comes out about 3 % shorter than the slider version.

I measured distinctness on the final audio, between voices, on three lines (calm, wary, urgent). Voices speak at different paces on purpose, so frames are not aligned. A pair’s distance is the mean cepstrum distance plus z-scored pitch median and spread, pause share, syllable rate, high-frequency share and spectral flatness. Across 31 candidates the closest two voices in the cast of twelve are 2.9 apart, against a median of 3.8 over all pairs. Two pairs differ almost only in pitch, and neither is in the cast.

A voice inside your head

The request for a telepathic voice rules out the obvious recipe, chorus and shimmer, which is the swarm sound again. The version I built goes the other way: close, dry and slow. A whisper of the same words rides under a soft-attacked voice at the same level. Before each phrase its own whisper plays reversed and rising, 0.32 s early, so the thought arrives before the sound, and a dark tail lets it linger without reading as a room. A stereo variant decorrelates only the breath between the ears and keeps the voice in the middle.

Voice design with a contract (v4)

The renderer doesn’t layer an effect on top of the voice. It generates harmonic and noise excitation and shapes them with the speech envelope, so consonants and vowels are both replaced, and at intensity ≥ 1 none of the original waveform or phase remains. Six characters (circuit, iron_oracle, prism_swarm, broken_terminal, deep_reactor, ghost_lattice) are presets over about fifteen continuous controls: pitch anchor and follow, formant ratio, pulse width, swarm, detune, inharmonicity, texture, motion, smear, fracture, sub and more. Cleanup is narrow and written down: whole-clip DC, isolated impulses of at most three samples, and whole-clip gain past a −2 dBFS oversampled peak. Every render’s metadata reports what cleanup did.

Kokoro runs as one ONNX session with 54 speakers. Switching speakers reuses the session, and blends interpolate speaker embeddings before synthesis instead of mixing finished recordings. Piper is an optional second backend that receives phoneme IDs directly. Model downloads are checksum-verified, and nothing is redistributed.

Status

v5 is done and frozen: the meaning layer, both languages, the game-format output and 86 tests. On 20 fresh lines written after the analyser was tuned, half still contained a word outside its lexicon, so the lexicon is the next thing to grow, not the grammar. v6 is a working prototype that I am tuning by ear: delivery per line, a casting sheet of twelve voices (two kept, seven new, three treatments of the psychic one) that I audition in the window, and the radio chain, a candidate to replace the game’s current band-pass filter. The measured distances show the voices differ in more than pitch; which ones I keep is decided by listening, and the extremes get pulled back after that.