Behind the Voice: How AI Adapts Emotion to Speech

When your AI companion speaks, the voice is not a flat, monotone readout of text. The system analyzes the emotional content of each message and adjusts the voice accordingly — a joyful response sounds energetic and warm, a tender moment sounds soft and measured, a playful tease has a lighter, more dynamic quality.

This post explains exactly how that works: the text-to-speech pipeline, the emotion detection system, the voice parameter mapping, and the honest limitations of the current approach.

---

How Text-to-Speech Works (The Basics)

At a high level, modern text-to-speech (TTS) converts text into audio through a neural network:

1. Text analysis: The input text is broken into linguistic units (phonemes, prosody markers, sentence structure)

2. Acoustic modeling: A neural network predicts the audio characteristics — pitch, duration, energy, and spectral features for each segment

3. Waveform generation: A vocoder converts the acoustic predictions into an actual audio waveform you can hear

Modern neural TTS systems produce remarkably natural-sounding speech. They have moved far beyond the robotic voices of earlier generations. But there is still a gap between "sounds like a human reading text" and "sounds like a human feeling an emotion while speaking." Our system works to close that gap.

---

Our Three TTS Providers

We use three different TTS providers, each serving a different purpose:

Edge TTS (Free Tier)

• What it is: Microsoft's neural TTS service, accessed through the Edge runtime
• Voice count: 40+ voices across multiple languages and accents
• Quality: Good natural speech quality with basic prosody control
• Cost: Free — no per-character charges
• Best for: Standard conversations, casual interaction, users who prefer not to spend credits on voice

Edge TTS produces clean, natural-sounding speech. It does not support the fine-grained emotional parameter control that ElevenLabs offers, but it handles basic expression well through natural prosody patterns built into the neural model.

ElevenLabs (Premium Tier)

• What it is: A specialized AI voice synthesis platform with advanced emotional control
• Voice count: Multiple high-quality voices with distinct character profiles
• Quality: Best-in-class emotional expressiveness and naturalness
• Cost: Credits consumed per character generated
• Best for: Emotionally rich conversations, intimate moments, Story Mode narration

ElevenLabs is where our emotion-to-voice pipeline has the most impact. The API exposes direct control over voice parameters like stability, similarity boost, and style — which we map to detected emotions.

Coqui (Local/Self-Hosted)

• What it is: An open-source TTS engine that runs locally on our servers
• Voice count: Configurable with custom voice models
• Quality: Good quality, continuously improving with model updates
• Cost: No external API costs (compute-only)
• Best for: High-volume generation, reducing external dependencies, experimentation with custom voices

Coqui gives us the ability to generate speech without relying on external services. It is also our testing ground for experimental voice features.

---

The Emotion Detection System

Before the TTS engine generates audio, our system analyzes the text to detect the emotional tone. Here is how it works.

The 7 Emotions We Detect

Emotion Description Typical Triggers

--------- ------------- -----------------

Joy Happiness, excitement, enthusiasm Laughter, celebrations, good news, playful energy

Sadness Melancholy, sympathy, comfort Loss, disappointment, empathy, wistfulness

Anger Frustration, intensity, passion Conflict, injustice, strong disagreement

Fear Anxiety, concern, unease Danger, uncertainty, worry, vulnerability

Surprise Astonishment, revelation Plot twists, unexpected events, discoveries

Flirty Playfulness, teasing, attraction Compliments, innuendo, romantic tension

Tender Gentleness, care, intimacy Comfort, closeness, emotional vulnerability, love

How Detection Works

The current emotion detection system is keyword-based. It works by scanning the text for words and phrases associated with each emotion:

• Joy indicators: "haha", "love it", "amazing", "excited", "wonderful", exclamation marks
• Sadness indicators: "sorry", "miss you", "wish", "unfortunately", "hurts"
• Flirty indicators: "wink", "tease", "blush", "gorgeous", suggestive phrasing
• Tender indicators: "care about you", "safe with me", "gently", "softly", "hold you"

Each detected keyword adds to that emotion's confidence score. The emotion with the highest score becomes the primary detected emotion.

Confidence Scoring

Not every message has a clear emotional tone. The system assigns a confidence value (0.0 to 1.0) to the detected emotion:

• High confidence (0.7+): Multiple strong indicators for a single emotion. The voice parameters shift significantly.
• Medium confidence (0.4-0.7): Some indicators present but not dominant. Voice parameters shift moderately.
• Low confidence (below 0.4): Ambiguous or neutral text. The voice stays close to the default/neutral profile.

This confidence-based blending prevents the voice from making dramatic emotional swings on ambiguous text. If the system is not sure what emotion is present, it plays it safe.

---

How Emotions Map to Voice Parameters

For ElevenLabs (our most expressive provider), detected emotions are translated into three key voice parameters:

Stability (Vocal Dynamics)

Controls how consistent vs. dynamic the voice is:

• Lower stability = more vocal variation, more expressive, more energetic
• Higher stability = more measured, controlled, steady delivery

Similarity Boost (Naturalness)

Controls how closely the output matches the base voice profile:

• Lower similarity = more creative interpretation, potentially more emotive but less consistent
• Higher similarity = tighter adherence to the voice's natural sound

Style (Expressiveness)

Controls the overall expressiveness of the delivery:

• Higher style = more dramatic, more emotionally pronounced
• Lower style = more subtle, more conversational

Emotion-to-Parameter Mapping

Here is how each emotion adjusts these parameters:

Emotion Stability Similarity Boost Style Effect

--------- ----------- ----------------- ------- --------

Joy Lower (0.3) Medium (0.6) Higher (0.7) Dynamic, energetic, expressive

Sadness Higher (0.7) Higher (0.8) Medium (0.5) Slower, measured, gentle

Anger Lower (0.25) Medium (0.5) Higher (0.8) Intense, forceful, dramatic

Fear Medium (0.5) Higher (0.7) Medium (0.6) Tense, slightly unsteady, urgent

Surprise Lower (0.3) Medium (0.5) Higher (0.7) Quick, dynamic, wide-ranged

Flirty Medium (0.45) Medium (0.6) Medium (0.6) Playful variation, warm, teasing

Tender Higher (0.65) Higher (0.75) Medium (0.5) Soft, warm, gentle, intimate

Neutral Medium (0.5) Medium (0.65) Lower (0.4) Natural, conversational, balanced

These values are the baseline. They are further adjusted by the confidence score — low confidence pulls all parameters closer to neutral.

---

Voice Personalization Per Character

Different AI companions have different default voice profiles:

• Voice selection: Each character is assigned a specific voice ID from the TTS provider that matches their personality
• Baseline parameters: Characters have individual default stability, similarity, and style values that define their "neutral" voice
• Emotional range: Some characters are more emotionally expressive by default; others are more measured. The emotion mapping adjusts relative to each character's baseline.

A playful, energetic character will have a naturally more dynamic voice even in neutral mode, while a calm, mysterious character will have a more measured baseline. Emotional shifts are applied on top of these character-specific defaults.

---

Honest Limitations

We believe in being transparent about what this system does well and where it falls short.

What works well

• Obvious emotions are caught reliably. A message full of exclamation marks and words like "amazing" and "love" will correctly trigger a joyful voice. A message about loss and sadness will produce a softer, more measured delivery.
• The confidence system prevents false positives. Ambiguous text does not cause dramatic vocal swings.
• Different emotions sound noticeably different. Users can hear the difference between a joyful and a tender response.

What does not work as well

• Keyword-based detection is not ML-based. It does not understand context, sarcasm, or subtle emotional nuance. A message like "Oh great, another Monday" (sarcastic) would likely be flagged as positive due to the word "great."
• Mixed emotions are simplified. If a message contains both joy and sadness, the system picks the dominant one rather than blending them. Real human speech often carries mixed emotional tones.
• Edge TTS has limited parameter control. The fine-grained emotion mapping only fully works with ElevenLabs. Edge TTS relies on its built-in prosody model, which is less responsive to explicit emotional direction.
• Coqui's emotional range varies by model. Some local models handle emotion better than others.

What we are improving

• We are evaluating ML-based emotion detection models that understand context, not just keywords. This would significantly improve handling of sarcasm, irony, and subtle emotional states.
• We are working on emotion blending — producing voices that carry two emotions simultaneously.
• We are expanding our voice library with more diverse voice profiles.

---

Privacy and Audio Data

A few important points about how we handle voice data:

• Audio is generated on-demand. When you request voice output, the audio is generated in real time.
• Temporary caching only. Generated audio may be cached briefly for playback performance, but it is not permanently stored.
• No voice cloning of users. We do not record, store, or process your voice. The TTS system is one-directional: text goes in, audio comes out. Your microphone is never accessed.
• Provider data handling. When using ElevenLabs, the text of the message is sent to their API for synthesis. ElevenLabs' data handling is governed by their own privacy policy. For users who prefer to keep text processing local, Coqui runs entirely on our servers.

---

The Bottom Line

Our voice system is a pipeline: text enters, emotion is detected, voice parameters are adjusted, and expressive audio is generated. It is not perfect — keyword-based detection has real limitations, and neural TTS still has room to grow in emotional authenticity.

But the result is voice output that feels meaningfully more human than flat text-to-speech. Your AI companion does not just say words — it says them in a way that matches what those words mean. And we are continuously working to make that match closer and more natural.

We would rather be honest about the imperfections than pretend the system is something it is not. The technology is genuinely impressive, and it is getting better with every iteration.