Building an AI Voice Chatbot in Flutter: From STT to LLM to TTS

Building a fluid, real-time AI voice chatbot in Flutter is an exciting challenge that blends audio processing, network calls, and state management into a single, seamless experience. The core loop—listen, think, speak—sounds simple, but each stage (Speech-to-Text, Large Language Model, Text-to-Speech) introduces its own complexities, especially around accuracy, latency, and error handling.

The most common pain point? Speech-to-Text (STT) accuracy in real-world conditions. Users expect the bot to understand them perfectly, but background noise, accents, and fast speech can trip up even the best services. While no solution is 100% perfect (humans aren’t either!), we can architect our app to maximize accuracy and provide a robust user experience.

Core Architecture: A Three-Stage Pipeline

Think of your app as a pipeline with three key services:

1. STT (Speech-to-Text): Converts user’s microphone input to text.
2. LLM (Large Language Model): Processes the text and generates a thoughtful response.
3. TTS (Text-to-Speech): Converts the LLM’s text response back into audible speech.

For this example, we’ll use a popular combination: OpenAI’s Whisper for STT, their Chat Completions API as the LLM, and a service like Eleven Labs for TTS. The principles apply to any service provider.

Step 1: Capturing Speech with Robust STT

Flutter’s microphone stream gives us raw audio data. Instead of processing it locally for complex STT, we’ll send chunks to a dedicated service for higher accuracy. A critical optimization here is implementing a basic Voice Activity Detection (VAD) to only send audio when the user is actually speaking, saving bandwidth and cost.

Here’s a simplified service class for handling audio capture and Whisper API calls:

import 'dart:io';
import 'package:http/http.dart' as http;
import 'package:path_provider/path_provider.dart';
import 'package:record/record.dart';

class STTService {
  final Record _audioRecord = Record();
  bool _isRecording = false;

  Future<String> transcribeAudio() async {
    // 1. Check and request permissions (not shown for brevity)
    if (await _audioRecord.hasPermission() == false) {
      throw Exception('Microphone permission not granted');
    }

    // 2. Start recording to a temporary file
    final tempDir = await getTemporaryDirectory();
    final audioPath = '${tempDir.path}/voice_input.wav';
    
    await _audioRecord.start(
      path: audioPath,
      encoder: AudioEncoder.wav, // Use WAV for better compatibility
    );
    _isRecording = true;

    // 3. In a real app, you would stop recording based on VAD/silence detection.
    // For simplicity, we record for a fixed duration.
    await Future.delayed(const Duration(seconds: 3));
    await _audioRecord.stop();
    _isRecording = false;

    // 4. Send the audio file to the Whisper API
    final file = File(audioPath);
    final bytes = await file.readAsBytes();

    final request = http.MultipartRequest(
      'POST',
      Uri.parse('https://api.openai.com/v1/audio/transcriptions'),
    );
    request.headers['Authorization'] = 'Bearer YOUR_OPENAI_API_KEY';
    request.files.add(
      http.MultipartFile.fromBytes('file', bytes, filename: 'audio.wav'),
    );
    request.fields['model'] = 'whisper-1';
    request.fields['response_format'] = 'text';

    final response = await request.send();
    final transcribedText = await response.stream.bytesToString();

    // 5. Clean up
    await file.delete();

    return transcribedText;
  }

  // Call this when the user stops speaking (e.g., via a button press)
  Future<void> stopRecording() async {
    if (_isRecording) {
      await _audioRecord.stop();
      _isRecording = false;
    }
  }
}

Key Takeaway: Always handle recording permissions gracefully and clean up temporary files. For production, replace the fixed delay with proper VAD logic, perhaps using a package like voice_activity_detector to stop recording after a period of silence.

Step 2: Processing with the LLM

Once you have the transcribed text, sending it to an LLM is relatively straightforward. The main challenge here is managing the asynchronous nature of the call and updating your UI with a streaming response if desired.

import 'package:http/http.dart' as http;

class LLMService {
  Future<String> getChatResponse(String userMessage) async {
    final url = Uri.parse('https://api.openai.com/v1/chat/completions');

    final response = await http.post(
      url,
      headers: {
        'Content-Type': 'application/json',
        'Authorization': 'Bearer YOUR_OPENAI_API_KEY',
      },
      body: '''
      {
        "model": "gpt-4o-mini",
        "messages": [
          {"role": "system", "content": "You are a helpful and friendly voice assistant."},
          {"role": "user", "content": "$userMessage"}
        ],
        "temperature": 0.7
      }
      ''',
    );

    if (response.statusCode == 200) {
      final Map<String, dynamic> data = json.decode(response.body);
      return data['choices'][0]['message']['content'];
    } else {
      throw Exception('Failed to load LLM response: ${response.body}');
    }
  }
}

Step 3: Speaking Back with TTS

For the final step, we convert the LLM’s text response into speech. Services like Eleven Labs provide high-quality, natural-sounding voices. You’ll fetch an audio file (e.g., MP3) and play it back in your app.

import 'package:audioplayers/audioplayers.dart';
import 'package:http/http.dart' as http;
import 'package:path_provider/path_provider.dart';

class TTSService {
  final AudioPlayer _audioPlayer = AudioPlayer();

  Future<void> speak(String text, {String voiceId = '21m00Tcm4TlvDq8ikWAM'}) async {
    final url = Uri.parse('https://api.elevenlabs.io/v1/text-to-speech/$voiceId');
    
    final response = await http.post(
      url,
      headers: {
        'Accept': 'audio/mpeg',
        'xi-api-key': 'YOUR_ELEVEN_LABS_API_KEY',
        'Content-Type': 'application/json',
      },
      body: '''
      {
        "text": "$text",
        "model_id": "eleven_monolingual_v1",
        "voice_settings": {
          "stability": 0.5,
          "similarity_boost": 0.75
        }
      }
      ''',
    );

    if (response.statusCode == 200) {
      // Save the audio bytes to a temporary file
      final tempDir = await getTemporaryDirectory();
      final audioFile = File('${tempDir.path}/tts_response.mp3');
      await audioFile.writeAsBytes(response.bodyBytes);

      // Play the audio file
      await _audioPlayer.play(DeviceFileSource(audioFile.path));
      
      // Optionally, delete the file after playing
      _audioPlayer.onPlayerComplete.listen((_) {
        audioFile.delete();
      });
    } else {
      throw Exception('TTS API call failed: ${response.statusCode}');
    }
  }

  void dispose() {
    _audioPlayer.dispose();
  }
}

Common Pitfalls and Optimization Strategies

1. Latency is the Enemy: The round-trip (STT -> LLM -> TTS) can feel sluggish. Optimize by:

   • Streaming STT: Use services that support real-time streaming transcription instead of sending full audio clips.
   • LLM Choice: Smaller, faster models might be sufficient for a voice assistant context.
   • Pre-cache TTS: If your app’s responses are predictable, consider pre-generating common TTS audio.

2. Error Handling is Non-Negotiable: Network calls will fail. Microphone access will be denied. Always implement try-catch blocks, provide user-friendly fallback messages (e.g., “I didn’t catch that, could you repeat it?”), and have a visual loading/error state in your UI.

3. State Management Complexity: Your UI state will cycle through: Idle -> Listening -> Processing -> Speaking. Use a state management solution (like Riverpod or Bloc) to cleanly manage these transitions and prevent race conditions—like trying to start a new recording while one is already in flight.

4. Audio Format & Quality: Ensure your recorded audio format (sample rate, bit depth) matches the STT service’s requirements. Poor quality input guarantees poor transcription.

By structuring your app as a well-orchestrated pipeline of these three services and paying close attention to error states and latency, you can build an AI voice chatbot in Flutter that feels responsive, reliable, and truly conversational. Start with the simple pipeline above, then iterate by adding VAD, response streaming, and more sophisticated error recovery.