---
name: webaudio-gemini-live
description: >-
  Guides the integration, troubleshooting, and optimization of real-time WebAudio capture,
  Gemini Multimodal Live API WebSocket streaming, PCM encoding, and frequency visualizers.
---

# WebAudio & Gemini Multimodal Live API Skill

Use this skill when developing, debugging, or optimizing real-time duplex voice conversations with Google Gemini's native audio models (`gemini-2.5-flash-native-audio-preview-09-2025` or future native audio models).

---

## 1. WebAudio Architecture & Data Flow

```mermaid
sequenceDiagram
    participant UserMic as Microphone (MediaStream)
    participant InAudioCtx as Input AudioContext (16kHz)
    participant LiveWS as Gemini Live WebSocket
    participant OutAudioCtx as Output AudioContext (24kHz)
    participant Visualizer as AnalyserNode & Visualizer

    UserMic->>InAudioCtx: Raw Audio Stream
    InAudioCtx->>InAudioCtx: Float32 -> Int16 PCM (16kHz)
    InAudioCtx->>LiveWS: sendRealtimeInput({ media: { data: Base64, mimeType: 'audio/pcm;rate=16000' } })
    LiveWS-->>OutAudioCtx: serverContent.modelTurn (PCM 24kHz Base64)
    OutAudioCtx->>Visualizer: Frequency Data Array (FFT)
    OutAudioCtx->>OutAudioCtx: Schedule AudioBufferSourceNode (Monotonic Queue)
    LiveWS-->>LiveWS: serverContent.interrupted -> Stop All Audio
```

---

## 2. Critical Implementation Guidelines

### 2.1 Audio Context Configuration
- **Input Sample Rate**: Must be initialized at `16000` Hz:
  ```typescript
  const inputAudioCtx = new (window.AudioContext || (window as any).webkitAudioContext)({ sampleRate: 16000 });
  ```
- **Output Sample Rate**: Must be initialized at `24000` Hz to match Gemini's native output audio sample rate:
  ```typescript
  const outputAudioCtx = new (window.AudioContext || (window as any).webkitAudioContext)({ sampleRate: 24000 });
  ```

### 2.2 PCM Conversion Math
Convert 32-bit floating point audio (`-1.0` to `+1.0`) into 16-bit signed PCM integers (`-32768` to `+32767`):
```typescript
export function floatTo16BitPCM(float32Array: Float32Array): Int16Array {
  const int16Array = new Int16Array(float32Array.length);
  for (let i = 0; i < float32Array.length; i++) {
    const s = Math.max(-1, Math.min(1, float32Array[i]));
    int16Array[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;
  }
  return int16Array;
}
```

### 2.3 Monotonic Playback Scheduling
To prevent audio overlap or gaps:
```typescript
const nextStartTimeRef = useRef<number>(0);
const audioCtx = outputAudioCtxRef.current!;

// Ensure scheduled start is never in the past
nextStartTimeRef.current = Math.max(nextStartTimeRef.current, audioCtx.currentTime);

const source = audioCtx.createBufferSource();
source.buffer = buffer;
source.connect(analyserNode);
source.start(nextStartTimeRef.current);
nextStartTimeRef.current += buffer.duration;
```

### 2.4 Handling Barge-in / Interruptions
When Gemini detects user speech while generating speech, it emits `message.serverContent.interrupted`:
```typescript
if (message.serverContent?.interrupted) {
  // Stop all currently playing audio source nodes immediately
  audioSourcesRef.current.forEach(source => {
    try { source.stop(); } catch (e) {}
  });
  audioSourcesRef.current.clear();
  nextStartTimeRef.current = 0;
  setIsAiSpeaking(false);
}
```

---

## 3. Visualizer Engineering
- Connect `AnalyserNode` with `fftSize = 256` between source and destination.
- Use `analyser.getByteFrequencyData(dataArray)` to sample normalized amplitude (0-255).
- Use `requestAnimationFrame` for continuous rendering with smooth falloff.