clarity-helper/.agents/skills/webaudio-gemini-live/SKILL.md

name, description

name	description
webaudio-gemini-live	Guides the integration, troubleshooting, and optimization of real-time WebAudio capture, Gemini Multimodal Live API WebSocket streaming, PCM encoding, and frequency visualizers.

WebAudio & Gemini Multimodal Live API Skill

Use this skill when developing, debugging, or optimizing real-time duplex voice conversations with Google Gemini's native audio models (gemini-2.5-flash-native-audio-preview-09-2025 or future native audio models).

---

1. WebAudio Architecture & Data Flow

sequenceDiagram
    participant UserMic as Microphone (MediaStream)
    participant InAudioCtx as Input AudioContext (16kHz)
    participant LiveWS as Gemini Live WebSocket
    participant OutAudioCtx as Output AudioContext (24kHz)
    participant Visualizer as AnalyserNode & Visualizer

    UserMic->>InAudioCtx: Raw Audio Stream
    InAudioCtx->>InAudioCtx: Float32 -> Int16 PCM (16kHz)
    InAudioCtx->>LiveWS: sendRealtimeInput({ media: { data: Base64, mimeType: 'audio/pcm;rate=16000' } })
    LiveWS-->>OutAudioCtx: serverContent.modelTurn (PCM 24kHz Base64)
    OutAudioCtx->>Visualizer: Frequency Data Array (FFT)
    OutAudioCtx->>OutAudioCtx: Schedule AudioBufferSourceNode (Monotonic Queue)
    LiveWS-->>LiveWS: serverContent.interrupted -> Stop All Audio

---

2. Critical Implementation Guidelines

2.1 Audio Context Configuration

• Input Sample Rate: Must be initialized at 16000 Hz:

  const inputAudioCtx = new (window.AudioContext || (window as any).webkitAudioContext)({ sampleRate: 16000 });
  

• Output Sample Rate: Must be initialized at 24000 Hz to match Gemini's native output audio sample rate:

  const outputAudioCtx = new (window.AudioContext || (window as any).webkitAudioContext)({ sampleRate: 24000 });
  

2.2 PCM Conversion Math

Convert 32-bit floating point audio (-1.0 to +1.0) into 16-bit signed PCM integers (-32768 to +32767):

export function floatTo16BitPCM(float32Array: Float32Array): Int16Array {
  const int16Array = new Int16Array(float32Array.length);
  for (let i = 0; i < float32Array.length; i++) {
    const s = Math.max(-1, Math.min(1, float32Array[i]));
    int16Array[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;
  }
  return int16Array;
}

2.3 Monotonic Playback Scheduling

To prevent audio overlap or gaps:

const nextStartTimeRef = useRef<number>(0);
const audioCtx = outputAudioCtxRef.current!;

// Ensure scheduled start is never in the past
nextStartTimeRef.current = Math.max(nextStartTimeRef.current, audioCtx.currentTime);

const source = audioCtx.createBufferSource();
source.buffer = buffer;
source.connect(analyserNode);
source.start(nextStartTimeRef.current);
nextStartTimeRef.current += buffer.duration;

2.4 Handling Barge-in / Interruptions

When Gemini detects user speech while generating speech, it emits message.serverContent.interrupted:

if (message.serverContent?.interrupted) {
  // Stop all currently playing audio source nodes immediately
  audioSourcesRef.current.forEach(source => {
    try { source.stop(); } catch (e) {}
  });
  audioSourcesRef.current.clear();
  nextStartTimeRef.current = 0;
  setIsAiSpeaking(false);
}

---

3. Visualizer Engineering

• Connect AnalyserNode with fftSize = 256 between source and destination.
• Use analyser.getByteFrequencyData(dataArray) to sample normalized amplitude (0-255).
• Use requestAnimationFrame for continuous rendering with smooth falloff.