jpremill
1feea40684
- Re-architect VoiceSession into full-bleed studio cockpit with overlay slide-over companion drawer - Eliminate width squishing defect on LiveInsightPanel and add native dark/light theme support - Integrate authentic Google Gemini 3.8 Flash TTS with Fenrir persona and fallback audio pipeline - Implement comprehensive Playwright E2E visual regression suite across Desktop, iPad, and iPhone (21 tests, 45 snapshots) - Migrate core application to Svelte 5 runes (, , ) with reactive stores - Maintain .NET 8 Azure Functions serverless endpoints and clean build hygiene
96 lines
3.5 KiB
Markdown
96 lines
3.5 KiB
Markdown
---
|
|
name: webaudio-gemini-live
|
|
description: >-
|
|
Guides the integration, troubleshooting, and optimization of real-time WebAudio capture,
|
|
Gemini Multimodal Live API WebSocket streaming, PCM encoding, and frequency visualizers.
|
|
---
|
|
|
|
# WebAudio & Gemini Multimodal Live API Skill
|
|
|
|
Use this skill when developing, debugging, or optimizing real-time duplex voice conversations with Google Gemini's native audio models (`gemini-2.5-flash-native-audio-preview-09-2025` or future native audio models).
|
|
|
|
---
|
|
|
|
## 1. WebAudio Architecture & Data Flow
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant UserMic as Microphone (MediaStream)
|
|
participant InAudioCtx as Input AudioContext (16kHz)
|
|
participant LiveWS as Gemini Live WebSocket
|
|
participant OutAudioCtx as Output AudioContext (24kHz)
|
|
participant Visualizer as AnalyserNode & Visualizer
|
|
|
|
UserMic->>InAudioCtx: Raw Audio Stream
|
|
InAudioCtx->>InAudioCtx: Float32 -> Int16 PCM (16kHz)
|
|
InAudioCtx->>LiveWS: sendRealtimeInput({ media: { data: Base64, mimeType: 'audio/pcm;rate=16000' } })
|
|
LiveWS-->>OutAudioCtx: serverContent.modelTurn (PCM 24kHz Base64)
|
|
OutAudioCtx->>Visualizer: Frequency Data Array (FFT)
|
|
OutAudioCtx->>OutAudioCtx: Schedule AudioBufferSourceNode (Monotonic Queue)
|
|
LiveWS-->>LiveWS: serverContent.interrupted -> Stop All Audio
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Critical Implementation Guidelines
|
|
|
|
### 2.1 Audio Context Configuration
|
|
- **Input Sample Rate**: Must be initialized at `16000` Hz:
|
|
```typescript
|
|
const inputAudioCtx = new (window.AudioContext || (window as any).webkitAudioContext)({ sampleRate: 16000 });
|
|
```
|
|
- **Output Sample Rate**: Must be initialized at `24000` Hz to match Gemini's native output audio sample rate:
|
|
```typescript
|
|
const outputAudioCtx = new (window.AudioContext || (window as any).webkitAudioContext)({ sampleRate: 24000 });
|
|
```
|
|
|
|
### 2.2 PCM Conversion Math
|
|
Convert 32-bit floating point audio (`-1.0` to `+1.0`) into 16-bit signed PCM integers (`-32768` to `+32767`):
|
|
```typescript
|
|
export function floatTo16BitPCM(float32Array: Float32Array): Int16Array {
|
|
const int16Array = new Int16Array(float32Array.length);
|
|
for (let i = 0; i < float32Array.length; i++) {
|
|
const s = Math.max(-1, Math.min(1, float32Array[i]));
|
|
int16Array[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;
|
|
}
|
|
return int16Array;
|
|
}
|
|
```
|
|
|
|
### 2.3 Monotonic Playback Scheduling
|
|
To prevent audio overlap or gaps:
|
|
```typescript
|
|
const nextStartTimeRef = useRef<number>(0);
|
|
const audioCtx = outputAudioCtxRef.current!;
|
|
|
|
// Ensure scheduled start is never in the past
|
|
nextStartTimeRef.current = Math.max(nextStartTimeRef.current, audioCtx.currentTime);
|
|
|
|
const source = audioCtx.createBufferSource();
|
|
source.buffer = buffer;
|
|
source.connect(analyserNode);
|
|
source.start(nextStartTimeRef.current);
|
|
nextStartTimeRef.current += buffer.duration;
|
|
```
|
|
|
|
### 2.4 Handling Barge-in / Interruptions
|
|
When Gemini detects user speech while generating speech, it emits `message.serverContent.interrupted`:
|
|
```typescript
|
|
if (message.serverContent?.interrupted) {
|
|
// Stop all currently playing audio source nodes immediately
|
|
audioSourcesRef.current.forEach(source => {
|
|
try { source.stop(); } catch (e) {}
|
|
});
|
|
audioSourcesRef.current.clear();
|
|
nextStartTimeRef.current = 0;
|
|
setIsAiSpeaking(false);
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Visualizer Engineering
|
|
- Connect `AnalyserNode` with `fftSize = 256` between source and destination.
|
|
- Use `analyser.getByteFrequencyData(dataArray)` to sample normalized amplitude (0-255).
|
|
- Use `requestAnimationFrame` for continuous rendering with smooth falloff.
|