feat(ui): redesign voice cockpit with clinical companion drawer, Svelte 5 migration, and Gemini 3.8 voice
- Re-architect VoiceSession into full-bleed studio cockpit with overlay slide-over companion drawer - Eliminate width squishing defect on LiveInsightPanel and add native dark/light theme support - Integrate authentic Google Gemini 3.8 Flash TTS with Fenrir persona and fallback audio pipeline - Implement comprehensive Playwright E2E visual regression suite across Desktop, iPad, and iPhone (21 tests, 45 snapshots) - Migrate core application to Svelte 5 runes (, , ) with reactive stores - Maintain .NET 8 Azure Functions serverless endpoints and clean build hygiene
This commit is contained in:
1 parent
c82b112624
commit
1feea40684
148 files changed
+11230
-1706
No files matched your search
@@ -0,0 +1,95 @@
|
||||
---
|
||||
name: webaudio-gemini-live
|
||||
description: >-
|
||||
Guides the integration, troubleshooting, and optimization of real-time WebAudio capture,
|
||||
Gemini Multimodal Live API WebSocket streaming, PCM encoding, and frequency visualizers.
|
||||
---
|
||||
|
||||
# WebAudio & Gemini Multimodal Live API Skill
|
||||
|
||||
Use this skill when developing, debugging, or optimizing real-time duplex voice conversations with Google Gemini's native audio models (`gemini-2.5-flash-native-audio-preview-09-2025` or future native audio models).
|
||||
|
||||
---
|
||||
|
||||
## 1. WebAudio Architecture & Data Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant UserMic as Microphone (MediaStream)
|
||||
participant InAudioCtx as Input AudioContext (16kHz)
|
||||
participant LiveWS as Gemini Live WebSocket
|
||||
participant OutAudioCtx as Output AudioContext (24kHz)
|
||||
participant Visualizer as AnalyserNode & Visualizer
|
||||
|
||||
UserMic->>InAudioCtx: Raw Audio Stream
|
||||
InAudioCtx->>InAudioCtx: Float32 -> Int16 PCM (16kHz)
|
||||
InAudioCtx->>LiveWS: sendRealtimeInput({ media: { data: Base64, mimeType: 'audio/pcm;rate=16000' } })
|
||||
LiveWS-->>OutAudioCtx: serverContent.modelTurn (PCM 24kHz Base64)
|
||||
OutAudioCtx->>Visualizer: Frequency Data Array (FFT)
|
||||
OutAudioCtx->>OutAudioCtx: Schedule AudioBufferSourceNode (Monotonic Queue)
|
||||
LiveWS-->>LiveWS: serverContent.interrupted -> Stop All Audio
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Critical Implementation Guidelines
|
||||
|
||||
### 2.1 Audio Context Configuration
|
||||
- **Input Sample Rate**: Must be initialized at `16000` Hz:
|
||||
```typescript
|
||||
const inputAudioCtx = new (window.AudioContext || (window as any).webkitAudioContext)({ sampleRate: 16000 });
|
||||
```
|
||||
- **Output Sample Rate**: Must be initialized at `24000` Hz to match Gemini's native output audio sample rate:
|
||||
```typescript
|
||||
const outputAudioCtx = new (window.AudioContext || (window as any).webkitAudioContext)({ sampleRate: 24000 });
|
||||
```
|
||||
|
||||
### 2.2 PCM Conversion Math
|
||||
Convert 32-bit floating point audio (`-1.0` to `+1.0`) into 16-bit signed PCM integers (`-32768` to `+32767`):
|
||||
```typescript
|
||||
export function floatTo16BitPCM(float32Array: Float32Array): Int16Array {
|
||||
const int16Array = new Int16Array(float32Array.length);
|
||||
for (let i = 0; i < float32Array.length; i++) {
|
||||
const s = Math.max(-1, Math.min(1, float32Array[i]));
|
||||
int16Array[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;
|
||||
}
|
||||
return int16Array;
|
||||
}
|
||||
```
|
||||
|
||||
### 2.3 Monotonic Playback Scheduling
|
||||
To prevent audio overlap or gaps:
|
||||
```typescript
|
||||
const nextStartTimeRef = useRef<number>(0);
|
||||
const audioCtx = outputAudioCtxRef.current!;
|
||||
|
||||
// Ensure scheduled start is never in the past
|
||||
nextStartTimeRef.current = Math.max(nextStartTimeRef.current, audioCtx.currentTime);
|
||||
|
||||
const source = audioCtx.createBufferSource();
|
||||
source.buffer = buffer;
|
||||
source.connect(analyserNode);
|
||||
source.start(nextStartTimeRef.current);
|
||||
nextStartTimeRef.current += buffer.duration;
|
||||
```
|
||||
|
||||
### 2.4 Handling Barge-in / Interruptions
|
||||
When Gemini detects user speech while generating speech, it emits `message.serverContent.interrupted`:
|
||||
```typescript
|
||||
if (message.serverContent?.interrupted) {
|
||||
// Stop all currently playing audio source nodes immediately
|
||||
audioSourcesRef.current.forEach(source => {
|
||||
try { source.stop(); } catch (e) {}
|
||||
});
|
||||
audioSourcesRef.current.clear();
|
||||
nextStartTimeRef.current = 0;
|
||||
setIsAiSpeaking(false);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Visualizer Engineering
|
||||
- Connect `AnalyserNode` with `fftSize = 256` between source and destination.
|
||||
- Use `analyser.getByteFrequencyData(dataArray)` to sample normalized amplitude (0-255).
|
||||
- Use `requestAnimationFrame` for continuous rendering with smooth falloff.
|
||||
Reference in new issue
Block a user