feat(ui): redesign voice cockpit with clinical companion drawer, Svelte 5 migration, and Gemini 3.8 voice
- Re-architect VoiceSession into full-bleed studio cockpit with overlay slide-over companion drawer - Eliminate width squishing defect on LiveInsightPanel and add native dark/light theme support - Integrate authentic Google Gemini 3.8 Flash TTS with Fenrir persona and fallback audio pipeline - Implement comprehensive Playwright E2E visual regression suite across Desktop, iPad, and iPhone (21 tests, 45 snapshots) - Migrate core application to Svelte 5 runes (, , ) with reactive stores - Maintain .NET 8 Azure Functions serverless endpoints and clean build hygiene
This commit is contained in:
jpremill committed
1 parent
c82b112624
commit
1feea40684
148 files changed
+11230
-1706
No files matched your search
@@ -0,0 +1,84 @@
|
||||
---
|
||||
name: azure-functions-genai
|
||||
description: >-
|
||||
Guides the design, implementation, and debugging of Azure Functions v4 (.NET 8/10 Isolated Worker)
|
||||
using the official Google.GenAI C# SDK for serverless AI endpoints.
|
||||
---
|
||||
|
||||
# Azure Functions & Google.GenAI C# SDK Skill
|
||||
|
||||
Use this skill when developing, updating, or testing the C# serverless API endpoints located in `api/`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Solution Architecture
|
||||
|
||||
The backend consists of an **Azure Functions v4 Isolated Worker** targeting **.NET 8.0**:
|
||||
- Project file: `api/api.csproj`
|
||||
- Runtime: `Microsoft.Azure.Functions.Worker` 2.50+ & `Google.GenAI` 1.20+
|
||||
- Host configuration: `api/host.json`
|
||||
- Application Insights integration: `Microsoft.ApplicationInsights.WorkerService`
|
||||
|
||||
---
|
||||
|
||||
## 2. Google.GenAI C# SDK Integration Patterns
|
||||
|
||||
### 2.1 Namespaces & Types
|
||||
```csharp
|
||||
using Google.GenAI;
|
||||
using Google.GenAI.Types;
|
||||
using Type = Google.GenAI.Types.Type; // Disambiguate with System.Type
|
||||
```
|
||||
|
||||
### 2.2 Client Initialization
|
||||
```csharp
|
||||
var apiKey = System.Environment.GetEnvironmentVariable("GEMINI_API_KEY");
|
||||
var client = new Client(apiKey: apiKey);
|
||||
```
|
||||
|
||||
### 2.3 Structured Output Generation (JSON Schema)
|
||||
```csharp
|
||||
var schema = new Schema
|
||||
{
|
||||
Type = Type.Object,
|
||||
Properties = new Dictionary<string, Schema>
|
||||
{
|
||||
{ "distortionsIdentified", new Schema { Type = Type.Array, Items = new Schema { Type = Type.String } } },
|
||||
{ "techniquesUsed", new Schema { Type = Type.Array, Items = new Schema { Type = Type.String } } },
|
||||
{ "clinicalInsight", new Schema { Type = Type.String } },
|
||||
{ "feedbackType", new Schema { Type = Type.String, Enum = new List<string> { "Constructive", "Positive", "Neutral" } } }
|
||||
},
|
||||
Required = new List<string> { "distortionsIdentified", "techniquesUsed", "clinicalInsight", "feedbackType" }
|
||||
};
|
||||
|
||||
var response = await client.Models.GenerateContentAsync(
|
||||
model: "gemini-2.5-flash",
|
||||
contents: promptContent,
|
||||
config: new GenerateContentConfig
|
||||
{
|
||||
SystemInstruction = new Content { Parts = new List<Part> { new Part { Text = systemInstruction } } },
|
||||
ResponseMimeType = "application/json",
|
||||
ResponseSchema = schema
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Endpoints Matrix
|
||||
|
||||
| Route | HTTP Method | Function Name | Purpose |
|
||||
|:---|:---|:---|:---|
|
||||
| `/api/chat` | `POST` | `ChatProgression` | Headless roleplay progression with strict anti-meta guardrails |
|
||||
| `/api/analysis` | `POST` | `ClinicalAnalysis` | Real-time clinical supervisor turn evaluation |
|
||||
| `/api/scenario/generate` | `POST` | `GenerateScenario` | Dynamic CBT roleplay scenario generation |
|
||||
|
||||
---
|
||||
|
||||
## 4. Local Build & Test Verification
|
||||
```bash
|
||||
# Build the C# API project
|
||||
dotnet build api/api.csproj
|
||||
|
||||
# Run local functions host (requires Azure Functions Core Tools)
|
||||
func start --csharp --prefix api
|
||||
```
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
name: cbt-clinical-evaluation
|
||||
description: >-
|
||||
Provides clinical supervision protocols, real-time distortion detection rules,
|
||||
psychometric scoring rubrics, and post-session diagnostic report formatting.
|
||||
---
|
||||
|
||||
# CBT Clinical Evaluation & Supervision Skill
|
||||
|
||||
Use this skill when refining the clinical analysis supervisor model prompts, tuning distortion detection thresholds, or formatting final session case reports.
|
||||
|
||||
---
|
||||
|
||||
## 1. Real-Time Turn-by-Turn Supervision Protocol
|
||||
|
||||
During a roleplay session, every completed user turn is evaluated out-of-band by an asynchronous clinical supervisor prompt.
|
||||
|
||||
### 1.1 Clinical Schema
|
||||
```typescript
|
||||
export interface AnalysisResult {
|
||||
analysis: {
|
||||
distortions: string[]; // Detected distortions (e.g. "Mind Reading", "Catastrophizing")
|
||||
isAdaptive: boolean; // True if the response demonstrates cognitive restructuring
|
||||
evidence: string; // Direct quote or behavioral evidence from user's response
|
||||
};
|
||||
rating: {
|
||||
score: number; // 1-10 clinical effectiveness score
|
||||
reason: string; // Clinical justification for the score
|
||||
};
|
||||
suggestions: {
|
||||
alternativeThought: string; // Rational reframe of the user's statement
|
||||
behavioralStep: string; // Actionable communication or behavioral adjustment
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Clinical Evaluation Rubric (1-10 Scoring Scale)
|
||||
|
||||
| Score Range | Clinical Category | Behavioral & Cognitive Characteristics |
|
||||
|:---|:---|:---|
|
||||
| **1 - 3** | **Maladaptive / Reactive** | User falls into cognitive distortions without awareness; demonstrates passive-aggressive, defensive, or submissive behavior; abdicates personal boundaries. |
|
||||
| **4 - 6** | **Emerging Awareness / Partial Restructuring** | User attempts to challenge distortions or state boundaries, but tone remains emotionally reactive, apologetic, or hesitant. |
|
||||
| **7 - 8** | **Adaptive / Effective Restructuring** | Clear cognitive restructuring; assertive, non-blaming boundary setting; validates other person's emotions while holding firm limits. |
|
||||
| **9 - 10** | **Mastery / Radical Flexibility** | Demonstrates emotional regulation under high friction; employs sophisticated techniques (e.g., DEAR MAN, Gray Rock, Radical Acceptance); de-escalates without concession. |
|
||||
|
||||
---
|
||||
|
||||
## 3. Post-Session Comprehensive Case Report
|
||||
|
||||
At the end of a session, a final summary prompt generates a macro assessment:
|
||||
1. **Primary Cognitive Distortions Identified**: Frequency and contexts of observed distortions.
|
||||
2. **Techniques Successfully Applied**: Count and effectiveness of behavioral tools used.
|
||||
3. **Core Cognitive Shifts**: Evolution of the user's responses across the dialogue arc.
|
||||
4. **Targeted Clinical Homework**: 1-2 practical real-world exercises to reinforce learning.
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
name: cbt-scenario-design
|
||||
description: >-
|
||||
Designs, calibrates, and validates Cognitive Behavioral Therapy (CBT) roleplay scenarios,
|
||||
difficulty ladders, character psychological profiles, and internal CBT logic rules.
|
||||
---
|
||||
|
||||
# CBT Scenario Design & Calibration Skill
|
||||
|
||||
Use this skill when creating new simulation scenarios, tuning AI character difficulty levels, adding cognitive distortion targets, or modifying the scenario catalog.
|
||||
|
||||
---
|
||||
|
||||
## 1. Clinical Scenario Architecture
|
||||
|
||||
Every scenario in Cognitive Clarity conforms to the following schema:
|
||||
|
||||
```typescript
|
||||
export interface Scenario {
|
||||
id: string; // Unique kebab-case slug (e.g. 'workplace-critical-review')
|
||||
title: string; // Human-readable evocative title
|
||||
description: string; // Context setting the premise for the user
|
||||
category: 'Social Anxiety' | 'Workplace' | 'Relationships' | 'Self-Talk';
|
||||
difficulty: 'Beginner' | 'Intermediate' | 'Advanced';
|
||||
characterName: string; // Name or archetype of the AI partner
|
||||
initialMessage: string; // The opening dialogue from the AI character
|
||||
cbtFocus: string[]; // Target distortions and techniques
|
||||
systemInstruction: string; // Comprehensive prompt with character persona & logic
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Difficulty Calibration Rubric
|
||||
|
||||
| Difficulty | Character Resistance | Cognitive Distortion Salience | User Task | Recommended Category |
|
||||
|:---|:---|:---|:---|:---|
|
||||
| **Beginner** | Highly receptive to assertive communication; explicitly hints at expectations. | Overt, obvious distortions (e.g., explicit All-or-Nothing statements). | Identify and dispute basic cognitive distortions; maintain calm presence. | `Social Anxiety`, `Self-Talk` |
|
||||
| **Intermediate** | Ambiguous motivations; subtle passive-aggression; holds reasonable opposing perspective. | Subtle, nuanced distortions (e.g., Mind Reading, Fortune Telling, Maladaptive Guilt). | Set boundaries, de-escalate without defensiveness, practice Cognitive Reframing. | `Workplace`, `Relationships` |
|
||||
| **Advanced** | Strong emotional activation; high clinical resistance; persistent invalidation or defensiveness. | Layered cognitive distortions combined with emotional triggers. | Apply Radical Acceptance, Gray Rock Method, multi-turn emotional regulation under stress. | `Relationships`, `Self-Talk` |
|
||||
|
||||
---
|
||||
|
||||
## 3. System Instruction Design Blueprint
|
||||
|
||||
When writing `systemInstruction` for a scenario, always follow this four-part structure:
|
||||
|
||||
```markdown
|
||||
1. CHARACTER IDENTITY & CONTEXT:
|
||||
- Name, relationship to user, current emotional state, and immediate goal.
|
||||
- Tone markers (e.g., "curt, hurried, stressed" or "skeptical, passive-aggressive").
|
||||
|
||||
2. INTERNAL CBT LOGIC:
|
||||
- IF the user becomes defensive or aggressive -> Escalate emotional distance or defensiveness.
|
||||
- IF the user validates emotions while establishing clear boundaries -> De-escalate and show grudging respect.
|
||||
- IF the user asks clarifying questions without mind reading -> Clarify directly and remove ambiguity.
|
||||
|
||||
3. VOCAL & DIALOGUE STYLE:
|
||||
- Sentence length: Short and conversational (1-3 sentences per turn).
|
||||
- Natural conversational elements: Hesitations, pauses, tone inflections.
|
||||
|
||||
4. ANTI-META GUARDRAILS:
|
||||
- NEVER break character.
|
||||
- NEVER provide feedback, scores, advice, or therapy tips.
|
||||
- NEVER say "Good job" or "That was a great reframing".
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Scenario Validation Checklist
|
||||
|
||||
- [ ] `id` is unique and formatted as lowercase kebab-case.
|
||||
- [ ] `category` is one of the four clinical categories.
|
||||
- [ ] `difficulty` is properly calibrated (`Beginner`, `Intermediate`, `Advanced`).
|
||||
- [ ] `cbtFocus` lists at least 2 valid entries from `CBT_GLOSSARY`.
|
||||
- [ ] `initialMessage` immediately immerses the user without preamble.
|
||||
- [ ] `systemInstruction` contains explicit `INTERNAL CBT LOGIC` rules.
|
||||
- [ ] Zero clinical meta-feedback or congratulatory phrases in persona instructions.
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
name: webaudio-gemini-live
|
||||
description: >-
|
||||
Guides the integration, troubleshooting, and optimization of real-time WebAudio capture,
|
||||
Gemini Multimodal Live API WebSocket streaming, PCM encoding, and frequency visualizers.
|
||||
---
|
||||
|
||||
# WebAudio & Gemini Multimodal Live API Skill
|
||||
|
||||
Use this skill when developing, debugging, or optimizing real-time duplex voice conversations with Google Gemini's native audio models (`gemini-2.5-flash-native-audio-preview-09-2025` or future native audio models).
|
||||
|
||||
---
|
||||
|
||||
## 1. WebAudio Architecture & Data Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant UserMic as Microphone (MediaStream)
|
||||
participant InAudioCtx as Input AudioContext (16kHz)
|
||||
participant LiveWS as Gemini Live WebSocket
|
||||
participant OutAudioCtx as Output AudioContext (24kHz)
|
||||
participant Visualizer as AnalyserNode & Visualizer
|
||||
|
||||
UserMic->>InAudioCtx: Raw Audio Stream
|
||||
InAudioCtx->>InAudioCtx: Float32 -> Int16 PCM (16kHz)
|
||||
InAudioCtx->>LiveWS: sendRealtimeInput({ media: { data: Base64, mimeType: 'audio/pcm;rate=16000' } })
|
||||
LiveWS-->>OutAudioCtx: serverContent.modelTurn (PCM 24kHz Base64)
|
||||
OutAudioCtx->>Visualizer: Frequency Data Array (FFT)
|
||||
OutAudioCtx->>OutAudioCtx: Schedule AudioBufferSourceNode (Monotonic Queue)
|
||||
LiveWS-->>LiveWS: serverContent.interrupted -> Stop All Audio
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Critical Implementation Guidelines
|
||||
|
||||
### 2.1 Audio Context Configuration
|
||||
- **Input Sample Rate**: Must be initialized at `16000` Hz:
|
||||
```typescript
|
||||
const inputAudioCtx = new (window.AudioContext || (window as any).webkitAudioContext)({ sampleRate: 16000 });
|
||||
```
|
||||
- **Output Sample Rate**: Must be initialized at `24000` Hz to match Gemini's native output audio sample rate:
|
||||
```typescript
|
||||
const outputAudioCtx = new (window.AudioContext || (window as any).webkitAudioContext)({ sampleRate: 24000 });
|
||||
```
|
||||
|
||||
### 2.2 PCM Conversion Math
|
||||
Convert 32-bit floating point audio (`-1.0` to `+1.0`) into 16-bit signed PCM integers (`-32768` to `+32767`):
|
||||
```typescript
|
||||
export function floatTo16BitPCM(float32Array: Float32Array): Int16Array {
|
||||
const int16Array = new Int16Array(float32Array.length);
|
||||
for (let i = 0; i < float32Array.length; i++) {
|
||||
const s = Math.max(-1, Math.min(1, float32Array[i]));
|
||||
int16Array[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;
|
||||
}
|
||||
return int16Array;
|
||||
}
|
||||
```
|
||||
|
||||
### 2.3 Monotonic Playback Scheduling
|
||||
To prevent audio overlap or gaps:
|
||||
```typescript
|
||||
const nextStartTimeRef = useRef<number>(0);
|
||||
const audioCtx = outputAudioCtxRef.current!;
|
||||
|
||||
// Ensure scheduled start is never in the past
|
||||
nextStartTimeRef.current = Math.max(nextStartTimeRef.current, audioCtx.currentTime);
|
||||
|
||||
const source = audioCtx.createBufferSource();
|
||||
source.buffer = buffer;
|
||||
source.connect(analyserNode);
|
||||
source.start(nextStartTimeRef.current);
|
||||
nextStartTimeRef.current += buffer.duration;
|
||||
```
|
||||
|
||||
### 2.4 Handling Barge-in / Interruptions
|
||||
When Gemini detects user speech while generating speech, it emits `message.serverContent.interrupted`:
|
||||
```typescript
|
||||
if (message.serverContent?.interrupted) {
|
||||
// Stop all currently playing audio source nodes immediately
|
||||
audioSourcesRef.current.forEach(source => {
|
||||
try { source.stop(); } catch (e) {}
|
||||
});
|
||||
audioSourcesRef.current.clear();
|
||||
nextStartTimeRef.current = 0;
|
||||
setIsAiSpeaking(false);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Visualizer Engineering
|
||||
- Connect `AnalyserNode` with `fftSize = 256` between source and destination.
|
||||
- Use `analyser.getByteFrequencyData(dataArray)` to sample normalized amplitude (0-255).
|
||||
- Use `requestAnimationFrame` for continuous rendering with smooth falloff.
|
||||
Reference in new issue
Block a user