# Variable Crossfade σ — Design Plan

**Status**: design only — implementation deferred to next session.
**Date**: 2026-05-22
**Reviewers needed**: CEO sign-off on UX target, ML lead on training cost.

---

## Problem

Current `smooth_cond_sigma_emotion` is a global constant (12 frames @ 30 fps ≈ 400 ms) applied uniformly to every emotion transition in every scenario. CEO feedback identified three downstream symptoms (brows too even, no pupil motion, no overall jitter) — the brow/jitter pieces are tackled in the naturalness viewer. The remaining model-side issue is that **emotion transitions feel uniform** when they should not be:

- **joy → sadness** is a high-arousal, valence-flip transition. Real human prosody decays into sadness over ~600–800 ms (Ekman affect-decay literature). A 400 ms fade looks abrupt and reads as "fake."
- **neutral → joy** is a low-amplitude ramp-in. People snap into joy quickly (~200 ms onset). A 400 ms fade feels sluggish.
- **short utterance** (<1.5 s): the 400 ms fade consumes a third of the clip, eliminating steady-state expression.
- **long utterance** (>4 s): the same 400 ms fade is too small relative to the emotional arc; the transition looks like a "click."

The current cond-smoothing pipeline can't address this because it operates with one σ for the whole sequence.

---

## Pipeline today (from code-map agent)

Two-stage smoothing already exists:

1. **Preprocessing** (`scripts/compiler/data_pipeline.py:565-587`): per-scenario cross-turn Gaussian σ=30 on both VAD and emotion one-hot, before the .npz is compiled. ALSO `crossfade_turn_boundaries()` at lines 341-369 cosine-fades the *target* blendshapes over `fade_frames=96`.
2. **Training/Inference** (`dataset.py:262-265`, `infer.py:322-326`): per-sample Gaussian σ=`smooth_cond_sigma_emotion` (currently 12) on `cond[:, :16]` only. VAD already continuous.

**Critical**: any variable-σ scheme is a *third* smoothing layer composing with two existing ones. Must either:

- replace the training-time σ entirely (cleanest), or
- replace stage 1 with variable σ and disable stage 2 (also clean), or
- accept double-smoothing and pick σ values that account for it (risky — the convolution of two Gaussians is a Gaussian with σ = √(σ₁² + σ₂²), but only if applied non-overlapping; we're stacking which makes the math empirical).

**Recommendation**: replace the training-time σ block with a *per-segment* variable-σ block. Preprocessing stage stays. Compose carefully.

---

## Formula (from research agent)

Multiplicative, bounded:

```
σ(s, t, L) = clip(σ_base · f_pair(s, t) · f_length(L) · f_vad(Δvad),  σ_min, σ_max)
```

where
- `σ_base` ≈ 12 (current value, acts as anchor)
- `σ_min` ≈ 4 (~133 ms — below this the model can't learn the transition; quantization noise dominates)
- `σ_max` ≈ 40 (~1.3 s — above this the next emotion leaks into the current one)
- `s, t` = source, target emotion indices ∈ [0..15]
- `L` = turn duration in seconds
- `Δvad` = VAD vector difference

Component definitions:

### `f_pair(s, t)` — 16×16 lookup table

Hand-seeded from psychophysics. Bands:

| Transition type | Factor |
|---|---|
| Antipodal (joy↔sadness, anger↔calm) | 1.4–1.8 |
| Cross-valence non-antipodal | 1.1–1.3 |
| Same-valence, same-arousal | 0.7–0.9 |
| Neutral → strong emotion (ramp-in) | 0.5–0.7 |
| Strong emotion → neutral (ramp-out) | 1.2–1.5 |
| Self-loop (e.g. joy→joy) | 1.0 |

Asymmetric on neutral; symmetric elsewhere as a starting point.

### `f_length(L)` — turn-duration scaling

```
f_length(L) = clip(α · L, β_min, β_max)
```

with α≈0.15, β_min=0.4, β_max=1.6. This caps the fade at ~15% of turn length but doesn't go below 40% of `σ_base` even on very short turns (otherwise transitions vanish entirely on snappy interjections).

### `f_vad(Δvad)` — VAD-distance scaling

```
Δ = sqrt(0.3·(v_t-v_s)² + 0.5·(a_t-a_s)² + 0.2·(d_t-d_s)²)
f_vad(Δ) = 1.0 + 0.4 · clip(Δ/2.0, 0, 1)
```

Arousal-weighted (0.5) > valence (0.3) > dominance (0.2) — matches Russell's circumplex findings on perceived transition speed. Adds up to 40% on top of `σ_base · f_pair` for high-Δvad pairs (the antipodal cases already covered by `f_pair` get further boosted; this is intentional).

### Asymmetric Gaussian kernel at boundaries

Critical for causal TCN: don't let the kernel pull the *next* turn's emotion in **before** that turn's audio is audible. At each turn boundary:

- left half of the kernel uses σ_left (the prev-turn fade-out)
- right half uses σ_right (the next-turn fade-in)
- `σ_left = σ(prev, curr) · 0.7` (ramp-out a bit shorter than full)
- `σ_right = σ(curr, next) · 0.8`

Actually simpler version for v1: use full symmetric σ but **truncate** the kernel at the boundary by 1.5σ. Catches the worst leakage without per-half asymmetry math.

---

## Implementation plan (when we start)

### Phase 1 — data infrastructure (1-2 hr)

- Extract per-turn frame ranges from existing pipeline. `data_pipeline.py:497-543` already computes `c["T"]` per turn cumulatively — surface this into the .npz as a new field `turn_frames: [int, ...]` so dataset.py doesn't need scenario JSON access.
- Add per-turn `emotion_index` and `vad` fields to .npz (already known per turn, just save them).
- Regenerate .npz cache for all scenarios. Estimate ~10 min for all training data.

### Phase 2 — σ computation in dataset.py (2 hr)

- Implement `compute_variable_sigma(s, t, L)` matching the formula above.
- Replace global `gaussian_filter1d(cond[:, :16], sigma=...)` with a loop:
  - For each transition (i.e. each pair of adjacent turns), compute σ.
  - Apply a per-region truncated Gaussian on cond[:, :16] over a window centered at the turn boundary, of half-width 3σ.
  - Leave the steady-state interiors (>3σ from any boundary) untouched.
- Add `smooth_cond_variable: bool` config knob to switch between old global and new variable modes for A/B.

### Phase 3 — mirror in infer.py (1 hr)

- Same logic. Reads turn_frames + emotion_indices from .npz (needed for inference too). For inference on scenarios *without* turn metadata (rare), fall back to global σ=σ_base.

### Phase 4 — train v19 variant (4 min runtime, plus eval setup)

- Resume from v18m's checkpoint with `--smooth-cond-variable` flag.
- 70 epochs, same other hyperparams as v18n/v18o.

### Phase 5 — evaluation

Metrics from research agent:
1. **Blendshape velocity P95 within fade windows** — should fall in the range observed in held-out teacher data.
2. **Emotion classifier confidence trajectory** — monotonic decay of source, monotonic onset of target. No oscillation. Use a small pretrained classifier on rendered output.
3. **Lip-sync preservation (LSE-D/LSE-C)** — should not degrade vs v18m.
4. **Subjective A/B in viewer**: load v18m and v19 in compare viewer, observe transitions in long_* scenarios — explicit comparison on joy↔sadness, fluster→neutral, surprise→sadness pairs.

---

## Risks / open questions

1. **Two-stage smoothing interaction**: preprocessing already does σ=30 cross-turn smoothing. If we apply variable σ on top at training, effective smoothing is dominated by σ=30. **Mitigation**: reduce preprocessing σ to 8–10 in a new preprocessing variant, OR disable preprocessing emotion smoothing entirely (leave only VAD), let training-time variable σ own the entire emotion-smoothing budget.
2. **Target-side cosine fade at boundaries** (`crossfade_turn_boundaries()`, 96 frames = 3.2s) is its own thing — operates on targets not cond. It interacts with cond smoothing through the loss surface. May need to also make that fade variable, or shorten the 96-frame default.
3. **Hand-tuned LUT is 240 numbers** (16×16, symmetric ≈ 136). Need a way to express this concisely and validate. **Suggested format**: small JSON table `pair_factors.json` checked in, with a script that prints the matrix in human-readable form for review.
4. **Per-turn VAD vs smoothed VAD**: turns[].vad is per-turn (single point). The smoothed VAD from preprocessing is per-frame. Use per-turn raw VAD for the formula (the formula needs source/target *points*, not trajectories).
5. **What about no-turn-boundary segments?** Solo + daily scenarios with one turn → no transitions → no variable σ applies. They train as-is, but model still needs to handle them — the loss surface should remain stable for non-multi-turn data.

---

## Decision gates before implementation

1. **CEO approval** that the naturalness viewer's σ32+ feel is the *direction* (we're solving the right problem).
2. **Confirm** which preprocessing-stage smoothing we touch (recommendation: reduce or remove the σ=30 emotion smoothing in stage 1, keep the VAD smoothing).
3. **Confirm σ_base, σ_min, σ_max** values empirically — viewer-side variable σ approximation first (post-process; not a true retrain but lets us sanity-check the formula visually).

Once those three are settled, the implementation is ~5-6 hours of focused work plus eval time.

---

## References

- Research agent analysis (2026-05-22): multiplicative decomposition + LUT seeding from psychophysics
- Code map (2026-05-22): pipeline touchpoints in `data_pipeline.py`, `dataset.py`, `infer.py`
- Yang et al. 2023, "Emotional Speech-Driven Animation with Content-Emotion Disentanglement" — arousal-conditioned temporal kernels (related prior work)
- NVIDIA Audio2Face emotion-blend curves (industry-standard hand-tuned per-pair) — competitive benchmark
- Russell's circumplex / Ekman affect decay literature — psychophysical priors