TeezU WebRTC Implementation

🎯 Overview

WebRTC (Web Real-Time Communication) is de kern van alle real-time audio/video features op TeezU v7.0. Primaire ingest via WHIP, delivery via WHEP — met HLS als automatische fallback voor legacy clients. Signaling verloopt via Socket.IO (niet raw WebSocket).

📹 Livestream (WHIP/WHEP)

Sub-second latency via WebRTC Ingestion/Egress Protocol. OBS → WHIP → MediaSoup SFU → WHEP kijkers.

🎤 Voice & Collab

Opus audio met echo cancellation. Tot 4 simultane deelnemers in collab stream mode via SFU.

🖥️ Screen & Recording

getDisplayMedia() voor screen sharing. MediaRecorder → 10s segmenten → AWS S3 via Fastify presigned URLs.

📡 WHIP/WHEP Protocol

Standaard WebRTC ingest/egress protocol. Browser-native, geen plugin. <300ms end-to-end latency.

🔄 HLS Fallback

Automatische fallback voor legacy browsers & TV's. 5–10s latency, adaptive bitrate 360p/720p/1080p.

🎭 Freeze & Reveal

Server-side state machine via Fastify. LIVE → FROZEN (blauwe glow) → REVEAL (confetti animatie).

🏗️ Architecture

┌─────────────────────────────────────────────────┐
│              CLIENT A                           │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐     │
│  │ Browser  │→ │MediaStream│→ │RTCPeerCon│     │
│  │   API    │  │   API     │  │ nection  │     │
│  └──────────┘  └──────────┘  └────┬─────┘     │
└────────────────────────────────────┼───────────┘
                                     ↓
                    ┌────────────────────────────┐
                    │   SIGNALING SERVER         │
                    │   (Socket.io)              │
                    │   - Offer/Answer           │
                    │   - ICE Candidates         │
                    └────────────────────────────┘
                                     ↓
┌────────────────────────────────────┼───────────┐
│              CLIENT B              │           │
│  ┌──────────┐  ┌──────────┐  ┌───┴──────┐    │
│  │ Browser  │← │MediaStream│← │RTCPeerCon│    │
│  │   API    │  │   API     │  │ nection  │    │
│  └──────────┘  └──────────┘  └──────────┘    │
└─────────────────────────────────────────────────┘
                                     
        Direct P2P Connection (when possible)
                   Client A ↔ Client B
                   
        TURN Relay (when NAT/firewall blocks P2P)
           Client A ↔ TURN Server ↔ Client B

🔧 Core Components

RTCPeerConnection

De core WebRTC API voor peer-to-peer verbindingen.

• Creates connection between peers
• Handles media streams
• Manages ICE candidates
• Negotiates codecs
• Monitors connection state

MediaStream API

Toegang tot camera, microfoon en scherm.

• getUserMedia() - camera/mic
• getDisplayMedia() - screen share
• MediaRecorder - recording
• Audio/video tracks
• Constraints & settings

Signaling Server

Socket.io server voor SDP exchange en ICE candidates.

• Room management
• SDP offer/answer exchange
• ICE candidate relay
• Connection status updates
• Error handling

STUN/TURN Servers

NAT traversal en relay servers.

• STUN - discover public IP
• TURN - relay when P2P fails
• Google STUN (free)
• Coturn (self-hosted TURN)
• Fallback strategies

📊 Connection Flow

Step-by-Step Process

1

Initialize Connection

Create RTCPeerConnection with STUN/TURN config

2

Get Media Stream

Request camera/mic access via getUserMedia()

3

Add Tracks

Add audio/video tracks to peer connection

4

Create Offer

Caller creates SDP offer and sends via signaling

5

Create Answer

Callee receives offer, creates SDP answer

6

Exchange ICE Candidates

Both peers discover and exchange network candidates

7

Connection Established

P2P connection active, media streams flowing

⚙️ Configuration

// ICE Server Configuration (v7.0)
const config = {
  iceServers: [
    // Google STUN servers (free, discover public IP)
    { urls: 'stun:stun.l.google.com:19302' },
    { urls: 'stun:stun1.l.google.com:19302' },
    // Custom TeezU STUN server
    { urls: 'stun:stun.teezu.online:3478' },
    // TURN relay server (NAT traversal fallback)
    {
      urls: [
        'turn:turn.teezu.online:3478',
        'turn:turn.teezu.online:5349?transport=tcp'  // TCP fallback
      ],
      username: 'teezu',
      credential: process.env.TURN_SECRET
    }
  ],
  iceTransportPolicy: 'all',   // Try P2P first, TURN as fallback
  bundlePolicy: 'max-bundle',
  rtcpMuxPolicy: 'require'
};

const peerConnection = new RTCPeerConnection(config);

// Signaling via Socket.IO (niet raw WebSocket)
socket.emit('webrtc:offer',  { sdp: offer,     roomId });
socket.on('webrtc:answer',   ({ sdp })       => pc.setRemoteDescription(sdp));
socket.on('webrtc:ice',      ({ candidate }) => pc.addIceCandidate(candidate));

📈 Quality & Performance

Adaptive Bitrate

Automatische aanpassing van kwaliteit op basis van netwerk condities.

• Bandwidth estimation
• Quality scaling (1080p → 480p)
• Frame rate adjustment
• Audio fallback

Audio Processing

Advanced audio features voor betere spraakkwaliteit.

• Echo cancellation
• Noise suppression
• Auto gain control
• Opus codec

📡 Streaming Protocol: WHIP/WHEP + HLS Fallback

✅ Primary: WebRTC (WHIP/WHEP)

WebRTC Ingestion Protocol (WHIP) voor uploaden, WebRTC Egress Protocol (WHEP) voor kijkers — sub-second latency.

• Latency <300ms (near real-time)
• Browser-native, geen plugin vereist
• OBS → WHIP endpoint → MediaSoup SFU
• Kijker ontvangt via WHEP endpoint
• Ondersteund in Chrome, Firefox, Edge

🔄 Fallback: HLS (HTTP Live Streaming)

Automatische fallback voor kijkers die geen WebRTC ondersteunen (oudere browsers, smart TV's).

• 5–10s latency (CDN-vriendelijk)
• Adaptive bitrate: 360p / 720p / 1080p
• Segment caching via CloudFront CDN
• Compatibel met alle browsers & smart TV's
• Auto-select wanneer WebRTC niet beschikbaar

🎬 Codec Priorities

🥇

VP9 — Preferred Video

• Betere compressie dan H.264
• Open-source, royalty-free
• Chrome / Firefox / Edge native
• Hardware-encoding op moderne GPUs

🥈

H.264 — Fallback Video

• Universele hardware-ondersteuning
• Safari & iOS compatibel
• Hardware encoding op mobiel
• Actief wanneer VP9 niet beschikbaar

🎵

Opus — Audio

• Standaard WebRTC audio codec
• 6–510 kbps adaptief bitrate
• Echo cancellation ingebouwd
• Noise suppression + auto gain control

🖥️ Screen Sharing & Recording

Screen Sharing (getDisplayMedia)

Via getDisplayMedia() API — kies scherm, venster of browser-tab.

// Screen share initialiseren
const screenStream = await navigator.mediaDevices
  .getDisplayMedia({
    video: {
      cursor: 'always',
      displaySurface: 'browser',
      frameRate: { ideal: 30 }
    },
    audio: true  // System audio (Chrome only)
  });

// Vervang video track in peer connection
const sender = pc.getSenders()
  .find(s => s.track?.kind === 'video');
await sender.replaceTrack(
  screenStream.getVideoTracks()[0]
);

Recording → AWS S3

MediaRecorder neemt 10s segmenten op en uploadt naar S3 via Fastify presigned URLs.

// MediaRecorder setup
const recorder = new MediaRecorder(stream, {
  mimeType: 'video/webm;codecs=vp9,opus'
});

let segmentId = 0;
// Segment elke 10s uploaden naar S3
recorder.ondataavailable = async (e) => {
  if (e.data.size > 0) {
    await uploadSegmentToS3(
      e.data,
      segmentId++
    );
  }
};

recorder.start(10_000); // 10s segmenten

👥 Multi-stream & Freeze / Reveal

Multi-stream — max 4 deelnemers

Collaborative livestreams met tot 4 gelijktijdige participant-streams via MediaSoup SFU.

• Max 4 simultane video-streams (collab mode)
• SFU routeert efficiënt — geen P2P mesh overhead
• Elke deelnemer publiceert 1 stream, ontvangt 3
• Bandwidth-bewust: auto quality degradation
• Room-based signaling via Socket.IO rooms
• Host kan deelnemers muten / verwijderen

Freeze & Reveal State Machine

Server-side state machine beheerd via Fastify. Creator controleert stream visibility in real-time.

// Stream states (server-side Fastify)
// LIVE    → Normale stream
// FROZEN  → Video gepauzeerd (blauwe glow)
//           Audio blijft actief voor kijkers
// REVEAL  → FROZEN → LIVE (confetti animatie)
// OFFLINE → Stream beëindigd

// Client emits (creator controls)
socket.emit('stream:freeze', { streamId });
socket.emit('stream:reveal', { streamId });

// Server broadcast → alle kijkers
socket.on('stream:state', ({ state }) => {
  if (state === 'FROZEN') applyBlueGlow();
  if (state === 'REVEAL') triggerConfetti();
  updateStreamOverlay(state);
});