AI Metadata Pipeline

RUSEON Core provides an end-to-end pipeline for AI video analytics.

The Loop

1. Ingestion: An external AI worker connects to the gRPC Service (StreamFrames), receiving raw NAL units.
2. Processing: The worker decodes the frame, runs inference, and generates bounding boxes.
3. Injection: The worker sends the bounding boxes back via PushMetadata, matching the pts (Presentation Timestamp) of the analyzed frame.
4. Broadcasting: The internal MetadataBroadcaster receives the data and distributes it to connected clients.

Delivery Mechanisms

The metadata is automatically formatted and delivered alongside the video stream through three distinct channels:

1. HLS WebVTT

In HLS streams, metadata is packaged as WebVTT subtitles in subs.m3u8. Video players can parse these cues to render overlays perfectly synchronized with the video playback buffer.

2. WebRTC DataChannels

For WebRTC WHEP streams, metadata is sent over a dedicated WebRTC DataChannel named "metadata".

3. MQTT

Metadata is also published to the configured MQTT broker for consumption by databases or rule engines.

Frontend Rendering

To display the bounding boxes in a web interface, RUSEON provides a standard approach:

• Extract the JSON metadata from the DataChannel or WebVTT cue.
• Use a <canvas> element overlaid on top of the <video> element.
• Use requestAnimationFrame to draw the boxes (x, y, width, height) perfectly synced with screen refreshes.