AI Metadata Pipeline & gRPC Ingestion

RUSEON Core provides a high-throughput AI Metadata Pipeline designed to bridge computer vision inference engines (e.g. YOLOv11, TensorRT, OpenVINO) with user-facing video streams with sub-50ms overlay synchronization and zero transcoding overhead.

---

1. The gRPC Protocol Buffer Definition

The interface is defined in api/proto/frame.proto:

syntax = "proto3";

package ruseon.core.frame;

option go_package = "github.com/RUSEGAL/ruseon-core/pkg/grpc/pb";

service FrameService {
  rpc StreamFrames(StreamRequest) returns (stream FrameResponse) {}
  rpc PushMetadata(stream MetadataRequest) returns (MetadataResponse) {}
}

message BoundingBox {
  float x = 1;          // Top-left X normalized (0.0 - 1.0)
  float y = 2;          // Top-left Y normalized (0.0 - 1.0)
  float width = 3;      // Width normalized (0.0 - 1.0)
  float height = 4;     // Height normalized (0.0 - 1.0)
  string label = 5;     // Class name (e.g. "person")
  float confidence = 6; // Confidence (0.0 - 1.0)
}

message MetadataRequest {
  string camera_id = 1;
  int64 pts = 2;
  repeated BoundingBox objects = 3;
}

---

2. Multi-Protocol Egress Broadcasting

Once metadata is ingested into RUSEON Core, the Metadata Broadcaster automatically fans it out across multiple delivery channels without modifying the underlying video stream:

1. WebRTC DataChannel (sub-50ms):
   • Sent directly over WebRTC SCTP DataChannels (label: "metadata").
   • The browser UI receives JSON events and renders animated SVG/Canvas overlays directly over the live <video> element with zero video latency penalty.
2. HLS WebVTT Track (GET /stream/hls/:id/subs.m3u8):
   • Packaged into time-aligned WebVTT cue segments (.vtt) in the HLS media playlist.
   • Allows standard HLS players and mobile apps to display detections synchronized during live playback.

---

3. Python AI Integration Example

import grpc
import time
import frame_pb2
import frame_pb2_grpc

def run_ai_worker():
    channel = grpc.insecure_channel('localhost:50051')
    stub = frame_pb2_grpc.FrameServiceStub(channel)
    
    def generate_requests():
        while True:
            req = frame_pb2.MetadataRequest(
                camera_id="cam-01",
                pts=int(time.time() * 1000),
                objects=[
                    frame_pb2.BoundingBox(
                        x=0.20,
                        y=0.15,
                        width=0.35,
                        height=0.60,
                        label="person",
                        confidence=0.95
                    )
                ]
            )
            yield req
            time.sleep(0.04) # 25 FPS
            
    response = stub.PushMetadata(generate_requests())
    print(f"Push response: {response.success}")

if __name__ == "__main__":
    run_ai_worker()

Related Documentation

• Integrations Overview
• gRPC Frame Service API Reference
• WebRTC & WHEP Architecture
• MQTT Integration