This technical article was originally published at V-Nova’s website. It is reprinted here with the permission of V-Nova.
TL;DR
This post describes a real-time AI video analytics pipeline built on the SMPTE VC-6 hierarchical image format. Instead of decoding full-resolution frames for every model, the system decodes and fetches a low-resolution layer for inference and selectively refines only the regions and resolutions requested by the models. The video is decoded once and the decoded output is reused by multiple AI models, avoiding repeated decoding and rescaling. The result is lower compute cost, higher throughput, reduced memory bandwidth and I/O data transfer, and scalable multi-model inference from a single video stream.
A developer-focused walkthrough of the VC-6 AI Blueprint
The VC-6 AI Blueprint is a real-time demonstration platform that showcases how the SMPTE VC-6 codec can be tightly integrated into AI-driven video analytics workflows to enable multi-model inference. Designed for developers, researchers, and systems engineers, the demo illustrates how efficient image decoding, intelligent region extraction, and multi-model inference can coexist within a unified, high-performance architecture.
This post explains how the Blueprint leverages VC-6 to enable adaptive, multi-model inference from a single video stream. It introduces the core VC-6 concepts used in the demo, Levels of Quality (LoQ) and Region-of-Interest (ROI) decoding, and then explains the multi-inference pipeline and its feedback loop. It shows how model outputs dynamically trigger higher-detail decoding for selected regions. The post also describes the developer-facing controls and real-time metrics used to tune LoQ, ROI, and confidence thresholds, and concludes with practical examples made possible by this architecture, such as event-based clip extraction.
VC-6 as the Backbone of Adaptive AI Processing
The VC-6 hierarchical data format introduces a paradigm shift in how visual data is handled in AI workflows. Unlike traditional formats that require full-frame decoding, VC-6 intra-codec supports multi-resolution decoding of any frame through Levels of Quality (LoQ). Each LoQ corresponds to a progressively refined representation of the same image, enabling the system to decode frames at different granularities based on context.
This layered design enables:
- Progressive rendering: Decode and display low-LoQ data first for instant feedback while higher LoQs are fetched as needed.
- Selective ROI decoding: Target specific areas for high-resolution analysis without processing the entire frame across any of the available LoQs.
- Partial bitstream access: Fetch only the necessary sections (either LoQs or ROIs) of the compressed bit-stream, minimizing I/O and compute overhead, optimizing network load and memory bandwidth requirements.
These capabilities directly translate into lower latency, higher throughput, and reduced bandwidth usage when paired with AI models, especially in real-time or resource-constrained environments, such as embedded vision systems, edge devices, and live monitoring platforms.
Pipeline Architecture and Multi-Inference Design
The core of the VC-6 AI Blueprint is an orchestrated multi-inference pipeline managed by a central Orchestrator component. The workflow operates as follows:
- Input: A VC-6 encoded video stream is provided to the Orchestrator.
- Decoding: Frames are decoded from a single VC-6 decoder state at a selected LoQ, ensuring that all downstream models operate on the same frame instance. Enabling ROI refinement to continue from that same state without redundant re-decode or rescaling.
- Inference: Decoded frames, or model-specific portions thereof, are passed in parallel to multiple AI models.
- Feedback loop: The output of each model (e.g., bounding boxes, meshes) informs the decoding process for ROIs at higher LoQs.
Workflow for the current implementation of the VC-6 AI Blueprint.
Currently, the demo integrates two key models, YOLOv5 and MediaPipe FaceMesh:
- YOLOv5 for object detection: Fast, scalable, and well-suited for identifying multiple classes in real time.
The blueprint uses detections to define dynamic ROIs, optionally filters them by class, and can trigger event-based extraction such as generating short vertical clips from selected detections.
- Google MediaPipe FaceMesh for dense facial landmark mapping: Enabling ROI-based mesh extraction for facial features such as eyes, nose, and lips.
Face detection and landmark based extraction of ROI for face, nose, left-eye and right-eye are performed alongside object detection models and extraction of dynamic ROIs from bounding boxes.
Because all inferences share a common decoder state, the pipeline achieves zero redundant decoding and maintains synchronization across model outputs.
Efficient Region and LoQ Management
Each detected object or face mesh triggers the console’s ROI decoding mechanism, allowing regions to be decoded at higher LoQs while continuing from the previous decoder state. For example, the pipeline may decode an entire frame at LoQ 5 (low resolution) for fast object detection, and then selectively re-decode detected bounding boxes at LoQ 0 (full resolution) for detailed inspection.
What to measure when tuning LoQ and ROI
When lowering the base LoQ (for example LoQ 0 to LoQ 2 or LoQ4), developers should validate two things in parallel:
- Model output stability: whether detections (counts, classes, boxes) and FaceMesh landmarks remain consistent for the target use case at the chosen confidence threshold.
- Throughput and latency: changes in Decode FPS and Inference FPS reported by the stats panel.
A typical goal is to reduce decode cost by lowering the base LoQ while confirming that model outputs are unchanged for the scenes and classes that matter.
A practical example using an NVIDIA-based system is given below.
This dynamic decoding strategy ensures:
- Optimal GPU and CPU utilization
- Reduced memory footprint
- Scalable inference throughput even under variable frame-rate conditions.
The same mechanism powers the “Extract Shorts” feature, which automatically generates high-resolution video clips from specific detection events, demonstrating real-world automation use cases like event tagging or forensic replay.
UI Controls and Developer Configurability and Stats
From a development perspective, the interactive control interface is not merely cosmetic, it serves as a live tuning tool for inference pipelines. Developers can adjust:
- LOQ decoding levels to balance performance vs. quality
- Confidence thresholds for detection models
- ROI LOQ settings to experiment with different decoding hierarchies in real time for selected ROIs.
In practice, the key KPIs to track are: Decode FPS, Inference FPS, and the detection/ landmark outputs at a fixed threshold. When experimenting with base LoQ and ROI LoQ, treat the model outputs as the correctness signal and FPS as the performance signal. 
The immediate visual feedback accelerates iterative testing and makes it easy to evaluate the codec’s behavior under varying inference loads.
Configuration Panel – Adjustable input LoQ and Adjustable extraction ROI LoQ, Model confidence thresholds controls and Shorts extraction object type and size of fixed ROI for shorts.
There is also a real time stats panel that shows the performance of the decode and the inference for the current workflow and the currently selected LOQ-ROI configuration.
End-to-End Integration and Encoding Tools
The demonstration includes an encoding utility that converts standard video inputs into VC-6 format, ensuring end-to-end compatibility. This allows engineers to benchmark and experiment with different encoding profiles and analyze their impact on inference performance, latency, and visual fidelity.
Encode any video file to VC-6 format to be used in the VC-6 AI Blueprint.
The design philosophy behind the Blueprint emphasizes single-state decoding as the foundation for efficient multi-model inference, a concept that could extend to distributed AI pipelines and hybrid edge-cloud architectures.
Conclusion: Enabling Smarter AI with Smarter Codecs
The VC-6 AI Blueprint demonstrates that the codec layer is not just a pre-processing step but a key enabler of efficient AI. By integrating VC-6’s adaptive decoding directly into the inference workflow, developers can significantly reduce redundant compute, accelerate processing pipelines, and enable new forms of inference use cases.
For engineers working in real-time vision, video surveillance, or autonomous perception, this architecture shows how codec-aware AI can redefine visual data processing, bridging the gap between compression efficiency and inference intelligence through a unique data format that complements training and inference workflows, enabling dynamic, scale-specific, and region-based image access.
Demonstration Use Cases
Machine: 13th Gen Intel(R) Core(TM) i9-13900K
Memory: 125.55 GiB
GPU: NVIDIA RTX A6000 – 49140 MiB – driver 575.51.03
Subject Tracking
In this use case, we demonstrate person tracking on warehouse surveillance footage with a resolution of 1842 × 1080. The workflow begins by enabling the Extract Shorts option in the UI, which provides an interface for selecting the target label to track, person in this example, along with a target crop size of 400 × 500.
The frame is decoded using VC-6 at a specified Level of Quality (LoQ) and rendered in the UI if enabled, and fed to object detection using YOLOv8. The resulting bounding boxes are then mapped to a corresponding Region-of-Interest (ROI) LoQ. Rather than decoding the entire frame at higher quality, only the decoded ROI is extracted from the VC-6 frame and rendered in the Extracted Shorts view, enabling efficient, high-detail tracking of the subject.
The following inference statistics are exposed in real time:
- Input LoQ: The level of quality of the decoded image provided to the object detection model.
- ROI LoQ: The level of quality extracted for the region of interest based on the object detection bounding box.
- FPS: End-to-end throughput, measured from decode to inference, ROI extraction, and rendering.
- No. of models used of inference
- Main Render: Rendering of the input frame used for object detection.
- ROI Render: Rendering of the decoded region of interest at the target quality level.
| Input LOQ | ROI LOQ | FPS | Models | Main Render | ROI Render |
| 0 (Full Quality) 1842×1080 |
0 (Full Quality) | 50 | 1 | Yes | Yes |
| 1 (Lower) 921×540 |
0 (Full Quality) | 90 | 1 | Yes | Yes |
| 2 (Lower) 461×270 |
0 (Full Quality) | 108 | 1 | Yes | Yes |
| 3 (Lower) 231×135 |
0 (Full Quality) | 115 | 1 | Yes | Yes |
| 4 (Lower)
116×68 |
0 (Full Quality) | 120 | 1 | Yes | Yes |
| 4 (Lower) 116×68 |
0 (Full Quality) | 150 | 1 | No | Yes |
This experiment demonstrates that a “person” can be successfully detected even at “LoQ 4” quality, while increasing the throughput by 2.4 times.
The demo demonstrates how extracted ROIs are maintained and tracked. Rendering can be disabled to display a list of detected labels instead.
Face Detection
In this use case, we showcase facial feature extraction and rendering using Google FaceMesh. The video which is 1920×1080 is decoded using VC-6 at a specified Level of Quality (LoQ) and rendered in the UI (if enabled) and fed to detect and extract the face region. Using the landmark metadata provided by FaceMesh, we identify key facial features—such as the left eye, right eye, nose, and lips based on the predefined legend indices.
Each detected feature is mapped to a corresponding Region of Interest (ROI) and selectively decoded from the VC-6 frame. This allows individual facial components to be extracted and rendered independently, alongside the full face, without requiring higher-quality decoding of the entire frame for the target ROI LoQ.
To view this workflow in action, enable the Face Manipulation option in the UI. The interface displays the decoded full face as well as separate, high-detail renders of each facial feature, demonstrating how landmark-driven metadata can be used to drive fine-grained ROI decoding and visualization.
This use case also demonstrates multi-inference pipelines which means that there are inference coming from multiple models for extraction of regions of interest and all of that is handled with single VC-6 decoder state.
You can also see the rendering of all the ROIs of the Object detection model along with the FaceMesh outputs being rendered.
Multi Subject Tracking
This use case highlights multi-subject detection and extraction by decoding and rendering multiple regions of interest (ROIs) from a single decoder state for each frame.
As multiple subjects are detected, their corresponding ROIs are independently extracted and rendered, allowing simultaneous visualization of each subject on screen. The same decoded ROIs are also reused for Shorts extraction, demonstrating how a single video stream can efficiently support multiple ROI workflows, both real-time rendering and downstream clip generation, without redundant decoding or rescaling.
Also in this use case, the user can experiment how a reduction in LoQ or ROI LoQ decoding can improve throughput, while preserving the effectiveness of the AI detection.
You can find the source code here: https://github.com/v-novaltd/vc6-ai-blueprint
SMPTE VC-6 Samples https://github.com/v-novaltd/vc6-samples
Have fun!
Vinod Balakrishnan
Principal Software Engineer, V-Nova
