Data Redis Ingestion

Overview

Data Redis Ingestion is a specialised ingestion service that bridges external real-time data stores into the OR platform's data pipeline. Its primary use case is ingesting computer vision analytics from the AutoEye system — specifically, tracker data generated by camera devices. This data is written by the AutoEye analysis pipeline to an external data store, and Data Redis Ingestion reads it and forwards it to the Data Transformer for normalisation and spatial tagging.

Unlike Data Ingestion (which polls HTTP/FTP endpoints) and Data Stream Ingestion (which subscribes to message topics), Data Redis Ingestion operates through data subscription notifications to detect when new data is written, then reading and forwarding it. This approach is optimised for the high-frequency, sub-second tracker data that flows from the computer vision pipeline.

Within the broader ingestion layer, this service provides the dedicated pathway for camera-derived traffic analytics — a critical capability for the OR platform's AI-native vision of combining physical sensors with computer vision for real-time traffic intelligence.

Architecture

The Data Redis Ingestion service operates as a dedicated real-time data bridge that subscribes to external data stores and forwards computer vision analytics to the platform's transformation layer.

Key Capabilities

• Data subscription — Subscribes to data notification patterns to detect when new tracker data is written to external data stores.
• Dual connection management — Maintains separate connections for subscribing to data events and for reading the actual data values (required to support both subscription and data retrieval operations).
• Configuration-driven connections — Connection details (endpoint, client identifier) are defined in configuration and resolved from environment variables at runtime.
• Data forwarding — Retrieved tracker data is forwarded to the Data Transformer, following the same pattern as other ingestion services.

Data Flow

AutoEye camera devices perform computer vision analysis and write tracker data to an external data store. The Data Redis Ingestion service subscribes to data notification patterns, detects when new tracker data arrives, retrieves the data, and forwards it to the Data Transformer. The Data Transformer then processes the tracker data and sends it to Data Fusion for AutoEye-specific processing including critical stoppage detection.

AutoEye Tracker Data

The primary data ingested is AutoEye tracker output from Panorama camera devices. Each message contains:

• Device URI — Unique identifier for the camera device
• Timestamp — Sub-second precision timestamp of the analysis frame
• Device ID — Logical device identifier
• Trackers — Array of detected objects, each with:
  • Bounding box coordinates (x1, y1, x2, y2)
  • Tracker ID
  • Object class and confidence
  • Speed and heading estimates
  • Grid cell position
  • Additional tracking metadata

The tracker data is generated at approximately 6 frames per second per camera device, with Data Transformer aggregating this into 1-second summaries before forwarding to Data Fusion for critical stoppage detection and way-level analytics.

Configuration

Connection configuration is managed through environment variables that specify:

• Data store endpoints — Connection details for external real-time data stores
• Key patterns — Data notification patterns that identify which data to ingest
• Client identifiers — Logical names for each connection to support multiple concurrent data sources

This configuration-driven approach enables operational teams to add or modify real-time data feeds without code changes, making it straightforward to integrate new computer vision sources or adjust subscription patterns as requirements evolve.

Related Services

• Data Transformer — Downstream consumer that processes tracker data into OR-compliant formats
• Data Fusion — Performs AutoEye-specific fusion including critical stoppage detection
• Data Ingestion — Sibling ingestion service for HTTP/FTP sources
• Data Stream Ingestion — Sibling ingestion service for event-driven message streams
• Experiment Manager — Central orchestration service