Data Redis Ingestion

Overview

Data Redis Ingestion is a specialised ingestion service that bridges external Redis-based data stores into the OR platform's data pipeline. Its primary use case is ingesting computer vision analytics from the AutoEye system — specifically, tracker data generated by Panorama camera devices. This data is written by the AutoEye analysis pipeline to an Elasticache instance in the external component store, and Data Redis Ingestion reads it and forwards it to the Data Transformer for normalisation and spatial tagging.

Unlike Data Ingestion (which polls HTTP/FTP endpoints) and Data Stream Ingestion (which subscribes to AMQP topics), Data Redis Ingestion operates at the Redis protocol level — subscribing to keyspace notifications to detect when new data is written, then reading and forwarding it. This approach is optimised for the high-frequency, sub-second tracker data that flows from the computer vision pipeline.

Within the broader ingestion layer, this service provides the dedicated pathway for camera-derived traffic analytics — a critical capability for the OR platform's AI-native vision of combining physical sensors with computer vision for real-time traffic intelligence.

Architecture

• Port: :7010
• Language: Julia
• Scaling: Singleton

Key Components

• Redis keyspace subscription — Uses Redis PSUBSCRIBE on keyspace notification patterns (e.g. *keyspace*panorama*) to detect when new tracker data is written to the external Elasticache instance.
• Dual Redis clients — Maintains separate Redis clients for subscribing to keyspace events and for reading the actual data values (required because a subscribed client cannot issue standard commands).
• Configuration-driven connections — Connection details (endpoint, client name) are defined in config and resolved from environment variables at runtime.
• Data forwarding — Retrieved tracker data is forwarded to the Data Transformer /data endpoint, following the same pattern as other ingestion services.

Data Flow

AutoEye / Panorama Devices
        ↓ (computer vision analysis)
External Elasticache (component store)
        ↓ (Redis keyspace notification)
Data Redis Ingestion [:7010]
  ├── PSUBSCRIBE to keyspace patterns
  ├── Detect new tracker data
  └── GET data by key
        ↓ (HTTP POST to /data endpoint)
Data Transformer [:5800]
        ↓
Data Fusion [:7004] (AutoEye processing + critical stoppage detection)

AutoEye Tracker Data

The primary data ingested is AutoEye tracker output from Panorama camera devices. Each message contains:

• Device URI — Unique identifier for the camera device
• Timestamp — Sub-second precision timestamp of the analysis frame
• Device ID — Logical device identifier
• Trackers — Array of detected objects, each with:
  • Bounding box coordinates (x1, y1, x2, y2)
  • Tracker ID
  • Object class and confidence
  • Speed and heading estimates
  • Grid cell position
  • Additional tracking metadata

The tracker data is generated at approximately 6 frames per second per camera device, with Data Transformer aggregating this into 1-second summaries before forwarding to Data Fusion for critical stoppage detection and way-level analytics.

Configuration

Connection configuration follows the pattern:

CONNECTIONS:
  AUTOEYE_TRACKERS:
    client_name: AUTOEYE_REDIS_ENDPOINT
    key_pattern: "*keyspace*panorama*"

The client_name resolves to an environment variable containing the Redis endpoint connection string.

Related Services

• Data Transformer — Downstream consumer that processes tracker data into OR-compliant formats
• Data Fusion — Performs AutoEye-specific fusion including critical stoppage detection
• Data Ingestion — Sibling ingestion service for HTTP/FTP sources
• Data Stream Ingestion — Sibling ingestion service for AMQP event streams
• Experiment Manager — Central coordination service (GraphQL on :5100)