Data Recorder

Overview

Data Recorder bridges the gap between the OR platform's real-time data layer and its historical data stores. Live operational data in the platform is cached in memory with short time-to-live values — ranging from a minute to a couple of hours — which means it would be lost without an active mechanism to persist it. Data Recorder fills this role by periodically reading live data from the cache and writing it to the primary database, ensuring that operational data is preserved for historical analysis, reporting, and validation.

The service listens for session events from the central orchestration service (on a configurable interval, typically every 5 minutes) and uses each session tick as a trigger to snapshot multiple data types simultaneously. For each data type, it queries the live data from the cache through the platform API, formats the results, and writes them to the corresponding actuals tables in the primary database.

Within the data pipeline lifecycle, Data Recorder sits downstream of Data Fusion — it consumes the fused, time-batched live data that Fusion has written to Redis, and persists it for longer-term use by the Data Archiver and other analytical services.

Architecture

The Data Recorder service operates as a dedicated data persistence system that coordinates the recording of live operational data into historical storage. It follows an event-driven approach triggered by platform session updates.

Key Capabilities

• Session-triggered recording — Subscribes to session updates from the central orchestration service. Each session tick (e.g. every 5 minutes) triggers the recording cycle.
• Parallel data collection — On each trigger, multiple concurrent operations simultaneously collect and save different types of live data, maximizing throughput within the recording window.
• Performance monitoring — Tracks recording duration and performance metrics for monitoring and operational visibility.
• Health monitoring — Provides health check endpoints for deployment orchestration and monitoring systems.

Data Flow

The Session Manager triggers the central orchestration service at regular intervals (typically every 5 minutes). The Data Recorder service subscribes to these session events and, for each data type, queries live data from the cache through the platform API, formats the response into structured data, writes it to the appropriate database tables, and posts a recording completion notice back to the cache.

Sequence of Operations

1. Service deploys — Initializes and starts the health monitoring server
2. Subscription starts — Subscribes to session events through the platform API
3. Session tick received — For each data type, a concurrent operation:
   • Queries live data from the cache through the platform API
   • Formats the response into a structured format
   • Writes the data to the appropriate database actuals table
   • Posts a recording completion notice back to the cache
   • Stores internal performance metrics

Recorded Data Types

Data Recorder captures snapshots of all live data types maintained by Data Fusion and other real-time services, including:

• Way actuals (road speed, volume, flow, travel time)
• Environment object actuals (device states, signal status)
• Event data (incidents, closures, planned works)
• Congestion tail data
• Public transport data

Related Services

• Data Fusion — Upstream source of fused live data in the cache
• Data Archiver — Archives recorded database data to cloud storage for long-term retention
• Experiment Manager — Central orchestration service; provides session triggers and data queries