Skip to content

MDK Data Schema

Platform Users — Engineers & Low-code Ops Users (ORA / Panel Builder) OR Platform ORA — AI Planning Interface Agent Workflows Plan Visualisation ADK Integration SDK UI — Frontend Shell FDK Architecture Low code Config-driven DDK Schema Definition Code Generator Generated Server MDK WEM DAL Experiment Manager Nexus Deployment Control Live Monitoring Registry Browser SCDK Source Control Pipeline Mgmt Azure DevOps deploys ↓ SDK API — GraphQL Federation Gateway Federation Gateway Component Resolvers Auth & Licensing Plugins: gql-autogeneration Migrator Helm KinD Boilerplate GenAI ··· Microservices — Domain IP Services Data Pipeline Core Platform Metrics & Analytics Spatial & Geo Simulation Event Detection Camera & Device Fire & Resource Opt. Satellite Modelling ↓ Nexus deploys Deployed OR Applications Rail Ops Dashboard Mine Mgmt Dashboard Port Ops Dashboard ··· FDK-built · DDK-backed · MDK-powered · deployed via Nexus ↑ Application Users — Operations Teams (shift managers, analysts, planners)

About This Guide

This document helps you understand the MDK data schema and how workflow orchestration entities are structured and related. It focuses on the data models that power Studies, Experiments, Trials, Workflows, Models, and their execution states.

Who this is for: Developers building MDK integrations, platform engineers extending the MDK, and technical users who need to understand how workflow data is organized and persisted.

What you'll learn:

  • The complete entity hierarchy from Studies down to TrialTaskOutputs
  • How the OR HTTP contract standardizes model communication
  • How configuration precedence works across experiments and tasks
  • The workflow execution model and trial lifecycle
  • How DDK directives define the schema and generate the API layer

What we don't cover: To keep this guide clear and accessible, we don't cover the internal implementation of the schema generation process, proprietary query optimization techniques, or complete resolver implementation details. For advanced schema extensions beyond the documented entities, please contact our support team.


Overview

The MDK data schema defines the structure and relationships of all workflow orchestration entities — studies, experiments, trials, workflows, models, and their execution states. The schema serves as the foundation for both data storage and API interactions, ensuring consistent data structures across the platform.


Schema Layers

The schema is organized into distinct logical layers:

Persistent Data Layer

Core data types that represent the fundamental workflow entities are persisted to the database. These form the foundation of the orchestration model.

The schema follows a hierarchical structure where Workflows contain WorkflowTasks, which reference ModelConfigs, which in turn reference Models with their input and output configurations. Models may also reference ModelRegistryVersions that link to ModelRegistry entries for containerized deployments.

The schema is defined using DDK directives that control entity behavior, relationships, and database constraints. These directives generate CRUD resolvers, create database tables, define primary keys and unique constraints, and establish one-to-many and back-reference relationships between entities.

DTO (Data Transfer Objects)

Types that are not persisted to the database — they exist only as API response types returned by custom resolvers:

  • WorkflowExecuteResponse — Response from workflow/experiment execution
  • WorkflowState / WorkflowTaskState — Runtime execution state
  • WorkflowTaskOutput — Task output data and visualisation metadata
  • CompatibilityResult — Input/output compatibility check results
  • FunctionTemplate — Generated code for operators and data connections
  • WorkflowAlert / ModelAlert — Validation alerts for workflows and models

Custom (Resolvers)

Custom business logic operations beyond generated CRUD — workflow execution, DAG generation, operator management, import/export, versioning, and deployment.


OR Types System

All OR model services communicate using a standardised HTTP contract. Understanding this contract is essential for working with the MDK schema.

Core Request/Response Types

ORHttpRequest

Every request to an OR model contains:

  • version — Optional API version for compatibility tracking
  • input — The primary data the model processes (either typed domain objects or DAL data inputs)
  • params — Configuration/tuning parameters
  • output_path — Where to store results (workflow ID, storage backend)

ORHttpResponse

Every response from an OR model contains:

  • output — Results from the model (either typed domain objects or DAL data outputs)
  • metadata — Optional runtime metrics (execution time, version, etc.)
  • logs — Optional debug/info logs from execution

The Three-Type Pattern: Input, Params, Output

Every OR model endpoint defines three custom types that flow through the request/response cycle:

Input + Parameters → [Model Logic] → Output

Input Types

Purpose: The data your model processes — domain objects your algorithm operates on.

Two forms:

  1. Direct Input — Plain data structures for testing, simple workflows, or same-language services
  2. DAL Input — Data abstraction layer format for cross-language communication

Params Types

Purpose: Configuration that controls how the model behaves — tuning knobs, not the data being processed.

Key distinction: Inputs = "what to process" (changes every request), Params = "how to process it" (configuration, often stable).

Best practice: Always provide sensible defaults so params are optional.

Output Types

Purpose: Results the model produces, consumed by downstream tasks or users.

Compatibility: If the output type is equivalent to an input type (matching field names and types), it can be passed directly as an output dependency to the matching model/task.

Params vs Input in Swagger

The Swagger/OpenAPI schema must clearly separate data inputs from params so the MDK can introspect and consume them for workflow use. The inputs field can be either a user-provided type (for manual/default input via UI) or a file input type (when data comes from a preceding task via DAL).

Summary

The OR HTTP contract standardises model communication through:

  1. ORHttpRequest — Wraps input (data), params (config), and output_path (storage)
  2. ORHttpResponse — Wraps output (results), metadata (metrics), and logs (debug)
  3. Three-Type Pattern — Every endpoint defines Input, Params, and Output types
  4. DAL Integration — Seamless cross-language data exchange via data abstraction layer

Schema Hierarchy

The MDK schema follows a hierarchical structure where Studies contain multiple Experiments, each of which executes a Workflow. Experiments define per-task configuration overrides through ExperimentTaskConfig records. When executed, Experiments create Trials representing individual execution runs. Each Trial captures snapshotted configuration via TrialTaskValue records and stores execution results in TrialTaskOutput records.

Workflows are composed of WorkflowTasks, which establish dependencies between tasks via WorkflowTaskDependency records. Each WorkflowTask references a ModelConfig that configures a Model, and may optionally reference a Function for custom logic. Models define their input schemas through ModelConfigInput entities containing ModelConfigInputField records, and their output schemas through ModelConfigOutput entities containing ModelConfigOutputField records.

This hierarchy enables:

  • Experimentation — Studies contain multiple experiments with different configurations
  • Reusability — Workflows can be versioned and reused across experiments, or executed in isolation
  • Modularity — Tasks are independently defined with clear inputs/outputs
  • Traceability — Dependencies track data flow between tasks

Core Schema Components

1. Study and Experiment

Study

A container for related experiments that share a common research goal.

Purpose: Studies organize experiments that explore variations of the same workflow or research question. Each study can contain multiple experiments with different configurations, enabling systematic comparison and optimization.

Key Capabilities:

  • Group related experiments under a single research objective
  • Track experiment history and evolution over time
  • Enable comparison across different workflow configurations
  • Provide organizational structure for complex optimization tasks

Relationships:

  • Contains multiple Experiments (one-to-many)
  • Each experiment shares the same high-level research goal but may differ in configuration

Experiment

A configured experiment that references a workflow and contains trials (individual execution runs).

Purpose: Experiments represent a specific configuration of a workflow within a study. They enable systematic testing of different parameter values, data inputs, or task configurations while maintaining full traceability of execution results.

Key Capabilities:

  • Execute workflows with experiment-specific configuration overrides
  • Track multiple trial runs with different inputs or parameters
  • Compare trial results to identify optimal configurations
  • Maintain immutable snapshots of configuration at execution time
  • Support both manual experiments and automated optimization workflows

Relationships:

  • Belongs to one Study (many-to-one)
  • Executes one Workflow (many-to-one)
  • Contains multiple Trials (one-to-many) — each trial represents an individual execution
  • Defines task configuration overrides via ExperimentTaskConfig (one-to-many)

Configuration System: Experiments use a three-tier configuration precedence model:

  1. ExperimentTaskConfig overrides (highest priority) — experiment-specific settings
  2. WorkflowTask configuration — task-level defaults
  3. ModelConfig configuration (lowest priority) — model defaults

Note

Experiments are read-only via auto-generated API resolvers. Creation, duplication, and execution are handled by custom business logic resolvers.

ExperimentTaskConfig

Per-experiment overrides for individual task configurations.

Purpose: Allows experiments to customize specific task parameters without modifying the underlying workflow definition. This enables systematic parameter exploration and A/B testing while keeping the base workflow reusable.

Key Capabilities:

  • Override task parameters for specific experiments
  • Enable parameter sweeps across multiple trials
  • Maintain workflow reusability across different experiments
  • Support automated hyperparameter tuning

Configuration Precedence (highest to lowest):

  1. ExperimentTaskConfig.configValue — Experiment-specific override
  2. WorkflowTask.configValue — Task-level configuration
  3. ModelConfig.configValue — Default model configuration

Trial

An individual execution run within an experiment.

Purpose: Trials represent single execution instances of an experiment, capturing complete configuration snapshots and execution results. Each trial is immutable after creation, ensuring full reproducibility and traceability.

Key Capabilities:

  • Execute workflows with immutable configuration snapshots
  • Track execution status through complete lifecycle
  • Store per-task outputs and execution metadata
  • Enable result comparison across multiple trial runs
  • Support promotion of successful trials to production workflows

Trial Status Lifecycle:

  • PENDING — Trial created but not yet executed
  • RUNNING — Currently executing
  • COMPLETED — Successfully finished
  • FAILED — Execution failed
  • EXPIRED — Trial results expired from temporary storage
  • PROMOTED — Selected as best result and promoted to production

Relationships:

  • Belongs to one Experiment (many-to-one)
  • Contains configuration snapshots via TrialTaskValue (one-to-many)
  • Produces outputs via TrialTaskOutput (one-to-many)

Execution Flow:

  1. Trial created with PENDING status (auto-created by execution or pre-created)
  2. On execution: status → RUNNING, execution timestamp recorded
  3. Configuration values snapshotted into TrialTaskValue records (immutable after creation)
  4. On completion: status → COMPLETED or FAILED, completion timestamp recorded
  5. Optionally promoted via promotion flag for production deployment

TrialTaskValue

A snapshot of the configuration used for a specific task in a specific trial.

Purpose: Captures the exact configuration used for each task at trial execution time, ensuring full reproducibility even if experiment or workflow configurations change later.

Key Capabilities:

  • Provide immutable configuration snapshots for each trial execution
  • Enable exact reproduction of trial results
  • Support configuration auditing and compliance
  • Preserve historical execution context

Snapshot Resolution Logic: At trial execution time, configuration is resolved using the precedence model:

  1. Check ExperimentTaskConfig override (highest priority)
  2. Fall back to WorkflowTask.configValue
  3. Fall back to ModelConfig.configValue The resolved value is persisted as an immutable TrialTaskValue record.

TrialTaskOutput

Stores the output data from a task execution within a specific trial.

Purpose: Captures the complete output data, execution logs, and performance metrics for each task in a trial run.

Key Capabilities:

  • Store task execution results with complete metadata
  • Track execution state and performance metrics
  • Support rich data visualization (GeoJSON, charts, tables)
  • Enable debugging via execution logs
  • Facilitate result comparison across trials

Output Types:

  • JSON — Structured data output
  • CSV — Tabular data output
  • OTHER — Binary or unstructured output

Captured Information:

  • Raw output data from task execution
  • Execution logs for debugging and auditing
  • Final execution state (success, failure, skipped)
  • Execution duration and performance metrics
  • Visualization metadata for rendering results

2. Workflow and WorkflowTask

Workflow

A directed acyclic graph (DAG) defining the execution flow of interconnected tasks.

Purpose: Workflows orchestrate complex computational processes by defining tasks, their dependencies, and data flow. They serve as reusable templates that can be versioned, deployed, and executed across multiple experiments.

Key Capabilities:

  • Define task execution order via dependency relationships
  • Support parallel and sequential task execution patterns
  • Enable conditional execution and iterative loops
  • Provide version control for workflow evolution
  • Support both test execution and production deployment
  • Enable variable-driven parameterization across experiments

Execution States:

  • PENDING — Workflow created but not yet executed
  • IN_PROGRESS — Currently executing
  • SUCCESSFUL — All tasks completed successfully
  • FAILED — One or more tasks failed
  • SKIPPED — Task skipped due to conditional logic

Relationships:

  • Contains multiple WorkflowTasks (one-to-many)
  • Executed by multiple Experiments (one-to-many)
  • Deployed as multiple WorkflowDeployments (one-to-many)
  • Versioned via parent-child workflow chain

Versioning Strategy:

  • Parent-child relationships create version chains between workflow revisions
  • Version flag marks the current production version
  • Support rollback to previous versions
  • Enable version comparison and conflict detection

WorkflowTask

An individual computational unit within a workflow.

Purpose: WorkflowTasks represent discrete execution steps within a workflow, encapsulating model invocation, data transformation, or external system integration. Each task is independently configurable and connects to other tasks via dependencies.

Key Capabilities:

  • Execute models, operators, or custom functions
  • Accept inputs from upstream task outputs
  • Pass outputs to downstream task inputs
  • Support variable-driven configuration substitution
  • Enable result caching for improved performance
  • Define execution priorities for loop cycle-breaking

Configuration System: Tasks support a variable substitution mechanism where configuration values can reference workflow-level variables. This enables dynamic parameterization across experiments:

  • configValue contains the base configuration with nested JSON structures
  • configVariables maps variable keys to dot-notation paths in the configuration
  • At execution time, variable values are substituted into the configuration

For example, a variable $user_name mapped to path "name" would replace the name field in the configuration with the variable's runtime value.

Relationships:

  • Belongs to one Workflow (many-to-one)
  • Executes one ModelConfig (many-to-one)
  • May use one Function for custom logic (optional, many-to-one)
  • Depends on upstream tasks via WorkflowTaskDependency (many-to-many)
  • Feeds into downstream tasks via WorkflowTaskDependency (many-to-many)
  • Overridden by ExperimentTaskConfig in specific experiments
  • Snapshotted in TrialTaskValue for each trial execution
  • Produces TrialTaskOutput results for each trial execution

3. WorkflowTaskDependency

Defines the relationship between two tasks in the workflow DAG.

Purpose: WorkflowTaskDependencies establish execution order and data flow between tasks, enabling both sequential orchestration and data passing patterns. They support advanced control flow including conditional execution and iterative loops.

Dependency Types:

  1. Output Dependency — Downstream task consumes upstream task's output data. Data is transferred via file system or Redis cache. The target input parameter is specified via parameter mapping.

  2. Sequential Dependency — Downstream task waits for upstream completion without data transfer. Used for execution ordering only.

Key Capabilities:

  • Define task execution order in the DAG
  • Enable data flow between tasks via output-to-input connections
  • Support conditional execution based on upstream task results
  • Enable iterative loops with configurable termination conditions
  • Provide cycle-breaking via task priorities

Loop Configuration:

Dependencies can define iterative behavior through loop configuration:

  • maxIterations — Maximum number of loop iterations
  • succeedAfterMaxIterations — Whether to continue workflow or fail when max iterations reached
  • continueCondition — Conditional expression evaluated on each iteration
  • succeedAfterContinueCondition — Whether to continue or fail when condition is met

Loop Behavior:

  • Max iterations reached + succeed flag true → Skip downstream, continue workflow
  • Max iterations reached + succeed flag false → Fail workflow
  • Continue condition met + succeed flag true → Continue loop
  • Continue condition met + succeed flag false → Fail workflow
  • Continue condition not met → Skip downstream, continue workflow

Condition System:

Supports rich conditional expressions with:

  • Comparison operators: equal, not equal, greater than, less than, etc.
  • String operators: contains, starts with, ends with, matches (regex)
  • Collection operators: in, not in
  • Logical operators: AND, OR for nested conditions
  • Field access via dot-notation paths (e.g., "data.status")

Supported operators: EQ, NEQ, GT, GTE, LT, LTE, CONTAINS, STARTS_WITH, ENDS_WITH, MATCHES, IN, NOT_IN


4. ModelConfig and Model

ModelConfig

A configured instance of a model ready for use in a workflow task.

Purpose: ModelConfigs provide reusable, pre-configured model instances that can be shared across multiple workflow tasks. They enable separation of model definition from model configuration, allowing the same base model to be used with different default settings.

Key Capabilities:

  • Define reusable model configurations with default parameter values
  • Enable model configuration validation before workflow execution
  • Support multiple configurations of the same base model
  • Allow optional custom function overrides for operators and connectors
  • Provide templates for consistent model usage across workflows

Relationships:

  • Configures one Model (many-to-one)
  • May override with one Function (optional, many-to-one)
  • Used by multiple WorkflowTasks (one-to-many)

Configuration Hierarchy: ModelConfig provides the base configuration layer that can be overridden by WorkflowTask and ExperimentTaskConfig settings.

Model

The blueprint of a computational component.

Purpose: Models define the interface and behavior of computational components that can be orchestrated in workflows. They represent the schema, not the execution instance — capturing input/output structures, API endpoints, and component metadata.

Component Types:

  1. MODEL — ML models or microservices deployed as Docker containers, invoked via HTTP endpoints

  2. OPERATOR — Custom Python/Julia functions for data transformation, executed by the PyRunner service

  3. IMPORT_DDK — Data connectors that import data from external GraphQL APIs

  4. EXPORT_DDK — Data connectors that export data to external systems

Key Capabilities:

  • Define input and output schemas with type validation
  • Support multiple component types (models, operators, connectors)
  • Enable version tracking and upgrade paths
  • Provide API endpoint definitions for invocation
  • Support both containerized services and custom functions
  • Enable input/output compatibility validation

Relationships:

  • Deployed as ModelRegistryVersion (optional, for containerized models)
  • Implemented by Function (optional, for operators and connectors)
  • Instantiated as multiple ModelConfigs (one-to-many)
  • Defines ModelConfigInputs for input schema (one-to-many)
  • Defines ModelConfigOutputs for output schema (one-to-many)
  • Versioned via ModelVersionLink for version chains

Model Versioning: Version chains track relationships between model revisions, enabling version conflict detection and providing upgrade paths when models evolve.


5. ModelConfigInput and ModelConfigInputField

ModelConfigInput

Defines a parameter or input group for a model.

Purpose: ModelConfigInputs define the input schema for models, specifying what data and parameters a model accepts. They enable type validation, compatibility checking, and automatic UI generation for model configuration.

Input Types:

  1. INPUT — Actual data consumed by the model (files, objects, arrays, primitives). May come from upstream task outputs or user-provided data.

  2. PARAM — Configuration parameters that control model behavior. Typically static values set in configuration, but can be overridden by workflow variables.

Key Capabilities:

  • Define structured input schemas with type validation
  • Support both data inputs and configuration parameters
  • Enable input/output compatibility validation between tasks
  • Provide default values for optional inputs
  • Support native type mapping for cross-language compatibility
  • Enable data connector configuration for IMPORT_DDK models

Relationships:

  • Belongs to one Model (many-to-one)
  • Contains ModelConfigInputFields for structured types (one-to-many)
  • Compatible with ModelConfigOutputs via Compatibility (many-to-many)

Type System:

  • Native field types preserve language-specific type information
  • Generic field types enable cross-language type validation
  • Default values support optional inputs with fallbacks

ModelConfigInputField

Defines individual fields within an input's data structure, supporting nested hierarchies.

ModelConfigInputField entities define the individual fields within a structured input, with each field containing a unique identifier, field name, optional description, and both generic and native field types. Fields can be marked as required or optional, and may have default values. The structure supports nested hierarchies through a parent-child relationship, where fields can reference a parent field to create complex nested objects. For example, a customer data input might contain simple fields like name and age, as well as a nested address object containing street and city fields that reference the address as their parent.

FieldType Enum — Comprehensive type system for schema validation:

CategoryTypes
ScalarSTRING, NUMBER, NUMBER_FLOAT, NUMBER_DOUBLE, INTEGER, INTEGER_32, INTEGER_64, BOOLEAN, OBJECT
ArrayARRAY, ARRAY_STRING, ARRAY_NUMBER, ARRAY_NUMBER_FLOAT, ARRAY_NUMBER_DOUBLE, ARRAY_INTEGER, ARRAY_INTEGER_32, ARRAY_INTEGER_64, ARRAY_BOOLEAN, ARRAY_OBJECT
2D ArrayARRAY2D_STRING, ARRAY2D_NUMBER, ARRAY2D_NUMBER_FLOAT, ARRAY2D_NUMBER_DOUBLE, ARRAY2D_INTEGER, ARRAY2D_INTEGER_32, ARRAY2D_INTEGER_64, ARRAY2D_BOOLEAN, ARRAY2D_OBJECT

6. ModelConfigOutput and ModelConfigOutputField

Mirror the input structure but define what a model produces. Same field pattern as ModelConfigInput/ModelConfigInputField with outputConfig replacing inputConfig.

FieldDescription
paramNameParameter name in the model's response
outputConfigFor EXPORT_DDK, contains external API configuration (resolver URL, mutations)
modelConfigOutputFieldsStructured field definitions for output data

7. Function

Custom source code executed by the PyRunner executor service.

Functions encapsulate custom source code that can be written in Python, Golang, or Julia. Each function has a unique identifier, name, optional description, and the actual source code contents. Functions include a creation timestamp used for versioned file naming, and a validation flag indicating whether the function passes validation checks. Functions can be associated with Models (for operators and connectors), ModelConfigs (for configuration overrides), and WorkflowTasks (for execution).

Key fields:

  • Secondary function identifier for reference tracking
  • Language specification supporting Python, Golang, or Julia
  • Full source code contents stored as a string
  • Creation timestamp for immutable versioning
  • Validation status flag

Usage Pattern:

  1. Function is stored in the database
  2. During DAG generation, function is written to a .py file as {name}_{dtTimestamp}.py
  3. File path and function name are passed to the Python executor (PyRunner)
  4. Executor loads and executes the function

Versioning via Timestamp: The dtTimestamp field ensures immutable function versions. Multiple experiments can use different function versions concurrently.


8. ModelRegistry and ModelRegistryVersion

ModelRegistry

Registry information for containerised model collections (microservices).

ModelRegistry entities store deployment configuration for containerized models, including the AWS ECR registry URL and repository name where Docker images are stored. Each registry specifies the application port the containerized model listens on and the programming language used. A registry can have multiple versions tracked through ModelRegistryVersion records.

ModelRegistryVersion

ModelRegistryVersion entities track specific versions of containerized models within a registry. Each version references its parent ModelRegistry and includes the Docker image tag (such as version numbers or "latest"). A boolean flag indicates whether this represents the latest deployed version. Multiple Models can reference the same ModelRegistryVersion for their deployment configuration.

Purpose: Connect models to their Docker container deployments and enable version control for deployed model collections.


9. Compatibility

Defines which outputs can connect to which inputs — a join table for type-safe task connections.

Compatibility records establish allowed connections between ModelConfigOutputs and ModelConfigInputs. Each compatibility record links one output to one input, using both identifiers as a composite primary key. This junction table enables bidirectional navigation between compatible inputs and outputs.

Validation Logic:

  • Only allows connections between type-compatible inputs and outputs
  • Prevents runtime type errors in workflow dependencies
  • Enforced during workflow design in the UI
  • The modifyCompatibilityTable mutation recalculates compatibility entries

10. Workflow Test Execution

A lightweight execution mode for validating workflow logic and dependencies without running the full experiment lifecycle. Workflows have one latest state stored in Redis, updated on each test execution. This allows for rapid iteration during development without creating multiple trials in the database.

The resolvers to retrieve workflow execution output/state/logs are always based on the latest execution, so the architecture is the same as the full experiment execution model, just without trial/task output persistence in the database. This means the UI can switch seamlessly between test execution and full experiment execution modes.

Test execution results are temporary and are never stored in the database. Technically, you can allow multiple concurrent test executions of the same workflow as each run has a unique dag run ID, but the UI only ever renders the latest execution results for simplicity.

Execution Flow Summary:

  1. User triggers test execution via testExecuteWorkflow mutation.
  2. A DAG is generated using the workflow definition, and a unique dag run ID is created.
  3. The WEM executes the workflow, storing execution state, outputs, and logs in Redis under keys namespaced by the dag run ID.
  4. The UI polls for execution results using the workflow ID, rendering the latest state, outputs, and logs from Redis/WEM state.
  5. Results eventually expire in Redis after a configurable TTL (e.g., 1 hour) to prevent stale data.

11. Experiment Execution (Trial System)

The experiment execution model uses a Trial-based approach. See Study and Experiment above for the full Experiment, ExperimentTaskConfig, Trial, TrialTaskValue, and TrialTaskOutput type definitions.

Execution Flow Summary:

  1. ExperimentTaskConfig records define per-experiment config overrides for each task
  2. When executeExperiment is called, a Trial is created (or an existing PENDING trial is used)
  3. Config values are resolved (ExperimentTaskConfig → WorkflowTask → ModelConfig) and snapshotted into TrialTaskValue records
  4. The WEM executes the workflow using the snapshotted config, recording outputs/states/logs into Redis.
  5. Redis data is temporary until saved back to the database as TrialTaskOutput records upon by the user calling saveTrial.
  6. Trial status transitions: PENDINGRUNNINGCOMPLETED/FAILED
  7. Trial's can be promoted into a new workflow by using promoteTrial, to persist the configuration and outputs as a new version of the workflow.

Supporting Entities

WorkflowDeployment

Deployment configuration for scheduled or event-driven workflow execution.

WorkflowDeployment entities configure how workflows are automatically triggered and executed. Each deployment has a unique identifier, name, optional description, and references a specific workflow. Deployments can be enabled or disabled, and include a deployment type that determines the trigger mechanism. Configuration details are stored as JSON, along with workflow variables and a creation timestamp.

Deployment Types:

TypeDescription
ONCE_OFF_SCHEDULERun at a specific unix timestamp
CRON_SCHEDULERun on a recurring cron expression
DDK_RESOLVERTriggered by default DDK resolver events
DDK_RESOLVER_CRUDTriggered by DDK CRUD events (CREATE/UPDATE/DELETE) with variable mappings
DDK_RESOLVER_CUSTOMTriggered by custom DDK resolver (not yet implemented)
STREAMING_REDIS_STREAMSRun on Redis Streams data (not yet implemented)
ON_STARTUPRun once when the Experiment Manager starts

WorkflowTemplate

Reusable workflow blueprints that can be instantiated as new workflows.

WorkflowTemplate entities store reusable workflow configurations that serve as blueprints for creating new workflows. Each template has a unique name and identifier, an optional description, and the complete template definition stored as JSON. Templates track creation and update timestamps to support version management.

Join table tracking version chains between models.

ModelVersionLink entities create directed relationships between model versions, linking a current model to its next version. The link uses both model identifiers as a composite primary key, establishing a version chain that can be traversed to detect model version conflicts in workflows and enable upgrade option queries.


See Also

  • Component Types — how the component type on WorkflowTask determines execution behaviour
  • DDK Schema Guide — the DDK directives used to define this schema and generate the CRUD layer underneath it

User documentation for Optimal Reality