Roadmap to v1.0

This page outlines planned work to bring Cassiopeia to version 1.0: stabilizing the architecture, improving the developer experience, and expanding real-time data support.

Hybrid Pipeline Architecture

The current pipeline uses Rayon for heavy parallelization, which is well-suited for batch processing but inefficient for low-frequency live streams. The plan is a two-channel system:

  • Batch flow — Retains heavy parallelization within each stage for maximum throughput on large datasets

  • Single-entry flow — Tuned for live data; stages run in parallel with each other but process records sequentially, with lower overhead and better latency for individual events

Source-Agnostic Ingestion and MQTT Support

Ingestors are currently tied to how data is fetched. Adding a dedicated collector stage before ingestion will decouple protocol from format:

  • Protocol independence — Native support for MQTT, HTTP, and local file systems

  • Data type handling — Once a source delivers data, it is routed to the appropriate ingestor (JSON, CSV, and so on) regardless of transport

  • Extensibility — New transport protocols can be added without modifying core logic

Data Profiler and UX Improvements

A Data Profiler module will simplify configuration by auto-detecting data formats and structure:

  • Format detection — Recognizes file formats (CSV, JSON, MsgPack) and dialects (delimiters, quoting)

  • Schema discovery — Infers field names and types to improve the mapping configuration wizard

  • Simplified CLI — For most standard datasets, specifying input types will become optional

Mapping Configuration Repository

Moving from purely file-based mapping management to a repository-backed approach:

  • Storage engine — A lightweight engine (possibly SQLite) to manage configurations

  • Reliability — Versioning, metadata, and fewer errors than editing raw files

  • Standalone mode — Cassiopeia will remain fully usable without external dependencies during and after the transition

API Stability (v2)

The current REST API was a proof-of-concept. For v1.0, a stable v2 API will replace it:

  • Standardization — Clean, predictable endpoints based on lessons learned in v1

  • Documentation — Full OpenAPI specifications for all stable endpoints

Lynx Ecosystem Integration

Tighter integration with Lynx for advanced context generation and configuration:

  • Shared repositories — Lynx as a central store for mapping configurations

  • Context awareness — Improved @context generation for NGSI-LD entities

Scheduling

Support for running conversions on a schedule instead of only on-demand:

  • Interval-based — Run specific conversions at fixed intervals (for example, every 5 minutes or every hour)

  • Cron-based — Schedule conversions via cron expressions (for example, daily at 02:00, every Monday at midnight) for batch jobs and periodic syncs

Continuous Mode

Support for pipelines that run continuously rather than processing finite batches:

  • Live streams — MQTT, WebSocket, and other push-based sources feeding data into pipelines in real time

  • Long-running pipelines — Cassiopeia can keep one or more pipelines active, processing events as they arrive