| Owner: HIRO |
Instances hosted by: HIRO, ABI |
Role in MYRTUS: Data management across the edge-fog-cloud continuum |
Description
To create a continuum of information, all the layers of the MYRTUS infrastructure share a unified knowledge base. The Distributed Knowledge Base (DKB) stores and shares knowledge across several distributed databases, keeping data consistent and available while acting as a single logical repository for all the data within MYRTUS.
All the metrics monitored on the infrastructure are stored in the DKB. The Resource Registry, a specialised interface on top of it, turns this data into the latest view of the available resources, which the managers of the MIRTO cognitive engine use, together with other data stored in the DKB, to decide how workloads are deployed and redistributed.
The DKB is built on MinIO object storage, accessed through a Python control-plane API, and remains compatible with ETCD- and MongoDB-based components.
From the infrastructure to the orchestration: how data flows through the DKB.
DKB architecture
The DKB runs as several MinIO instances deployed in different clusters of the MYRTUS federated infrastructure, at HIRO and at ABI, kept in sync through site replication. Clients, whether project partners or MYRTUS components, reach it through a single DKB API over HTTPS.
Multi-instance DKB across two sites of the MYRTUS federated infrastructure.
The control plane regularly checks the health of each instance and classifies it as healthy, degraded or unhealthy. Requests go to the primary instance; if it fails, the control plane redirects traffic to the next healthy one, and moves it back once the primary has recovered and caught up. Endpoints, check intervals and thresholds are set in configuration, so each deployment can tune them.
Interfaces
The DKB and the Resource Registry expose standard REST interfaces, specified with OpenAPI, so that components can be developed independently and still exchange data.
DKB interface
The single entry point to store and retrieve data, organised in namespaces: logical containers of objects such as files, logs or model files.
- Namespaces: create and delete a namespace.
- Objects: upload, retrieve and delete objects in a namespace.
- Health: status of the storage instances, used for routing and failover.
Resource Registry interface
A structured view of the infrastructure for the MIRTO agents, independent of how the data is stored in the DKB.
- Resources: current status of the infrastructure.
- Network: list of the networks and details of each one.
- Compute: list of the compute nodes and details of each one.
