KCL, HIRO – Library of Multi-agent RL algorithms and hierarchical and asynchronous FL algorithms across cloud, fog, and edge
| ID: R9 | Licence: To be defined |
Owner: KCL HIRO |
Contacts: yansha.deng@kcl.ac.uk na.2.yan@kcl.ac.uk |
| Short Description | Intelligent resource management and distributed learning algorithms for the cloud, fog and edge continuum. Deep reinforcement learning (DRL) drives the workload placement decisions of the DRL-based Workload Manager of the MIRTO cognitive engine, in coordination with the bandwidth allocation of the Network Manager. A heterogeneity-resilient federated learning algorithm, with adaptive local training round configuration, bandwidth-aware coordination and robust aggregation mechanisms, supports distributed model training; it can be extended to semi-asynchronous scenarios. The same DRL approach is used for the bandwidth allocation of the Network Manager (R12). | |
| Deep reinforcement learning | Federated learning | |
|---|---|---|
| Key features | – Decision, for each service, between execution on its edge node and offloading to the fog – PPO policies trained offline on simulated service chains and resource conditions – Coordination with the Network Manager for the bandwidth of the offloaded services – Same interface as the other MIRTO Workload Managers | – Adaptive adjustment of training frequency based on node capabilities and network state – Channel-aware bandwidth allocation mechanisms – Tolerant to non-IID training data |
| Require | A pre-trained model for the size of the service chain; Resource Registry and Network Manager. | Python 3.9, PyTorch. |
| Provide | Workload placement decisions and the communication-flow requirements of the offloaded services. | Source code, configuration scripts. |
| Input | Service chain (for each service: source node, input and output data, computing demand, latency requirement) and the available compute and network resources. | Training data and simulated channel conditions. |
| Output | Allocation map (edge node or fog for each software component); bandwidth allocation and estimated latency from the Network Manager. | Trained model weights and accuracy plot. |
| User | MIRTO Agent and Deployer. | AI modules of the MIRTO system that need to evolve their models in a distributed way. |
| Benefits for the user | Communication-aware placement, which considers both computation and communication costs. | A lightweight and efficient federated learning solution adapted to heterogeneous system conditions, enabling better model performance in resource-constrained environments. |
| Download | Please contact the developers listed above. | Please contact the developers listed above. |
| Role in MYRTUS | DRL-based Workload Manager of the MIRTO cognitive engine. | Provides federated learning methods that can be used in the MIRTO cognitive engine, to improve training and communication efficiency when models are trained in a distributed way. |
| TRL@M0 | 1 | |
| TRL@M36 | 3 | |
General description
Deep reinforcement learning
The DRL-based Workload Manager optimizes workload placement in resource-constrained environments. The allocation of each service of a service chain is modelled as an offloading problem: the service either runs locally, on the edge node where the request originates, or is offloaded to another compute node such as the fog. A deep reinforcement learning agent observes the available computing and network resources and the demands of the services, and selects the placement that best meets their latency requirements. When offloading is selected, the Workload Manager translates the decision into communication-flow requirements and submits them to the Network Manager, which allocates the available bandwidth among the flows and returns the estimated latencies.
Federated learning
The federated learning algorithm supports distributed model training across heterogeneous cloud, fog and edge nodes. It adapts the number of local training rounds to the capabilities of each node and to the network state, coordinates the use of the available bandwidth, and uses robust aggregation mechanisms that tolerate non-IID training data. It is designed to improve training efficiency, communication efficiency and model aggregation in MYRTUS-like systems.
MYRTUS Extension/Contribution
Deep reinforcement learning
The DRL policy is trained offline with Proximal Policy Optimization (PPO), on simulated pairs of service chain requests and resource snapshots; the reward reflects how the estimated total latency, including computation and communication, compares with the latency requirement of each service. Since the size of the problem depends on the number of services, a separate model is trained for each supported service chain size. The pre-trained models are integrated in the Workload Manager, which exposes the same REST interfaces as the other MIRTO Workload Managers, retrieves the available resources from the Resource Registry and coordinates with the Network Manager. In the evaluation, the trained policy increased the share of services meeting their latency requirements, combining local execution and fog offloading.
Federated learning
Within MYRTUS, federated learning strategies have been designed to let the AI modules of MIRTO evolve their models in a distributed way, and to improve communication efficiency when the DRL-based Workload Manager and the Network Manager are used together.
Technology Benefits and Innovations
The DRL-based Workload Manager considers computation and communication costs together in a single decision framework, and coordinates with the Network Manager so that bandwidth allocation becomes part of workload placement. Policies trained offline can be refined with validated deployment results, and new models can be trained for different service chain sizes and resource conditions. The federated learning algorithm provides lightweight distributed training that adapts to heterogeneous nodes and network conditions.
Users
The DRL-based Workload Manager is suited to operators and integrators of continuum systems where long-term optimization and adaptive learning are important, and latency-sensitive tasks require a joint optimization of placement and resource allocation, provided that computational resources are available for training. The federated learning algorithm is relevant for developers of distributed AI applications that need to train models across heterogeneous, resource-constrained devices.
Application Domains
Latency-sensitive applications distributed between edge and fog, where the available bandwidth limits offloading. The approach is less suited when a system must be deployed immediately, without a training phase.
