Federated Reinforcement Learning (FRL) refers to a framework that combines federated learning with reinforcement learning (RL) to train policies across multiple distributed agents without centralizing local data.
Formally, each agent i optimizes its local policy \(\pi_i\) based on local interactions with the environment, while a global model \(\pi_G\) is formed by aggregating local updates, for example through a weighted average of gradients or parameters:
\[ \pi_G = \sum_{i=1}^N w_i \pi_i \quad \text{with} \quad \sum_{i=1}^N w_i = 1 \]
The aggregation typically occurs in communication rounds, where algorithms like FedAvg are adapted to combine policy gradients or value functions.
FRL addresses challenges such as non-IID data distributions, dynamic environments, and asynchronous updates.
It utilizes techniques from multi-agent RL, federated learning, and optimizes communication strategies to ensure efficiency and privacy.
Definition:
“Federated Reinforcement Learning is a learning paradigm in which multiple distributed agents cooperate through local reinforcement learning processes and aggregate their models without exchanging raw data, to jointly learn optimal policies.”
Source:
Li, T., Sahu, A. K., Talwalkar, A., & Smith, V. (2020). Federated Learning: Challenges, Methods, and Future Directions. IEEE Signal Processing Magazine.