Federated Reinforcement Learning

Warning

The code has been updated in order to use modern tools such as uv and make it more accessible with the lastest python versions. The original code version is on tag 2023.02.

Introduction

The goal of this project is to simulate an environment for Federated Learning and experiment different algorithms using Reinforcement Learning for better aggregation or to select in a better way the next participants.

Installation

1. First, download the repository

git clone https://github.com/bourbonut/reinforcedFL.git 

2. Then create your own environment. For instance, with uv:

uv venv
uv pip install -r requirements.txt
uv pip install -r pytorch-requirements.txt

Note

Check if the version of CUDA in pytorch-requirements.txt is compatible with your CUDA version (nvidia-smi).

Warning

On my machine, the total size of dependencies is ~4.5GB.

Usage

Configuration file

You should create a config.toml file at the root of the project.

[environment.env20]
nexps = 20        # number of experiments
rounds = 10       # number of rounds
epochs = 3        # number of epochs
nworkers = 20     # number of workers
dataset = "MNIST" # name of the dataset

[environment.env100]
nexps = 20
rounds = 10
epochs = 3
nworkers = 100
dataset = "MNIST"

[distribution.iid]
label_distrb = "iid"
minlabels = 3
volume_distrb = "iid"
balanced = true
k = 5

[distribution.noniid]
label_distrb = "noniid"   # labels are non independent and identically distributed
minlabels = 3             # there are at least 3 labels per worker
volume_distrb = "noniid"  # volume of data is non independent and identically distributed
balanced = true           # True for "per worker" else each worker is a cluster of labels
k = 5                     # coefficient for data augmentation (example MNIST : 60_000 x 5 = 300_000 samples)

[model.fedavg]
worker_class = "Worker"             # name of the worker class
server_class = "FederatedAveraging" # name of the algorithm for aggregation
scheduler_class = "RandomScheduler" # name of the scheduler class
task_model = "mnist"                # name of the task model for workers
batch_size = 64                     # batch size

Run a simulation

Run the following command :

uv run python main.py <env_name> <distrb_name> <model_name>

Add the flag --cpu to run on CPU and the flag --refresh to refresh the distribution of data if you need. By default, the program will choose the GPU if there is one, else it will run on CPU.

For instance :

uv run python main.py env20 iid fedavg --cpu --refresh

Results

For each experiment, data are distributed and saved in ./experiments folder as results. If the data was already generated, it can be generated again with the flag --refresh else, the program will skip the generation of data. The index of the name of the directory ./experiments/experiment-{k} increases by increments between each experiment. Almost all results are saved as .pkl file (see pickle) and can be opened easily :

import pickle
with open("data.pkl", "rb") as file:
    data = pickle.load(file)

print(data)

Report

The report summarizes :

1. A state of art of Federated Learning focused on Reinforcement Learning
2. More details on algorithms implemented
3. Results of several experiments