Flower Federated Learning

../_images/flower-logo.png ../_images/huggingface-1.svg

In this example we will be deploying a Federated Learning (FL) experiment with the Flower FL framework. In particular we will deploy this quickstart example leveraging Hugging Face’s Transformers.

If you are not familiar with the concept of Federated Learning, you can read this quick introduction: link.

Here is what the topology of the experiment that we are going to deploy looks like:

graph "Grid5000 deployment graph" { subgraph cluster_grid5000 { label="Grid5000" subgraph cluster_cloud { label="Cloud" Superlink [ label="cloud.server.1.1\nSuperlink" ] } subgraph cluster_edge { label="Edge" Supernode1 [ label="edge.client.1.2\nSupernode" ] Supernode2 [ label="edge.client.1.1\nSupernode" ] } Superlink -- Supernode1 Superlink -- Supernode2 } }

In this example you will learn to:

  • Configure python virtual environments

  • Check end-to-end execution

  • Connect to nodes with E2Clab’s ssh command

  • Use E2Clab’s .e2c_env env file

Experiment Artifacts

Download the experimental setup for this Flower Federated Learning experiment:

$ git clone https://gitlab.inria.fr/E2Clab/examples/flower-fl
$ cd flower-fl/

The structure of the experimental setup looks like this:

flower-fl/
├─quickstart-huggingface/   # ARTIFACTS_DIR
├─g5k/                      # SCENARIO_DIR
│   ├─.e2c_env
│   ├─layers_services.yaml
│   ├─network.yaml
│   └─workflow.yaml
├─README.md
└─...

Defining the Experimental Environment

Layers & Services Configuration

 1environment:
 2  job_name: flower
 3  walltime: 01:00:00
 4  g5k:
 5    job_type: ["deploy"]
 6    cluster: parasilo
 7    env_name: "debian12-x64-big"
 8layers:
 9- name: cloud
10  services:
11  - name: Server
12    quantity: 1
13- name: edge
14  services:
15  - name: Client
16    # If you change the quantity, also think to change the
17    # number of partitions in workflow.yaml edge section
18    quantity: 4

Here we do not define new user-defined services and instead use the nodes as provided by the provider.

When the services names do not match any defined user defined services, you will see in the logs that E2Clab’s deploys Default services which basically that you get bare nodes as they were configured by the providers.

Network Configuration

In this experiment we do not define new network constraints, but you could add some if you want by taking inspiration from the comments.

1networks:
2# Layer-level network configuration
3# - src: edge
4#   dst: cloud
5#   delay: "5ms"
6#   rate: "1gbit"
7#   loss: "2%"

Workflow Configuration

You can go through the file to understand how it works but here are the key steps:

  • Download experiment artifacts to the nodes (the quickstart-huggingface directory)

  • Install the dependencies

  • Start the supernodes and the superlink which are flower’s servers and clients that help orchestrate the experiment, in a tmux session to make sure they keep running

  • And in the launch step we juste have to run flwr run to start the whole experiment

  • Once it’s done, we backup the logs and final model from the nodes to our experiment’s result directory ({{ working_dir }})

Note

Notice that we pass the --insecure argument to supernode and superlink, this means that we run those server without HTTPS. Of course this makes communications between the server unsafe. Here we don’t work with sensitive data or models and just provide an example workflow and we stay within Grid5000’s network.

If you want to run experiments in a way that is more secure, take a look a flower’s documentation and adapt this example.

 1# Cloud
 2- hosts: cloud.server.*
 3  prepare:
 4    - name: Copy artifacts
 5      copy:
 6        src: "{{ working_dir }}/"
 7        dest: /tmp/flower_scenario/
 8    - name: Create venv
 9      shell:
10        cmd: "python -m venv venv"
11        chdir: /tmp/
12    - name: Update pip
13      command:
14        cmd: "./venv/bin/python -m pip install -U pip"
15        chdir: /tmp/
16    - name: Install dependencies
17      command:
18        cmd: "./venv/bin/python -m pip install -e ./flower_scenario"
19        chdir: /tmp/
20        creates: /tmp/venv/bin/flower-superlink
21    - name: Start flower superlink
22      shell: 
23        cmd: tmux new-session -d -s flower '. /tmp/venv/bin/activate && flower-superlink --insecure >> /tmp/superlink.log 2>&1'
24        chdir: /tmp/
25        creates: /tmp/superlink.log 
26    - name: Ensure tmux session is running
27      command: tmux has-session -t flower
28      register: result
29      failed_when: result.rc != 0
30  launch:
31    - name: Starting Flower run
32      command:
33        argv:
34          - ./venv/bin/flwr
35          - run
36          - ./flower_scenario
37          - deployment
38          - --stream
39        chdir: /tmp/
40  finalize:
41    - name: Stopping superlink
42      shell:
43        cmd: tmux kill-session -t flower
44    - name: Backing up logs
45      fetch:
46        src: "/tmp/superlink.log"
47        dest: "{{ working_dir }}/"
48    - name: Backing up model
49      fetch:
50        src: "/tmp/final_model.pt"
51        dest: "{{ working_dir }}/"
52
53# Edge
54- hosts: edge.client.*
55  depends_on:
56    - service_selector: "cloud.server.*"
57      grouping: "round_robin"
58      prefix: "server"
59  prepare:
60    - name: Copy artifacts
61      copy:
62        src: "{{ working_dir }}/"
63        dest: /tmp/flower_scenario/
64    - name: Create venv
65      shell:
66        cmd: "python -m venv venv"
67        chdir: /tmp/
68    - name: Update pip
69      command:
70        cmd: "./venv/bin/python -m pip install -U pip"
71        chdir: /tmp/
72    - name: Install dependencies
73      command:
74        cmd: "./venv/bin/python -m pip install -e ./flower_scenario"
75        chdir: /tmp/
76        creates: /tmp/venv/bin/flower-supernode
77    - name: Run superlink
78      shell:
79        cmd: tmux new-session -d -s flower '. /tmp/venv/bin/activate && flower-supernode --insecure --superlink {{ server.__address__ }}:9092 --node-config "partition-id={{ _self.machine_id | int - 1 }} num-partitions=4" >> supernode.log 2>&1'
80        chdir: /tmp/
81        creates: /tmp/supernode.log 
82    - name: Ensure tmux session is running
83      command: tmux has-session -t flower
84      register: result
85      failed_when: result.rc != 0
86  finalize:
87    - name: Stop supernode
88      shell:
89        cmd: tmux kill-session -t flower
90    - name: Backup supernode logs
91      fetch:
92        src: "/tmp/supernode.log"
93        dest: "{{ working_dir }}/"

Running & Verifying Experiment Execution

You can deploy the whole experiment with a single deploy command:

$ cd g5k
$ e2clab deploy

This will reserver the experiment’s computing resources, deploy the artifacts and run the whole workflow.

Note

You might notice that we did not specify the paths for ARTIFACTS_DIR or SCENARIO_DIR in the deploy command. They are actually specified in environment variables defined in the .e2c_env env file, as noted when you run e2clab --help.

1E2C_SCENARIO_DIR="./"
2E2C_ARTIFACTS_DIR="../quickstart-huggingface/"
3
4# Examples
5# E2C_DEBUG=false
6# E2C_MUTE_ANSIBLE=false
7# E2C_MUTE_ENOSLIB=false

Deployment Validation & Experiment Results

The experiment may take a few dozens of minutes to run on those CPU nodes.

While the launch step is running, you can ssh to some of the nodes to check the logs from the supernodes or the superlinks.

$ cd g5k
$ e2clab ssh
# You can the select which host you want to ssh to
# cat the current logs
$ tail -f /tmp/super*.log | less

Once the finalize step is done, logs and the finished model are downloaded to the result folder.

Free resources

Once you are absolutely done, you can free the compute nodes by killing the G5K job:

$ cd g5k
$ e2clab destroy .