Flower Federated Learning on Scaleway

In this example we will be deploying a Federated Learning (FL) experiment with the Flower FL framework. In particular we will deploy this quickstart example leveraging Hugging Face’s Transformers.

If you are not familiar with the concept of Federated Learning, you can read this quick introduction: link.

This example is an extension of the previous flower example. But this time we will be running the clients on Scaleway instances equipped with GPUs.

Here is what the topology of the experiment that we are going to deploy looks like:

Warning

This example will deploy GPU resources on Scaleway, which can be expensive, please be mindful of your usage and keep an eye on the billing.

In this example you will learn to:

• Configure python virtual environments

• Check end-to-end execution

• Connect to nodes with E2Clab’s ssh command

• Run multiprovider experiments with Grid5000 and Scaleway Instances

• Use E2Clab’s .e2c_env env file

Experiment Artifacts

Download the experimental setup for this Flower Federated Learning experiment:

$ git clone https://gitlab.inria.fr/E2Clab/examples/flower-fl
$ cd flower-fl/

The structure of the experimental setup looks like this:

flower-fl/
├─quickstart-huggingface/   # ARTIFACTS_DIR
├─scaleway_g5k/             # SCENARIO_DIR
│   ├─.e2c_env
│   ├─layers_services.yaml
│   ├─network.yaml
│   └─workflow.yaml
├─README.md
└─...

Defining the Experimental Environment

Layers & Services Configuration

To communicate between both providers, we need to activate IPv6 and setup the firewall rules for Grid5000: https://www.grid5000.fr/w/Reconfigurable_Firewall

To setup the Scaleway provider we also need to activate IPv6 on the instances that we deploy so that we can communicate with the G5k node. We also set architecture as standalone as we don’t want to setup our instances in a private network behind a gateway but rather directly access the G5K server.

For the client nodes we request 2 nodes equipped with a single Nvidia L4 accelerator. We also need to specify that we want to deploy and OS with Nvidia drivers pre-installed: ubuntu_noble_gpu_os_13_nvidia.

You may need to spend some time to read Scaleway’s documentation to understand how to properly set those parameters.

You can explore how to change those those paramters by checking Scaleway’s API with the cli:

# List available OS image
$ scw marketplace image list
# List available commercial types
$ scw instance server-type list

Note

It is also important that you properly setup you credentials. If you used scw init to setup your credentials, E2Clab should be able to find your configuration file without the path. Also make sure to set the profile that you want to use from your configuration, if not E2Clab will default to the default profile and fail if it doesn’t exist.

You can learn more on how to configure scalewayinstances here.

environment:
  job_name: flower
  walltime: 01:00:00
  g5k:
    job_type: ["deploy"]
    cluster: parasilo
    env_name: "debian12-x64-big"
    ipv6: true
    # Supernode related ports
    firewall_rules:
      - services: ["Server"]
        ports: [9091, 9092, 9093]
  scalewayinstances:
    # Scaleway profile to use
    # config_file: path_to_config
    # profile: profilename
    architecture: standalone
    ipv6: true
layers:
- name: cloud
  services:
  - name: Server
    quantity: 1
    environment: g5k
- name: edge
  services:
  - name: Client
    # If you change the quantity, also think to change the
    # number of partitions in workflow.yaml edge section
    quantity: 2
    # Instance commercial type
    commercial_type: l4-1-24G
    # Deployed OS 
    image: ubuntu_noble_gpu_os_13_nvidia
    environment: scalewayinstances

Here we do not define new user-defined services and instead use the nodes as provided by the provider.

When the services names do not match any defined user defined services, you will see in the logs that E2Clab’s deploys Default services which basically that you get bare nodes as they were configured by the providers.

Network Configuration

In this experiment we do not define new network constraints.

networks:

Workflow Configuration

You can go through the file to understand how it works but here are the key steps:

• Download experiment artifacts to the nodes (the quickstart-huggingface directory)

• Install the dependencies

• Start the supernodes and the superlink which are flower’s servers and clients that help orchestrate the experiment, in a tmux session to make sure they keep running

• And in the launch step we juste have to run flwr run to start the whole experiment

• Once it’s done, we backup the logs and final model from the nodes to our experiment’s result directory ({{ working_dir }})

Notice that here we pass the address of the server deployed in Grid5000 to the clients in Scaleway with {{ server.url6 }} as Grid5000 resources are accessible from the outside only through IPv6. We also need to install cuda on the client nodes as Scaleway provides GPU OSs without it.

Note

Notice that we pass the --insecure argument to supernode and superlink, this means that we run those server without HTTPS. Of course this makes communications between the server unsafe. Here we don’t work with sensitive data or models and just provide an example workflow.

If you want to run experiments in a way that is more secure, take a look a flower’s documentation and adapt this example.

# Cloud
- hosts: cloud.server.*
  prepare:
    - name: Copy artifacts
      copy:
        src: "{{ working_dir }}/"
        dest: /tmp/flower_scenario/
    - name: Create venv
      shell:
        cmd: "apt install -y python3-venv tmux && python3 -m venv venv"
        chdir: /tmp/
    - name: Update pip
      command:
        cmd: "./venv/bin/python -m pip install -U pip"
        chdir: /tmp/
    - name: Install dependencies
      command:
        cmd: "./venv/bin/python -m pip install -e ./flower_scenario"
        chdir: /tmp/
        creates: /tmp/venv/bin/flower-superlink
    - name: Start flower superlink
      shell: 
        cmd: tmux new-session -d -s flower '. /tmp/venv/bin/activate && flower-superlink --insecure >> /tmp/superlink.log 2>&1'
        chdir: /tmp/
        creates: /tmp/superlink.log 
    - name: Ensure tmux session is running
      command: tmux has-session -t flower
      register: result
      failed_when: result.rc != 0
  launch:
    - name: Starting Flower run
      command:
        argv:
          - ./venv/bin/flwr
          - run
          - ./flower_scenario
          - deployment
          - --stream
        chdir: /tmp/
  finalize:
    - name: Stopping superlink
      shell:
        cmd: tmux kill-session -t flower
    - name: Backing up logs
      fetch:
        src: "/tmp/superlink.log"
        dest: "{{ working_dir }}/"
    - name: Backing up model
      fetch:
        src: "/tmp/final_model.pt"
        dest: "{{ working_dir }}/"

# Edge
- hosts: edge.client.*
  depends_on:
    - service_selector: "cloud.server.*"
      grouping: "round_robin"
      prefix: "server"
  prepare:
    - name: Install cuda
      shell:
        cmd: "apt-get update && apt-get -y install nvidia-cuda-toolkit"
    - name: Install venv
      shell:
        cmd: "apt-get update && apt-get install -y python3-venv"
    - name: Copy artifacts
      copy:
        src: "{{ working_dir }}/"
        dest: /tmp/flower_scenario/
    - name: Create venv
      shell:
        cmd: "python3 -m venv venv"
        chdir: /tmp/
    - name: Update pip
      command:
        cmd: "./venv/bin/python -m pip install -U pip"
        chdir: /tmp/
    - name: Install dependencies
      command:
        cmd: "./venv/bin/python -m pip install -e ./flower_scenario"
        chdir: /tmp/
        creates: /tmp/venv/bin/flower-supernode
    - name: Run superlink
      shell:
        cmd: tmux new-session -d -s flower '. /tmp/venv/bin/activate && flower-supernode --insecure --superlink {{ server.url6 }}:9092 --node-config "partition-id={{ _self.machine_id | int - 1 }} num-partitions=2" >> supernode.log 2>&1'
        chdir: /tmp/
        creates: /tmp/supernode.log 
    - name: Ensure tmux session is running
      command: tmux has-session -t flower
      register: result
      failed_when: result.rc != 0
  finalize:
    - name: Stop supernode
      shell:
        cmd: tmux kill-session -t flower
    - name: Backup supernode logs
      fetch:
        src: "/tmp/supernode.log"
        dest: "{{ working_dir }}/"

Running & Verifying Experiment Execution

You can deploy the whole experiment with a single deploy command:

$ cd scaleway_g5k
$ e2clab deploy

This will reserver the experiment’s computing resources, deploy the artifacts and run the whole workflow:

• prepare

• launch

• finalize

In that order

Note

You might notice that we did not specify the paths for ARTIFACTS_DIR or SCENARIO_DIR in the deploy command. They are actually specified in environment variables defined in the .e2c_env env file, as noted when you run e2clab –help.

E2C_SCENARIO_DIR="./"
E2C_ARTIFACTS_DIR="../quickstart-huggingface/"

# Examples
# E2C_DEBUG=false
# E2C_MUTE_ANSIBLE=false
# E2C_MUTE_ENOSLIB=false

Deployment Validation & Experiment Results

Compared to the previous example, this experiment will run in a few minutes thanks to the GPU computing power used.

While the launch step is running, you can ssh to some of the nodes to check the logs from the supernodes or the superlinks.

$ cd scaleway_g5k
$ e2clab ssh
# You can the select which host you want to ssh to
# cat the current logs
$ tail -f /tmp/super*.log | less

If you ssh to client nodes, you can check how the GPUs are used

Once the finalize step is done, logs and the finished model are downloaded to the result folder.

Free resources

Once you are absolutely done, you can free the compute nodes by killing the G5K job and terminating the Scaleway instances:

$ cd scaleway_g5k
$ e2clab destroy .