Experimental workflow

../_images/E2Clab-proposal-v2.png

Once your experiment setup is defined (see documentation) you are ready to run your experiment.

Enforcing your Layers & Services

Enforcing your experiment’s Layers & Services is possible using a single command that will pass your layers_services.yaml file to e2clab’s Layers and Services manager.

E2clab’s Layers and Services manager will interpret your layers_services.yaml file to reserve the physical resources on the defined testbeds and to install, configure and launch all services defined by the user.

Run the following command to initiate your experiment and enforce your layers and services :

e2clab layers-services /path/to/SCENARIO_DIR /path/to/ARTIFACTS_DIR

Once the command is done, e2clab will have created a env binary file in your SCENARIO_DIR that is used to store the current state of your experiment.

Note

That means that you now must always pass the same SCENARIO_DIR parameter to the following command or else e2clab won’t be able to retreive your experiment’s status and it will result in an error.

If you want to run an another experiment, you will need to re-run this command passing the SCENARIO_DIR pointing to your new experiment’s status.

Experiment folder

The command will also initiate your experiment’s result folder inside your SCENARIO_DIR with a name like YYYYmmdd-hhmmss where e2clab will output verification file and where you can output your experiment’s result using the working_dir variable in your workflow.yaml (doc). Your SCENARIO_DIR should look something like this

SCENARIO_DIR/
├─layers_services.yaml
├─network.yaml
├─workflow.yaml
├─env                   # experiment status
└─"YYYYmmdd-hhmmss"     # result folder
  └─...

layers_services-validate

A layers_services-validate.yaml file will be created in your result folder, detailing the deployment of your services (hosts tags and addresses, monitoring dashboard…).

Example layers_services.yaml file

 1---
 2environment:
 3  job_name: monitoring-tpg-g5k-iotlab
 4  walltime: "00:59:00"
 5  g5k:
 6    cluster: paradoxe
 7    job_type: ["allow_classic_ssh"]
 8  iotlab:
 9    cluster: grenoble
10monitoring:
11  type: tpg
12  provider: g5k
13  cluster: paradoxe
14  network: shared
15  ipv: 6
16layers:
17- name: cloud
18  services:
19  - name: MyServer
20    environment: g5k
21    roles: [monitoring]
22    quantity: 1
23- name: edge
24  services:
25  - name: MyClientA
26    environment: iotlab
27    cluster: grenoble
28    archi: a8:at86rf231
29    roles: [monitoring]
30    quantity: 1
31  - name: MyClientB
32    environment: iotlab
33    cluster: grenoble
34    archi: rpi3:at86rf233
35    roles: [monitoring]
36    quantity: 1

Example layers_services-validate.yaml file generated from the deployment of the previous file:

 1# '* * * * * * * Monitoring Service (started during workflow ''launch'' step)'
 2# 'Available at: http://localhost:3000'
 3# 'Access from your local machine: ssh -NL 3000:localhost:3000 paradoxe-2.rennes.grid5000.fr'
 4# 'username: admin / password: admin'
 5# '* * * * * * * Configure network.yaml and workflow.yaml using the information below!'
 6cloud:
 7- cloud.myserver.1.1:
 8  - paradoxe-7.rennes.grid5000.fr
 9edge:
10- edge.myclienta.1.1:
11  - node-a8-103.grenoble.iot-lab.info
12- edge.myclientb.2.1:
13  - node-rpi3-1.grenoble.iot-lab.info

Your SCENARIO_DIR should look something like this:

SCENARIO_DIR/
├─layers_services.yaml
├─network.yaml
├─workflow.yaml
├─env
├─...
└─"YYYYmmdd-hhmmss"
  ├─layers_services-validate.yaml
  └─...

CLI reference

Layers & Service CLI documentation

e2clab layers-services

Enforce Layers & Services in experiment environment

Usage

e2clab layers-services [OPTIONS] SCENARIO_DIR ARTIFACTS_DIR

Arguments

SCENARIO_DIR

Required argument

ARTIFACTS_DIR

Required argument

Environment variables

E2C_SCENARIO_DIR

Provide a default for SCENARIO_DIR

E2C_ARTIFACTS_DIR

Provide a default for ARTIFACTS_DIR

For more information on the whole command-line interface for e2clab check the full documentation.

Enforcing your Network

Once your have initiated your experiment and enforced your layers and services, you can enforce your network emulation defined in network.yaml with the following command :

e2clab network /path/to/SCENARIO_DIR

Once the command is done, you should find network validating files inside the network-validate folder located in your experiment’s result folder.

SCENARIO_DIR/
├─layers_services.yaml
├─network.yaml
├─workflow.yaml
├─env
├─...
└─"YYYYmmdd-hhmmss"
  ├─...
  └─network-validate
    └─...

CLI reference

Network CLI documentation

e2clab network

Enforce communication rules in experiment environment.

Usage

e2clab network [OPTIONS] SCENARIO_DIR

Arguments

SCENARIO_DIR

Required argument

Environment variables

E2C_SCENARIO_DIR

Provide a default for SCENARIO_DIR

For more information on the whole command-line interface for e2clab check the full documentation.

Enforcing your Workflow

Once your experiment’s Layers & Services and Network have been enforced, you are ready to launch your experiment’s Workflow.

In your workflow.yaml file, you should have defined up to three tasks for your experiment: prepare, launch and finlaize.

You can run each of these steps indpendently with the following commands:

prepare

To run the prepare task of your workflow (e.g. install dependencies, import configuration files):

e2clab workflow /path/to/SCENARIO_DIR prepare

launch

To run the launch task of your workflow (e.g. run a bash script):

e2clab workflow /path/to/SCENARIO_DIR launch

Note

When your launch task is done if you use the --finalize option, e2clab will automatically run the finalize task of your workflow where you should have defined the steps to back-up your experiment’s data. This allows you to run your launch task multiple times without having to worry about backing-up your data after each run.

e2clab workflow /path/to/SCENARIO_DIR launch --finalize

finalize

To run the finalize task of your workflow (e.g. backup your data):

e2clab workflow /path/to/SCENARIO_DIR finalize

workflow_validate

A workflow-validate.out file will be generated in your experiment’s result folder exposing ansible’s logging for each task defined in your workflow.yaml file.

Your SCENARIO_DIR should look something like this:

SCENARIO_DIR/
├─layers_services.yaml
├─network.yaml
├─workflow.yaml
├─env
├─...
└─"YYYYmmdd-hhmmss"
  ├─workflow-validate.out
  └─...

CLI reference

Workflow CLI documentation

e2clab workflow

Enforce Workflow in experiment environment

Usage

e2clab workflow [OPTIONS] SCENARIO_DIR {prepare|launch|finalize}

Options

--app_conf <app_conf>

Application configurations separated by comma.

--finalize

Automatically run finalize after launch

Arguments

SCENARIO_DIR

Required argument

TASK

Required argument

Environment variables

E2C_SCENARIO_DIR

Provide a default for SCENARIO_DIR

For more information on the whole command-line interface for e2clab check the full documentation.

Finalizing your experiment

The finalize command is a useful utility to back-up all your experiment’s information.

  • The Layers and Services manager will backup all monitoring and provenance data in your experiment folder.

  • The Workflow manager will run your workflow’s finalize tasks.

e2clab finalize /path/to/SCENARIO_DIR

CLI reference

Finalize CLI documentation

e2clab finalize

Finalize workflow and backup experiment data

Usage

e2clab finalize [OPTIONS] SCENARIO_DIR

Options

--destroy

Run destroy after finalize

Arguments

SCENARIO_DIR

Required argument

Environment variables

E2C_SCENARIO_DIR

Provide a default for SCENARIO_DIR

For more information on the whole command-line interface for e2clab check the full documentation.

Freeing your experiment’s resources

Once you are done with your experiment and your reservations our your testbeds have not reached the defined walltime you might want to free all the resources (e.g. stop your OAR job on grid5000). You can run the following command:

e2clab destroy /path/to/SCENARIO_DIR

CLI reference

Finalize CLI documentation

e2clab destroy

Frees experiment testbed resources

Usage

e2clab destroy [OPTIONS] SCENARIO_DIR

Options

--network

To only destroy network emulation

Arguments

SCENARIO_DIR

Required argument

Environment variables

E2C_SCENARIO_DIR

Provide a default for SCENARIO_DIR

For more information on the whole command-line interface for e2clab check the full documentation.

Automating your whole experiment

Running or automating all those previous commands can be cumbursome. The deploy command runs the Layers and Services and Network managers, and then enforces the prepare and launch tasks from your workflow, then waits for the provided --duration beforme finalizing the experiment (i.e. backing-up data and running the finalize task). deploy automatically runs your whole experiment.

You can repeat you experiment automatically using the --repeat option.

# Running 3 experiments with a duration of 2 minutes after 'launch'
e2clab deploy /path/to/SCENARIO_DIR /path/to/ARTIFACTS_DIR --repeat 3 -duration 120

CLI reference

Deploy CLI documentation

e2clab deploy

Deploys scenarios or list of scenarios. When using ‘app_conf’, ‘prepare’ task from the workflow is only enforced once, independently from the ‘app_conf’ parameter

Usage

e2clab deploy [OPTIONS] SCENARIO_DIR ARTIFACTS_DIR

Options

--scenarios_name <scenarios_name>

Scenario names separated by comma.

--app_conf <app_conf>

Application configurations separated by comma.

--repeat <repeat>

Number of times to repeat the experiment.

--duration <duration>

Duration of each experiment in seconds.

--destroy

Run destroy after successful deploy

--destroy-on-fail

Run destroy if there is a workflow error

-p, --pause

Pause after launch step and duration completed, waiting for user input.

Arguments

SCENARIO_DIR

Required argument

ARTIFACTS_DIR

Required argument

Environment variables

E2C_SCENARIO_DIR

Provide a default for SCENARIO_DIR

E2C_ARTIFACTS_DIR

Provide a default for ARTIFACTS_DIR

For more information on the whole command-line interface for e2clab check the full documentation.

Automating workflow configurations

app_conf parameter

You might find the way that the Workflow manager deals with the workflow.yaml configuration file too rigid to test different application flavors or deploying multiple experiments to test different parameters too long and redundant.

For example, you want to run a stress command on your hosts but you want to change the stress type between ‘cpu’ or ‘io’ when launching your workflow.

  • You don’t want to have to modify your workflow.yaml between e2clab workfow launch commands.

  • You don’t want to have to configure several experiments and have to re-deploy the infrastructure.

The e2clab deploy and e2clab workflow commands have a –app_conf arguments that allows the user to dynamically pass application configurations to the Workflow manager in the workflow.yaml in the {{ app_conf }} variable.

The following command and workflow.yaml file allows you to chain one experiment stressing the hosts cpus and ios:

# here we finalize (backup) the experiment between each launch with --finalize
e2clab workflow /path/to/SCENARIO_DIR/ launch --app_conf cpu,io --finalize

Also possibile using the deploy command:

e2clab deploy /path/to/SCENARIO_DIR /path/to/ARTIFACTS_DIR --app_conf cpu,io

 1# FLINK
 2- hosts: cloud.*
 3  depends_on:
 4    - service_selector: fog.*
 5      grouping: "round_robin"
 6      prefix: "_fog"
 7  vars:
 8    stress_name: "stress"
 9  prepare:
10    - debug:
11        msg: "Installing {{ stress_name }}"
12    - shell: "sudo apt-get update && sudo apt-get install -y stress"
13  launch:
14    - debug:
15        # {{ app_conf }} will first take the value 'cpu' during the first launch
16        # and 'io' during the second
17        msg: "Starting stress {{ app_conf }}"
18    - shell: "stress --{{ app_conf }} 4 --timeout 60s > /tmp/exp.txt"
19  finalize:
20    - fetch:
21        src: "/tmp/exp.txt"
22        dest: "{{ working_dir }}/experiment_results/"
23        validate_checksum: no
24- hosts: fog.*
25  prepare:
26    - debug:
27        msg: "Installing stress"
28    - shell: "sudo apt-get update && sudo apt-get install -y stress"
29  launch:
30    - debug:
31        msg: "Starting {{ app_conf }} stress"
32    - shell: "stress --{{ app_conf }} 4 --timeout 60s"

workflow_env.yaml and application configuration

You might also find the app_conf too limiting and want to dynamically set multiple variables in your workflow.yaml.

The values passed to app_conf can be used as keys to fetch parameters in an extra configuration file called workflow_env.yaml that will be available with the env_ prefix.

For example you want to run a stress command on your hosts but you want to change the number of stress “workers” deployed in our experiment and the “timeout” of the command.

Our experiment setup might look something like this:

SCENARIO_DIR/
├─layers_services.yaml
├─network.yaml
├─workflow.yaml
├─workflow_env.yaml
└─...

workflow_env.yaml:

1# app_conf parameters must match either 'cpu' or 'io'
2cpu:
3  workers: 4    # {{ env_workers }} in 'workflow.yaml'
4  timeout: 60s  # {{ env_timeout }} in 'workflow.yaml'
5
6io:
7  workers: 2    # {{ env_workers }} in 'workflow.yaml'
8  timeout: 120s # {{ env_timeout }} in 'workflow.yaml'

workflow.yaml:

 1# FLINK
 2- hosts: cloud.*
 3  depends_on:
 4    - service_selector: fog.*
 5      grouping: "round_robin"
 6      prefix: "_fog"
 7  vars:
 8    stress_name: "stress"
 9  prepare:
10    - debug:
11        msg: "Installing {{ stress_name }}"
12    - shell: "sudo apt-get update && sudo apt-get install -y stress"
13  launch:
14    - debug:
15        # {{ app_conf }} will first take the value 'cpu' during the first launch
16        # and 'io' during the second
17        msg: "Starting stress {{ app_conf }}"
18    - shell: "stress --{{ app_conf }} {{ env_workers }} --timeout {{ env_timeout }} > /tmp/exp.txt"
19  finalize:
20    - fetch:
21        src: "/tmp/exp.txt"
22        dest: "{{ working_dir }}/experiment_results/"
23        validate_checksum: no
24- hosts: fog.*
25  prepare:
26    - debug:
27        msg: "Installing stress"
28    - shell: "sudo apt-get update && sudo apt-get install -y stress"
29  launch:
30    - debug:
31        msg: "Starting {{ app_conf }} stress"
32    - shell: "stress --{{ app_conf }} {{ env_workers }} --timeout {{ env_timeout }}"