Experimental setup

E2Clab’s experimental setup follows a 3-step methodology.

../_images/E2Clab-methodology.png

Figure 1: E2Clab experiment methodology.

  • Layers & Services

    • Services represent any system that provides a specific functionality or action in the scenario workflow.

    • Layers define the hierarchy between services and groups them with different granularities. They can also be used to reflect the geographical distribution of the compute resources. In the context of the Computing Continuum, layers refers to Edge, Fog and Cloud for instance.

  • Network

    • Specifies the network communication rules between layers and between services

  • Workflow

    • Specifies all the execution logic and rules of the software, algorithms and applications running on services.

Those 3 steps are respectively defined in the following YAML files :

Each of these configuration files will be parsed by e2clab and used by the respective manager within e2clab.

Those three files must be present in the same folder, which later in the documentation will be refered to as SCENARIO_DIR or scenario.

SCENARIO_DIR/
├─layers_services.yaml
├─network.yaml
├─workflow.yaml
└─...

To know more about YAML files and their syntax, you can check Ansible’s YAML Syntax guide.

Schemas

JSON schemas (jsonschema documentation) are utilized within e2clab to validate the syntax and structure of the YAML configuration files. They are a powerful way to ensure that those YAML files adhere to the expected format.

You don’t need to understand them but you can use them as a reference for the YAML files syntax and for parameter meanings and possible values.

Layers & Services

The layers and services of the experiment are defined in a layers_services.yaml file. In the computing-continuum field, those layers are usually called ‘cloud’, ‘fog’ and ‘edge’ each hosting several services.

Environments

The environment section of our layers_services.yaml file describes the testbeds used to run our e2clab experiment on. Single-testbed or mumtiple testbeds experiments are possible on the available testbeds :

For a multi-testbed development setup, you can check this example.

Layers

The layers section of our layers_services.yaml file describes the hierarchy between services and groups them.

Services

To know more about how services work in e2clab and how you can define you own services, check the documentation.

If you have user-defined services installed that you wish to use in your Layer & Services definiton, the name of your service in the layers_services.yaml must match (case-sensitive) the name of your user-defined service. If the name of your service in your layers_services.yaml file doesn’t match any user-defined service, a Default service will be deployed, meaning a bare-metal node from your configured testbed.

You can deploy services on different testbeds in a same experiment (grid5000 for cloud services and FIT IoT-LAB for edge services for example). Check the documentation to see how to deploy a user-defined service and how to pass metadata to your services.

Monitoring

Check the monitoring documentation

Provenance

Check the provenance documentation

Syntax

Here is a dummy layers_services.yaml file for a e2clab experiment exposing several configuration parameters and their meaning. For more information on each parameter, check the schema definition.

  1---
  2########################################
  3# Environment to run the experiment on #
  4########################################
  5environment:
  6  job_name: my-experiment
  7  walltime: 01:00:00
  8  # keyword 'reservation' is only used by g5k but can be defined here for example
  9  reservation: "2022-02-23 19:00:00"
 10  # Definition of each provider / testbed
 11  # grid5000 testbed
 12  g5k:
 13    job_type: [deploy, exotic]
 14    env_name: debian10-x64-big
 15    cluster: chiclet
 16    # Reconfigurable firewall on G5K allows you to open some ports of some of your Services.
 17    # Allows connections from other environments (e.g., FIT IoT, Chameleon, etc.) to G5K.
 18    firewall_rules:
 19    - services: ["Master", "Producer"]
 20      ports: [22000]
 21  # FIT IoT-LAB testbed
 22  iotlab:
 23    cluster: "saclay"
 24  # ChameleonCloud testbed
 25  chameleoncloud:
 26    key_name: my-chameleonloud-key
 27    rc_file: "/path/to/my-app-cred-cloud-openrc.sh"
 28    image: CC-Ubuntu20.04
 29    cluster: "gpu_rtx_6000"
 30  # ChameleonEdge testbed
 31  chameleonedge:
 32    cluster: "jetson-nano"
 33    rc_file: "/path/to/my-app-cred-edge-openrc.sh"
 34############################## monitoring
 35monitoring:
 36  provider: g5k OR chameleoncloud  # provider is 1 machine [ui,collector]
 37  ########
 38  cluster: paradoxe
 39  # OR
 40  servers: ["chifflot-7.lille.grid5000.fr"]
 41  ########
 42  network: shared OR private (in G5K)
 43  ipv: 6
 44  type: tig OR tpg OR dstat
 45  agent_conf: telegraf.conf.j2 # optional for tig
 46  options: "-m -c -n"  # optional for dstat
 47monitoring_iotlab:
 48  profiles:
 49    - name: test_profile_a8
 50      archi: a8  # ['a8', 'custom']
 51      current: True
 52      power: True
 53      voltage: True
 54      period: 8244  # [140, 204, 332, 588, 1100, 2116, 4156, 8244]
 55      average: 4  # [1, 4, 16, 64, 128, 256, 512, 1024]
 56############################## provenance
 57provenance:
 58  provider: g5k OR chameleoncloud  # provider is 1 machine [ui,collector]
 59  ########
 60  cluster: paradoxe
 61  # OR
 62  servers: [ "chifflot-7.lille.grid5000.fr" ]
 63  ########
 64  dataflow_spec: path/to/dataflow-specification.py
 65  ipv: 6
 66  parallelism: 1
 67########################################################
 68# Define the hierarchy between services and group them #
 69# in different granularities                           #
 70########################################################
 71layers:
 72- name: cloud
 73  services:
 74  - name: Master
 75    environment: g5k # Environment to deploy the service on
 76    roles: [monitoring]
 77############################## coarse-grained submission ##############################
 78    cluster: paradoxe
 79    quantity: 2
 80    repeat: 2
 81############################## fine-grained submission ###########################
 82    servers: ["chifflot-7.lille.grid5000.fr", "chifflot-8.lille.grid5000.fr"]
 83#----------------------------------------------------------------------------------------
 84  - name: Server
 85    environment: chameleoncloud
 86    roles: [monitoring]
 87############################## coarse-grained submission ##############################
 88    cluster: gpu_rtx_6000  # flavour
 89    quantity: 1            # number
 90    repeat: 1
 91#----------------------------------------------------------------------------------------
 92- name: edge
 93  services:
 94  - name: Producer
 95    environment: iotlab
 96    image: data/firmware/custom/contikimac/er-example-server.iotlab-m3  # for 'm3' nodes
 97    repeat: 2
 98############################## monitoring
 99    profile: prov_capture_a8  # OR
100    roles: [monitoring]  # Telegraf/Prometheus/Grafana OR Telegraf/InfluxDB/Grafana stack
101############################## coarse-grained submission ##############################
102    cluster: grenoble     # site
103    archi: a8:at86rf231   # archi
104    quantity: 2           # number
105############################## fine-grained submission ###########################
106    servers: ["a8-1.grenoble.iot-lab.info", "a8-2.grenoble.iot-lab.info"]  # hostname
107#----------------------------------------------------------------------------------------

Schema

configuration jsonschema

Experiment Layers & Services description

type

object

properties

  • environment

environment

  • layers

Experiment layers definition.

type

array

items

type

object

properties

  • name

Name of the layer, e.g. edge, fog, cloud…

type

string

  • services

service_list

  • provenance

Provenance manager

  • monitoring

Monitoring manager

  • monitoring_iotlab

MonitoringIoTManager

  • kwollect

Grid5000 kwollect monitoring manager

additionalProperties

False

service_list

Description of the service to be deployed on our layer.

type

array

items

type

object

properties

  • name

Name of the service. If the name matches a user-defined service name, this service will be deployed.

type

string

  • quantity

Number of nodes the service will be deployed on

type

number

  • roles

type

array

  • repeat

Number of times the service definition will be duplicated.

type

number

  • env

Service metadata.

type

object

  • environment

Environment on which the service will be deployed.If set, you can specify other environment-specific propertiesfrom the environment you chose

type

string

enum

g5k, iotlab, chameleoncloud, chameleonedge

  • cluster

Cluster to deploy the service to

type

string

  • servers

Server to deploy services, overwrites cluster definition

examples

chifflot-7.lille.grid5000.fr

chifflot-8.lille.grid5000.fr

  • archi

Host architecture (used for FIT IoT-LAB)

type

string

environment

Definition of experiment environments

type

object

properties

  • job_name

Name of our experiment

type

string

  • walltime

Walltime for our experiment, in format hh:mm:ss

type

string

pattern

^([01]\\d|2[0-3]):([0-5]\\d):([0-5]\\d)$

  • cluster

Which cluster to deploy on

type

string

  • g5k

Grid5000 configuration

g5k

  • iotlab

FIT IoT-LAB configuration

iotlab

  • chameleoncloud

ChameleonCloud configuration

chameleoncloud

  • chameleonedge

ChameleonEdge configuration

chameleonedge

additionalProperties

False

g5k

type

object

properties

  • job_name

Name of the job on the testbed

type

string

default

E2Clab

  • walltime

Walltime for our experiment, in format hh:mm:ss

type

string

pattern

^([01]\\d|2[0-3]):([0-5]\\d):([0-5]\\d)$

  • queue

Grid5000 queue to use

type

string

enum

default, production, besteffort, testing

default

default

  • reservation

reservation date in YYYY-mm-dd HH:MM:SS format

type

string

pattern

^\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}$

  • job_type

OAR job type

type

array

default

  • env_name

The kadeploy3 environment to use (deploy only)

type

string

  • cluster

Which G5k cluster to use

type

string

  • key_name

SSH public key to use

type

string

default

/root/.ssh/id_rsa.pub

  • firewall_rules

G5k firewall rules

type

array

items

type

object

properties

  • services

List of services to open ports for

type

array

  • ports

List of ports to open

type

array

  • proto

Protocol to use (tcp, udp)

type

string

enum

tcp+udp, all

default

tcp+udp

  • monitor

Activate on demand metrics (e.g.`prom_.*`)

type

string

  • ipv6

Enable IPv6 on g5k hosts

type

boolean

iotlab

type

object

properties

  • job_name

Name of the job on the testbed

type

string

default

E2Clab

  • walltime

Walltime for our experiment, in format hh:mm:ss

type

string

pattern

^([01]\\d|2[0-3]):([0-5]\\d):([0-5]\\d)$

  • cluster

Iotlab cluster to use

type

string

default

saclay

chameleoncloud

type

object

properties

  • job_name

Name of the job on the testbed

type

string

default

E2Clab

  • walltime

Walltime for our experiment, in format hh:mm:ss

type

string

pattern

^([01]\\d|2[0-3]):([0-5]\\d):([0-5]\\d)$

  • rc_file

Openstack environment rc file path

type

string

  • key_name

SSH pub key

type

string

  • image

Cloud image to use

type

string

default

CC-Ubuntu20.04

  • cluster

type

string

chameleonedge

type

object

properties

  • job_name

Name of the job on the testbed

type

string

default

E2Clab

  • walltime

Walltime for our experiment, in format hh:mm:ss

type

string

pattern

^([01]\\d|2[0-3]):([0-5]\\d):([0-5]\\d)$

  • rc_file

Openstack environment rc file path

type

string

  • key_name

SSH pub key

type

string

  • image

Chameleon edge image to use

type

string

default

ubuntu

  • cluster

type

string

default

jetson-nano

Provenance manager

Definition of the provenance data capture capabilities

type

object

properties

  • provider

Testbed to deploy the provenance service: G5k only

type

string

enum

g5k, iotlab, chameleoncloud, chameleonedge

  • cluster

Cluster where the provenence server will be running

type

string

  • servers

Machine where the provenance server will be running

type

array

items

type

string

maxItems

1

  • dataflow_spec

User-defined dataflow specifications

type

string

default

  • ipv

IP network version to transmit provenance data

type

number

enum

4, 6

default

4

  • parallelism

Parallelizes the prov data translator and broker topic

type

number

default

1

Monitoring manager

Definition of the monitoring capabilities

type

object

properties

  • type

Type of monitoring deployed on experiment

type

string

enum

dstat, tpg, tig

  • provider

Dedicated machine hosting InfluxDB and Grafana

type

string

enum

g5k, chameleoncloud

  • cluster

Cluster on which to deploy the machine

type

string

  • servers

Optional if cluster defined. Machine on which to deploy the machine

type

array

  • network

Define network for the monitoring service.

type

string

enum

shared, private

default

shared

  • agent_conf

Config file in ‘artifacts_dir’ for the monitor agent

type

string

  • options

Dstat monitoring options

type

string

default

-m -c -n

  • ipv

Type of network the monitoring provider will use.

type

number

enum

4, 6

oneOf

properties

  • type

enum

tpg, tig

oneOf

properties

  • provider

enum

g5k

properties

  • provider

enum

chameleoncloud

properties

  • type

enum

dstat

MonitoringIoTManager

FIT IoT-Lab monitoring profiles

type

object

properties

  • profiles

FIT IoT-LAB monitoring manager

https://www.iot-lab.info/testbed/resources/monitoring

type

array

items

type

object

properties

  • name

Name of the monitoring profile.

type

string

  • archi

type

string

enum

a8, m3, custom

  • current

Enable current monitoring

type

boolean

default

False

  • power

Enable power monitoring

type

boolean

default

False

  • voltage

Enable voltage monitoring

type

boolean

default

False

  • period

Sampling period (µs)

type

number

enum

140, 204, 332, 588, 1100, 2116, 4156, 8244

default

8244

  • average

Monitoring samples averaging window.

type

number

enum

1, 4, 16, 64, 128, 256, 512, 1024

default

4

Grid5000 kwollect monitoring manager

type

object

properties

  • metrics

Metrics to pull from job, ‘[all]’ to pull all metrics

type

array

items

type

string

  • step

Workflow step to monitor

type

string

enum

prepare, launch, finalize, launch

default

launch

  • start

Workflow step to start monitor at. Mandatory if ‘end’ is set

type

string

enum

prepare, launch, finalize, launch

  • end

Workflow step to end monitor at

type

string

enum

prepare, launch, finalize, launch

dependencies

  • end

start

Network

e2clab has the capacity to emulate network constraints between layers, or any subset of services. Those constraints are defined in a single network.yaml configuration file. For each network, users may set parameters like:

  • delay

  • rate

  • loss

This is a usefull functionality to simulate lossy networks between edge and cloud services for example.

This network emulation relies on EnOSlib’s network emulation service.

Syntax

Here is a dummy network.yaml file for a e2clab experiment. For more information on each parameter, check the schema definition.

 1---
 2#########################################
 3# Network definition for the experiment #
 4#########################################
 5networks:
 6# List of network emulations
 7- src: cloud  # Required
 8  dst: fog    # Required
 9  delay: 10ms
10  rate: 1gbit
11  loss: 1%
12  symmetric: false
13- src: fog.gateway.*    # Required
14  dst: edge   # Required
15  delay: 100ms
16  rate: 50kbits
17  loss: 10%
18  symmetric: true

The network topology and emulation is defined in the network.yaml file.

Schema

configuration jsonschema

Experiment Network description

type

object

properties

  • networks

type

array / null

items

Network emulation

additionalProperties

False

Network emulation

type

object

properties

  • src

Source services name

type

string

examples

cloud

cloud.kafka.*

  • dst

Destination services name

type

string

  • delay

The delay to apply

type

string

examples

10ms

1ms

  • rate

The rate to apply

type

string

examples

1gbit

100mbit

  • loss

The percentage of loss

type

string

examples

1%

5%

pattern

\\d*.?\\d*%

  • symmetric

True for symmetric rules to be applied

type

boolean

additionalProperties

False

Workflow

The workflow of the experiment (i.e. the orchestration of your application) is defined in the workflow.yaml file. e2clab relies heavily on ansible to enforce commands on the remote hosts.

Hosts

Hosts refer to remote hosts used to deploy the experiment services (libraries, applications…) and to enforce the workflow. They can be selected according to their tags. You can also find those tags in the layers_services-validate file.

Depends on

The depends-on attribute allows users to map services in a reproducible and scalable way. The mapping strategy is described by the grouping type. To know more about how those mapping strategies work and how services can access other services parameters (such as a url), check the services relationships documentation

Deployment

Users can manage the deployment of their application in three well defined steps

  • prepare: install software, dependencies or populate hosts with the experiment’s input .

  • launch: launch experiment.

  • finalize: stop experiment, clean environment and fetch experiment artifacts from the hosts.

Each of those steps are an ansible task definition like you would in a playbook, so if you are already familiar with the syntax this should be easy.

Variables

Several variables are exposed by e2clab inside your workflow.yaml file to help you manage you experiment:

  • scenario_dir: points to SCENARIO_DIR passed in the command line

  • artifacts_dir: points to ARTIFACTS_DIR passed in the command line, usefull to manade input data and load hosts with scripts.

  • working_dir:

    • points to SCENARIO_DIR during prepare and launch steps

    • points to your experiment’s output dir during finalize step. Helpfull to backup output data.

  • app_conf: see this section

Other variables exposing services to each-other are available, check how to access variables from other services and how to access service’s own variables.

Syntax

Here is an example workflow.yaml file for a e2clab experiment. For more information on each parameter, check the schema definition.

 1---
 2#########################################
 3# Workflow to run during the experiment #
 4#########################################
 5# SERVER
 6- hosts: cloud.server.*
 7  # 'prepare' ansible task definition
 8  prepare:
 9    - copy:
10        src: "{{ working_dir }}/artifacts_cloud/"
11        dest: /tmp
12  # 'launch' ansible task definition
13  launch:
14    - debug:
15        msg: "Running the server"
16    # runnig shell script
17    - shell: cd /tmp && bash cloud_worker.sh > cloud-log.log
18    - shell: python3 /tmp/predict_loop.py
19      async: 3120
20      poll: 0
21  # 'finalize' ansible task definition
22  finalize:
23    - debug:
24        msg: "Saving results"
25    # fetching experiment results
26    - fetch:
27        src: /tmp/predict.log
28        dest: "{{ working_dir }}/experiment-results/"
29        validate_checksum: no
30# CLIENT
31- hosts: edge.client.*
32  # Define service interconnection
33  depends_on:
34    - service_selector: cloud.server.*
35      grouping: "round_robin"
36      prefix: server
37  # 'prepare' ansible task definition
38  prepare:
39    - copy:
40        src: "{{ working_dir }}/artifacts_edge/"
41        dest: /
42  # 'launch' ansible task definition
43  launch:
44    - debug:
45        msg: "Running the client"
46    # running script taking the 'server.gateway' parameter defined from the 'depends_on' section.
47    - shell: "bash edge_worker.sh edge_data 100 {{ server.gateway }} True"

Schema

configuration jsonschema

Non-described properties will be passed to ansible in a play.

type

array

items

type

object

properties

  • hosts

hosts description on which to execute workflow

type

string

  • depends_on

depends_on

  • prepare

Ansible task definition.

type

array

  • launch

Ansible task definition.

type

array

  • finalize

Ansible task definition.

type

array

depends_on

Description of hosts interconnections

type

array

items

type

object

properties

  • service_selector

type

string

  • grouping

Grouping strategy between hosts, defaults: round_robin

type

string

enum

round_robin, asarray, aggregate, address_match

  • prefix

Prefix to access linked hosts parameters

type

string

  • array_info

Information passed in the array

type

string

enum

id, hostname, g5k_ipv6_hostname

default

id