Getting Started

Installation

Setting up a virtual environment

# In your workstation
$ cd git/
$ virtualenv -p python3 venv
$ source venv/bin/activate
$ git clone https://gitlab.inria.fr/E2Clab/e2clab.git
$ cd e2clab
$ pip install -U -e .

Note

It’s a good practice to use a virtualenv or python version manager like pyenv.

Create your ssh keys

Skip this step if your keys already exist in ~/.ssh/.

$ ssh-keygen -q -t rsa -f ~/.ssh/id_rsa -N ""

Setting up testbeds

Please, refer to the tutorials below to setup connectivity to the different testbeds:

Checking Connectivity to testbeds

This command gives you a view of:

  • the installed providers (dependency check)

  • the connectivity to the various testbeds (connectivity check)

$ e2clab check-testbeds

E2Clab Command Line Interface

The commands allowed in E2Clab are presented below. Note that you can use just a single command to deploy your application deploy or you can deploy it separately using the commands layers-services, network, workflow (prepare, launch, or finalize), and finalize, respectively.

$ e2clab --help
Usage: e2clab [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  check-testbeds     Checks the connectivity to the various testbeds.
  deploy             Manage experiment deployment, execution, and backup.
    layers-services  Enforce layers and services in experiment environment.
    network          Enforce communication rules in experiment environment.
    workflow         Enforce workflow in experiment environment.
    finalize         Backup experiment data.
  optimize           Optimize application workflow.

Running Multiple Experiments

$ e2clab deploy --help
Usage: e2clab deploy [OPTIONS] SCENARIO_DIR ARTIFACTS_DIR

  Manage experiment deployment, execution, and backup.

Options:
  --scenarios_name TEXT  Scenario names separated by comma.
  --app_conf TEXT        Application configurations separated by comma.
  --repeat INTEGER       Number of times to repeat the experiment.
  --duration INTEGER     Duration of each experiment in seconds.
  --logging_level TEXT   Logging level.
  --help                 Show this message and exit.

  • Example: the following experiment will consist of four iterations --repeat 3, each one with a duration of four minutes --duration 240.

$ e2clab deploy --repeat 3 --duration 240 /path/to/scenario/ /path/to/artifacts/

Incremental Experiment Configuration

  • Layers & Services: Enforce layers and services in experiment environment.

$ e2clab layers-services /path/to/scenario/ /path/to/artifacts/

  • Network: Enforce network communication rules in experiment environment.

$ e2clab network /path/to/scenario/

  • Workflow: Enforce workflow in experiment environment prepare, launch, or finalize.

$ e2clab workflow /path/to/scenario/ prepare   # prepare workflow (i.e. copy dataset and libraries, create a Kafka topic, etc.)
$ e2clab workflow /path/to/scenario/ launch    # execute workflow (i.e. start data producers, submit Flink job, etc.)
$ e2clab workflow /path/to/scenario/ finalize  # backup workflow data (i.e. data generated from processes that compose the workflow)

Note

E2Clab first runs on all machines the prepare tasks. Then, the launch tasks on all machines, and finally the finalize tasks. Regarding the hosts order, it is top to down as defined by the users in the workflow.yaml file.

  • Finalize: Backup all experiment data, such as log files, monitoring data, etc.

$ e2clab finalize /path/to/scenario/