Getting Started

Installation

Installation from the package

You can install e2clab directly from pip:

pip install e2clab

Installing from the git repository

You can install e2clab directly from the git repository to get the latest updates.

cd <your_working_dir>
git clone https://gitlab.inria.fr/E2Clab/e2clab.git
cd e2clab
pip install -U -e .

Note

It’s a good practice to use a virtualenv or python version manager like pyenv or virtualenv to install e2clab and its dependencies.

Setting up a virtual environment

Here is an example using virtualenv.

# In your workstation
cd <your_working_dir>
virtualenv -p python3 venv
source venv/bin/activate
pip install e2clab

Create your ssh keys

To get access to the testbeds abailable in e2clab and their APIs, you will need to have a SHH key pair. Each testbed provides documentation and tutorials about how to setup your SSH keys with your accounts.

If you are not familiar with ssh keys and their usage, you can read about it [online](https://www.geeksforgeeks.org/introduction-to-sshsecure-shell-keys/).

Note

For examples, to generate a new rsa ssh keypair:

ssh-keygen -t rsa -f ~/.ssh/id_rsa -C "your.email@here.com"

Setting up testbeds

Please, refer to the tutorials below to setup connectivity to the different testbeds:

Checking Connectivity to testbeds

Once you have setup your access to the desired testbeds, you might want to check your setup.

e2clab gives you access to a command called check-testbeds which gives you a view of:

  • the installed providers (dependency check)

  • the connectivity to the various testbeds (connectivity check)

e2clab check-testbeds

If the connectivity to your configured testbed is valid, you’re ready to go !

Check the different tutorials.

Deploying your experiments

Below are some useful commands to deploy your first experiments usign e2clab. For a thorough documentation of the command-line interface, check the full documentation.

$ e2clab --help
Usage: e2clab [OPTIONS] COMMAND [ARGS]...

  Work with your ``e2clab`` experiment defined in SCENARIO_DIR and experiment
  artifacts in ARTIFACTS_DIR

Options:
  -d, --debug         Enable debug logging.
  -e, --mute_enoslib  Mute EnOSlib logging.
  -a, --mute_ansible  Mute Ansible spinning callback
  --help              Show this message and exit.

Commands:
  check-configuration  Checks configuration files syntax
  check-testbeds       Checks the connectivity to the various testbeds.
  deploy               Deploys scenarios or list of scenarios.
  destroy              Frees experiment testbed resources
  finalize             Finalize workflow and backup experiment data
  layers-services      Enforce Layers & Services in experiment environment
  network              Enforce communication rules in experiment...
  optimize             Optimize application workflow
  services             Manage E2clab services
  workflow             Enforce Workflow in experiment environment

Running an Experiment

e2clab deploy

Deploys scenarios or list of scenarios. When using ‘app_conf’, ‘prepare’ task from the workflow is only enforced once, independently from the ‘app_conf’ parameter

e2clab deploy [OPTIONS] SCENARIO_DIR ARTIFACTS_DIR

Options

--scenarios_name <scenarios_name>

Scenario names separated by comma.

--app_conf <app_conf>

Application configurations separated by comma.

--repeat <repeat>

Number of times to repeat the experiment.

--duration <duration>

Duration of each experiment in seconds.

--destroy

Run destroy after successful deploy

--destroy-on-fail

Run destroy if there is a workflow error

-p, --pause

Pause after launch step and duration completed, waiting for user input.

Arguments

SCENARIO_DIR

Required argument

ARTIFACTS_DIR

Required argument

Environment variables

E2C_SCENARIO_DIR

Provide a default for SCENARIO_DIR

E2C_ARTIFACTS_DIR

Provide a default for ARTIFACTS_DIR

  • Example: the following experiment will consist of four iterations --repeat 3, each one with a duration of four minutes --duration 240.

$ e2clab deploy --repeat 3 --duration 240 /path/to/scenario/ /path/to/artifacts/

Incremental Experiment Configuration

  • Layers & Services: Enforce layers and services in experiment environment.

$ e2clab layers-services /path/to/scenario/ /path/to/artifacts/

  • Network: Enforce network communication rules in experiment environment.

$ e2clab network /path/to/scenario/

  • Workflow: Enforce workflow in experiment environment prepare, launch, or finalize.

$ e2clab workflow /path/to/scenario/ prepare   # prepare workflow (i.e. copy dataset and libraries, create a Kafka topic, etc.)
$ e2clab workflow /path/to/scenario/ launch    # execute workflow (i.e. start data producers, submit Flink job, etc.)
$ e2clab workflow /path/to/scenario/ finalize  # backup workflow data (i.e. data generated from processes that compose the workflow)

Note

E2Clab first runs on all machines the prepare tasks. Then, the launch tasks on all machines, and finally the finalize tasks. Regarding the hosts order, it is top to down as defined by the users in the workflow.yaml file.

  • Finalize: Backup all experiment data, such as log files, monitoring data, etc.

$ e2clab finalize /path/to/scenario/

Your first deployment

Now that you have setup your e2clab installation, you can run your first e2clab experiment on grid5000 with this Flink Click Event Count example.