********************************************** Savanna Edge-to-Cloud: Grid'5000 + FIT IoT LAB ********************************************** .. figure:: savanna-edge-to-cloud/g5k+fitiot-logo.png :width: 60% :align: center .. .. list-table:: * - .. figure:: savanna-edge-to-cloud/grid5000.png :width: 60% :align: center - .. figure:: savanna-edge-to-cloud/logo-fit-iotlab.png :width: 60% :align: center .. contents:: :depth: 2 In this tutorial, we show how to analyze the performance of a real-life Edge-to-Cloud application deployed in the African savanna (illustrated in :ref:`xp_setup`). This application is composed of distributed Edge devices monitoring animal migration in the Serengeti region. Devices at the Edge collect and compress wildlife images, then the image is sent to the Cloud where the animal classification happens using a pre-trained Neural Network model. Finally, classified data helps conservationists to learn what management strategies work best to protect species. The **goals of these experiments** are to understand the impact on performance of `Cloud-centric` and `Hybrid (Edge+Cloud)` processing approaches. In this example **you will learn how to**: - Configure a cloud server on Grid'5000 testbed (animal classification using Deep Learning) - Configure an edge device on FIT IoT LAB testbed (collect and compress wildlife images, then send images to the cloud server) - Configure the network between the edge device and cloud server. - Execute experiments and analyze results .. _xp_setup: .. figure:: savanna-edge-to-cloud/savanna-setup.png :width: 70% :align: center Figure 1: Edge-to-Cloud application Experiment Artifacts ==================== .. code-block:: bash $ cd ~/git/ $ git clone https://gitlab.inria.fr/E2Clab/examples/savanna $ cd savanna/ $ ls artifacts # contains artifacts for the cloud server and the edge device g5k-fitiot # contains layers_services.yaml, network.yaml, and workflow.yaml Defining the Experimental Environment ===================================== Layers & Services Configuration ------------------------------- This configuration file presents the **layers** and **services** that compose this example. The **Cloud Server** (one machine ``quantity: 1`` in Grid'5000 ``environment: g5k``). The **Edge Client** (one Raspberry Pi 3 ``quantity: 1`` in FIT IoT LAB ``environment: iotlab``). .. literalinclude:: savanna-edge-to-cloud/g5k_fitiot/layers_services.yaml :language: yaml :linenos: .. **User-Defined Service:** add the following services in the ``~/git/e2clab/e2clab/services/`` directory. - `Server `_ - `Client `_ .. note:: We create a firewall rule on Grid'5000 to allow the **Cloud Server** to receive data from the **Edge Client** (FIT IoT LAB) in ``port 1883`` MQTT (Message Queuing Telemetry Transport) protocol. Network Configuration --------------------- The file below presents the network configuration between the ``cloud`` and ``edge`` infrastructures ``delay: 30ms, loss: 0.1%, rate: 512kbit``. .. literalinclude:: savanna-edge-to-cloud/g5k_fitiot/network.yaml :language: yaml :linenos: Workflow Configuration ---------------------- This configuration file presents the application workflow configuration. - **The Cloud Server** ``cloud.server.*``: ``prepare`` copies from the local machine to the remote machine the artifacts ``launch`` executes the Python application (animal classification using Deep Learning) ``finalize`` after experiment ends, copies the results from the remote to the local machine - **The Edge Client** ``edge.client.*``: ``prepare`` copies from the local machine to the remote machine the artifacts ``launch`` executes the client application (collect and compress wildlife images, then send images to the cloud server) .. literalinclude:: savanna-edge-to-cloud/g5k_fitiot/workflow.yaml :language: yaml :linenos: Understanding the parameters of **edge_worker.sh**: - **edge_data** is the topic name - **100** is the number of times the edge device will send images to the Cloud (every 30 seconds) - **True** means *Edge+Cloud processing approach* (**False** means *Cloud-only processing approach*) .. note:: Using ``depends_on`` on the **Edge Client** ``edge.client.*`` we can access the IPv6 address of the **Cloud Server** as follows ``{{ server.__address6__ }}``. Running & Verifying Experiment Execution ======================================== Find below the commands to deploy this application and check its execution. .. code-block:: bash $ e2clab deploy ~/git/savanna/g5k-fitiot/ ~/git/savanna/artifacts/ .. note:: For the first deployment, a good practice is to do it incrementally, as `explained here `_. **Firewall rule** - You can see the firewall rules created at https://api.grid5000.fr/stable/sites/````/firewall/````. **ssh to the Cloud server** .. code-block:: bash $ ssh root@paravance-40.rennes.grid5000.fr $ root@paravance-40:~# tail -f /tmp/predict.log **You may also check the mosquitto topic** You can use the Python script at ``~/git/savanna/artifacts/utils/mosquitto_sub_img.py`` to download images received. Images will be downloaded in the directory you run the script. In the example below, ``2001:660:4406:700:1::28`` is the IPv6 address of the cloud server. .. code-block:: bash $ python mosquitto_sub_img.py --topic edge_data --mqtt_broker 2001:660:4406:700:1::28 Deployment Validation & Experiment Results ========================================== Find below the files generated after the execution of each experiment. It consists of: - **validation files:** ``layers_services-validate.yaml``, ``network-validate/``, and ``workflow-validate.out`` - **experiment resutls:** for each experiment a new directory is generated ``20230623-113827/``. .. code-block:: bash $ ls ~/git/savanna/g5k-fitiot/20230623-113827/ layers_services-validate.yaml # Mapping between layers and services with physical machines network-validate/ # Network configuration for each physical machine workflow-validate.out # Commands used to deploy application (prepare, launch, and finalize) experiment-results/ # Experiment results - **Plotting the results:** you can use the ``~/git/savanna/artifacts/utils/plot_processing-time.py`` Python script to plot the results as in :ref:`plot_g5k`. Update the script according to your needs. .. _plot_g5k: .. figure:: ./savanna-edge-to-cloud/g5k_fitiot/plot-g5k-fit.png :width: 80% :align: center Figure 2: Cloud-centric vs Edge+Cloud processing