User-Defined Services (relationships)
In distributed infrastructures, the various services that compose the application workflow
may need to communicate with each other to exchange data and execute tasks. Therefore, to
interconnect such services and define their relationships, E2Clab provides the idea of
depends_on
and various Grouping
strategies.
depends_on
The structure of depends_on
is as follows:
1 - hosts: # refer to the services that need information from other services
2 depends_on:
3 - service_selector: # refers to the other services
4 grouping: # refers to the strategy used to interconnect services
5 prefix: # refers to the prefix used to access information from other services
Users may use depends_on
to access information from another service. In the
example below, using the prefix
named myservice_master
all workers
cloud.myservice.1.worker.*
can access the extra information from the master
cloud.myservice.1.master.1
, such as the IPv4 address __address4__
.
1- hosts: cloud.myservice.1.worker.*
2 depends_on:
3 - service_selector: "cloud.myservice.1.master.1"
4 grouping: "round_robin"
5 prefix: "myservice_master"
6 launch:
7 - shell: docker run --name {{ _self.container_name }} IMAGE
8 --master {{ myservice_master.__address4__ }}
Accessing information from several services may be done as follows:
1- hosts: cloud.myservice.1.worker.*
2 depends_on:
3 - service_selector: "cloud.myservice.1.master.1"
4 grouping: "round_robin"
5 prefix: "myservice_master"
6 - service_selector: "edge.anotherservice.*"
7 grouping: "round_robin"
8 prefix: "anotherservice"
9 launch:
10 - shell: docker run --name {{ _self.container_name }} IMAGE
11 --master-ip {{ myservice_master.__address4__ }}
12 --device {{ anotherservice.__url6__ }}
Note that, in the example above each worker of MyService cloud.myservice.1.worker.*
is accessing the URL (ip:port) {{ anotherservice.__url6__ }}
of another service
edge.anotherservice.*
in round_robin
strategy. Meaning that, the 1st worker
has the URL of the 1st host in the anotherservice
service; the 2nd worker the
URL of the 2nd host in anotherservice
, and so on. Therefore, workers of
MyService
connect to hosts in anotherservice
in a round robin fashion.
Grouping
There are 3 grouping strategies provided by E2Clab, they are:
RoundRobin
Asarray
Aggregate
Note
Users may extend the /e2clab/e2clab/grouping/__init__.py
file to implement their
own grouping strategy (in case the existing ones do not fit their needs).
Next, we explain each grouping strategy.
RoundRobin
Groups Services in round-robin. For instance, considering that:
we have 2 Cloud servers and 4 Edge devices; and
Edge devices depend on metadata from the Cloud servers:
1- hosts: edge.*
2 depends_on:
3 - service_selector: cloud.*
4 grouping: "round_robin"
5 prefix: server
Edge devices 1 and 3 will have access to metadata from cloud server 1
.
Edge device 2 and 4 will have access to metadata from cloud server 2
.
It allows the following in the workflow.yaml
file:
1- hosts: edge.*
2 depends_on: ...
3 prepare:
4 - shell: command... {{ server.__address__ }}
This will be translated to:
[edge device 1]: --> command... 10.52.2.226
[edge device 2]: --> command... 192.168.87.254
Asarray
Groups Services id
. For instance, considering that:
we have 2 Cloud servers and 4 Edge devices; and
Edge devices depend on metadata from the Cloud servers:
1- hosts: edge.*
2 depends_on:
3 - service_selector: cloud.*
4 grouping: "asarray"
5 prefix: server
All edge devices will have access to the id
of all cloud servers.
It allows the following in the workflow.yaml
file:
1- hosts: edge.*
2 depends_on: ...
3 prepare:
4 - shell: command... {{ server }}
5 - shell: command... {{ server[0] }} {{ server[1] }}
This will be translated to:
[all devices]:
--> command... ['111', '112']
--> command... 111 112
Aggregate
Aggregates metadata from various Services. For instance, considering that we have:
2 Cloud servers (1 SERVICE named
horovod
) and4 Edge devices (4 SERVICES named
a
,b
,c
,d
);
CASE 1: Aggregates metadata from a SINGLE Service (metadata from all machines that compose this Service)
Edge devices depend on metadata from the Cloud servers (the horovod
service):
1- hosts: edge.*
2 depends_on:
3 - service_selector: cloud.horovod.*
4 grouping: "aggregate"
5 prefix: horovod
All edge devices will have access to the metadata from all cloud servers in
horovod
service.
It allows the following in the workflow.yaml
file:
1- hosts: edge.*
2 depends_on: ...
3 prepare:
4 - shell: command... {{ horovod1_1.url }} {{ horovod1_2.url }}
This will be translated to:
[all devices]: --> command... 10.52.3.90:9999 10.52.3.47:9999
Note
In horovod1_1
, 1_1
is the last part of the Service ID. The Service ID is
defined as LayerID_ServiceID_MachineID
. For instance:
horovod1_1
–>1_1
meansServiceID_MachineID
: machine 1 of service 1.horovod1_2
–>1_2
meansServiceID_MachineID
: machine 2 of service 1.
CASE 2: Aggregates metadata from VARIOUS Services (metadata from all machines that compose each Service)
All the services in the cloud depends on metadata from the Edge devices. Each device is
placed in a different service (named a
, b
, c
, d
).
1- hosts: cloud.*
2 depends_on:
3 - service_selector: edge.*
4 grouping: "aggregate"
5 prefix: client
All cloud services will have access to the metadata from all edge servers in a
,
b
, c
, d
services.
It allows the following in the workflow.yaml
file:
1- hosts: edge.*
2 depends_on: ...
3 prepare:
4 - shell: command... {{ client1_1.url }} {{ client2_1.url }} {{ client3_1.url }} {{ client4_1.url }}
This will be translated to:
[all cloud services]: --> command... 10.152.23.90:9999 10.12.2.40:9999 10.152.24.10:9999 10.152.113.147:9999
Note
In client1_1
, 1_1
is the last part of the Service ID. The Service ID is defined
as LayerID_ServiceID_MachineID
. For instance:
client1_1
–>1_1
meansServiceID_MachineID
: machine 1 of servicea
.client4_1
–>4_1
meansServiceID_MachineID
: machine 1 of serviced
.