How to configure spark cluster in ecs?

Navigation
1

I have a multimaster spark configuration with zookeeper like this:

2 spark-master registered in zookeeper and two workers who register with the leader of the spark master, in addition the jobs are sent through livy which is an api rest.

My question is: how load balancing would be handled in ecs-fargate-elb taking into account that each component (spark-master, zookeeper, spark-workers, livy) is deployed in an independent ecs service.

    
asked by Alejandro Quintero 27.02.2018 в 22:11
source

1 answer

0

Load balancing is not possible because spark master spark workers and livy are connected through the configuration files and the connection is created in compile time. The way in which containers can communicate is through an interface that exposes the task definition called loopback interface that is fixed and is ip 127.0.0.1 and hostname localhost, so the worker would register with the master so spark: //127.0.0.1: 7077

    
answered by 16.03.2018 / 13:34
source
Error installing wordpress. Shows setup after installing I can not use a WebViewClient