Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (3)
New Contributor

Docker on YARN is relatively easy to set up on an existing cluster, but full clusters are not always available.

My Ember Project was created to simplify dev/test for technologies managed by Ambari and Cloudera Manager. It provides utilities for creating dockerized clusters that use far fewer resources than a full bare metal or VM-based cluster. Additionally, by using pre-built images, the time it takes to get a cluster up and running can be reduced to less than 10 minutes.

The following four commands are all that is necessary to download and run a ~5GB image that comes preinstalled with Ambari, Zookeeper, HDFS, and YARN with Docker on YARN pre-configured. Docker containers spawned by YARN will be created on the host machine as peers to the container with YARN inside. All containers are launched into the "ember" docker network by default. Once the container is downloaded, it takes less than 5 minutes to start all services.

curl -L https://github.com/SamHjelmfelt/Ember/archive/v1.1.zip -o Ember_1.1.zip
unzip Ember_1.1.zip
cd Ember-1.1/
./ember.sh createFromPrebuiltSample samples/yarnquickstart/yarnquickstart-sample.ini

The Ambari UI can be found at http://localhost:8080

The YARN Resource Manager UI can be found at http://localhost:8088


Usage

The YARN service REST API documentation can be found here:

https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html

The YARN app CLI documentation can be found here:

https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#application_or_...

Testing

Place the following service definition into a file (e.g. redis.json)

{
  "name": "redis-service",
  "version": "1.0.0",
  "description": "redis example",
  "components" :
    [
      {
        "name": "redis",
        "number_of_containers": 1,
        "artifact": {
          "id": "library/redis",
          "type": "DOCKER"
        },
        "launch_command": "",
        "resource": {
          "cpus": 1,
          "memory": "256"
        },
        "configuration": {
          "env": {
            "YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE": "true"
          }
        }
      }
    ]
}

Submit the service with the following curl command. YARN should respond back with the applicationId

curl -X POST -H "Content-Type: application/json" http://localhost:8088/app/v1/services?user.name=ambari-qa -d @redis.json

The service status can be viewed on the YARN UI or through the REST API (python makes it easier to read):

curl http://localhost:8088/app/v1/services/redis-service?user.name=ambari-qa | python -m json.tool

The service name must be unique in the cluster. If you need to delete your service, the following command can be used:

curl -X DELETE http://localhost:8088/app/v1/services/redis-service?user.name=ambari-qa
571 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎01-09-2019 02:31 AM
Updated by:
 
Contributors
Top Kudoed Authors