Docker on YARN - Sandbox

[#1]

Docker on YARN is relatively easy to set up on an existing cluster, but full clusters are not always available.

My Ember Project was created to simplify dev/test for technologies managed by Ambari and Cloudera Manager. It provides utilities for creating dockerized clusters that use far fewer resources than a full bare metal or VM-based cluster. Additionally, by using pre-built images, the time it takes to get a cluster up and running can be reduced to less than 10 minutes.

The following four commands are all that is necessary to download and run a ~5GB image that comes preinstalled with Ambari, Zookeeper, HDFS, and YARN with Docker on YARN pre-configured. Docker containers spawned by YARN will be created on the host machine as peers to the container with YARN inside. All containers are launched into the "ember" docker network by default. Once the container is downloaded, it takes less than 5 minutes to start all services.

curl -L https://github.com/SamHjelmfelt/Ember/archive/v1.1.zip -o Ember_1.1.zip
unzip Ember_1.1.zip
cd Ember-1.1/
./ember.sh createFromPrebuiltSample samples/yarnquickstart/yarnquickstart-sample.ini

The Ambari UI can be found at http://localhost:8080

The YARN Resource Manager UI can be found at http://localhost:8088

Usage

The YARN service REST API documentation can be found here:

https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html

The YARN app CLI documentation can be found here:

https://hadoop.apache.org/docs/r3.1.1/hadoop-yarn/hadoop-yarn-site/YarnCommands.html#application_or_...

Testing

Place the following service definition into a file (e.g. redis.json)

{
  "name": "redis-service",
  "version": "1.0.0",
  "description": "redis example",
  "components" :
    [
      {
        "name": "redis",
        "number_of_containers": 1,
        "artifact": {
          "id": "library/redis",
          "type": "DOCKER"
        },
        "launch_command": "",
        "resource": {
          "cpus": 1,
          "memory": "256"
        },
        "configuration": {
          "env": {
            "YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE": "true"
          }
        }
      }
    ]
}

Submit the service with the following curl command. YARN should respond back with the applicationId

curl -X POST -H "Content-Type: application/json" http://localhost:8088/app/v1/services?user.name=ambari-qa -d @redis.json

The service status can be viewed on the YARN UI or through the REST API (python makes it easier to read):

curl http://localhost:8088/app/v1/services/redis-service?user.name=ambari-qa | python -m json.tool

The service name must be unique in the cluster. If you need to delete your service, the following command can be used:

curl -X DELETE http://localhost:8088/app/v1/services/redis-service?user.name=ambari-qa