Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Cloudera Director support for new AWS i3 instances

avatar
Explorer

When will the new i3 instances be supported in Cloudera Director? When I tried to launch a cluster with the i3 instances, it threw a generic error about how that instance type is not supported by the AMI I selected. I then made up a non-existent instance type and got the same error.

1 ACCEPTED SOLUTION

avatar
Super Collaborator

Hi Garren,

 

You can configure Director to work with a new EC2 instance type without waiting for a new Director release. We implemented the AWS plugin this way because new instance types appear all the time, and you shouldn't have to wait for us to be able to use them.

 

https://www.cloudera.com/documentation/director/latest/topics/director_aws_new_instance_types.html

 

Even once you configure for a new instance type, though, it's still possible that the AMI you chose is not available for it. For example, my favorite in us-east-1, ami-6d1c2007, can't use an i3 instance type when I try to launch one from the AWS console: "The instance configuration for this AWS Marketplace product is not supported."

View solution in original post

6 REPLIES 6

avatar
Super Collaborator

Hi Garren,

 

You can configure Director to work with a new EC2 instance type without waiting for a new Director release. We implemented the AWS plugin this way because new instance types appear all the time, and you shouldn't have to wait for us to be able to use them.

 

https://www.cloudera.com/documentation/director/latest/topics/director_aws_new_instance_types.html

 

Even once you configure for a new instance type, though, it's still possible that the AMI you chose is not available for it. For example, my favorite in us-east-1, ami-6d1c2007, can't use an i3 instance type when I try to launch one from the AWS console: "The instance configuration for this AWS Marketplace product is not supported."

avatar
Explorer

Thanks Bill!

 

Oddly, I had no problem launching an i3.8xlarge with that same AMI (ami-6d1c2007)

avatar
Super Collaborator

Oh weird! I tried an i3.large and it balked.

avatar
Explorer

 

 

Hey @Bill Havanki I edited the files and added these lines:

 

ec2.virtualizationmappings.properties

hvm=i3.large,\
  i3.xlarge,\
  i3.2xlarge,\
  i3.4xlarge,\
  i3.8xlarge,\
  i3.16xlarge,\

 

ec2.ephemeraldevicemappings.properties:

i3.large=1
i3.xlarge=1
i3.2xlarge=1
i3.4xlarge=2
i3.8xlarge=4
i3.16xlarge=8

 

Unfortunately, while the provisioning of the instances was successful, the NVMe drives were NOT mounted. This is likely because they are at non-default locations like "/dev/nvme0n1"

avatar
Rising Star

Hey Garren,

 

Good catch, due to the way the NVMe volume devices are named, they unintentionally get skipped during Director's mount script. We will look into fixing the mount script for these volumes in a future release when we add the i3 instances.

 

For now it's still possible to get this working by manually mounting the volumes through an instance bootstrap script. For example, the following script worked for me using i3.4xlarge instances. 

 

bootstrapScripts: ["""#!/bin/sh

      prepare_disk()
      {
        mount=$1
        device=$2

        FS=ext4
        FS_OPTS="-E lazy_itable_init=1"

        echo "Warning: ERASING CONTENTS OF $device"
        mkfs.$FS -F $FS_OPTS $device -m 0

        echo "Mounting $device on $mount"
        if [ ! -e "${mount}" ]; then
          mkdir "${mount}"
        fi

        mount -o defaults,noatime "${device}" "${mount}"
        echo "$device $mount $FS defaults,noatime 0 0" >> /etc/fstab
      }

      prepare_disk /data0 /dev/nvme0n1
      prepare_disk /data1 /dev/nvme1n1
  """]

 The above is a simplified version of Director's mount script, it will mount the 2 NVMe volumes for i3.4xlarge

avatar
Explorer

Thanks @aarman; that worked wonderfully!