question Re: Cloudera Director support for new AWS i3 instances in Archives of Support Questions (Read Only)

Cloudera Director support for new AWS i3 instances

Garren — Fri, 16 Sep 2022 11:25:46 GMT

When will the new i3 instances be supported in Cloudera Director? When I tried to launch a cluster with the i3 instances, it threw a generic error about how that instance type is not supported by the AMI I selected. I then made up a non-existent instance type and got the same error.

Re: Cloudera Director support for new AWS i3 instances

Bill Havanki — Fri, 07 Apr 2017 20:18:38 GMT

Hi Garren,

You can configure Director to work with a new EC2 instance type without waiting for a new Director release. We implemented the AWS plugin this way because new instance types appear all the time, and you shouldn't have to wait for us to be able to use them.

https://www.cloudera.com/documentation/director/latest/topics/director_aws_new_instance_types.html

Even once you configure for a new instance type, though, it's still possible that the AMI you chose is not available for it. For example, my favorite in us-east-1, ami-6d1c2007, can't use an i3 instance type when I try to launch one from the AWS console: "The instance configuration for this AWS Marketplace product is not supported."

Re: Cloudera Director support for new AWS i3 instances

Garren — Fri, 07 Apr 2017 21:05:25 GMT

Thanks Bill!

Oddly, I had no problem launching an i3.8xlarge with that same AMI (ami-6d1c2007)

Re: Cloudera Director support for new AWS i3 instances

Bill Havanki — Fri, 07 Apr 2017 22:05:22 GMT

Oh weird! I tried an i3.large and it balked.

Re: Cloudera Director support for new AWS i3 instances

Garren — Thu, 20 Apr 2017 22:20:22 GMT

Hey @Bill Havanki I edited the files and added these lines:

ec2.virtualizationmappings.properties

hvm=i3.large,\
i3.xlarge,\
i3.2xlarge,\
i3.4xlarge,\
i3.8xlarge,\
i3.16xlarge,\

ec2.ephemeraldevicemappings.properties:

i3.large=1
i3.xlarge=1
i3.2xlarge=1
i3.4xlarge=2
i3.8xlarge=4
i3.16xlarge=8

Unfortunately, while the provisioning of the instances was successful, the NVMe drives were NOT mounted. This is likely because they are at non-default locations like "/dev/nvme0n1"

Re: Cloudera Director support for new AWS i3 instances

aarman — Tue, 25 Apr 2017 06:12:15 GMT

Hey Garren,

Good catch, due to the way the NVMe volume devices are named, they unintentionally get skipped during Director's mount script. We will look into fixing the mount script for these volumes in a future release when we add the i3 instances.

For now it's still possible to get this working by manually mounting the volumes through an instance bootstrap script. For example, the following script worked for me using i3.4xlarge instances.

bootstrapScripts: ["""#!/bin/sh

      prepare_disk()
      {
        mount=$1
        device=$2

        FS=ext4
        FS_OPTS="-E lazy_itable_init=1"

        echo "Warning: ERASING CONTENTS OF $device"
        mkfs.$FS -F $FS_OPTS $device -m 0

        echo "Mounting $device on $mount"
        if [ ! -e "${mount}" ]; then
          mkdir "${mount}"
        fi

        mount -o defaults,noatime "${device}" "${mount}"
        echo "$device $mount $FS defaults,noatime 0 0" >> /etc/fstab
      }

      prepare_disk /data0 /dev/nvme0n1
      prepare_disk /data1 /dev/nvme1n1
  """]

The above is a simplified version of Director's mount script, it will mount the 2 NVMe volumes for i3.4xlarge

Re: Cloudera Director support for new AWS i3 instances

Garren — Tue, 25 Apr 2017 19:15:56 GMT

Thanks @aarman; that worked wonderfully!