cloudera director c5d instance type support

[#1]

Hi,

 I am not able to use AWS instance type c5d for cluster template. Is it related to the cloudera director version? How can I include the template to this instance type?

Thanks

[#2]

These new instance types aren't in Director yet. For now however, you should be able to update Director's AWS plugin to include them. You can refer to the documentation for the steps: https://www.cloudera.com/documentation/director/latest/topics/director_aws_new_instance_types.html

[#3]

I did configure the c5d instance types, but it is still failing:

Process logs can be found at /root/.cloudera-director/logs/application.log
Plugins will be loaded from /var/lib/cloudera-director-plugins
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256M; support was removed in 8.0
Cloudera Director 2.8.0 initializing ...
Connecting to http://localhost:7189
Current user roles: [ROLE_READONLY, ROLE_ADMIN]
Configuration file passes all validation checks.
Creating a new environment...
Creating external database servers if configured...
Creating a new Cloudera Manager...
Unexpected internal error (see logs): {
  "validationErrors" : [ "Incompatible AMI virtualization type. Instance type c5d.2xlarge does not support hvm virtualization type of AMI ami-3548444c." ],
  "validationWarnings" : [ ],
  "conditions" : [ {
    "scope" : "aws.com.cloudera.director.aws.ec2.EC2Provider.template",
    "key" : "image",
    "exceptionInfo" : {
      "@class" : "com.cloudera.launchpad.common.error.ErrorInfo",
      "code" : [ "com.cloudera.launchpad.pluggable.common.ProviderErrorCode", "PROVIDER_EXCEPTION" ],
      "properties" : {
        "message" : "Incompatible AMI virtualization type. Instance type c5d.2xlarge does not support hvm virtualization type of AMI ami-3548444c."
      },
      "causes" : [ ],
      "message" : "Incompatible AMI virtualization type. Instance type c5d.2xlarge does not support hvm virtualization type of AMI ami-3548444c."
    },
    "message" : "Incompatible AMI virtualization type. Instance type c5d.2xlarge does not support hvm virtualization type of AMI ami-3548444c."
  } ],
  "formatWarnings" : [ ]
}
[centos@ip-10-197-19-68 ~]$
[centos@ip-10-197-19-68 ~]$ cat /var/lib/cloudera-director-plugins/aws-provider-1.5.0/etc/ec2.customvirtualizationmappings.properties
hvm=c5d.xlarge,c5d.2xlarge,c5.large,m5.xlarge
[centos@ip-10-197-19-68 ~]$ cat /var/lib/cloudera-director-plugins/aws-provider-1.5.0/etc/ec2.ephemeraldevicemappings.properties
# (c) Copyright 2015 Cloudera, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Comprehensive list of EC2 ephemeral device mappings at:
# http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#StorageOnInstanceTypes

c5d.2xlarge=1
[centos@ip-10-197-19-68 ~]$

[#4]

Hmm, so etc/ec2.virtualizationmappings.properties is the default location to specify the virtualization mapping. You can use etc/ec2.customvirtualizationmappings.properties as shown in your example but then you would have to specify this custom location in etc/aws-plugin.conf (as mentioned in the docs). Can you double check to see if that's the issue.

[#5 · Accepted solution]

I resolved the issue, the problem was:

- the listing of hvm instance types has to be in one row separated by comma (I dont know why the escape on new lines did not worked)

Here is the working configuration:

cat << EOF | sudo tee -a /var/lib/cloudera-director-plugins/aws-provider-1.5.0/etc/aws-plugin.confx
virtualizationMappings {
    customMappingsPath: ec2.customvirtualizationmappings.properties
}
EOF

cat << EOF | sudo tee /var/lib/cloudera-director-plugins/aws-provider-1.5.0/etc/ec2.customvirtualizationmappings.propertiesx
hvm=c5.large,c5.xlarge,c5.2xlarge,c5.4xlarge,c5.9xlarge,c5.18xlarge, m5.large,m5.xlarge,m5.2xlarge,m5.4xlarge,m5.12xlarge,m5.24xlarge,c5d.large,c5d.xlarge,c5d.2xlarge,c5d.4xlarge,c5d.9xlarge,c5d.18xlarge,r5.large,r5.xlarge,r5.2xlarge,r5.4xlarge,r5.12xlarge,r5.24xlarge
EOF

cat << EOF | sudo tee -a /var/lib/cloudera-director-plugins/aws-provider-1.5.0/etc/aws-plugin.confx
ephemeralDeviceMappings {
    customMappingsPath: ec2.customephemeraldevicemappings.properties
}
EOF

cat << EOF | sudo tee /var/lib/cloudera-director-plugins/aws-provider-1.5.0/etc/ec2.customephemeraldevicemappings.propertiesx
c5d.large=1
c5d.xlarge=1
c5d.2xlarge=1
c5d.4xlarge=1
c5d.9xlarge=1
c5d.18xlarge=2
EOF

[#6]

Glad you got it working.

Just to note something like this did work for me for hvm instance types:

hvm=c5.large,\
c5.xlarge,\
c5.2xlarge,\
c5d.2xlarge

There should be a ec2.virtualizationmappings.properties.example file you can look at, modify as an example.