Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

Cannot resize AWS cluster with error: Two non-terminal instances with virtual instance id ..

Highlighted

Cannot resize AWS cluster with error: Two non-terminal instances with virtual instance id ..

New Contributor

Hi,

 

I was trying to resize the cluster by following instructions on https://www.cloudera.com/documentation/director/latest/topics/director_updates.html

 

But the effort failed with error: 

[2017-10-11 17:35:08.038 +0000] ERROR [p-d7b6e18a6630-DefaultUpdateClusterJob] - c.c.director.aws.ec2.EC2Provider: Two non-terminal instances with virtual instance id 90e56cce-f713-4584-989c-06a2a2c63e2c exist: {InstanceId: i-0a10d39f28363c05f,ImageId: ami-6f68cf0f,State: {Code: 16,Name: running},PrivateDnsName: ip-10-0-0-201.us-west-2.compute.internal,PublicDnsName: ec2-34-214-154-101.us-west-2.compute.amazonaws.com,StateTransitionReason: ,KeyName: impala,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: d2.8xlarge,LaunchTime: Tue Oct 10 22:23:43 UTC 2017,Placement: {AvailabilityZone: us-west-2c,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-49b9e412,VpcId: vpc-e691d780,PrivateIpAddress: 10.0.0.201,PublicIpAddress: 34.214.154.101,Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [{DeviceName: /dev/sda1,Ebs: {VolumeId: vol-0de63b831ed6fe2c3,Status: attached,AttachTime: Tue Oct 10 22:23:44 UTC 2017,DeleteOnTermination: true}}],VirtualizationType: hvm,ClientToken: 2ac0932f-3edb-41a2-9cee-29203f185110,Tags: [{Key: Cloudera-Director-Id,Value: 90e56cce-f713-4584-989c-06a2a2c63e2c}, {Key: Cloudera-Director-Template-Name,Value: nodes}, {Key: owner,Value: root}, {Key: Name,Value: cdh5-90e56cce-f713-4584-989c-06a2a2c63e2c}],SecurityGroups: [{GroupName: cdh5,GroupId: sg-20f87d5d}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-77875a75,SubnetId: subnet-49b9e412,VpcId: vpc-e691d780,Description: ,OwnerId: 026847283572,Status: in-use,MacAddress: 0a:59:a2:20:d5:72,PrivateIpAddress: 10.0.0.201,PrivateDnsName: ip-10-0-0-201.us-west-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: cdh5,GroupId: sg-20f87d5d}],Attachment: {AttachmentId: eni-attach-2246ebc5,DeviceIndex: 0,Status: attached,AttachTime: Tue Oct 10 22:23:43 UTC 2017,DeleteOnTermination: true},Association: {PublicIp: 34.214.154.101,PublicDnsName: ec2-34-214-154-101.us-west-2.compute.amazonaws.com,IpOwnerId: amazon},PrivateIpAddresses: [{PrivateIpAddress: 10.0.0.201,PrivateDnsName: ip-10-0-0-201.us-west-2.compute.internal,Primary: true,Association: {PublicIp: 34.214.154.101,PublicDnsName: ec2-34-214-154-101.us-west-2.compute.amazonaws.com,IpOwnerId: amazon}}],Ipv6Addresses: []}],EbsOptimized: false,} and {InstanceId: i-0f7ee139d2a945207,ImageId: ami-6f68cf0f,State: {Code: 16,Name: running},PrivateDnsName: ip-10-0-0-178.us-west-2.compute.internal,PublicDnsName: ec2-52-40-171-101.us-west-2.compute.amazonaws.com,StateTransitionReason: ,KeyName: impala,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: r4.8xlarge,LaunchTime: Wed Oct 11 16:24:01 UTC 2017,Placement: {AvailabilityZone: us-west-2c,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-49b9e412,VpcId: vpc-e691d780,PrivateIpAddress: 10.0.0.178,PublicIpAddress: 52.40.171.101,Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [{DeviceName: /dev/sda1,Ebs: {VolumeId: vol-02d2541902af03afb,Status: attached,AttachTime: Wed Oct 11 16:24:02 UTC 2017,DeleteOnTermination: true}}],VirtualizationType: hvm,ClientToken: KHrAu1507739040599,Tags: [{Key: owner,Value: root}, {Key: Cloudera-Director-Id,Value: 90e56cce-f713-4584-989c-06a2a2c63e2c}, {Key: Cloudera-Director-Template-Name,Value: nodes}, {Key: Name,Value: cdh5-impala-01}],SecurityGroups: [{GroupName: cdh5,GroupId: sg-20f87d5d}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-2c6ab32e,SubnetId: subnet-49b9e412,VpcId: vpc-e691d780,Description: Primary network interface,OwnerId: 026847283572,Status: in-use,MacAddress: 0a:48:76:c7:b3:96,PrivateIpAddress: 10.0.0.178,PrivateDnsName: ip-10-0-0-178.us-west-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: cdh5,GroupId: sg-20f87d5d}],Attachment: {AttachmentId: eni-attach-d84be13f,DeviceIndex: 0,Status: attached,AttachTime: Wed Oct 11 16:24:01 UTC 2017,DeleteOnTermination: true},Association: {PublicIp: 52.40.171.101,PublicDnsName: ec2-52-40-171-101.us-west-2.compute.amazonaws.com,IpOwnerId: amazon},PrivateIpAddresses: [{PrivateIpAddress: 10.0.0.178,PrivateDnsName: ip-10-0-0-178.us-west-2.compute.internal,Primary: true,Association: {PublicIp: 52.40.171.101,PublicDnsName: ec2-52-40-171-101.us-west-2.compute.amazonaws.com,IpOwnerId: amazon}}],Ipv6Addresses: []}],EbsOptimized: false,}

 

Any help is appreciated.

1 REPLY 1

Re: Cannot resize AWS cluster with error: Two non-terminal instances with virtual instance id ..

Cloudera Employee
Hmm... this is weird. Do you still have application.log around? It could be helpful to figure out how it ran into the situation that two instances both running mapped to the same virtual instance id