Reply
Highlighted
New Contributor
Posts: 2
Registered: ‎10-09-2017

Cannot resize AWS cluster with error: Two non-terminal instances with virtual instance id ..

Hi,

 

I was trying to resize the cluster by following instructions on https://www.cloudera.com/documentation/director/latest/topics/director_updates.html

 

But the effort failed with error: 

[2017-10-11 17:35:08.038 +0000] ERROR [p-d7b6e18a6630-DefaultUpdateClusterJob] - c.c.director.aws.ec2.EC2Provider: Two non-terminal instances with virtual instance id 90e56cce-f713-4584-989c-06a2a2c63e2c exist: {InstanceId: i-0a10d39f28363c05f,ImageId: ami-6f68cf0f,State: {Code: 16,Name: running},PrivateDnsName: ip-10-0-0-201.us-west-2.compute.internal,PublicDnsName: ec2-34-214-154-101.us-west-2.compute.amazonaws.com,StateTransitionReason: ,KeyName: impala,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: d2.8xlarge,LaunchTime: Tue Oct 10 22:23:43 UTC 2017,Placement: {AvailabilityZone: us-west-2c,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-49b9e412,VpcId: vpc-e691d780,PrivateIpAddress: 10.0.0.201,PublicIpAddress: 34.214.154.101,Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [{DeviceName: /dev/sda1,Ebs: {VolumeId: vol-0de63b831ed6fe2c3,Status: attached,AttachTime: Tue Oct 10 22:23:44 UTC 2017,DeleteOnTermination: true}}],VirtualizationType: hvm,ClientToken: 2ac0932f-3edb-41a2-9cee-29203f185110,Tags: [{Key: Cloudera-Director-Id,Value: 90e56cce-f713-4584-989c-06a2a2c63e2c}, {Key: Cloudera-Director-Template-Name,Value: nodes}, {Key: owner,Value: root}, {Key: Name,Value: cdh5-90e56cce-f713-4584-989c-06a2a2c63e2c}],SecurityGroups: [{GroupName: cdh5,GroupId: sg-20f87d5d}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-77875a75,SubnetId: subnet-49b9e412,VpcId: vpc-e691d780,Description: ,OwnerId: 026847283572,Status: in-use,MacAddress: 0a:59:a2:20:d5:72,PrivateIpAddress: 10.0.0.201,PrivateDnsName: ip-10-0-0-201.us-west-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: cdh5,GroupId: sg-20f87d5d}],Attachment: {AttachmentId: eni-attach-2246ebc5,DeviceIndex: 0,Status: attached,AttachTime: Tue Oct 10 22:23:43 UTC 2017,DeleteOnTermination: true},Association: {PublicIp: 34.214.154.101,PublicDnsName: ec2-34-214-154-101.us-west-2.compute.amazonaws.com,IpOwnerId: amazon},PrivateIpAddresses: [{PrivateIpAddress: 10.0.0.201,PrivateDnsName: ip-10-0-0-201.us-west-2.compute.internal,Primary: true,Association: {PublicIp: 34.214.154.101,PublicDnsName: ec2-34-214-154-101.us-west-2.compute.amazonaws.com,IpOwnerId: amazon}}],Ipv6Addresses: []}],EbsOptimized: false,} and {InstanceId: i-0f7ee139d2a945207,ImageId: ami-6f68cf0f,State: {Code: 16,Name: running},PrivateDnsName: ip-10-0-0-178.us-west-2.compute.internal,PublicDnsName: ec2-52-40-171-101.us-west-2.compute.amazonaws.com,StateTransitionReason: ,KeyName: impala,AmiLaunchIndex: 0,ProductCodes: [],InstanceType: r4.8xlarge,LaunchTime: Wed Oct 11 16:24:01 UTC 2017,Placement: {AvailabilityZone: us-west-2c,GroupName: ,Tenancy: default,},Monitoring: {State: disabled},SubnetId: subnet-49b9e412,VpcId: vpc-e691d780,PrivateIpAddress: 10.0.0.178,PublicIpAddress: 52.40.171.101,Architecture: x86_64,RootDeviceType: ebs,RootDeviceName: /dev/sda1,BlockDeviceMappings: [{DeviceName: /dev/sda1,Ebs: {VolumeId: vol-02d2541902af03afb,Status: attached,AttachTime: Wed Oct 11 16:24:02 UTC 2017,DeleteOnTermination: true}}],VirtualizationType: hvm,ClientToken: KHrAu1507739040599,Tags: [{Key: owner,Value: root}, {Key: Cloudera-Director-Id,Value: 90e56cce-f713-4584-989c-06a2a2c63e2c}, {Key: Cloudera-Director-Template-Name,Value: nodes}, {Key: Name,Value: cdh5-impala-01}],SecurityGroups: [{GroupName: cdh5,GroupId: sg-20f87d5d}],SourceDestCheck: true,Hypervisor: xen,NetworkInterfaces: [{NetworkInterfaceId: eni-2c6ab32e,SubnetId: subnet-49b9e412,VpcId: vpc-e691d780,Description: Primary network interface,OwnerId: 026847283572,Status: in-use,MacAddress: 0a:48:76:c7:b3:96,PrivateIpAddress: 10.0.0.178,PrivateDnsName: ip-10-0-0-178.us-west-2.compute.internal,SourceDestCheck: true,Groups: [{GroupName: cdh5,GroupId: sg-20f87d5d}],Attachment: {AttachmentId: eni-attach-d84be13f,DeviceIndex: 0,Status: attached,AttachTime: Wed Oct 11 16:24:01 UTC 2017,DeleteOnTermination: true},Association: {PublicIp: 52.40.171.101,PublicDnsName: ec2-52-40-171-101.us-west-2.compute.amazonaws.com,IpOwnerId: amazon},PrivateIpAddresses: [{PrivateIpAddress: 10.0.0.178,PrivateDnsName: ip-10-0-0-178.us-west-2.compute.internal,Primary: true,Association: {PublicIp: 52.40.171.101,PublicDnsName: ec2-52-40-171-101.us-west-2.compute.amazonaws.com,IpOwnerId: amazon}}],Ipv6Addresses: []}],EbsOptimized: false,}

 

Any help is appreciated.

New Contributor
Posts: 15
Registered: ‎07-25-2017

Re: Cannot resize AWS cluster with error: Two non-terminal instances with virtual instance id ..

Hmm... this is weird. Do you still have application.log around? It could be helpful to figure out how it ran into the situation that two instances both running mapped to the same virtual instance id
Announcements