question Re: Horotonworks Data Cloud error autoscaling token expired in Support Questions

Horotonworks Data Cloud error autoscaling token expired

Carolyn — Sat, 29 Apr 2017 23:06:20 GMT

I am provisioning a cluster using version 1.14.0 of Hortonworks Data Cloud. The Autoscaling status slider is in the off position, but cluster provisioning fails with the following error on the auto scaling token:

Infrastructure creation failed. Reason: com.amazonaws.AmazonServiceException: The security token included in the request is expired (Service: AmazonAutoScaling; Status Code: 403; Error Code: ExpiredToken; Request ID: cf2aebb0-2cec-11e7-8985-4100dda4669c)

The autoscaling tab on the failed cluster in the cloud controller, shows the cluster trying to autoscale to 3 nodes:

Re: Horotonworks Data Cloud error autoscaling token expired

jeff1 — Mon, 01 May 2017 20:15:40 GMT

Hello,

So you are using Controller 1.14.0. A few more questions:

- What HDP version are you attempting to launch?

- What Cluster Type?

- What is your Worker + Compute count? Spot instances? Etc?

- Maybe just post the .json (that would be used for the cluster create CLI) and we can see all the options you selected.

Thanks!

Jeff

Re: Horotonworks Data Cloud error autoscaling token expired

Dominika — Sun, 18 Aug 2019 03:08:27 GMT

In case this helps, back in March I got this error when using HDP 2.5 EDW-ETL in us-east.

In my case, for some reason the infrastructure creation took too long (1 hour instead of 5 minutes) and the security token expired during that time.

In addition to what Jeff said, it may help to check the CloudFormation UI on the AWS console, find your stack and post the data from the “Events” and “Resources” tabs.

Re: Horotonworks Data Cloud error autoscaling token expired

dletendre — Tue, 02 May 2017 06:18:57 GMT

I tried HDP 2.6 on us-east, hive+spark2.1, 3 workers, pretty much standard everything

Re: Horotonworks Data Cloud error autoscaling token expired

dletendre — Tue, 02 May 2017 23:30:54 GMT

This happened twice yesterday. The third try in the afternoon worked (all AWS us-east) Tried to add one worker today and it failed again, same error. Something is really broken here hdp26secondfail.png hdp26addworkerfail.png

Re: Horotonworks Data Cloud error autoscaling token expired

dletendre — Tue, 02 May 2017 23:31:44 GMT

@jeff any ideas on this?

Re: Horotonworks Data Cloud error autoscaling token expired

Dominika — Wed, 03 May 2017 03:29:20 GMT

The error happens when you use default settings, just specify cluster name and Ambari password.

Re: Horotonworks Data Cloud error autoscaling token expired

dletendre — Wed, 03 May 2017 03:58:47 GMT

I think there is a bug where it is checking the spot bidding prices when you don't even request spot I tried resizing my cluster twice, both fails hdp26resizefail2.png

I have a feeling if I wait until after 5 or 6pm it will work

Re: Horotonworks Data Cloud error autoscaling token expired

Dominika — Wed, 03 May 2017 04:05:28 GMT

What worked for me last time is creating the cloud controller in a region different than us-east.

You could also try using newer version 1.14.1: http://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.14.1/index.html

Re: Horotonworks Data Cloud error autoscaling token expired

Dominika — Wed, 03 May 2017 04:27:34 GMT

@cdub

@DL

What worked for me is creating the cloud controller in a region different than us-east. When I use us-east with this specific cloud controller version, I seem to get various errors.

You could also try using the newer cloud controller version 1.14.1 with us-east: http://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.14.1/index.html

Re: Horotonworks Data Cloud error autoscaling token expired

dletendre — Wed, 03 May 2017 04:39:23 GMT

Thanks @Dominika Bialek I'm using 1.14.1. Unfortunately I'm stuck in US east since all our other infra is there.

Re: Horotonworks Data Cloud error autoscaling token expired

Dominika — Wed, 03 May 2017 04:46:40 GMT

Oh 😞 The last thing that I can think of is that this is related to the instance type so you could try using a different type than default.

@Marton Sereg @Attila Kanto I can reproduce this error. I don't see anything unusual in the CloudFormation console (It showed AutoScaling group creation in progress). Could you recommend which cloudbreak log to access and what to look for? I accessed /var/lib/cloudbreak-deployment/cbreak.log on the cloud controller instance but wasn't sure what to look for. Here is what I found:

/cbreak_identity_1 | [2017-05-02 20:09:45.851] cloudfoundry-identity-server - ???? [http-nio-8080-exec-5] .... DEBUG --- UaaRequestMatcher: [oauthAuthorizeRequestMatcher] Checking match of request : '/check_token'; '/oauth/authorize' with parameters={response_type=token, source=credentials} and headers {accept=[application/json, application/x-www-form-urlencoded]}
/cbreak_identity_1 | [2017-05-02 20:09:45.852] cloudfoundry-identity-server - ???? [http-nio-8080-exec-5] .... DEBUG --- UaaRequestMatcher: [oauthAuthorizeApiRequestMatcher] Checking match of request : '/check_token'; '/oauth/authorize' with parameters={response_type=code, client_id=} and headers {Authorization=[bearer ]}
/cbreak_identity_1 | [2017-05-02 20:09:45.853] cloudfoundry-identity-server - ???? [http-nio-8080-exec-5] .... DEBUG --- UaaRequestMatcher: [xOauthCallbackRequestMatcher] Checking match of request : '/check_token'; '/login/callback' with parameters={code=} and headers {}
/cbreak_identity_1 | [2017-05-02 20:09:45.853] cloudfoundry-identity-server - ???? [http-nio-8080-exec-5] .... DEBUG --- UaaRequestMatcher: [oauthAuthorizeRequestMatcherOld] Checking match of request : '/check_token'; '/oauth/authorize' with parameters={response_type=token, credentials={} and headers {accept=[application/json, application/x-www-form-urlencoded]}
/cbreak_identity_1 | [2017-05-02 20:09:45.851] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- UaaRequestMatcher: [loginAuthenticateRequestMatcher] Checking match of request : '/check_token'; '/authenticate' with parameters={} and headers {Authorization=[bearer ], accept=[application/json]}
/cbreak_identity_1 | [2017-05-02 20:09:45.854] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- UaaRequestMatcher: [loginAuthorizeRequestMatcher] Checking match of request : '/check_token'; '/oauth/authorize' with parameters={source=login} and headers {accept=[application/json]}
/cbreak_identity_1 | [2017-05-02 20:09:45.855] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- UaaRequestMatcher: [loginTokenRequestMatcher] Checking match of request : '/check_token'; '/oauth/token' with parameters={source=login, grant_type=password, add_new=} and headers {Authorization=[bearer ], accept=[application/json]}
/cbreak_identity_1 | [2017-05-02 20:09:45.855] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- UaaRequestMatcher: [loginAuthorizeRequestMatcherOld] Checking match of request : '/check_token'; '/oauth/authorize' with parameters={login={} and headers {accept=[application/json]}
/cbreak_identity_1 | [2017-05-02 20:09:45.856] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- UaaRequestMatcher: [passcodeTokenMatcher] Checking match of request : '/check_token'; '/oauth/token' with parameters={grant_type=password, passcode=} and headers {accept=[application/json, application/x-www-form-urlencoded]}
/cbreak_identity_1 | [2017-05-02 20:09:45.856] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- UaaRequestMatcher: [oauthAuthorizeRequestMatcher] Checking match of request : '/check_token'; '/oauth/authorize' with parameters={response_type=token, source=credentials} and headers {accept=[application/json, application/x-www-form-urlencoded]}
/cbreak_identity_1 | [2017-05-02 20:09:45.857] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- UaaRequestMatcher: [oauthAuthorizeApiRequestMatcher] Checking match of request : '/check_token'; '/oauth/authorize' with parameters={response_type=code, client_id=} and headers {Authorization=[bearer ]}
/cbreak_identity_1 | [2017-05-02 20:09:45.857] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- UaaRequestMatcher: [xOauthCallbackRequestMatcher] Checking match of request : '/check_token'; '/login/callback' with parameters={code=} and headers {}
/cbreak_identity_1 | [2017-05-02 20:09:45.858] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- UaaRequestMatcher: [oauthAuthorizeRequestMatcherOld] Checking match of request : '/check_token'; '/oauth/authorize' with parameters={response_type=token, credentials={} and headers {accept=[application/json, application/x-www-form-urlencoded]}
/cbreak_identity_1 | [2017-05-02 20:09:45.860] cloudfoundry-identity-server - ???? [http-nio-8080-exec-5] .... DEBUG --- DisableIdTokenResponseTypeFilter: Processing id_token disable filter
/cbreak_identity_1 | [2017-05-02 20:09:45.861] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- DisableIdTokenResponseTypeFilter: Processing id_token disable filter
/cbreak_identity_1 | [2017-05-02 20:09:45.862] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- DisableIdTokenResponseTypeFilter: pre id_token disable:false pathinfo:null request_uri:/check_token response_type:null
/cbreak_identity_1 | [2017-05-02 20:09:45.862] cloudfoundry-identity-server - ???? [http-nio-8080-exec-9] .... DEBUG --- DisableIdTokenResponseTypeFilter: post id_token disable:false pathinfo:null request_uri:/check_token response_type:null

Re: Horotonworks Data Cloud error autoscaling token expired

dletendre — Wed, 03 May 2017 07:51:46 GMT

Thanks @Dominika Bialek For some reason I can't post questions in the Data Cloud track even though we've been an HDP customer with an AWS cluster running since last Nov. Is there anything I can do to help figure out why this is happening?

Re: Horotonworks Data Cloud error autoscaling token expired

Dominika — Wed, 03 May 2017 09:59:34 GMT

Hi @DL Did you follow the instructions at http://docs.hortonworks.com/HDPDocuments/HDCloudAWS/HDCloudAWS-1.14.1/bk_hdcloud-aws/content/gethelp/index.html#getting-help to sign up for the Hortonworks Data Cloud track? If not, you need to do that first. If yes, then maybe we can create a separate question about this.

Re: Horotonworks Data Cloud error autoscaling token expired

dletendre — Wed, 03 May 2017 11:03:22 GMT

@Dominika BialekI did but it said I already had an account with my work email and wouldn't let me do anything. Do I have to somehow delete this account and start all over?

Re: Horotonworks Data Cloud error autoscaling token expired

Dominika — Wed, 03 May 2017 21:42:00 GMT

@jeff @Julia Ostrowski Can you help?

Re: Horotonworks Data Cloud error autoscaling token expired

msereg — Wed, 03 May 2017 22:37:53 GMT

@cduby @DL

If you're using spot instances, can you try to increase the bid price, or without spot instances first?

I can see on the screenshots that infrastructure creation is taking more than an hour (it should be around 5 mins with on-demand instances, and a few mins more with spot priced instances). You can try to check the spot requests page and see if there are any messages there, like "request is pending".

So the main problem is that infra creation is taking too much time and that's why the temporary credentials used by HDC are expired - it is still a bug because spot instance requests can sometimes take more than an hour to fulfill and HDC should be able to handle that.

But for now as a workaround you should find out what is taking too long (my guess is spot requests), and change the cluster configuration according to that - different instance types, bid price, etc..

Re: Horotonworks Data Cloud error autoscaling token expired

dletendre — Wed, 03 May 2017 23:01:46 GMT

I have spot instances unchecked so it shouldn't be using them at all, but it seems to be. I also tried to add a worker late last night and no luck (same timeout after an hour). I terminated the cluster (for the third time). Will try again and take screenshots

Re: Horotonworks Data Cloud error autoscaling token expired

dletendre — Wed, 03 May 2017 23:08:11 GMT

Trying now. Spot is unchecked, autoscaling unchecked, here is the json:

{ "ClusterName": "cluster-6", "HDPVersion": "2.6", "ClusterType": "EDW-ETL: Apache Hive 1.2.1, Apache Spark 2.1", "Master": { "InstanceType": "r3.xlarge", "VolumeType": "ephemeral", "VolumeSize": 80, "VolumeCount": 1 }, "Worker": { "InstanceType": "r3.xlarge", "VolumeType": "gp2", "VolumeSize": 500, "VolumeCount": 1, "InstanceCount": 4, "RecoveryMode": "AUTO" }, "Compute": { "InstanceType": "m3.xlarge", "VolumeType": "ephemeral", "VolumeSize": 40, "VolumeCount": 2, "InstanceCount": 0, "RecoveryMode": "AUTO" }, "SSHKeyName": "xxx-pem", "RemoteAccess": "x.x.x.x/32", "WebAccess": true, "HiveJDBCAccess": true, "ClusterComponentAccess": true, "ClusterAndAmbariUser": "admin", "ClusterAndAmbariPassword": "", "Tags": {}, "Autoscaling": { "Configurations": { "CooldownTime": 30, "ClusterMinSize": 3, "ClusterMaxSize": 100 } }, "InstanceRole": "CREATE" }

Re: Horotonworks Data Cloud error autoscaling token expired

Carolyn — Wed, 03 May 2017 23:45:26 GMT

@Marton Sereg I was not using spot instances but perhaps the setting is not read properly?