How to request a applicationID using YARN REST API

[#1]

Trying to execute the spark program using YARN RESTAPI in HORTONWORKS 2.5

i have followed this link: https://community.hortonworks.com/content/kbentry/28070/starting-spark-jobs-directly-via-yarn-rest-a...

Issue iam facing is :

In the 5step: Request an application ID from YARN

curl -ikvu "knoxuser:knoxpwd" -X POST "https://$KNOX_SERVER:8443/gateway/default/resourcemanager/v1/cluster/apps/new-application"

Error:

curl -ikvu "knoxuser:knoxpwd" -X POST "https://$KNOX_SERVER:8443/gateway/default/resourcemanager/v1/cluster/apps/new-application" * About to connect() to $KNOX_SERVER port 8443 (#0) * Trying 53.244.194.234... connected * Connected to $KNOX_SERVER (53.244.194.234) port 8443 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs/ * SSLv3, TLS handshake, Client hello (1): * error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol * Closing connection #0 curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

where do i find the url or i am going wrong in the url part in the command..

Please let me know, it would be great help to me.

[#2]

@shashi kumar The URL looks okay - try doing a curl directly to the ResourceManager (i.e. without Knox) to verify that it is working as expected. This will eliminate YARN as the issue.

The error 'SSL23_GET_SERVER_HELLO:unknown protocol' looks like there is an issue establishing an SSL connection to Knox so I think this is the source of your issue. Check that the Knox server is set up correctly and all the certificates are working properly.

[#3]

Thanks for replying, In our cluster, the SSL is disabled.

Now i executed a command, which curl directly to the ResourceManager.

COMMAND: curl -v -X POST "http://xxxxxxxxxx.xxxxx.xxxxxxxx.net:8088/ws/v1/cluster/apps/new-application" ERROR: * About to connect() to xxxxxxxxxx.xxxxx.xxxxxxxx.net port 8088 (#0) * Trying 53.244.194.222... connected * Connected to xxxxxxxxxx.xxxxx.xxxxxxxx.net (53.244.194.222) port 8088 (#0) > POST /ws/v1/cluster/apps/new-application HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-suse-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.7 libidn/1.10 > Host: xxxxxxxxxx.xxxxx.xxxxxxxx.net:8088 > Accept: */* > < HTTP/1.1 403 Forbidden < Cache-Control: no-cache < Expires: Tue, 24 Jan 2017 07:40:09 GMT < Date: Tue, 24 Jan 2017 07:40:09 GMT < Pragma: no-cache < Expires: Tue, 24 Jan 2017 07:40:09 GMT < Date: Tue, 24 Jan 2017 07:40:09 GMT < Pragma: no-cache < Content-Type: application/json < X-Frame-Options: SAMEORIGIN < Transfer-Encoding: chunked < Server: Jetty(6.1.26.hwx) < * Connection #0 to host xxxxxxxxxx.xxxxx.xxxxxxxx.net left intact * Closing connection #0 The default static user cannot carry out this operation.

is there any other issue ..i need to fix up?

[#4]

Hello @shashi kumar @Terry Stebbens have you found a solution to this? I am getting the same issue.

I am trying locally on a kerberized sandbox. I have kinit before I run:

curl --negotiate -u : -v -X POST 'http://<fqdn>:8088/ws/v1/cluster/apps/new-application'

getting:

* About to connect() to <fqdn> port 8088 (#0)
*   Trying 192.168.33.11... connected
* Connected to <fqdn> (192.168.33.11) port 8088 (#0)
> POST /ws/v1/cluster/apps/new-application HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: <fqdn>:8088
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Cache-Control: no-cache
< Expires: Wed, 17 May 2017 20:03:25 GMT
< Date: Wed, 17 May 2017 20:03:25 GMT
< Pragma: no-cache
< Expires: Wed, 17 May 2017 20:03:25 GMT
< Date: Wed, 17 May 2017 20:03:25 GMT
< Pragma: no-cache
< Content-Type: application/json
< X-Frame-Options: SAMEORIGIN
< Transfer-Encoding: chunked
< Server: Jetty(6.1.26.hwx)
<
* Connection #0 to host <fqdn> left intact
* Closing connection #0
The default static user cannot carry out this operation.

[#5]

Have you tried providing the user explicitly?

curl -v -X POST 'http://<RM>:8088/ws/v1/cluster/apps/new-application?user.name=yarn'

[#6]

In case anyone still need this and get the same problem as me, when using kerberos HDP.

You need to follow these steps to setup HTTP authentication before spnego would work for the YARN RE...

Once these are added in ambari and corresponding services restarted, I was able to get new application ID. However any previous anonymous access to Resource Manager now needs authentication!

[user@master hdfs]$ curl --negotiate -u : -v -X POST http://<fqdn>:8088/ws/v1/cluster/apps/new-application
* About to connect() to <fqdn> port 8088 (#0)
*   Trying 192.168.33.11... connected
* Connected to <fqdn> (192.168.33.11) port 8088 (#0)
> POST /ws/v1/cluster/apps/new-application HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: <fqdn>:8088
> Accept: */*
>
< HTTP/1.1 401 Authentication required
< Cache-Control: must-revalidate,no-cache,no-store
< Date: Thu, 18 May 2017 13:42:56 GMT
< Pragma: no-cache
< Date: Thu, 18 May 2017 13:42:56 GMT
< Pragma: no-cache
< Content-Type: text/html; charset=iso-8859-1
< WWW-Authenticate: Negotiate
< Set-Cookie: hadoop.auth=; Path=/; Domain=<my domain>; HttpOnly
< Content-Length: 1427
< Server: Jetty(6.1.26.hwx)
<
* Ignoring the response-body
* Connection #0 to host <fqdn> left intact
* Issue another request to this URL: 'http://<fqdn>:8088/ws/v1/cluster/apps/new-application'
* Re-using existing connection! (#0) with host <fqdn>
* Connected to <fqdn> (192.168.33.11) port 8088 (#0)
* Server auth using GSS-Negotiate with user ''
> POST /ws/v1/cluster/apps/new-application HTTP/1.1
> Authorization: Negotiate YIICaQYJKoZIhvcSAQICAQBuggJYMIICVKADAgEFoQMCAQ6iBwMFAAAAAACjggFtYYIBaTCCAWWgAwIBBaEJGwdMQkcuQ09NoikwJ6ADAgEDoSAwHhsESFRUUBsWd29ya2VyLnNhbmRib3gubGJnLmNvbaOCASYwggEioAMCARKhAwIBAaKCARQEggEQYSIFHaKqdpjxjhuBgc+K8swimOG+UeIxhgNYhOEQXgfFcyQoNRcPwFRS0nbXeLN8HT8S8QEib5/KXJoPj0On7r7gWNDlcYEI9ycAJ8xe11FE5WTMgSL2BDeiOtA6OLLYGj5rHFCwsWByLBwBu8jI5Bmmnx93jN+XkjPWxvrS3dBwU3qDiwbWfqze34JDfLBAWJBjke0KcFCrzA9an4fw7Evvflu9NtT/XixW7edfF0+anV/tcrBSPqj1UFKqqNr2bYOdes3pApixmohe9xAvCd4Wg6T5JLUwRlbfdt/beqwMwkY0a1WpnnFOeuOoB6ReUIcsufmRZGMkrIh63mIz/O13lbQlzXOhjBfwKyiMo/Kkgc0wgcqgAwIBEqKBwgSBvwZfOYFelpjopPr89JOyFtKzPC6xxCyLjNGAZHMFF/VKHKtdytbf7Dy5YNtcoCK1nu2D8Ihkum1hYaxH1ugK4i5sKU8xaAp0qNanc6Lu+Y7sUH/s5XKCqwVQM96mYC0ejpWIq8WDrB3CX5+MshSOnbeIEcMyG8puQ/5nHfUlNsOC7vhq4Qbs8yTTqG+9W7+79sl9fbhmVqIOx5UUfHXtq3qkKAtgmSoQhpDi4ERC/bYBIMYyubtPiXKC/k0JxSyn
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: <fqdn>:8088
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Expires: Thu, 18 May 2017 13:42:56 GMT
< Date: Thu, 18 May 2017 13:42:56 GMT
< Pragma: no-cache
< Expires: Thu, 18 May 2017 13:42:56 GMT
< Date: Thu, 18 May 2017 13:42:56 GMT
< Pragma: no-cache
< Content-Type: application/json
< Set-Cookie: hadoop.auth="u=user&p=user@<my domain>&t=kerberos&e=1495150976195&s=NxiE0Svo7+3QTPXC8L9aUlPN54c="; Path=/; Domain=<my domain>; HttpOnly
< X-Frame-Options: SAMEORIGIN
< Transfer-Encoding: chunked
< Server: Jetty(6.1.26.hwx)
<
* Connection #0 to host <fqdn> left intact
* Closing connection #0
{"application-id":"application_1495114416899_0001","maximum-resource-capability":{"memory":4096,"vCores":6}}[user@master hdfs]$