hello friends, We need your help in better understanding of the issue: The yarn application was failed (based on yarn application logs) due to "out of memory" and the application was submitted to 'default' queue (we are using Capacity Scheduler) and the default queue usage was for today as is below: +------------+----------------------+--------------------+--------------------+---------------+--------------------+--+ | queue | allocatedcontain | pendingcont | capacity allo | usedcapacity | datetime | +------------+----------------------+--------------------+--------------------+---------------+--------------------+--+ | default | 551 | 525 | 5.0 | 213.84296 | 08:20:18:17:00:01 | | default | 519 | 525 | 5.0 | 207.2314 | 08:20:18:16:45:01 | | default | 492 | 525 | 5.0 | 201.65288 | 08:20:18:16:35:01 | | default | 1356 | 739 | 5.0 | 380.16528 | 08:20:18:16:15:01 | | default | 543 | 525 | 5.0 | 212.19008 | 08:20:18:16:00:02 | | default | 633 | 525 | 5.0 | 230.78513 | 08:20:18:15:45:01 | | default | 527 | 525 | 5.0 | 208.8843 | 08:20:18:15:35:01 | | default | 571 | 1524 | 5.0 | 220.04132 | 08:20:18:15:15:02 | | default | 872 | 4082 | 5.0 | 279.75208 | 08:20:18:15:00:02 | | default | 1454 | 8716 | 5.0 | 400.41324 | 08:20:18:14:45:02 | | default | 1454 | 18615 | 5.0 | 400.41324 | 08:20:18:14:35:01 | | default | 1600 | 22973 | 5.0 | 430.57852 | 08:20:18:14:15:01 | If you see in the above, the default queue was allocated with only 5% cluster resources, and most of the time, queue was demanding (waiting) for containers (as indicated under: Pending Containers column) --> so finally the job was failed and when we looked into the application (yarn logs) we noticed that it was failed for 'out of memory' errors. Now the confusion for me is: 1. The job failed for 'out of memory' --> due to (it never got it's requested memory (as indicated above, the 'pending containers count never went down) --> it means, the jobs never got their requested memory --> so the job was failed for 'out of memory' (may be when they don't get their requested memory, they must be in stuck (state) but should not be failed right? 2. The job in fact was processing more data and may be actually looking for a bigger container size (the default container size was set to 8GB) I just wanted to get some clear picture here, hence i am looking for your advice. I greatly appreciate your time and help. thanks, hdpadmin ... View more

Friends, can anyone please help me with the following KNOX Read a File issue: This is currently working for "listing a file (using "ListStatus") with our Knox Gateway LOAD BALANCE R URL: $ curl -i -k -L -u <user id> 'https://knoxgateway.<CORP DOMAIN>:443/gateway/default/webhdfs/v1/user/<user id>/servers?op=ListStatus' Enter host password for user '<user id>': HTTP/1.1 200 OK Set-Cookie: JSESSIONID=9m4tcprbrs1eapxa0ljk5sfj;Path=/gateway/default;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: no-cache Expires: Mon, 24 Apr 2017 13:25:28 GMT Date: Mon, 24 Apr 2017 13:25:28 GMT Pragma: no-cache Expires: Mon, 24 Apr 2017 13:25:28 GMT Date: Mon, 24 Apr 2017 13:25:28 GMT Pragma: no-cache Server: Jetty(6.1.26.hwx) Content-Type: application/json Content-Length: 281 {"FileStatuses":{"FileStatus":[{"accessTime":1492803070763,"blockSize":134217728,"childrenNum":0,"fileId":219423467,"group":"hdfs","length":249,"modificationTime":1492803071085,"owner":"<user id>","pathSuffix":"","permission":"777","replication":3,"storagePolicy":0,"type":"FILE"} Now, i want to read this file using the same KNOX GATEWAY LB URL: $curl -i -k -L -u <user id> 'https://knoxgateway.<CORP DOMAIN>:443/gateway/test/webhdfs/v1/user/<user id>/servers?op=OPEN' < Enter host password for user '<user id>': HTTP/1.1 307 Temporary Redirect Set-Cookie: JSESSIONID=1sopqfdu53c61xutx0ufk0hij;Path=/gateway/test;Secure;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT Cache-Control: no-cache Expires: Mon, 24 Apr 2017 13:34:31 GMT Date: Mon, 24 Apr 2017 13:34:31 GMT Pragma: no-cache Expires: Mon, 24 Apr 2017 13:34:31 GMT Date: Mon, 24 Apr 2017 13:34:31 GMT Pragma: no-cache Location: https://knoxgateway.<CORP DOMAIN>/knox/test/webhdfs/data/v1/webhdfs/v1/user/<user id>/servers?_=AAAACAAAABAAAAEAlcGYLi4LTj7bhrrDPr1o2u6UIMEkO_aYiGAxiS4hu39uo-Homt5CbB2pwJ9p0Lkl2-7-l0vxINRjR70Ub7SA3D_ZKcoN46q0Bj97ceByV8hZgwEiIvyZmwSYEdKTVRCKV3VOhbuw1peDAJMhlS8SwYoPsRUOmPsdbmX5NLysp7mM7qktkmbHJyf_qXiAwNYuXmIhPBW_PZMmwjmQXckj7mDGAk61P-qWy1rSPoyPZ5oZ6y-7Uwijew0C3FNZzISDJICX6ePU2ptLEJOu1G8FaQonOUi37pvblYUuKSo-0wiLnBKRIvzrjfPzvh0tKrXi7FbCQnbn9sG0IyFjWssqlIoOlUVbf-Jo9eVF653ZyIqGjIYn9aX-7g Server: Jetty(6.1.26.hwx) Content-Type: application/octet-stream Content-Length: 0 HTTP/1.1 404 Not Found Cache-Control: must-revalidate,no-cache,no-store Content-Type: text/html;charset=ISO-8859-1 Content-Length: 1324 Server: Jetty(8.1.14.v20131031) <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 Not Found</title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /knox/default/webhdfs/data/v1/webhdfs/v1/user/<user id>/servers. Reason: <pre> Not Found</pre></p> <hr /><i><small>Powered by Jetty://</small></i> I am really confused with this Error: " Problem accessing /knox/default/webhdfs/data/v1/webhdfs/v1/user/<user id>/servers: --> For me this path is wrong and i guess it is redirecting to a wrong path using 'rewrite.xml'? but i don't think we ever modified this file during the KNOX setup. Can anyone please help me / guide me for fixing this issue? I greatly appreciate your help. thank you, ... View more

Good Morning Experts, I am currently facing the following issue: We have 200 nodes hadoop cluster and we configured rack awareness, suddenly we noticed one of the datanode was missing from Ambari, but we do have that datanode process on that particular node, when we looked at the logs, we have noticed the following error: Initialization failed for Block pool BP-3x84848-92929299 (Datanode Uuid 6048438486-d001-47af-a899-6493aca15c4c) service to hostname.com/<data node ip>:8020 Failed to add /default-rack/<datanode ip>:1019: You cannot have a rack and a non-rack node at the same level of the network topology. We have added the datanode again from ambari, but after starting the datanode, but it is still complaining with the above errors in the datanode logs. I didn't see any similar question in the community, so i am looking for your help. Since this is currently an issue in our production cluster, can anyone please help me quickly? I greatly appreciate your quick help. thanks, ~hdpadmin ... View more

Friends, I need your advise/help on the following issue: We have successfully configured Hiveserver2 / SSL Ranger Plugin / Kerberos, but i never tested earlier, but recently i found a hive / ranger plugin issue, that is @ when I tried to connect hiveserver2 through beeline, i was able to connect to hiveserver2, but when i typed 'show databases' i was not getting any result, but in hiveserver2 logs,i found the following errors: Here is the hiveserver2 log: 2017-03-11 22:46:56,026 ERROR [Thread-9]: util.RangerRESTClient (RangerRESTClient.java:getTrustManagers(342)) - Unable to read the necessary SSL Keystore and TrustStore Files java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:225) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) at java.security.KeyStore.load(KeyStore.java:1445) at org.apache.ranger.plugin.util.RangerRESTClient.getTrustManagers(RangerRESTClient.java:323) at org.apache.ranger.plugin.util.RangerRESTClient.buildClient(RangerRESTClient.java:190) at org.apache.ranger.plugin.util.RangerRESTClient.getClient(RangerRESTClient.java:177) at org.apache.ranger.plugin.util.RangerRESTClient.getResource(RangerRESTClient.java:157) at org.apache.ranger.admin.client.RangerAdminRESTClient.createWebResource(RangerAdminRESTClient.java:162) at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:70) at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:215) at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:183) at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:156) Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778) ... 13 more 2017-03-11 22:46:56,027 ERROR [Thread-9]: util.PolicyRefresher (PolicyRefresher.java:loadPolicyfromPolicyAdmin(238)) - PolicyRefresher(serviceName=TEST_hive): failed to refresh policies. Will continue to use last known version of policies (-1) java.lang.IllegalArgumentException: SSLContext must not be null I have verified the "java keystore & trust store password" as i was able to list those two stores (keytool command) with the passwords. Can anyone please help me on resolving this issue. Thank you, Subrah ... View more

Hello, Can anyone please explain me how Knox handover user information to Ranger, so that Ranger can apply it's policies on that particular user? based on user's group ( i believe Ranger pulls the user group using SSSD from AD). Here is brief summary: We are building 250+ node HDP Cluster at my work place and we have already implemented Kerberos and integrated with AD and established one way trust with local KDC, before that we have completed several steps (for joining our HDP nodes with AD using SSSD), we also added two knox gateway servers for perimeter security and we got a LB infront of Knox GW Servers and for authorization, we got Ranger. We have added LDAP/AD for authenticating Knox users. My question is: 1. Once Knox receives a request, it will be authenticated by LDAP. (i believe user login/pwd information is included in the URL, we have implemented SSL). Once Authenticated, then knox uses, user "knox" with Keytab to get TGT from Kerberos, right? My question here is: Where do we mention this one? (what properties, we should define to inform, Knox, to use "knox" user to go to local KDC for getting TGT. 2. Once Knox gets "TGT", suppose, user is trying to access a Hive DB, then, his request will be validated in Ranger (for authorization right? , at this point, Ranger retrieves "original User" information from the request and pulls the Group Information from AD using SSSD to validate the permissions for that user right? -> My question here is: how Ranger retrieves that User details? (what confuses me is: it gets "TGT" using "knox user", but when request goes to Ranger, how ranger retrieves that user details? --> Where is this configuration? (@ Properties? where we define this?) 3. When it goes to Ranger: How ranger pulls the Group details, belongs to that particular User? (is that through LDAP Sync) --> where do we define Properties to inform Ranger to pull the Group Details from AD using SSSD? I completely follow the overall flow, but i am actually looking the exact parameters (or) properties, for these connections. BTW, i read the documentation, but still didn't find the exact linking properties. I greatly appreciate your time and response. (If possible please explain in detail). Thanks, hdpadmin ... View more