Member since
07-05-2017
19
Posts
0
Kudos Received
0
Solutions
07-31-2017
07:06 AM
Hi, dynamic allocation is enabled and the value of max dynamic allocation executors is set to 10. My yarn container min. size is 4GB, executor size is 10 GB, overhead memory is 384MB, so the containers are launching with 12GB. We are ok with 12 GB container size and also we are clear on the memory utilization. But, we failed to understand how spark is launching more containers than the no. of executors. I think for driver also it uses AM container rather than launching separate container for the driver process.
... View more
07-29-2017
05:28 PM
Hi, Please correct me if I am wrong. As per my understanding Spark can run equal no. of containers to that of Executors. When we run a job with 600 executors, the spark trying to run more containers roughly around 700 causing the job fail as my queue allow ro run upto 620 containers. Please help me in this issue. As per my understanding all tasks are executed in spark executor containers. Then, how come these extra containers got launched
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
07-20-2017
08:37 AM
Hi, My cluster is kerbeorized with KMS. When I execute the following command I am getting the below error. Could anyone help me on this asap. curl --negotiate -u : -X GET http://<clustername>:9292/kms/v1/keyversion/<keyname@0/_eek?ee_op=decrypt { "RemoteException" : { "message" : null, "exception" : "WebApplicationException", "javaClassName" : "javax.ws.rs.WebApplicationException" } In the kms-audit.log I see the following error "ErrorMsg:'Authentication required' saying that Authentication required. I specified --negotiate option and -u option as well. But the below command is working curl --negotiate -u : -X GET http://<clustername>:9292/kms/v1/keyversion/<keyname@0 { "name" : "<keyname", "versionName" : "<keyname>@0", "material" : "<some value>" } The user with which the command is being executed has the valid Kerberos ticket and decrypt permissions also set with Ranger kms web ui
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Ranger
07-20-2017
07:13 AM
Hi, My cluster is kerbeorized with KMS. When I execute the following command I am getting the below error. Could anyone help me on this asap. curl --negotiate -u : -X GET http://<clustername>:9292/kms/v1/keyversion/<keyname@0/_eek?ee_op=decrypt {
"RemoteException" : {
"message" : null,
"exception" : "WebApplicationException",
"javaClassName" : "javax.ws.rs.WebApplicationException"
} In the kms-audit.log I see the following error "ErrorMsg:'Authentication required' saying that Authentication required. I specified --negotiate option and -u option as well. But the below command is working curl --negotiate -u : -X GET http://<clustername>:9292/kms/v1/keyversion/<keyname@0 {
"name" : "<keyname",
"versionName" : "<keyname>@0",
"material" : "<some value>"
} The user with which the command is being executed has the valid Kerberos ticket and decrypt permissions also set with Ranger kms web ui
... View more
Labels:
- Labels:
-
Apache Ranger
-
Kerberos
07-17-2017
01:23 PM
@Jay Thanks for your immediate reply. The above commands resolved the issue.
... View more
07-17-2017
01:09 PM
Hi, I am using hdp 2.4 with ambari 2.2. I am continuously seeing the following message in ambari server logs. Unable to run the ambari_server_stale_alerts alert: java.lang.NullPointerException WARN [qtp-ambari-client-62] servletHandler:563 - /api/v1/clusters/<ournameofCluster>/alerts The curl command to ambari alerts received status 500 server error. curl -u <user>:<password> -X GET http://<host>:8080/api/v1/clusters/<nameofourcluster>/alerts the same curl command is working for alert_definitions. Please help me on this asap. Thanks
... View more
Labels:
- Labels:
-
Apache Ambari