Is there a way to create Spark Dashboard in Activity Explorer. We don't have it by default. Just wondering, if it is possible to create one. ... View more

HiveServer2 Interactive UI is showing Launching Instances. This seems to be stale llap deamons. How can we find these deamon process running on which server as we have around 300 nodes?         ... View more

Hi Team, I am trying to find out Ambari API to Trun on Maintenance Mode for specific rack (rack awareness is configured). I could use Ambari API for same operation for specific host and set of hosts but it would be very useful to have this for specific rack.    curl -u admin:admin -H "X-Requested-By: ambari" -X PUT -d '{"RequestInfo":{"context":"Turn On Maintenance Mode for host","query":"Hosts/host_name.in(host1,host2)"},"Body":{"Hosts":{"maintenance_state":"ON"}}}' http://ambari:8080/api/v1/clusters/c1/hosts   I tried to capture the response from Developer Tools -> Network in Chrome and could see that it consider all the hosts in that specific rack.  Any help on this. ... View more

Hi Team,   I am running a select query with some condition in llap which shows no data. When I run exactly the same query in hive is giving output. Count shows 676566 rows to be returned.    Any help on this issue. ... View more

Hi Team, I am trying to create a custom metric and import NFS metrics into Grafana but it doesn't seems to be working. Followed, https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html and now metrics is  available at  http://<hostname>:50079/jmx  as a JSON document. <property> <name>nfs.metrics.percentiles.intervals</name> <value>100</value> </property> Also, tried to import json as mentioned in below link but no luck.  https://grafana.com/grafana/dashboards/6257 HDP - 2.6.5 Ambari - 2.6.2.2 Any help on this.   ... View more

Hi Team,   We have noticed Spark2 Thrift server encounters authentication failures and requires manual restarts.  19/08/29 04:43:27 WARN Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] Similar issue has been reported in https://community.cloudera.com/t5/Support-Questions/Spark-Thrift-Servers-are-failing-to-renew-TGTs-automatically/m-p/103833 I could see spark.yarn.keytab and spark.yarn.principal is set in custom spark2-thrift-sparkconf     HDP Version : 2.6.5 Spark Version : 2.3.0 ... View more

Hi Team,   I am trying to load data into Druid from Hive and currently it is failing with below error.    ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: Cannot load JDBC driver class 'com.mysql.jdbc.Driver' INFO : Resetting the caller context to HIVE_SSN_ID:d5bd4c20-1bee-4b6c-8447-e0e2675b9b31 INFO : Completed executing command(queryId=hive_20190828144836_480305b7-2c2a-4587-a5f3-8109d8aa4231); Time taken: 251.571 seconds Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: Cannot load JDBC driver class 'com.mysql.jdbc.Driver' (state=08S01,code=1)     As per below articles, issue seems to be related to mysql-connector-java.jar. I have performed ambari-server setup while installing druid and there was no issues during that time. https://groups.google.com/forum/m/#!msg/druid-user/tiRCi0P9mjw/YA9LIB_L_hgJ https://support.hortonworks.com/s/article/Loading-the-data-into-Druid-from-hive-is-failing   Verified below properties as well and it is present in Hive LLAP. hive.druid.metadata.username=${DRUID_USERNAME}; hive.druid.metadata.password=${DRUID_PASSWORD}; hive.druid.metadata.uri=jdbc:mysql://${DRUID_HOST}/druid; HDP Version : 2.6.5 Ambari Version: 2.6.2.2   Query ran in LLAP   CREATE TABLE default.druid_test_table (`__time` TIMESTAMP, `dimension1` STRING, `dimension2` STRING, `metric1` INT, `metric2` FLOAT) STORED BY 'org.apache.hadoop.hive.druid.DruidStorageHandler';   Any help on this issue?   Regards, Shyam ... View more

Hi All, We have observed that whenever VCores Total equals to VCores Used, total memory in RM UI shows incorrect memory. We have checked and all the NodeManagers are up. Is this a expected behavior/Bug? Regards, Shyam ... View more

Hi Team, I have a spark job which needs few extra jars to run. How to pass extra jars in Oozie Spark action. I tried to pass using --conf spark.executor.extraClassPath --conf spark.executor.driverClassPath but that is not working. I also tried to place the jars in lib folder of application path but no luck. Any help on this. ... View more

Hi Team, I have a Spark job which write to Phoenix table. I am able to run the job using spark-submit command but when I schedule the job using oozie spark action, it is failing. spark-submit --jars /usr/hdp/current/hbase-client/lib/hbase-client.jar,/usr/hdp/current/hbase-client/lib/hbase-common.jar,/usr/hdp/current/hbase-client/lib/hbase-server.jar,/usr/hdp/current/hbase-client/lib/guava-12.0.1.jar,/usr/hdp/current/hbase-client/lib/hbase-protocol.jar,/usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/current/hive-client/lib/hive-hbase-handler.jar --conf spark.speculation=false --master yarn --deploy-mode=cluster --conf spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/lib/phoenix-spark2-4.7.0.2.6.5.1052-6.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar --conf spark.driver.extraClassPath=/usr/hdp/current/phoenix-client/lib/phoenix-spark2-4.7.0.2.6.5.1052-6.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar --conf spark.yarn.queue=default --files "/home/shyam/test.properties,/etc/spark2/conf/hbase-site.xml" --py-files test.zip model_main.py Extra jars requires to run the job /usr/hdp/current/hbase-client/lib/hbase-client.jar,/usr/hdp/current/hbase-client/lib/hbase-common.jar,/usr/hdp/current/hbase-client/lib/hbase-server.jar,/usr/hdp/current/hbase-client/lib/guava-12.0.1.jar,/usr/hdp/current/hbase-client/lib/hbase-protocol.jar,/usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/current/hive-client/lib/hive-hbase-handler.jar Error Message in log Caused by: java.lang.RuntimeException: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:679) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:637) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:745) ... 17 more Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:611) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:156) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:737) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:734) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:734)  I am passing the keytab in workflow.xml as <file> tag. Extra jars I am passing in <spark-opts> <action name="model_main" cred="hive_auth">         <spark xmlns="uri:oozie:spark-action:0.2">             <job-tracker>${jobTracker}</job-tracker>             <name-node>${nameNode}</name-node>             <configuration>                 <property>                     <name>mapred.job.queue.name</name>                     <value>${queueName}</value>                 </property>             </configuration>             <master>${master}</master>             <mode>${mode}</mode>             <name>Model</name>             <jar>model_main.py</jar>             <spark-opts>${sparkOpts}</spark-opts>             <file>${nameNode}${filePath}/model_main.py#model_main.py</file>             <file>${filePath}/dist#dist</file>             <file>${filePath}/test.properties#test.properties</file>             <file>${filePath}/shyam.keytab#shyam.keytab</file>             <file>${filePath}/hbase-site.xml#hbase-site.xml</file>         </spark> nameNode=hdfs://testhdp filePath=/user/shyam/code/oozie_source/model_test oozie.wf.application.path=${nameNode}${filePath} jobTracker=yarn-cluster master=yarn mode=cluster oozie.action.sharelib.for.spark=spark2 oozie.use.system.libpath=true workflow_name=model queueName=default user=shyam kerb_auth=keytab python_path=/opt/anaconda2/envs/python3/bin/python3 phoenix_jars=/usr/hdp/current/phoenix-client/lib/phoenix-spark2-4.7.0.2.6.5.1052-6.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar libDir=${nameNode}${filePath}/lib sparkLib=${libDir}/hbase-client.jar,${libDir}/hbase-common.jar,${libDir}/hbase-server.jar,${libDir}/hbase-protocol.jar,${libDir}/hive-hbase-handler.jar #hbase_jars=/usr/hdp/current/hbase-client/lib/hbase-client.jar,/usr/hdp/current/hbase-client/lib/hbase-common.jar,/usr/hdp/current/hbase-client/lib/hbase-server.jar,/usr/hdp/current/hbase-client/lib/hbase-protocol.jar,/usr/hdp/current/hive-client/lib/hive-hbase-handler.jar sparkOpts=--jars file:///usr/hdp/current/hbase-client/lib/hbase-client.jar,file:///usr/hdp/current/hbase-client/lib/hbase-common.jar,file:///usr/hdp/current/hbase-client/lib/hbase-server.jar,file:///usr/hdp/current/hbase-client/lib/hbase-protocol.jar,file:///usr/hdp/current/hive-client/lib/hive-hbase-handler.jar --master yarn --deploy-mode cluster --conf spark.executor.extraClassPath=${phoenix_jars} --conf spark.driver.extraClassPath=${phoenix_jars} --conf spark.yarn.queue=${queueName} --conf spark.executorEnv.PYSPARK_PYTHON=${python_path} --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=${python_path} --conf spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON=${python_path} --files "test.properties,hbase-site.xml,shyam.keytab" --py-files dist/*.egg I tried to place the dependent jars in lib folder of application path and then pass it to <spark-opts> but that didn't worked. It failed with java.lang.IllegalStateException: Variable substitution depth too large: 20 java.lang.IllegalStateException: Variable substitution depth too large: 20 --jars ${sparkLib} --conf spark.speculation=false --master yarn --deploy-mode cluster --conf spark.executor.extraClassPath=${phoenix_jars} --conf spark.driver.extraClassPath=${phoenix_jars} --conf spark.yarn.queue=${queueName} --conf spark.executorEnv.PYSPARK_PYTHON=${python_path} --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=${python_path} --conf spark.yarn.appMasterEnv.PYSPARK_DRIVER_PYTHON=${python_path} --files "test.properties,hbase-site.xml,shyam.keytab" --py-files dist/*.egg Any help on this. ... View more

Hi All, We are trying to find out a way from where we can get the information of applications ran on YARN over a period of time. Let's say in a week. We tried with YARN/APP Timeline Server rest api's but that didn't helped much. We have AMS configured. In YARN - HOME, under Apps we can view the completed application on a day to day basis. We tried accessing hbase using phoenix client to capture the same but with no luck. Any help on this. ... View more

We have a Spark application on 2.2 with keeps failing with below error. The same application works fine on Spark 1.6.3. This seems more related to SPARK-12945. We tried by setting ("spark.ui.enabled", "false") as mentioned here but that didn't helped. Any help on this? [Stage 8:====> (17679 + 35) / 37645][Stage 10:> (0 + 0) / 11] [Stage 8:====> (17737 + 35) / 37645][Stage 10:> (0 + 0) / 11]18/04/26 07:42:22 ERROR LiveListenerBus: Listener JobProgressListener threw an exception java.lang.NullPointerException at org.apache.spark.ui.jobs.JobProgressListener$anonfun$onTaskEnd$1.apply(JobProgressListener.scala:361) at org.apache.spark.ui.jobs.JobProgressListener$anonfun$onTaskEnd$1.apply(JobProgressListener.scala:360) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32) at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45) at org.apache.spark.ui.jobs.JobProgressListener.onTaskEnd(JobProgressListener.scala:360) at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:42) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:55) at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:38) at org.apache.spark.util.AsynchronousListenerBus$anon$1$anonfun$run$1$anonfun$apply$mcV$sp$1.apply$mcV$sp(AsynchronousListenerBus.scala:87) at org.apache.spark.util.AsynchronousListenerBus$anon$1$anonfun$run$1$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:72) at org.apache.spark.util.AsynchronousListenerBus$anon$1$anonfun$run$1$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:72) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.apache.spark.util.AsynchronousListenerBus$anon$1$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:71) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1181) at org.apache.spark.util.AsynchronousListenerBus$anon$1.run(AsynchronousListenerBus.scala:70) ... View more