About mdh_raghavendra

mdh_raghavendra · ‎05-17-2019

Hi, We have preemption enabled for all queues in our prod cluster. We need to run a big spark job without preemption enabled. Can someone help me on running a spark job without preemption.

mdh_raghavendra · ‎05-13-2019

Hi, Can someone please help me on this.

mdh_raghavendra · ‎05-10-2019

Hi, We have Oozie SSH action that is taking longer time. We have HA enabled for oozie. The ssh commands are getting completed in 2 minutes. However, the actions is getting updated after 10 minutes. Below is one of the sample logs. 2019-05-09 23:28:12,454 INFO WorkflowNotificationXCommand:520 - SERVER[manag002] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0022791-190411111121510-oozie-oozi-W] ACTION[0022791-190411111121510-oozie-oozi-W@pull] No Notification URL is defined. Therefore nothing to notify for job 0022791-190411111121510-oozie-oozi-W@pull 2019-05-09 23:28:17,055 WARN ActionStartXCommand:523 - SERVER[manag004] USER[svc-prod] GROUP[-] TOKEN[-] APP[myapp] JOB[0022791-190411111121510-oozie-oozi-W] ACTION[0022791-190411111121510-oozie-oozi-W@pull] E0816: Action pending=[true], status=[RUNNING]. Skipping ActionStart Execution, Error Code: E0816 2019-05-09 23:39:23,203 INFO ActionStartXCommand:520 - SERVER[manag004] USER[svc-prod] GROUP[-] TOKEN[-] APP[myapp] JOB[0022791-190411111121510-oozie-oozi-W] ACTION[0022791-190411111121510-oozie-oozi-W@orc_build] Start action [0022791-190411111121510-oozie-oozi-W@orc_build] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] Please help us asap.

mdh_raghavendra · ‎11-15-2018

Hi, Could anyone please help me on this asap. We are facing frequent fail overs with this issue.

mdh_raghavendra · ‎11-15-2018

Hi, Today our active NN went down due to thread deadlock's issue. This is the sample from the thread dump. All the IPC server handlers for both 8020 and 8040 were hung due to the below issue. Could you please explain about the lock issue. org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:422) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) Thread 372 (IPC Server handler 83 on 8040): State: WAITING Blocked count: 20620 Waited count: 2798944 Waiting on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@4391b320 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)

mdh_raghavendra · ‎10-14-2018

Hi, HBase region server process is not creating its pid file under /tmp/hsperfdata folder. Could anyone help me on this.

mdh_raghavendra · ‎10-14-2018

Hi, Can any one explain me how HBase scan works internally ? What happens internally when we fire hbase scan command ? Could anyone also explain me, is the data locality in HBase depends upon Region server or Region ?

mdh_raghavendra · ‎08-06-2018

Check whether the time is in sync between knox server and ambari server. Check whether ntp service is running in both the machines

mdh_raghavendra · ‎08-06-2018

Add the below statement in /etc/ambari-agent/conf/ambari-agent.ini file under 'security' section and then restart the ambari agent force_https_protocol=PROTOCOL_TLSv1_2

mdh_raghavendra · ‎07-31-2018

Hi, We need to create data flow between two kafka clusters (cluster1 and cluster2). We are using Nifi sitting at cluster1 to send the messages from cluster1 kafka to cluster2 kafka. Here we wanted to use the external load balancer in front of cluster2 Kafka. However, it is not working properly. The Nifi is unable to send the data to cluster2 kafka if we use LB url in the kafka brokers section. If we put the actual kafka brokers, it is working properly. Can any one help me on this asap.

mdh_raghavendra · ‎07-24-2018

Hi, When I open the schema registry URL, it is stuck and loading forever as in attached image and also not showing the existing schemas. When, I try to create new schema, I am getting the exception like " TypeError: Cannot read property 'schemaVersionInfos'". Can any one please help me on this asap registry-error.jpg

mdh_raghavendra · ‎07-23-2018

Hi, Could any one help me on this asap.

mdh_raghavendra · ‎07-21-2018

Hi, I connected to Phoenix from Soap UI via JDBC connection, to access HBase tables. When I run any query from Soap UI to access HBase phoenix tables, I am getting the "java.sql.SQLFeatureNotSupportedException" error. Could anyone help me on this.

mdh_raghavendra · ‎07-13-2018

Hi, Is it possible to setup Ambari SSO with two Knox Servers. Here my requirement is that we have two Knox servers. If one of the Knox servers goes down, automatically Ambari SSO should redirect to other knox. Can any one help me on this

mdh_raghavendra · ‎07-11-2018

@Sandeep Nemuri Thanks

mdh_raghavendra · ‎07-11-2018

@Sandeep Nemuri Thanks for providing the information. Do we have any link or page that mentioned the same ? I need to show this to client

mdh_raghavendra · ‎07-11-2018

We are using HDP2.6.3 and Ambari 2.6.0

mdh_raghavendra · ‎07-11-2018

Hi, Can anyone let me know that is it possible to configure SSL for Spark1 history server ? Can any one help me on this asap

mdh_raghavendra · ‎07-04-2018

Hi, We set the same https webhdfs URL. But, still it is pointing to http url

mdh_raghavendra · ‎07-04-2018

Hi, Can anyone help me asap

mdh_raghavendra · ‎07-04-2018

Hi, We are trying to access Webhdfs from Knox REST API. Webhdfs is configured with https. When we tried to access the webhdfs from Knox, knox was redirecting to http Name node url (50070 instead of 50470) and saying connction refused. We put NN https urls in knox webhdfs section. Please can any one help me on this asap

mdh_raghavendra · ‎05-17-2018

Can any one help me on this.

mdh_raghavendra · ‎05-17-2018

We are running Platfora spark jobs scheduled for every 30 mins. The jobs are successful in alternate runs. For every alternate run, they are failing with the below error. Can any one help me on this kms.LoadBalancingKMSClientProvider (LoadBalancingKMSClientProvider.java:doOp(96)) - KMS provider at "" threw an IOException [HTTP status [403], message [Forbidden]]!! org.apache.http.HttpException: Failed to submit application_1526036886004_0450 to YARN : Failed to renew token: Kind: kms-dt, Service:

mdh_raghavendra · ‎05-03-2018

I tried to read the data in 'ATLAS_HOOK' topic also. But nothing is there in the topic. When I tried to run the 'import-hive.sh' script, it was failing with the below error. Caused by: org.apache.atlas.notification.NotificationException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:239) at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java:212) at org.apache.atlas.notification.AbstractNotification.send(AbstractNotification.java:116) at org.apache.atlas.notification.NotificationEntityChangeListener.notifyOfEntityEvent(NotificationEntityChangeListener.java:191) at org.apache.atlas.notification.NotificationEntityChangeListener.onEntitiesUpdated(NotificationEntityChangeListener.java:89) at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:168) ... 107 more Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:769) at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:490)

mdh_raghavendra · ‎05-03-2018

Hi, We are using hdp2.6.4 in our environment. We have recently installed Atlas and setup SSL. In the Atlas logs, we continuously see the below error messages just after trying to read the kafka events. Kafka is not enabled with SSL. Could any one help me on this. We have enabled ranger plugin for kafka and had given necessary policies. I tried to read the data in 'ATLAS_ENTITIES' kafka topic. But nothing is there in the topic. 2018-05-02 17:28:04,453 WARN - [main-EventThread:] ~ Instance is active from BECOMING_ACTIVE (ServiceState:80) 2018-05-02 17:28:05,444 INFO - [NotificationHookConsumer thread-0:] ~ Atlas Server is ready, can start reading Kafka events. (NotificationHookConsumer:492) 2018-05-02 17:28:31,456 WARN - [pool-2-thread-21:] ~ javax.net.ssl.SSLException: Received fatal alert: certificate_unknown (nio:720)

mdh_raghavendra · ‎04-27-2018

Hi, From Ranger web UI, in Audit tab, we are unable to see the Audits stored in Solr Cloud (Ambari Infra). We are getting an exception 'Error running solr query, please check solr configs. IOException occured when talking to server at <>'. In the Ranger admin xa_portal.log, we can see the error saying "Unrecoverable key exception: Keystore is tampered with incorrect password" . Can any one help me on this. We have SSL applied for both ambari infra solr cloud and Ranger.

mdh_raghavendra · ‎04-26-2018

Hi, Can any one post me the doc or help me how to setup SSL for Ambari infra Solr Cloud (with CA signed) asap. I couldn't find any doc on this.

mdh_raghavendra · ‎04-02-2018

Hi, Can any one update me asap.

mdh_raghavendra · ‎04-02-2018

Hi, Is preemption only for FAIR ordering policy or can be used for FIFO ordering policy ? The reason, here why I am asking this question is that we want to use Hive LLAP in our environment. However, to use LLAP, we need to enable 'YARN preemption'. We are using FIFO ordering policies. So, enabling yarn preemption will create any impact even if we are using FIFO ordering policies.

mdh_raghavendra · ‎03-27-2018

Hi, When dynamic allocation is enabled, most of the times we are facing issues while fetching the blocks RetryingBlockFetcher: Retrying fetch (1/3) for 1 outstanding blocks after 5000ms Error RetyringBlockFetcher: Exception while beginning fetch of 1 outstanding blocks (after 1 retries) java.io.IOException: Failed to connect to <host>:<some port> Caused by java.net.ConnectException: Connection refused: <host>:<some port> We are seeing these errors continuously in the executors when we run a big spark jobs. During this time nothing is being processed and after some time these errors are getting disappeared and the processing gets resumed. This is impacting our job SLAs. Can any one help me on this

Online	Offline
Last Visited	‎05-17-2019 01:41 PM

Date Registered	‎01-25-2017 07:04 PM
Last Visited	‎05-17-2019 01:41 PM
Total Messages Posted	76
Total Kudos Received	6

Disable preemption for a spark job

Re: Oozie SSH action status update is taking longe...

Oozie SSH action status update is taking longer ti...

Re: Namenode is in Hung state due to deadlock erro...

Namenode is in Hung state due to deadlock error

hbase regionserver is not creating pid file in /tm...

How HBase scan works internally ?

Re: KNOX SSO ambari login redirect Issue

Re: Heartbeat lost on Ambari 2.6.2.2

Kafka behind an external Load Balancer

Schema Registry UI is loading forever

Re: java.sql.SQLFeatureNotSupportedException Error

java.sql.SQLFeatureNotSupportedException Error

Ambari SSO with two Knox Servers

Re: Enable SSL for SparkHistory server (Spark1 ver...

Re: Enable SSL for SparkHistory server (Spark1 ver...

Re: Enable SSL for SparkHistory server (Spark1 ver...

Enable SSL for SparkHistory server (Spark1 version...

Re: Knox redirecting to HTTP webhdfs (we enabled h...

Re: Knox redirecting to HTTP webhdfs (we enabled h...

Knox redirecting to HTTP webhdfs (we enabled https...

Re: Ranger KMS issue

Ranger KMS issue

Re: Atlas 'certificate_unknown' error

Atlas 'certificate_unknown' error

Ranger Audit unable to query the Solr

SSL for Ambari Infra Solr Cloud (With CA signed)

Re: preemption for FIFO ordering policy

preemption for FIFO ordering policy

Spark shuffle is failing with connection exception...