About mdh_raghavendra

mdh_raghavendra · ‎12-12-2019

Hi, We have KMS keys that are in 128 bit length. Is there any way to change their length from 128 to 256 ? For new keys, we know that using "hadoop key <keyname> -size 256" will create a key with 256 bit length. But, for existing keys, is there any way to change the length ?

mdh_raghavendra · ‎05-17-2019

Hi, We have preemption enabled for all queues in our prod cluster. We need to run a big spark job without preemption enabled. Can someone help me on running a spark job without preemption.

mdh_raghavendra · ‎05-13-2019

Hi, Can someone please help me on this.

mdh_raghavendra · ‎05-10-2019

Hi, We have Oozie SSH action that is taking longer time. We have HA enabled for oozie. The ssh commands are getting completed in 2 minutes. However, the actions is getting updated after 10 minutes. Below is one of the sample logs. 2019-05-09 23:28:12,454 INFO WorkflowNotificationXCommand:520 - SERVER[manag002] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0022791-190411111121510-oozie-oozi-W] ACTION[0022791-190411111121510-oozie-oozi-W@pull] No Notification URL is defined. Therefore nothing to notify for job 0022791-190411111121510-oozie-oozi-W@pull 2019-05-09 23:28:17,055 WARN ActionStartXCommand:523 - SERVER[manag004] USER[svc-prod] GROUP[-] TOKEN[-] APP[myapp] JOB[0022791-190411111121510-oozie-oozi-W] ACTION[0022791-190411111121510-oozie-oozi-W@pull] E0816: Action pending=[true], status=[RUNNING]. Skipping ActionStart Execution, Error Code: E0816 2019-05-09 23:39:23,203 INFO ActionStartXCommand:520 - SERVER[manag004] USER[svc-prod] GROUP[-] TOKEN[-] APP[myapp] JOB[0022791-190411111121510-oozie-oozi-W] ACTION[0022791-190411111121510-oozie-oozi-W@orc_build] Start action [0022791-190411111121510-oozie-oozi-W@orc_build] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] Please help us asap.

mdh_raghavendra · ‎11-15-2018

Hi, Could anyone please help me on this asap. We are facing frequent fail overs with this issue.

mdh_raghavendra · ‎11-15-2018

Hi, Today our active NN went down due to thread deadlock's issue. This is the sample from the thread dump. All the IPC server handlers for both 8020 and 8040 were hung due to the below issue. Could you please explain about the lock issue. org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:422) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) Thread 372 (IPC Server handler 83 on 8040): State: WAITING Blocked count: 20620 Waited count: 2798944 Waiting on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@4391b320 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)

mdh_raghavendra · ‎08-06-2018

Check whether the time is in sync between knox server and ambari server. Check whether ntp service is running in both the machines

mdh_raghavendra · ‎08-06-2018

Add the below statement in /etc/ambari-agent/conf/ambari-agent.ini file under 'security' section and then restart the ambari agent force_https_protocol=PROTOCOL_TLSv1_2

mdh_raghavendra · ‎07-31-2018

Hi, We need to create data flow between two kafka clusters (cluster1 and cluster2). We are using Nifi sitting at cluster1 to send the messages from cluster1 kafka to cluster2 kafka. Here we wanted to use the external load balancer in front of cluster2 Kafka. However, it is not working properly. The Nifi is unable to send the data to cluster2 kafka if we use LB url in the kafka brokers section. If we put the actual kafka brokers, it is working properly. Can any one help me on this asap.

mdh_raghavendra · ‎07-23-2018

Hi, Could any one help me on this asap.

mdh_raghavendra · ‎07-21-2018

Hi, I connected to Phoenix from Soap UI via JDBC connection, to access HBase tables. When I run any query from Soap UI to access HBase phoenix tables, I am getting the "java.sql.SQLFeatureNotSupportedException" error. Could anyone help me on this.

mdh_raghavendra · ‎07-13-2018

Hi, Is it possible to setup Ambari SSO with two Knox Servers. Here my requirement is that we have two Knox servers. If one of the Knox servers goes down, automatically Ambari SSO should redirect to other knox. Can any one help me on this

mdh_raghavendra · ‎07-11-2018

@Sandeep Nemuri Thanks

mdh_raghavendra · ‎07-11-2018

@Sandeep Nemuri Thanks for providing the information. Do we have any link or page that mentioned the same ? I need to show this to client

mdh_raghavendra · ‎07-11-2018

We are using HDP2.6.3 and Ambari 2.6.0

mdh_raghavendra · ‎07-11-2018

Hi, Can anyone let me know that is it possible to configure SSL for Spark1 history server ? Can any one help me on this asap

mdh_raghavendra · ‎07-04-2018

Hi, We set the same https webhdfs URL. But, still it is pointing to http url

mdh_raghavendra · ‎07-04-2018

Hi, Can anyone help me asap

mdh_raghavendra · ‎07-04-2018

Hi, We are trying to access Webhdfs from Knox REST API. Webhdfs is configured with https. When we tried to access the webhdfs from Knox, knox was redirecting to http Name node url (50070 instead of 50470) and saying connction refused. We put NN https urls in knox webhdfs section. Please can any one help me on this asap

mdh_raghavendra · ‎05-17-2018

Can any one help me on this.

mdh_raghavendra · ‎05-17-2018

We are running Platfora spark jobs scheduled for every 30 mins. The jobs are successful in alternate runs. For every alternate run, they are failing with the below error. Can any one help me on this kms.LoadBalancingKMSClientProvider (LoadBalancingKMSClientProvider.java:doOp(96)) - KMS provider at "" threw an IOException [HTTP status [403], message [Forbidden]]!! org.apache.http.HttpException: Failed to submit application_1526036886004_0450 to YARN : Failed to renew token: Kind: kms-dt, Service:

mdh_raghavendra · ‎05-03-2018

I tried to read the data in 'ATLAS_HOOK' topic also. But nothing is there in the topic. When I tried to run the 'import-hive.sh' script, it was failing with the below error. Caused by: org.apache.atlas.notification.NotificationException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:239) at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java:212) at org.apache.atlas.notification.AbstractNotification.send(AbstractNotification.java:116) at org.apache.atlas.notification.NotificationEntityChangeListener.notifyOfEntityEvent(NotificationEntityChangeListener.java:191) at org.apache.atlas.notification.NotificationEntityChangeListener.onEntitiesUpdated(NotificationEntityChangeListener.java:89) at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:168) ... 107 more Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:769) at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:490)

mdh_raghavendra · ‎05-03-2018

Hi, We are using hdp2.6.4 in our environment. We have recently installed Atlas and setup SSL. In the Atlas logs, we continuously see the below error messages just after trying to read the kafka events. Kafka is not enabled with SSL. Could any one help me on this. We have enabled ranger plugin for kafka and had given necessary policies. I tried to read the data in 'ATLAS_ENTITIES' kafka topic. But nothing is there in the topic. 2018-05-02 17:28:04,453 WARN - [main-EventThread:] ~ Instance is active from BECOMING_ACTIVE (ServiceState:80) 2018-05-02 17:28:05,444 INFO - [NotificationHookConsumer thread-0:] ~ Atlas Server is ready, can start reading Kafka events. (NotificationHookConsumer:492) 2018-05-02 17:28:31,456 WARN - [pool-2-thread-21:] ~ javax.net.ssl.SSLException: Received fatal alert: certificate_unknown (nio:720)

mdh_raghavendra · ‎04-27-2018

Hi, From Ranger web UI, in Audit tab, we are unable to see the Audits stored in Solr Cloud (Ambari Infra). We are getting an exception 'Error running solr query, please check solr configs. IOException occured when talking to server at <>'. In the Ranger admin xa_portal.log, we can see the error saying "Unrecoverable key exception: Keystore is tampered with incorrect password" . Can any one help me on this. We have SSL applied for both ambari infra solr cloud and Ranger.

mdh_raghavendra · ‎04-02-2018

Hi, Can any one update me asap.

mdh_raghavendra · ‎04-02-2018

Hi, Is preemption only for FAIR ordering policy or can be used for FIFO ordering policy ? The reason, here why I am asking this question is that we want to use Hive LLAP in our environment. However, to use LLAP, we need to enable 'YARN preemption'. We are using FIFO ordering policies. So, enabling yarn preemption will create any impact even if we are using FIFO ordering policies.

mdh_raghavendra · ‎03-27-2018

Hi, When dynamic allocation is enabled, most of the times we are facing issues while fetching the blocks RetryingBlockFetcher: Retrying fetch (1/3) for 1 outstanding blocks after 5000ms Error RetyringBlockFetcher: Exception while beginning fetch of 1 outstanding blocks (after 1 retries) java.io.IOException: Failed to connect to <host>:<some port> Caused by java.net.ConnectException: Connection refused: <host>:<some port> We are seeing these errors continuously in the executors when we run a big spark jobs. During this time nothing is being processed and after some time these errors are getting disappeared and the processing gets resumed. This is impacting our job SLAs. Can any one help me on this

mdh_raghavendra · ‎03-06-2018

Hi, Can anyone help me on this asap.

mdh_raghavendra · ‎03-06-2018

Hi, In our environment we are using Spark dynamic execution. When the spark executors are finished with their tasks and idle for sometime (usually, 1 min) , the executor containers are being killed by the Spark AM. But here, when the new executors are launched that are depend upon the result set of the previously killed executors are throwing the lot of errors like "ERROR RetryingBlockFetcher: Exception while beginning fetch of 1 outstanding blocks" , "java.io.IOException: Failed to connect to <the killed executor host>" As per understanding, with spark dynamic execution, even if the idle executor container is killed, its result set should be accessible to other executors via external shuffle service. I don't understand why it is not working here. Please correct me if I am wrong. Can any one help me on this asap.

mdh_raghavendra · ‎03-05-2018

Hi, Can any one explain the property yarn.scheduler.capacity.node-locality-delay From the defintion, I couldn't understand the purpose of this property. Could any one explain about this.

Online	Offline
Last Visited	‎12-12-2019 11:10 AM

Date Registered	‎01-25-2017 07:04 PM
Last Visited	‎12-12-2019 11:10 AM
Total Messages Posted	77
Total Kudos Received	6

Change KMS Key Length

Disable preemption for a spark job

Re: Oozie SSH action status update is taking longe...

Oozie SSH action status update is taking longer ti...

Re: Namenode is in Hung state due to deadlock erro...

Namenode is in Hung state due to deadlock error

Re: KNOX SSO ambari login redirect Issue

Re: Heartbeat lost on Ambari 2.6.2.2

Kafka behind an external Load Balancer

Re: java.sql.SQLFeatureNotSupportedException Error

java.sql.SQLFeatureNotSupportedException Error

Ambari SSO with two Knox Servers

Re: Enable SSL for SparkHistory server (Spark1 ver...

Re: Enable SSL for SparkHistory server (Spark1 ver...

Re: Enable SSL for SparkHistory server (Spark1 ver...

Enable SSL for SparkHistory server (Spark1 version...

Re: Knox redirecting to HTTP webhdfs (we enabled h...

Re: Knox redirecting to HTTP webhdfs (we enabled h...

Knox redirecting to HTTP webhdfs (we enabled https...

Re: Ranger KMS issue

Ranger KMS issue

Re: Atlas 'certificate_unknown' error

Atlas 'certificate_unknown' error

Ranger Audit unable to query the Solr

Re: preemption for FIFO ordering policy

preemption for FIFO ordering policy

Spark shuffle is failing with connection exception...

Re: Spark external shuffle service is not working

Spark external shuffle service is not working

Any one explain yarn.scheduler.capacity.node-local...