Member since
01-25-2017
77
Posts
6
Kudos Received
0
Solutions
12-12-2019
07:50 AM
Hi, We have KMS keys that are in 128 bit length. Is there any way to change their length from 128 to 256 ? For new keys, we know that using "hadoop key <keyname> -size 256" will create a key with 256 bit length. But, for existing keys, is there any way to change the length ?
... View more
- Tags:
- cloudera navigator
05-17-2019
01:41 PM
Hi, We have preemption enabled for all queues in our prod cluster. We need to run a big spark job without preemption enabled. Can someone help me on running a spark job without preemption.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Spark
-
Apache YARN
05-13-2019
07:48 AM
Hi, Can someone please help me on this.
... View more
05-10-2019
02:29 PM
Hi, We have Oozie SSH action that is taking longer time. We have HA enabled for oozie. The ssh commands are getting completed in 2 minutes. However, the actions is getting updated after 10 minutes. Below is one of the sample logs. 2019-05-09 23:28:12,454 INFO WorkflowNotificationXCommand:520 - SERVER[manag002] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0022791-190411111121510-oozie-oozi-W] ACTION[0022791-190411111121510-oozie-oozi-W@pull] No Notification URL is defined. Therefore nothing to notify for job 0022791-190411111121510-oozie-oozi-W@pull 2019-05-09 23:28:17,055 WARN ActionStartXCommand:523 - SERVER[manag004] USER[svc-prod] GROUP[-] TOKEN[-] APP[myapp] JOB[0022791-190411111121510-oozie-oozi-W] ACTION[0022791-190411111121510-oozie-oozi-W@pull] E0816: Action pending=[true], status=[RUNNING]. Skipping ActionStart Execution, Error Code: E0816 2019-05-09 23:39:23,203 INFO ActionStartXCommand:520 - SERVER[manag004] USER[svc-prod] GROUP[-] TOKEN[-] APP[myapp] JOB[0022791-190411111121510-oozie-oozi-W] ACTION[0022791-190411111121510-oozie-oozi-W@orc_build] Start action [0022791-190411111121510-oozie-oozi-W@orc_build] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] Please help us asap.
... View more
Labels:
11-15-2018
04:46 PM
Hi, Could anyone please help me on this asap. We are facing frequent fail overs with this issue.
... View more
11-15-2018
02:13 PM
Hi, Today our active NN went down due to thread deadlock's issue. This is the sample from the thread dump. All the IPC server handlers for both 8020 and 8040 were hung due to the below issue. Could you please explain about the lock issue. org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:422)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
Thread 372 (IPC Server handler 83 on 8040):
State: WAITING
Blocked count: 20620
Waited count: 2798944
Waiting on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@4391b320
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
... View more
Labels:
- Labels:
-
Apache Hadoop
08-06-2018
09:11 AM
Check whether the time is in sync between knox server and ambari server. Check whether ntp service is running in both the machines
... View more
08-06-2018
09:07 AM
Add the below statement in /etc/ambari-agent/conf/ambari-agent.ini file under 'security' section and then restart the ambari agent force_https_protocol=PROTOCOL_TLSv1_2
... View more
07-31-2018
02:17 AM
Hi, We need to create data flow between two kafka clusters (cluster1 and cluster2). We are using Nifi sitting at cluster1 to send the messages from cluster1 kafka to cluster2 kafka. Here we wanted to use the external load balancer in front of cluster2 Kafka. However, it is not working properly. The Nifi is unable to send the data to cluster2 kafka if we use LB url in the kafka brokers section. If we put the actual kafka brokers, it is working properly. Can any one help me on this asap.
... View more
Labels:
07-23-2018
07:47 AM
Hi, Could any one help me on this asap.
... View more
07-21-2018
05:46 AM
Hi, I connected to Phoenix from Soap UI via JDBC connection, to access HBase tables. When I run any query from Soap UI to access HBase phoenix tables, I am getting the "java.sql.SQLFeatureNotSupportedException" error. Could anyone help me on this.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache HBase
-
Apache Phoenix
07-13-2018
07:30 PM
Hi, Is it possible to setup Ambari SSO with two Knox Servers. Here my requirement is that we have two Knox servers. If one of the Knox servers goes down, automatically Ambari SSO should redirect to other knox. Can any one help me on this
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hadoop
-
Apache Knox
07-11-2018
05:26 PM
@Sandeep Nemuri Thanks
... View more
07-11-2018
05:21 PM
@Sandeep Nemuri Thanks for providing the information. Do we have any link or page that mentioned the same ? I need to show this to client
... View more
07-11-2018
05:14 PM
We are using HDP2.6.3 and Ambari 2.6.0
... View more
07-11-2018
05:12 PM
Hi, Can anyone let me know that is it possible to configure SSL for Spark1 history server ? Can any one help me on this asap
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Spark
07-04-2018
06:30 PM
Hi, We set the same https webhdfs URL. But, still it is pointing to http url
... View more
07-04-2018
02:08 PM
Hi, Can anyone help me asap
... View more
07-04-2018
01:36 PM
Hi, We are trying to access Webhdfs from Knox REST API. Webhdfs is configured with https. When we tried to access the webhdfs from Knox, knox was redirecting to http Name node url (50070 instead of 50470) and saying connction refused. We put NN https urls in knox webhdfs section. Please can any one help me on this asap
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Knox
05-17-2018
06:43 AM
We are running Platfora spark jobs scheduled for every 30 mins. The jobs are successful in alternate runs. For every alternate run, they are failing with the below error. Can any one help me on this kms.LoadBalancingKMSClientProvider (LoadBalancingKMSClientProvider.java:doOp(96)) - KMS provider at "" threw an IOException [HTTP status [403], message [Forbidden]]!! org.apache.http.HttpException: Failed to submit application_1526036886004_0450 to YARN : Failed to renew token: Kind: kms-dt, Service:
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Ranger
-
Apache Spark
05-03-2018
06:25 AM
I tried to read the data in 'ATLAS_HOOK' topic also. But nothing is there in the topic. When I tried to run the 'import-hive.sh' script, it was failing with the below error. Caused by: org.apache.atlas.notification.NotificationException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. at org.apache.atlas.kafka.KafkaNotification.sendInternalToProducer(KafkaNotification.java:239) at org.apache.atlas.kafka.KafkaNotification.sendInternal(KafkaNotification.java:212) at org.apache.atlas.notification.AbstractNotification.send(AbstractNotification.java:116) at org.apache.atlas.notification.NotificationEntityChangeListener.notifyOfEntityEvent(NotificationEntityChangeListener.java:191) at org.apache.atlas.notification.NotificationEntityChangeListener.onEntitiesUpdated(NotificationEntityChangeListener.java:89) at org.apache.atlas.repository.store.graph.v1.AtlasEntityChangeNotifier.notifyListeners(AtlasEntityChangeNotifier.java:168) ... 107 more Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms. at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:769) at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:490)
... View more
05-03-2018
06:00 AM
Hi, We are using hdp2.6.4 in our environment. We have recently installed Atlas and setup SSL. In the Atlas logs, we continuously see the below error messages just after trying to read the kafka events. Kafka is not enabled with SSL. Could any one help me on this. We have enabled ranger plugin for kafka and had given necessary policies. I tried to read the data in 'ATLAS_ENTITIES' kafka topic. But nothing is there in the topic. 2018-05-02 17:28:04,453 WARN - [main-EventThread:] ~ Instance is active from BECOMING_ACTIVE (ServiceState:80) 2018-05-02 17:28:05,444 INFO - [NotificationHookConsumer thread-0:] ~ Atlas Server is ready, can start reading Kafka events. (NotificationHookConsumer:492) 2018-05-02 17:28:31,456 WARN - [pool-2-thread-21:] ~ javax.net.ssl.SSLException: Received fatal alert: certificate_unknown (nio:720)
... View more
Labels:
- Labels:
-
Apache Atlas
-
Apache Hadoop
-
Apache Kafka
04-27-2018
11:29 AM
Hi, From Ranger web UI, in Audit tab, we are unable to see the Audits stored in Solr Cloud (Ambari Infra). We are getting an exception 'Error running solr query, please check solr configs. IOException occured when talking to server at <>'. In the Ranger admin xa_portal.log, we can see the error saying "Unrecoverable key exception: Keystore is tampered with incorrect password" . Can any one help me on this. We have SSL applied for both ambari infra solr cloud and Ranger.
... View more
Labels:
04-02-2018
12:28 PM
Hi, Can any one update me asap.
... View more
04-02-2018
11:28 AM
Hi, Is preemption only for FAIR ordering policy or can be used for FIFO ordering policy ? The reason, here why I am asking this question is that we want to use Hive LLAP in our environment. However, to use LLAP, we need to enable 'YARN preemption'. We are using FIFO ordering policies. So, enabling yarn preemption will create any impact even if we are using FIFO ordering policies.
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache YARN
03-27-2018
10:30 AM
Hi, When dynamic allocation is enabled, most of the times we are facing issues while fetching the blocks RetryingBlockFetcher: Retrying fetch (1/3) for 1 outstanding blocks after 5000ms Error RetyringBlockFetcher: Exception while beginning fetch of 1 outstanding blocks (after 1 retries) java.io.IOException: Failed to connect to <host>:<some port> Caused by java.net.ConnectException: Connection refused: <host>:<some port> We are seeing these errors continuously in the executors when we run a big spark jobs. During this time nothing is being processed and after some time these errors are getting disappeared and the processing gets resumed. This is impacting our job SLAs. Can any one help me on this
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Spark
03-06-2018
10:10 AM
Hi, Can anyone help me on this asap.
... View more
03-06-2018
07:43 AM
Hi, In our environment we are using Spark dynamic execution. When the spark executors are finished with their tasks and idle for sometime (usually, 1 min) , the executor containers are being killed by the Spark AM. But here, when the new executors are launched that are depend upon the result set of the previously killed executors are throwing the lot of errors like "ERROR RetryingBlockFetcher: Exception while beginning fetch of 1 outstanding blocks" , "java.io.IOException: Failed to connect to <the killed executor host>" As per understanding, with spark dynamic execution, even if the idle executor container is killed, its result set should be accessible to other executors via external shuffle service. I don't understand why it is not working here. Please correct me if I am wrong. Can any one help me on this asap.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Spark
-
Apache YARN
03-05-2018
12:18 PM
Hi, Can any one explain the property yarn.scheduler.capacity.node-locality-delay From the defintion, I couldn't understand the purpose of this property. Could any one explain about this.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache YARN