Member since
01-19-2017
3681
Posts
633
Kudos Received
372
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1610 | 06-04-2025 11:36 PM | |
| 2071 | 03-23-2025 05:23 AM | |
| 984 | 03-17-2025 10:18 AM | |
| 3741 | 03-05-2025 01:34 PM | |
| 2573 | 03-03-2025 01:09 PM |
08-25-2019
12:20 PM
1 Kudo
@Manoj690 Going through your logs I can see that the Namenode is in SAFE MODE, and in this case it won't allow you to change the status of any file in the cluster including the logs. 2019-08-22 12:31:01,376 [server.Accumulo] INFO : Attempting to talk to zookeeper 2019-08-22 12:31:01,681 [server.Accumulo] INFO : ZooKeeper connected and initialized, attempting to talk to HDFS 2019-08-22 12:31:01,946 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode 2019-08-22 12:31:01,946 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 1.0 seconds 2019-08-22 12:31:02,950 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode 2019-08-22 12:31:02,950 [server.Accumulo] INFO : Backing off due to failure; current sleep period is 2.0 seconds 2019-08-22 12:31:04,954 [server.Accumulo] WARN : Waiting for the NameNode to leave safemode To resolve the issue can you do the following as hdfs user $ hdfs dfsadmin -safemode get Safe mode is OFF The above is the desired output but if you get ON then proceed like below First backup your FS Image & Edits $ hdfs dfsadmin -saveNamespace Then exit the safemode $ hdfs dfsadmin -safemode leave Once successful then revalidate $ hdfs dfsadmin -safemode get This time it should be off and you can now successfully restart the failed services from Ambari everything should succeed HTH
... View more
08-25-2019
11:59 AM
@iamabug There is a lot more than just kerberizing the cluster and you are good to go. Have you enabled SSL also? Can you share a tokenized version of the below files? Basically, the ACL in zk is the key to who can do what and usually the Kafka admin is the only one allowed! server.properties [listeners, advertised.listeners,authorizer.class.name,sasl.enabled.mechanism and super.users] Kafka_server_jaas.conf Kafka_client_jaas.conf kafka_client_kerberos.properties Hope that helps
... View more
08-18-2019
01:27 AM
@ray_teruya If you found this answer addressed your question, please take a moment to log in and click the "kudos" link on the answer. That would be a great help to Community users to find the solution quickly for these kinds of errors.
... View more
08-11-2019
06:16 PM
@Ray Teruya The error in BOLD below is what I stated in Question/Answer 2 in my former post. To avoid the split-brain decision you MUST install 3 zookeepers 2019-07-31 07:57:58,191 - WARN [main:QuorumPeerConfig@291] - No server failure will be tolerated. You need at least 3 servers. Solution Delete/remove the failed installation. Add 2 new zk using Ambari UI in your cluster using ADD SERVICE, start the new zookeepers if they ain't started, this should form a quorum where only one is a leader and the rest are followers. To identify a Zookeeper leader/follower, there are few possible options. Mentioning 2 for keeping this document simple. 2. Use "nc" command to listen to TCP communication on port 2181 and determine if the ZooKeeper server is a leader or a follower. 1. Check the zookeeper log file on each node, and grep as below: # grep LEAD /var/log/zookeeper/zookeeper-zookeeper-server-xyz.out Desired output 2019-08-10 22:33:47,113 - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:QuorumPeer@829] - LEADING 2019-08-10 22:33:47,114 - INFO [QuorumPeer[myid=3]/0:0:0:0:0:0:0:0:2181:Leader@358] - LEADING - LEADER ELECTION TOOK - 9066 After doing the above procedure you should be good to go. HTH
... View more
08-03-2019
11:07 AM
@Ray Teruya OutOfMemoryError is a subclass of java.lang.VirtualMachineError; it’s thrown by the JVM when it encounters a problem related to utilizing resources. More specifically, the error occurs when the JVM spent too much time performing Garbage Collection and was only able to reclaim very little heap space. According to Java docs, by default, the JVM is configured to throw this error if the Java process spends more than 98% of its time doing GC and when only less than 2% of the heap is recovered in each run. In other words, this means that our application has exhausted nearly all the available memory and the Garbage Collector has spent too much time trying to clean it and failed repeatedly. In this situation, users experience extreme slowness of the application. Certain operations, which usually complete in milliseconds, take more time to complete. This is because the CPU is using its entire capacity for Garbage Collection and hence cannot perform any other tasks. Solution: On HDP 3.x & 2.6.x depending on the memory available to the cluster check and increase the below You could throttle it to 2048 MB HTH
... View more
08-02-2019
11:18 PM
@FA Use a ReplaceText processor which replaces \A\n|\n*\s*(?=\n) with '' (empty replacement value). The search regex looks for: \n*\s*(?=\n) - newline (0 or more) followed by whitespace (0 or more) followed by a newline.
... View more
08-02-2019
10:18 PM
@Matas Mockus This is a duplicate posting I responded to the initial thread. http://community.hortonworks.com/answers/249938/view.html Did you check the response please either merge or delete this post as it will be difficult to follow the 2 threads!
... View more
08-02-2019
08:46 PM
@Matas Mockus The below error comes up in your ambari-agent logs , can you check your hiveServer2 hostname, you might need to adjust in Ambari UI Execute['! beeline -u 'jdbc:hive2://:10000/;transportMode=binary;auth=noSasl' It should look like this Execute['! beeline -u 'jdbc:hive2://{some_host_FQDN}:10000/;transportMode=binary;auth=noSasl' Please try that and revert
... View more
07-31-2019
01:36 PM
@Ray Teruya How many hosts do you have in your cluster? Can you share your zookeeper logs and your /etc/hosts? HTH
... View more