Member since
08-08-2017
1652
Posts
30
Kudos Received
11
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1914 | 06-15-2020 05:23 AM | |
| 15434 | 01-30-2020 08:04 PM | |
| 2055 | 07-07-2019 09:06 PM | |
| 8094 | 01-27-2018 10:17 PM | |
| 4561 | 12-31-2017 10:12 PM |
11-21-2017
11:52 AM
1 Kudo
If it is only one datanode failing but not all of them, it may be failing due to a volume failure. You can check Namenode web-ui to see if you are facing any volume failures: http://<active-namenode-host>:50070/dfshealth.html#tab-datanode-volume-failures or just http://<active-namenode-host>:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState For plain JMX data to see whether or not you're having volume failures (it is not physical disk error all the time. It may occur after logical write issues).
... View more
11-21-2017
12:09 AM
@Michael Bronson That's really a very good question. I can see there is still work going on it and one JIRA ( HDFS-107 ) in open state. So the answer to your first question is obvious NO. Formatting the namenode will surely be going to impact your whole cluster as it's the master node and contains metadata of all of the data nodes. So, formatting namenode is not a good idea in my view. I have to replicate some steps before answering your second question. But, it's the tricky one. I will try to answer it ASAP.
... View more
11-27-2017
07:37 PM
1 Kudo
Hi @Michael Bronson All services logs can be found at the location mentioned in the yarn-env.sh (can be found under Hadoop Conf files, usually /etc/hadoop/conf/) on the respective nodes. Hence please check location mentioned in yarn-env.sh on the master02 machine. Once we check the logs, it will be easier to figure out the exact reason for failure. Example yarn-env.sh export YARN_LOG_DIR=/grid/0/log/yarn/
... View more
12-02-2017
11:04 PM
thank you for the answer , but we create another new worker machine instead that machine , I think it was waste of time find the problem on that machine and better to create a new one
... View more
11-19-2017
11:21 AM
@Michael Bronson When we delete a service from ambari via Ambari UI then immediately after service deletion the Ambari UI reloads the page and then you won't see the deleted services in the ambari UI on the left panel. However when we delete services via ambari API then it does not involve the Ambari UI invocation rather it simply invokes the back end APIs that are responsible for deleting the Service details from the ambari DB directly and hence the ambari UI will still show the Deleted Service present there. . So after deleting the service using ambari APIs you will need to refresh the ambari UI (a hard browser refresh), which will make the ambari UI to reload the service list and then you should not see the deleted service there. Sometimes it also happens that some browsers (like Internet Explorer does a caching of the browser contents) caches the UI data so in that you might have to either disable the browser cache or doa hard refresh of the browser or reopen the browser after clearing the ambari UI cache from browser. You do not need to restart the ambari server, rather you will need to just refresh the browser or clear the browser cache and reopen it. . Unfortunately it is not possible to refresh the Ambari UI using the Rest API calls (or using any CLI calls).
... View more
11-22-2017
05:26 PM
f you don't run the filesystem checker, the apparent corruption in the filesystem may get worse. Unchecked, this can lead to data corruption or at the unlikely worst destruction of the filesystem. During the filesystem check, file structures within the filesystem will be checked, and if necessary repaired. The repair takes no account of content; it's all about making sure the filesystem is self-consistent. If you run e2fsck -y /dev/sdc you have no opportunity to validate the corrections being applied. On the other hand if you run e2fsck -n /dev/sdc you can see what would happen without it actually being applied, and if you run e2fsck /dev/sdc you will be asked each time a significant correction needs to be applied. In summary
If you ignore the warning and do nothing, over time you may lose your data If you run with -y you have no option to review the potentially destructive changes, and you may lose your data If you run with -n you will not fix any errors, and over time may lose your data, but you will get to review the set of changes that would be made If you run with no special flag you will be prompted to fix relevant errors, and you can decide for each whether you are going to need direct professional assistance Recommendation
Run e2fsck -n /dev/sdc to review the errors Decide whether this merits a subsequent e2fsck /dev/sdc (or possibly e2fsck -y /dev/sdc ) or whether you would prefer to obtain direct professional assistance
... View more
11-13-2017
05:43 AM
@Michael Bronson There are some limited predefined alerts available in terms of disk checking in ambari. However ambari allows users to register their own custom alert definitions to fulfill their need. Following is an article that explains how we can register our own custom ambari alert and register it with Ambari to get custom alerts: https://community.hortonworks.com/articles/38149/how-to-create-and-register-custom-ambari-alerts.html . Once the alert is setup and working we can either Setup Email Notification (OR) SNMP notification based on our requierment as described in : https://community.hortonworks.com/articles/40361/how-to-troubleshoot-ambari-alerts-notification.html https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/creating_or_editing_notifications.html .
... View more
11-12-2017
10:07 AM
@Michael Bronson, Yes. You need to add the service -> install the service and start the service. If you do it from Ambari , it will take care of all these after "Configure services" step
... View more
11-10-2017
01:22 PM
@Michael Bronson Yes. It will startup inspite the volume is bad. If you dont want this to happen you might have to replace your failed volume with a new volume (ie unmount old one and mount new one)
... View more
11-10-2017
01:09 PM
yes sure thank you so much
... View more