Member since
02-10-2017
56
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
366 | 04-28-2017 06:08 PM |
12-07-2017
03:09 PM
@Mudassar Hussain
When I posted my initial question it was pertaining to alerts that are triggered because of HDFS storage capacity exceeding a configurable threshold. Then there are some alerts that trigger once a week, month about capacity or if you restarted a service. In short these are trivial alerts that are just annoying but not detrimental to the health of the cluster. In your image you provided correct me if I am wrong, it appears you have an issue with your namenode. If that is accurate that is serious and that is an alert you do not want to ignore. Have you checked to see if your name node volumes are corrupted? A datanode is corrupted or exceded capacity? I found this article as well. Not sure if this is precisely related: https://community.hortonworks.com/questions/81840/node-in-maintenance-mode-throws-stale-alert-from-m.html
... View more
10-11-2017
08:49 PM
@Jay SenSharma Thank you very much for the reference. I will keep this approach in my notes in case the answer below fails to work.
... View more
10-11-2017
08:48 PM
@Jonathan Hurley Thank you Jonathan! Disabling and re-enabling worked just fine.
... View more
10-11-2017
06:53 PM
Hello I am using Ambari version 2.4.2. Can anyone provide the steps for purging alerts especially those that are stale without having to delete the alert or restart the individual components/services?
... View more
Labels:
- Labels:
-
Apache Ambari
09-11-2017
07:12 PM
Sorry @Scott Shaw, Can you please elaborate or be more explicit? Do you have to export the tables first then do dscp? Is the DDL the schematool or something else? My assumption is if all of the metadata and data are sent over isn't that able to be imported entirely at once? My apologies for my ignorance I just want to make sure I do this correctly the first time and without corrupting the database.
... View more
09-11-2017
05:48 PM
Hello, What is the most formal or preferable way to migrate a Hive table from one cluster to another cluster running HIVE of course. I have found this reference located here. My issue with this approach is why not use discp at all and just export directly to another HDFS instance over the network and then do an import from there? Please note this is not an production environment so the fact that it is "noisy" or bandwidth invasive is not an issue. Thanks for any advice!
... View more
- Tags:
- Hadoop Core
- HDFS
- Hive
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
08-30-2017
04:37 PM
@Rajesh Thank you! This actually much cleaner and simpler than the yum query and I was able to run this without root or sudo. I wish I could give your answer a check as well. I appreciate it.
... View more
08-30-2017
04:02 PM
@Dinesh Chitlangia Thanks Dinesh! I had no idea you could query the service components via yum.
... View more
08-30-2017
02:02 PM
I want to know is there a native way of querying what service components are installed and running on a given cluster without using Ambari Web UI or Ambari Shell? I do not have credentials for this cluster but I do have CLI access.
... View more
Labels:
- Labels:
-
Apache Ambari
08-26-2017
05:41 PM
@Geoffrey Shelton Okot No worries! Based on prior feedback from the community it turns out that there are bugs in IBM Open Platform (IOP) version 4.2.0 for packages that are now deprecated and or patched. I am going to use 4.2.5 and start from scratch in hopes this works. It turns out that starting IOP version 4.3 which is IBM latest version of Big Insights or their next release after that they are going to drop IOP all together and use "vanilla" HDP thank goodness. There must be a reason why they are doing this and I believe we have encountered it. I would go directly to version 4.3 but there is no community edition of that version and it requires a license etc. In any case I will let you know how 4.2.5 works out. The post of the colleague who fixed my HBASE issue told me of the bugs in the same thread if you are interested.
... View more
08-26-2017
05:41 PM
@Geoffrey Shelton Okot No worries! Based on other's feedback it appears that IBM's Open Platform (IOP) version 4.2.0 is using HDP packages with bugs that are fixed in later releases. I am going to start from scratch and use IOP version 4.2.5 in hopes that the issues we faced are a direct result of the deprecated version of their HDP fork I am using. I will let you know in any event if I am successful. It turns out that starting either in version 4.3 which is their latest or > 4.3 IBM is going to drop IOP all together and use "vanilla" HDP thank goodness. There has to be a reason for this and I believe I have encountered it because with all of the changes we have done and your advice there is no way this should not be working by now.
... View more
08-23-2017
08:13 PM
@Geoffrey Shelton Okot Same result unfortunately. The new logs are located here.
... View more
08-23-2017
06:22 PM
@Geoffrey Shelton Okot Same behavior unfortunately. Would it make sense to repost a new thread to see if others have run into this issue? I attached my screenshot as well.screen-shot-2017-08-23-at-22035-pm.png
... View more
08-23-2017
04:27 PM
@Geoffrey Shelton Okot The metrics still did not populate after a rebooting all nodes and restarting all services. The logs are located here. In the interim I will follow the instructions you have provided in the previous post. Thank you!
... View more
08-23-2017
03:53 PM
@Geoffrey Shelton Okot @Jay SenSharma With the help of other community colleagues I have resolved the HBase issue which is now running in distributed mode as opposed to embedded mode. I was hoping in doing so that would resolve the Ambari-Metrics to display properly which I am still seeing "n/a" and node data available. I am now going to shutdown all services, reboot the nodes and restart the services. I will update if that works or not.
... View more
08-16-2017
12:44 PM
@Ron Buckley IBM has been using IOP (IBM Open Platform) all the way through version 4.2.5. BI version 4.3 is not available for free at least as the others. IOP as you mostly know is HDP or their fork of it at least that they maintain in their own repository. Lastly our sponsor is unfortunately using version 4.1 and 4.2 which is why I am using this. I would much rather run the latest vanilla version of HDP with all of the bug fixes and support. I will start a new thread for the Ambari-Metrics in the interim. If you know something that refutes my statement above that I do not please let me know. Otherwise thank you very much for at least getting me through the Hbase hurdle.
... View more
08-16-2017
12:27 AM
@Sandeep Kumar Toopran Hello this was solved in the this thread below: https://community.hortonworks.com/questions/122793/hbase-regionserver-dies-and-shortly-after-hbase-ma-1.html?childToView=122807#answer-122807 Thank you!
... View more
08-15-2017
07:41 PM
@Ron Buckley Thank you very much Ron! That worked! Should I start another thread on why Ambari-Metrics is not updating? All of my values are "n/a" and "no data available" and I have all green on the Ambari console.
... View more
08-15-2017
05:07 PM
@Sandeep Kumar Toopran I tweaked the following parameters: HBase Master Maximum Memory HBase RegionServer Maximum Memory Before the RegionServer were dying first then the Hbase Master. Now it is vise versa. Sometime 1 of my 4 RegionServers will remain running while the Hbase Master died.
... View more
08-15-2017
04:23 PM
@Sandeep Kumar Toopran I attached a screenshot to show all services. In the interim I will try increasing the Hbase_regionserver_heapsize.screen-shot-2017-08-15-at-122059-pm.png
... View more
08-15-2017
04:08 PM
@Jay SenSharma @Geoffrey Shelton Okot Hbase shows green for approximately 1 minute and I have followed the fine tuning guide for HBASE for Big Insights that utilizes HBASE here. All server start green and the RegionServers begin to die one-by-one and then Hbase Master finally dies. I even tried disabling RANGER and disabling the HBASE plugin to see if that had something to with the following error: Connection failed: [Errno 111] Connection refused to x.x.x:16030 (region servers) Connection failed: [Errno 111] Connection refused to x.x.x:16000 (master) But that had no effect. My Hbase logs are located here.
... View more
08-15-2017
04:01 PM
@Jay SenSharma @Geoffrey Shelton Okot Ambari shows green for approximately 1 minute and I have followed the fine tuning guide for HBASE for Big Insights that utilizes HBASE here. All server start green and the RegionServers begin to die one-by-one and then Hbase Master finally dies. I even tried disabling RANGER and disabling the HBASE plugin to see if that had something to with the following error: Connection failed: [Errno 111] Connection refused to x.x.x:16030 (region servers) Connection failed: [Errno 111] Connection refused to x.x.x:16000 (master) But that had no effect. My Hbase logs are located here.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache HBase
08-15-2017
03:59 PM
@Jay SenSharma @Geoffrey Shelton Okot Ambari shows green for approximately 1 minute and I have followed the fine tuning guide for HBASE for Big Insights that utilizes HBASE here. All server start green and the RegionServers begin to die one-by-one and then Hbase Master finally dies. I even tried disabling RANGER and disabling the HBASE plugin to see if that had something to with the following error: Connection failed: [Errno 111] Connection refused to x.x.x:16030 (region servers) Connection failed: [Errno 111] Connection refused to x.x.x:16000 (master) But that had no effect. My Hbase logs are located here.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache HBase
08-15-2017
03:57 PM
@Jay SenSharma @Geoffrey Shelton Okot Ambari shows green for approximately 1 minute and I have followed the fine tuning guide for HBASE for Big Insights that utilizes HBASE here. All server start green and the RegionServers begin to die one-by-one and then Hbase Master finally dies. I even tried disabling RANGER and disabling the HBASE plugin to see if that had something to with the following error: Connection failed: [Errno 111] Connection refused to x.x.x:16030 (region servers) Connection failed: [Errno 111] Connection refused to x.x.x:16000 (master) But that had no effect. My Hbase logs are located here.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache HBase
08-11-2017
06:31 PM
@Geoffrey Shelton Okot I have the following configuration for each of the 4 nodes: 8 VCPUs 40GB memory 275GB storage The hbase Memory link you provided the page no longer exist. In addition to using the Ambari fine tuning I will change my settings according to the BigInsights HBase and your recommendations above and let you know the results. Thank you!
... View more
08-11-2017
04:00 PM
@Geoffrey Shelton Okot ams-site phoenix.query.maxGlobalMemoryPercentage The above parameter I set to 30. The default as you most likely know is 25 and the range is from 20 - 30.
... View more
08-11-2017
03:35 PM
@Geoffrey Shelton Okot The other point to make is HBASE did not have any issue until made the changes you provided below in an earlier reply: Can run "hbck" from your hbase install bin directory to check if hbase is corrupt? check the hbase-site.xml value for zookeeper.znode.parent # ambari-metrics-monitor status If any process is running # ambari-metrics-collector stop Do you see any errors in ambari-metrics-collector.log ? If so zip them and attach Can you do the following 1.Set hbase.zookeeper.property.clientPort to 2181. I see your ports even for in the zoo.cfg are different. The Zookeeper server also runs at the same port, so the hbase.zookeeper.property.clientPort should also be set to the same port. If these ports are different, then there will be a number of errors thrown by Ambari-metrics collector service is started. Therefore make sure this property is set to the same port as the zookeeper server. 2.Change the "Metrics Service operation mode" to distributed. 3.Set hbase.cluster.distributed to true 4 hbase.rootdir should be set to a hdfs folder if not create a new folder in the hdfs root and made hdfs the owner of the folder. 5 Restart the zookepper ensemble. 6 Restarted the Ambari Metric collector
... View more
08-11-2017
03:31 PM
screen-shot-2017-08-11-at-112136-am.pngscreen-shot-2017-08-11-at-112100-am.pngscreen-shot-2017-08-11-at-112037-am.png @Geoffrey Shelton Okot @Jay SenSharma I am aware of the fine tuning and tried to follow it as best as I can understand. I only 4 nodes in my cluster. Attached are screenshots that Geoffery asked for.
... View more
08-11-2017
01:04 PM
@Geoffrey Shelton Okot All nodes were restarted after changing /etc/host from my previous post FYI. Do you think memory attributes for heap-size and XMN are the culprit for HBASE dying? Do you have any updates? Regards
... View more
08-10-2017
08:06 PM
@Geoffrey Shelton Okot I commented out that line on every node in the hosts file and the behavior did not change. The logs are located here.
... View more