Member since
10-01-2015
65
Posts
42
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2137 | 01-31-2018 06:29 PM | |
883 | 09-27-2017 06:32 PM | |
1338 | 03-01-2017 06:26 PM | |
635 | 01-09-2017 06:42 PM | |
7996 | 07-09-2016 06:28 PM |
10-11-2018
06:43 PM
You can use Ambari to deploy AMS and use pre-built Grafana dashboards to monitor your cluster. Note: Since the cluster is not Ambari managed, sizing AMS might need manual work, the Apache Amabri wiki has good amount of detail. The catch is configuring daemons to write to AMS, which is basically editing the configs files, I would go about actually creating a dummy sandbox cluster and copying over and modifying the configs like: hadoop-metrics2.properties. Can you file an Apache Jira for missing documentation around this? Myself / @Aravindan Vijayan can add details on it which can be added to wiki. Feel free to contribute to the same.
... View more
10-11-2018
06:32 PM
You need to select only the latest config, so the above update is dangerous if more than one row in clusterconfig. You should be able to sort by create_timestamp and choose the latest config_id Alternatively, you can delete and re-add the service if the above seems complicated to do. The re-add should get you in clean state. Make sure to take database backup at every stage. The metrics data can be preserved by taking a backup of hbase.rootdir in ams-hbase-site configuration and after re-add you can stop AMS replace the HBase data and restart.
... View more
01-31-2018
06:29 PM
Can you upgrade Ambari to 2.5.2 or AMS only to 2.5.2? This most likely a known issue: https://issues.apache.org/jira/browse/AMBARI-21640
... View more
09-27-2017
06:32 PM
You should be able to find DEBUG level messages in the individual Hadoop service logs; messages starting with org.apache.hadoop.metrics2.* One config missing is: # default sampling period *.period=10
... View more
09-11-2017
06:32 PM
1 Kudo
Hi Ali, the communication between HDP daemons an AMS is only on the HTTP connection on port 6188. The connections that you are seeing on the client side are just tcp connections that client will use a random port for while opening a tcp connection. On the client side, the one initiating the connection to the server, it has to pick a source port to fulfil its part of that 4-tuple needed to uniquely identify a connection. Clients usually pick a random port in the 1025-65535 range. The server-side port is still 6188 which should be used to design the firewall rule.
... View more
05-24-2017
06:26 PM
If you see the drop down populated with your appId that means the metrics were received by the backend. Note: When we receive data that is older that the timeline.metrics.service.outofband.time.allowance.millis we discard that data, as long as your timesatmps are not older than 5 minute which is default setting the data should be persisted as well.
... View more
05-23-2017
07:13 PM
Make to reduce your query interval to 1 hour / 2 hours or select Precision = seconds. Default precision is minutes which means AMS will be sending data aggregated at minute boundary (5 min default). You have sent 1 point and it would take at least 5 minutes for first aggregation to pick it up. https://cwiki.apache.org/confluence/display/AMBARI/Metrics+Collector+API+Specification
... View more
05-05-2017
09:42 PM
The metric monitors send data to collector on 6188 port. Did you check the log on the metric monitors? /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out HMaster port UI port is 61310.
... View more
05-04-2017
07:16 PM
@Jon Page How big is your cluster? It could be that AMS is not properly configured per cluster size or AMS HBase is writing to a contended disk. As a quick solution you could try whitelisting AMS data which reduces the write load and will give perf improvement. https://issues.apache.org/jira/browse/AMBARI-16969 Configs: (make sure the whitelist file exists (came with the rpm)) ams-site :: timeline.metrics.whitelist.file = /etc/ambari-metrics-collector/conf/metrics_whitelist ams-site : timeline.metrics.precision.table.durability = ASYNC_WAL ams-site : timeline.metrics.aggregate.tables.durability = ASYNC_WAL
... View more
05-01-2017
11:04 PM
9 Kudos
Problem summary: Users can end up in a situation where the SQL Engine for all Ambari tables is set to MyISAM instead of InnoDB which is required for Ambari. There are few ways to land up in this situation. Prior to MySQL version 5.5.5 the default engine was MyISAM. It could also be a configured default as a global default in my.cnf which affected Ambari versions before 2.5. AMBARI-18951 addressed this issue by making InnoDB default for new databases (2.5+ Ambari deployments) as well as throwing an explicit error preventing Ambari upgrade if the wrong engine is set in order to avoid landing up in an intermediate non-upgraded state. Following is an sample error that would be encountered on an upgrade path from 2.2.2 to 2.4.2 ERROR [main] DBAccessorImpl:830 - Error executing query: ALTER TABLE hostcomponentdesiredstate ADD CONSTRAINT hstcmpnntdesiredstatecmpnntnme FOREIGN KEY (component_name, service_name, cluster
_id) REFERENCES servicecomponentdesiredstate (component_name, service_name, cluster_id)
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate key name 'hstcmpnntdesiredstatecmpnntnme' Reason for failure: When MyISAM is chosen as database engine FK create statements doesn't
create any object in DB which describes FK, but creates corresponding
index. This is the reason of failure, the Ambari UpgradeCatalogs check existence of FK which is absent, but there is an index with same name. Also there is an issue with migration. Because there is no information about FK stored in DB, a simple engine change won't fix this problem. Suggested approach: MySQL docs suggest steps for migration from MyISAM to InnoDB by creating a new InnoDB table and doing a select and insert. Although, this sounds doable for 106 tables it needs renaming of all Foreign keys for new tables to prevent collision with existing objects in the database and then again renamed back after dropping old tables since this will be required for smooth upgrades of Ambari in the future. Instead the approach described below takes a more direct but simpler approach and avoids any hand edits of the DDL to make the migration. The recommendation is to create a blank schema using DDL script and insert data using the db dump. Note 1: This has been tried and tested on 2.2.2 Ambari version and upgrading to 2.4.2. Although the approach and steps are generic enough to apply to any 2.X version and upwards. Note 2: If you are in the middle of upgrade and hit this error related to the storage engine, please make sure to revert back to the last backup of the database as well as the ambari bits (yum donwgrade) Steps:
1. Edit /var/lib/ambari-server/Ambari-DDL-MySQL-CREATE.sql from your ambari-server host to add the following lines that will set the appropriate engine on re-creating the database. These should be added right at the beginning of the file. -- Set default_storage_engine to InnoDB
-- storage_engine variable should be used for versions prior to MySQL 5.6
set @version_short = substring_index(@@version, '.', 2);
set @major = cast(substring_index(@version_short, '.', 1) as SIGNED);
set @minor = cast(substring_index(@version_short, '.', -1) as SIGNED);
set @engine_stmt = IF(@major >= 5 AND @minor>=6, 'SET default_storage_engine=INNODB', 'SET storage_engine=INNODB');
prepare statement from @engine_stmt;
execute statement;
DEALLOCATE PREPARE statement; 2. Take a backup of your good working condition non-upgraded Ambari database, lets call it, dbdump.sql 3. Create the INSERT only SQL, lets call it insert_final.sql without the CREATE statements from your dbdump file. (Optional) Purge the dbdump.sql of historical information to make the database dump of manageable size, getting rid of historical items. Use the following cat / grep statement to achieve this: cat dbdump.sql | grep "INSERT INTO" | grep -v "INSERT INTO \`alert" | grep -v "INSERT INTO \`host_role_command\`" | grep -v "INSERT INTO \`execution_command\`" | grep -v "INSERT INTO \`request" | grep -v "INSERT INTO \`stage" | grep -v "INSERT INTO \`topology" | grep -v "INSERT INTO \`blueprint" | grep -v "INSERT INTO \'qrtz" | grep -v "INSERT INTO \`hostgroup" > insert_final.sql (Without purge option use only the first grep statement) 4. Drop Ambari database and Recreate it using the /var/lib/ambari-server/Ambari-DDL-MySQL-CREATE.sql mysql> drop database ambari;
mysql> create database ambari;
mysql> use ambari;
mysql> source /var/lib/ambari-server/Ambari-DDL-MySQL-CREATE.sql 5. Make sure the new storage engine is set to InnoDB SELECT `ENGINE` FROM `information_schema`.`TABLES` WHERE `TABLE_SCHEMA`='ambari'; 6. Add the following statements to the beginning of the insert_final.sql. These will delete the default INSERT(s) coming from the DDL script which would be correctly re-inserted from your db dump later on in the file. Without the deletes you would get duplicate constraint violations. The silencing of constraints allows out of order inserts without constraint violations. SET unique_checks=0;
SET FOREIGN_KEY_CHECKS=0;
DELETE FROM `adminpermission`;
DELETE FROM `adminprincipal`;
DELETE FROM `adminpermission`;
DELETE FROM `adminprincipaltype`;
DELETE FROM `adminprivilege`;
DELETE FROM `adminresource`;
DELETE FROM `adminresourcetype`;
DELETE FROM `viewentity`;
DELETE FROM `metainfo`;
DELETE FROM `users`;
DELETE FROM `ambari_sequences`; 7. Turn on the constraints at the end of the dump file. Append the following at the end of the insert_final.sql file. SET unique_checks=1;
SET FOREIGN_KEY_CHECKS=1; 8. Execute the insert_final.sql on the ambari database. mysql> use ambari;
mysql> source insert_final.sql (These instruction should not fail with any error messages. If you get primary key constraint violations, it is in all likelyhood some seed data inserted by the DDL script. Just add the DELETE statement at the beginning of the file.) 9. With no errors on step 8, start the Ambari server and verify everything looks good and are no SQL errors in the server logs. If step 8 fails, make the minor adjustments and just rinse and repeat using Step 4 and Step 8 until there are no errors. 10. Proceed with Ambari upgrade to the new desired version.
... View more
- Find more articles tagged with:
- Ambari
- ambari-server
- ambari-upgrade
- Cloud & Operations
- Issue Resolution
- issue-resolution
- MySQL
Labels:
03-09-2017
09:53 PM
hdp242-s1.openstacklocal: Doesn't resolve for me. If the metric is not in metadata that means that HBase has never sent this metric to AMS. Did you perform a scan / run Ambari Smoke test to see if any data is sent? Could be this metric is not sent by the HBase version you are using?
... View more
03-01-2017
06:26 PM
2 Kudos
Are you able to see other RegionServer metrics on this dashboard? I checked out my test cluster with HDP 2.6 and Ambari 2.5 and I am able to see them. One way to verify if these metrics were ever sent to AMS is by making a call to AMS metadata API in your browser: http://<ams-collector-host>:6188/ws/v1/timeline/metrics/metadata You can look for metrics name in the response, example: regionserver.Server.ScanTime
... View more
02-10-2017
03:53 PM
In embedded mode we combine region server and hbase heapsize since there is only 1 daemon running.
Xmn settings:
regionserver_xmn_size
hbase_master_xmn_size
JDK8 perm setting is unnecessary since it is offheap, so effectively the setting itself doesn't matter.
... View more
02-10-2017
05:56 AM
XMN should be 15 % of Xmx, suggest setting this value to at 1 GB. maxperm does not need to be more than 128M
... View more
02-09-2017
06:06 PM
Comment from @Jay SenSharma regarding Region metrics is also important and applicable. Note: Additionally make sure Xmn settings = 15 % of Xmx in ams-env and ams-hbase-env
... View more
02-09-2017
05:55 PM
If this is a production cluster switch to distributed mode will make use of cluster Zookeeper which will make the system a lot more stable. Embedded mode works perfectly fine for cluster size of 40 nodes albeit, memory and disk are not heavily contended. AMS HBase will write to 1 disk and talk to the embedded zookeeper so starightforward recommendations without looking at the full logs and configs and without changing the mode: ams-env :: metrics_collector_heapsize = 1024 ams-hbase-env :: hbase_regionserver_heapsize = 4096 Make sure hbase.rootdir and hbase.tmp.dir are not pointing to the same location. Key is to put hbase.rootdir on a non-contended disk. If you switch ti distributed mode the disk settings do not matter: https://cwiki.apache.org/confluence/display/AMBARI/AMS+-+distributed+mode
... View more
02-07-2017
07:11 PM
Please also look at: https://cwiki.apache.org/confluence/display/AMBARI/Troubleshooting+Guide How big is your cluster?
... View more
02-03-2017
07:01 PM
The backup function is called from a couple of places and it is not a recent addition based on git history. You can manually disable it by commenting out the copy call in the python code at: /usr/lib/python2.6/site-packages/ambari_server/serverConfiguration.py def backup_file_in_temp(filePath):
... View more
02-02-2017
10:44 PM
amabri.properties file does not store any passwords. Is any other data classified as sensitive on your end?
... View more
01-27-2017
06:41 PM
https://cwiki.apache.org/confluence/display/AMBARI/Build+Instructions
... View more
01-09-2017
06:42 PM
If you are trying to build AMS artifacts here is useful links: https://cwiki.apache.org/confluence/display/AMBARI/Build+Instructions General build instructions are on this page: https://cwiki.apache.org/confluence/display/AMBARI/Ambari+Development
... View more
12-29-2016
07:00 PM
Grafana 3.0+ has a bosun plugin and Ambari should be able to support this. Care to open a Apache Jira for this as desired feature?
... View more
10-11-2016
05:53 PM
Actually, Grafana allow dashboard import. You might be able to just get the dashboard definition from Ambari source and import into you existing Grafana. Here is link to trunk dashboard locations: https://github.com/apache/ambari/tree/trunk/ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/files/grafana-dashboards/HDP Import is pretty straightforward: http://docs.grafana.org/reference/export_import/
... View more
08-25-2016
06:29 PM
Make sure to use Ambari-2.2.2. A related issue was fixed in 2.2.2 https://issues.apache.org/jira/browse/AMBARI-15774
... View more
07-29-2016
09:49 PM
1 Kudo
We have basic stuff documented on the wiki: https://cwiki.apache.org/confluence/display/AMBARI/Metrics
... View more
07-09-2016
06:28 PM
@Angel Kafazov Were you able to verify the AMS keytabs work? Most of the config changes performed above were not needed, example changes to zookeeper and znode settings : For distributed mode only config changes needed are these: https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.1.0/bk_ambari_reference_guide/content/_configuring_ambari_metrics_for_distributed_mode.html When you enable security through Ambari the keytabs and principals are generated by Ambari and applied to AMS configs. Before looking into ambari-metrics-collector.log or ambari-metrics-monitor.out, the ams-hbase daemon should be up and running fine, if not the connection timeouts are of no help since these are expected. Based on the hbase logs posted the HBase daemon tried to login and failed, so we need to figure out why it did fail. Note: If the collector was moved older keytabs would become invalid because hostname changed and would have to be re-generated. Example of keytab commands: http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDP1/HDP-1.2.0/bk_installing_manually_book/content/rpm-chap14-1-4.html
... View more
07-07-2016
07:03 PM
1 Kudo
Both Master and Regions server logs indicate unable to login to Kerberos. Login failure for amshbasemaster/m2.DOMAIN@DOMAIN from keytab /etc/security/keytabs/ams-hbase.regionserver.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user Can you check if you can manually login with the ams keytabs under /etc/security/keytabs/ams*.keytab Example: kinit -kt /etc/security/keytabs/ams-hbase.regionserver.keytab amshbasemaster/m2.DOMAIN@DOMAIN
... View more
07-07-2016
06:02 AM
@Geoffrey Shelton Okot The document suggests setting the zookeeper.znode.parent to the same value as HBase service which is somewhat incorrect for versions > 2.1.2.1. Prior to 2.2, the value was not set so it defaulted to /hbase, this was ok because AMS started its own ZK. Post 2.2, AMS talks to cluster ZK and thereby the znode is set to /ams-hbase-(unsecure/secure). Logs would indicate whether TGT was acquired correctly and if the problem is something totally different.
... View more
07-07-2016
04:19 AM
1 Kudo
@Angel Kafazov We need to look at the Master, RS and ZK logs to identify the issue. Can you upload them from /var/log/ambari-metrics-collector/ ?
... View more
07-06-2016
06:21 PM
1 Kudo
Please revert back the znode setting to default, if cluster is not kerberized: /ams-hbase-unsecure Also, make sure the quorum value in ams-hbase-site is: hbase.zookeeper.quorum {{zookeeper_quorum_hosts}}
... View more