Member since
09-29-2015
63
Posts
107
Kudos Received
13
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1327 | 05-04-2017 12:32 AM | |
1379 | 12-09-2016 07:58 PM | |
4611 | 12-09-2016 07:53 PM | |
590 | 06-08-2016 09:26 PM | |
1067 | 06-08-2016 09:01 PM |
07-31-2017
09:09 PM
This is missing, hdfs dfs -chown admin:hadoop /user/admin
... View more
05-08-2017
09:07 PM
@mayki wogno, this could be because the View work folder was not expanded properly.
Can you try these steps, ambari-server stop
rm -rf /var/lib/ambari-server/resources/views/work/
ambari-server start
Once Ambari Server starts, it will unzip the views again and recreate those artifacts.
... View more
05-04-2017
06:30 PM
@Avijeet Dash, you may find this presentation useful, http://events.linuxfoundation.org/sites/events/files/slides/ambari_big_data_2017.pdf
Custom Services can be added to the stack, as long as they support lifecycle commands like stop, start, restart, status, etc., and can also contain Custom Commands. Technology-wise, Ambari Server is all Java and supports multiple DB types (Postgres, MySQL, Oracle, Azure).
The Agents use Python for running, heartbeating, and executing the scripts for managing the Hadoop Services.
The Agents send heartbeat messages to the Server every 4-10 seconds (configurable). The web UI uses AngularJS and Node.js to responsiveness and auto-refresh, and relies on the RESTful web service for all data.
... View more
05-04-2017
12:32 AM
2 Kudos
@Avijeet Dash At its core, Ambari is able to perform the same common ops that DevOps are used to with Chef and Puppet.
Chef, Puppet, and Ansible are great for small clusters and limited operations, but as soon as you need to do perform complex operations like Kerberize, HA, Stack Upgrade, then those tools are not sufficient; this is where Ambari shines since it has deep knowledge about the cluster and Hadoop in order to perform things like:
Rolling Restart (pick any component, like DataNodes, and pick your batch size, sleep time, and failures tolerated) High Availability for NameNode, Hive, Oozie, Resource Manager, etc (wizards to configure these and restart affected services) Config Management (this is where it truly brings value; if a config changes, Ambari can suggest other configs to change, and validate dependent configs) Kerberos Security: create and distribute keytabs, also handle use cases like adding hosts and more services Rolling and Express Upgrade: the orchestration and config changes here alone, plus handling a downgrade, would easily take weeks to develop and test. Ambari has all of the tools to visualize the orchestration, auto-retry, handle failures, skip optional steps, etc. Metrics: dashboards to visualize and analyze metrics, trends Alerts: notifications, SNMP, port checks, liveliness checks, and UI to enable/disable Maintenance Mode: put hosts and services in maintenance mode to suppress alerts and operations from running on certain hosts LDAP: active directory for enterprise clusters Service Auto-Start: automatically restart down services and more. Ambari was developed with scale in mind to minimize to support clusters with several thousand nodes. The agents heartbeat to the server with the status of commands and daemons; while the server sends commands down to the agents. Blueprints is the mechanism to export an existing cluster and re-create it (or several others just like it) with a predefined topology, configs, etc. This is extremely useful in cloud environments. I hope this answers your question.
... View more
05-03-2017
10:01 PM
Problem: There's a known bug in Ambari 2.4 and 2.5 that causes "ambari-server upgrade" to fail if the agent RPM is not upgraded first. E.g. stack trace: Using python /usr/bin/python Setup ambari-server Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 33, in from ambari_server.dbConfiguration import DATABASE_NAMES, LINUX_DBMS_KEYS_LIST File "/usr/lib/python2.6/site-packages/ambari_server/dbConfiguration.py", line 28, in from ambari_server.serverConfiguration import decrypt_password_for_alias, get_ambari_properties, get_is_secure, \ File "/usr/lib/python2.6/site-packages/ambari_server/serverConfiguration.py", line 36, in from ambari_commons.os_utils import run_os_command, search_file, set_file_permissions, parse_log4j_file ImportError: cannot import name parse_log4j_file Cause: This occurs because os_utils.py and other python files inside of /usr/lib/ambari-agent/lib/ambari_commons are upgraded by the agent's RPM and are used by the server's scripts to find which database to use.
Solution:
Note: Always back up your Ambari database before the upgrade. If ambari-agent is also present on the Ambari Server host, run "yum upgrade ambari-agent" (or equivalent for your OS).
... View more
- Find more articles tagged with:
- Ambari
- ambari-upgrade
- Cloud & Operations
- Issue Resolution
Labels:
03-14-2017
04:49 PM
It's possible that the feature has not been maintained in the version of Ambari that you are using.
... View more
01-31-2017
01:47 AM
10 Kudos
Whether you're creating an Ambari cluster from scratch, taking over an existing cluster, or growing your cluster over time, it is imperative to tune Ambari and MySQL to work at a large scale of 1000-3000 Ambari Agents. Ambari Server Configs
First, increase the memory used by Ambari. For large clusters, 8 GB of memory should be sufficient. If you have more than 10 concurrent users, increase it to 16 GB.
Edit /var/lib/ambari-server/ambari-env.sh and change the -Xmn setting.
export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Xms2048m -Xmx8192m
Edit /etc/ambari-server/conf/ambari.properties with the following configs
# The size of the Jetty connection pool used for handling incoming Ambari Agent requests.
# 10 hosts => 25
# 50 hosts => 35
# 100 hosts => 75
# 500 hosts => 100
agent.threadpool.size.max=100
# Determines whether current alerts should be cached.
# Enabling this can increase performance on large cluster, but can also result in lost alert data
# if the cache is not flushed frequently.
alerts.cache.enabled=true
# The size of the alert cache.
# Less than 50 hosts => 50000
# More than 50 hosts => 100000
alerts.cache.size=100000
# The number of threads used to handle alerts received from the Ambari Agents.
# The value should be increased as the size of the cluster increases.
# Less than 50 hosts => 2
# More than 50 hosts => 4
alerts.execution.scheduler.maxThreads=4
After performing these changes, restart Ambari Server. Move an existing Ambari DB from a spinning disk to a SSD
It is highly suggested to use a Solid State Drive for the Ambari Database since this will be much faster.
Check the throughput of the disk in which Ambari’s database (Postgres, MySQL, MariaDB, or Oracle) is on.Ideally, it should be a Solid State Drive or support at least 200 IOPS and be either on the same host as Ambari or only a 1-2 hops away.
Type
Details
IOPS
Throughput
HDD
10,000 rpm SAS drive
175-210
100 MB/s
SSD
solid-state
500+
500+ MB/s
1. ambari-server stop
2. Take a backup of the Ambari database,
mysqldump -u root ambari > /tmp/ambari.sql
3. Stop MySQL server, copy its data, and change the directory.
service mysqld stop
cp -R -p /var/lib/mysql /mnt/disks/ssd/mysql
cat /etc/my.cnf
sed -ie 's/\/var\/lib\/mysql/\/mnt\/disks\/ssd\/mysql/g' /etc/my.cnf
4. Create symlink for sock file and start MySQL
ln -s /mnt/disks/ssd/mysql/mysql.sock /var/lib/mysql/mysql.sock
service mysqldstart
5. Ensure Ambari DB is accessible.
mysql -u root -p
show databases;
use ambari;
show tables;
select count(*) from hosts;
MySQL Optimizations
First and foremost, if you're on an older version of MySQL, you can try to update it to MySQL 5.6 or 5.7, which has a lot of performance improvements.
Connect to the MySQL DB and inspect these variables. E.g.,
SHOW VARIABLES LIKE 'name';These suggested values assume that only Ambari Database’s is on the MySQL Server.If you have other databases in the same MySQL Server, increment by these values.
WARNING: Never stop MySQL server while Ambari Server is running.
Variable
Suggested Value
innodb_log_buffer_size
512M
innodb_buffer_pool_size
16G
innodb_file_io_threads (deprecated in MySQL 5.5)
16
innodb_log_file_size
5M
innodb_thread_concurrency
32
join_buffer_size
512M
key_buffer_size
16G
max_connections
500
max_allowed_packet
1024M
max_heap_table_size
64M
query_cache_limit
16M
query_cache_size
512M
read_rnd_buffer_size
128M
sort_buffer_size
128M
table_open_cache
1024
thread_cache_size
128
thread_stack
256K
To change these values.1. Stop MySQL: service mysqld stop
2. Edit the configs in /etc/my.cnf , under the “[mysqld]” section (note, it may be in a different location).
3. Start MySQL: service mysqld start
... View more
- Find more articles tagged with:
- Ambari
- ambari-server
- Cloud & Operations
- How-ToTutorial
- MySQL
- optimization
- performance
Labels:
12-09-2016
07:58 PM
2 Kudos
Thanks wbu, What version of Ambari are you using? I'm guessing 2.4.x I created https://issues.apache.org/jira/browse/AMBARI-19155 on your behalf.
... View more
12-09-2016
07:53 PM
6 Kudos
There's an API to remove older versions from the hosts.
Take a look at https://issues.apache.org/jira/browse/AMBARI-18435 E.g., curl 'http://c6401.ambari.apache.org:8080/api/v1/clusters/cl1/requests' -u admin:admin -H "X-Requested-By: ambari" -X POST -d'{"RequestInfo":{"context":"remove_previous_stacks", "action" : "remove_previous_stacks", "parameters" : {"version":"2.5.0.0-1245"}}, "Requests/resource_filters": [{"hosts":"c6403.ambari.apache.org, c6402.ambari.apache.org"}]}' That removes all versions older than "2.5.0.0-1245"; the version mentioned in request remains intact. We also have checks that prevent removing the current version and the operation is idempotent. That will not remove the version from Ambari's database, but will free up space.
... View more
09-23-2016
07:07 PM
The only relevant columns that are varchar(256) in the hosts table are host_name, cpu_info, os_arch, os_type, and rack_info. You can try to figure out which one is exceeding the limit by printing out these values on the agent, or setting up wireshark to look at the traffic. ambari=> select host_name, cpu_info, os_arch, os_type, rack_info from hosts;
host_name | cpu_info | os_arch | os_type | rack_info
-------------------------+----------+---------+---------+---------------
c6401.ambari.apache.org | | x86_64 | centos6 | /default-rack
c6402.ambari.apache.org | | x86_64 | centos6 | /default-rack
c6403.ambari.apache.org | | x86_64 | centos6 | /default-rack
(3 rows)
Or you can take a DB backup, increase the size of all of the varchar columns to 512 to find the offending value, fix it, and then restore your DB to perform the agent registration again. http://sniptools.com/databases/resize-a-column-in-a-postgresql-table-without-changing-data
... View more
06-08-2016
09:26 PM
2 Kudos
The custom commands are actually defined at the service-level in the metainfo.xml file. E.g., HDFS: DECOMMISSION, REBALANCEHDFS
Hbase: DECOMMISSION
YARN: REFRESHQUEUES, CREATE_YARN_DIRECTORIES (needed for LLAP in a Kerberized cluster)
Knox: STARTDEMOLDAP, STOPDEMOLDAP
Hive: CLEAN, RESTART_LLAP
Kerberos: SET_KEYTAB, REMOVE_KEYTAB
... View more
06-08-2016
09:01 PM
For this command, find the correct /var/lib/ambari-agent/data/command-*.json file
and find the section, e.g., "hostLevelParams": {
...,
"ambari_db_rca_password": "mapred"
} This was introduced by AMBARI-4149, which tries to access the element at index 0, however, it's a string instead of a list.
This is controlled by the "server.jdbc.rca.user.passwd" property in the /etc/ambari-server/conf/ambari.properties file. You should file a Jira on Apache since this is a bug.
... View more
05-31-2016
08:18 PM
1 Kudo
That conf-select call tries to create the following symlink. /usr/hdp/current/zookeeper-server/conf -> /etc/zookeeper/2.4.0.0-169/0 1. Make sure zookeeper server is actually installed and Ambari shows it as a component for that host.
rpm -qa | grep zookeeper_.*server 2. Set its symlink.
hdp-select set zookeeper-server 2.4.0.0-169
# this will create symlink /usr/hdp/current/zookeeper-server -> /usr/hdp/2.4.0.0-169/zookeeper 3. Restart ZK Server on that host via the UI.
... View more
05-31-2016
08:07 PM
2 Kudos
ambari-server setup controls the database used by Ambari, which I don't think you want to change.
You can see the DB name, username, and jdbc string in /etc//etc/ambari-server/conf/ambari.properties
Ambari doesn't actually perform any mysql restarts on that database. Instead, you may have to delete the MySQL component from the HIVE service in Ambari since you're managing it yourself.
http://server:8080/api/v1/clusters/$NAME/services/HIVE/components/MYSQL_SERVER
will show which host contains that component. (note: you may have to stop the DB server before running this). If I understood your question correctly, this is what you want to do to manage MySQL server on your own.
You should also ask on the Hive mailing list. curl -u admin:admin -X DELETE -H 'X-Requested-By:admin' http://server:8080/api/v1/clusters/$NAME/hosts/$FQDN/host_components/MYSQL_SERVER
curl -u admin:admin -X DELETE -H 'X-Requested-By:admin' http://server:8080/api/v1/clusters/$NAME/services/HIVE/components/MYSQL_SERVER
... View more
04-25-2016
11:40 PM
7 Kudos
When performing a
Rolling or Express Upgrade, failures can naturally happen because large clusters are bound to have problematic hosts.
Here are 10 easy tips to prevent, diagnose and fix errors. Before upgrading the stack ...
1. Always upgrade Ambari to the most recent version, even if it's a dot release.
Often, there are fixes and optimizations that make the stack upgrade smoother.
2. Ensure all services are up, service checks are passing, there are no critical alerts, etc.
This helps ensure that the cluster is fully operational and helps to isolate any failures.
3. Pre-Install the bits and make sure all hosts have enough disk space. You can check that the version is found on all hosts. E.g.,
hdp-select versions | grep 2.5.0.0 | sort | tail -1
4. Do not ignore warnings. Starting in Ambari 2.2.2, there's a flag in ambari.properties file that allows users to bypass PreCheck errors, make sure it is either not present or set to false,
stack.upgrade.bypass.prechecks=false
5. Take a backup of the Ambari database. E.g.,
pg_dump -U ambari ambari > /tmp/ambari_bk.psql
mysqldump -u ambari ambari > /tmp/ambari_bk.mysql
In the middle of Upgrade ...
6. Rolling Upgrade will pause after 30% of the DataNodes have been upgraded. This allows the customer to run additional jobs and ensure that the partial upgrade is still healthy.
7. If a failure occurs, click on "Retry" and make sure that all other dependent services and masters are up.
Often, a retry will work if the previous command failed due to a timeout, network glitch, host goes down and then comes back up, etc. Capture any logs from both the component that failed and the ambari-agent at /var/lib/ambari-agent/data/output-*.txt and /var/lib/ambari-agent/data/errors-*.txt
8. If the failure requires changing configs or restarting a component on a host, then click on the "Pause" button. This will temporarily suspend the Upgrade/Downgrade and allow the user to change configs, execute other commands, such as restarting services, running service checks, etc. Once done, click on the "Resume" button.
CAUTION: do not ever add or move hosts, add or delete services, enable High Availability, or change topology while the upgrade is in progress.
If cannot Finalize ...
9. Find out the problematic hosts and components. In Ambari 2.0 - 2.2, you can run
SELECT repo_version_id, version, display_name FROM repo_version;
-- The state for your version may be in UPGRADING, UPGRADED.-- UPGRADING: some component on a host is still not on the newer version
-- UPGRADED: all components on all hosts are on the newer version
SELECT version, state FROM cluster_version cv JOIN repo_version rv ON cv.repo_version_id = rv.repo_version_id ORDER BY version DESC;
-- Find how many hosts are in each state
SELECT version, state, COUNT(*) FROM host_version hv JOIN repo_version rv ON hv.repo_version_id = rv.repo_version_id GROUP BY version, state ORDER BY version DESC, state;
-- Find components on hosts still not on the newer version
SELECT service_name, component_name, version, host_name FROM hostcomponentstate hcs JOIN hosts h ON hcs.host_id = h.host_id WHERE service_name NOT IN ('AMBARI_METRICS', 'KERBEROS') and component_name NOT IN ('ZKFC') ORDER BY version, service_name, component_name, host_name;
On these hosts, run the following,
1. hdp-select set all <new_version>
2. Restart any components still on the older version (you may have to click on the "Pause" button first).
Once all hosts are on the newer version, then the Cluster Version status should transition to UPGRADED; this will allow you to Finalize the upgrade.
10. If you still run into problems, gather all of the logs, result of the SQL queries, and either email Hortonworks Support or the mailing list of the component it failed on.
Here's another useful query.
Postgres:
SELECT u.upgrade_id, u.direction, u.from_version, u.to_version, hrc.request_id, hrc.task_id, substr(g.group_title, 0, 30), substr(i.item_text, 0, 30), hrc.status
FROM upgrade_group g JOIN upgrade u ON g.upgrade_id = u.upgrade_id
JOIN upgrade_item i ON i.upgrade_group_id = g.upgrade_group_id
JOIN host_role_command hrc ON hrc.stage_id = i.stage_id AND hrc.request_id = u.request_id
ORDER BY hrc.task_id;
MySQL:
SELECT u.upgrade_id, u.direction, u.from_version, u.to_version, hrc.request_id, hrc.task_id, left(g.group_title, 30), left(i.item_text, 30), hrc.status
FROM upgrade_group g JOIN upgrade u ON g.upgrade_id = u.upgrade_id
JOIN upgrade_item i ON i.upgrade_group_id = g.upgrade_group_id
JOIN host_role_command hrc ON hrc.stage_id = i.stage_id AND hrc.request_id = u.request_id
ORDER BY hrc.task_id;
Have fun upgrading.
... View more
- Find more articles tagged with:
- Ambari
- Cloud & Operations
- express
- Issue Resolution
- rolling
- upgrade
Labels:
04-07-2016
11:14 PM
1 Kudo
This happened because a record may be missing in the repo_version table. If you insert the record before performing the Ambari Upgrade it should work.
You can make sure you have a record for each one of the versions.
SELECT from_version, to_version FROM upgrade;
-- Assuming 2.2.0.0-2041 is missing
INSERT INTO repo_version (repo_version_id, version, display_name, upgrade_package, repositories, stack_id) VALUES (1, '2.2.0.0-2041', 'HDP-2.2.0.0-2041', 'upgrade-2.2', '', (SELECT stack_id FROM stack WHERE stack_version = '2.2'));
... View more
04-06-2016
06:14 PM
2 Kudos
This is happening during the upgrade to Ambari 2.2.0
https://github.com/apache/ambari/blob/release-2.2.1-rc2/ambari-server/src/main/java/org/apache/ambari/server/upgrade/UpgradeCatalog220.java#L508-L598 What is the output of running, SELECT upgrade_id, cluster_id, from_version, to_version, direction, upgrade_package, upgrade_type FROM upgrade;
Basically, you have to populate the upgrade_type and upgrade_pack columns based on the versions used.
... View more
04-06-2016
05:44 PM
5 Kudos
Whenever you see a property enclosed by {{ }} it means that the value is interpreted dynamically by reading params.py or params_linux.py E.g., https://github.com/apache/ambari/blob/release-2.2.1-rc2/ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py#L120-L148
To change it, you can either modify params*.py to use different logic, edit the value in the Config tab of the UI from the Service Dashboard page, or use configs.sh to edit directly (which calls the API).
... View more
03-31-2016
09:12 PM
2 Kudos
Ambari doesn't support that yet. We have a Jira for Ambari 3.0.0 https://issues.apache.org/jira/browse/AMBARI-14714 It will allow you to have multiple instances of the same service, and potentially at different stack versions, e.g., Spark 1.6.1, 1.7.0, etc.
... View more
03-31-2016
01:32 AM
I believe that in either Ambari 2.1.2 or 2.2.0 we introduced a button to "Pause" the upgrade. This essentially aborts all of the pending tasks from the upgrade so that you can perform operations on your own, such as restarting services. In this "paused" state, we only recommended to start/stop/restart services and change configs; anything that involves changing the topology (adding hosts, adding services, HA, etc.) is likely to result in problems. Because the RU/EU is still technically active (although in an ABORTED state), there will be a button to "Resume" it.
... View more
03-28-2016
06:53 PM
If you see this error, make sure to upgrade Ambari to version 2.1.2 or higher, https://github.com/apache/ambari/blob/release-2.1.2-rc3/ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py#L83-L89
... View more
03-14-2016
05:58 PM
2 Kudos
On that host, try running, hdp-select versions
hdp-select status
Make sure that /usr/hdp/ dir only contains the version folder and current
List all symlinks in /usr/hdp/current and make sure they point to the correct location. Note: if some packages are not installed, it is expected to have dead symlinks.
... View more
03-03-2016
01:44 AM
1 Kudo
What version of Ambari are you using?
Can you check this URI with the exact version, e.g., http://server:8080/views/ADMIN_VIEW/2.2.1.1/INSTANCE/#/ Can you check the API, http://server:8080/api/v1/users ? Any errors in /var/log/ambari-server/ambari-server.log ?
... View more
03-02-2016
07:06 PM
1 Kudo
Ambari 2.2 had to change some default configs for YARN in HDP 2.3.4.0 that are not compatible out-of-the-box with HDP 2.3.0.0.
You need to change these configs in yarn-site and then restart YARN.
yarn.timeline-service.version=1.0
yarn.timeline-service.store-class=org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore
... View more
02-04-2016
11:50 PM
What's the output of /var/log/hadoop/hdfs/hadoop-hdfs-namenode-pp-hdp-m.log ?
... View more
02-04-2016
07:20 PM
5 Kudos
$ grep "password" /etc/ambari-server/conf/ambari.properties
server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat
$ echo "bigdata_custom" > /etc/ambari-server/conf/password.dat
$ sudo -u postgres psql
postgres=# ALTER USER ambari WITH PASSWORD 'bigdata_custom';
postgres=# \q
$ ambari-server restart
... View more
01-28-2016
06:55 PM
Try just,
netstat -tupln | grep -i 10000 Any firewall rules?
... View more
01-19-2016
08:51 PM
2 Kudos
Actually, that issue is fixed in Ambari 2.1.2 by https://issues.apache.org/jira/browse/AMBARI-12979
... View more
01-19-2016
06:39 PM
2 Kudos
As of HDP 2.2.4, I believe that Knox changed its data folder to be a symlink to a versioned folder,
so /usr/hdp/current/knox-server/data -> /var/lib/knox/data/${version} (or something like this) This means that Knox in Ambari should always follow the symlink to get the right path instead of trying to access /var/lib/knox/data directly. I opened AMBARI-14726 to track a fix.
... View more
01-12-2016
09:44 PM
This should be against the requests endpoint, e.g.,
http://ambariserver:8080/api/v1/clusters/horton/requests
... View more