Member since
05-03-2016
18
Posts
5
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2614 | 08-31-2016 01:50 PM |
06-23-2017
01:17 PM
We had kerberised the HDP cluster and storm UI was not opening and we need to troubleshoot StormUI issue. To put DEBUG on for STormUI follow this steps: -------------------------------------------------------------- In Ambari - Storm - Config - STORM UI Server section add below : ui.childopts(original)= -Xmx768m_JAAS_PLACEHOLDER ui.childopts = -Xmx768m_JAAS_PLACEHOLDER -Dsun.security.krb5.debug=true we wanted to troubleshoot zookeeper as well To put DEBUG on for Zookeeper follow this steps: ------------------------------------------------------------------ In Amabri - Zookeeeper - Advanced Zookeeper-log4j add below: #log4j.rootLogger=INFO,CONSOLE ***Comment this line log4j.rootLogger=DEBUG,CONSOLE,ROLLINGFILE ***uncomment this line The log file will be generated by default to /home/zookeeper/zookeeper.log if below property is set as default. log4j.appender.ROLLINGFILE.File=zookeeper.log
... View more
Labels:
03-30-2017
01:39 PM
When you install HDP cluster and if you select 'MySQL' database to store ambari or hive metastore or Ranger or oozie databases. The important aspect is to setup MYSQL replication and below step will illustrate how to setup MYSQL database replication. Mysql Master-Master-Active-Passive Replication :
================================================
step 1- Configure Master-Slave Replication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On server A:
------------ 1) Shutdown the mysql server process if its running.
#mysqladmin -u root shutdown -p 2) Edit my.cnf file with following values: # cat /etc/my.cnf
[mysqld]
datadir=/hadoop/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# New parameters for Ranger partitioned tables
innodb_file_per_table=1
innodb_buffer_pool_size=4G
innodb_flush_log_at_trx_commit=2
innodb_flush_method=O_DIRECT log_bin=mysql-bin
binlog_format=ROW
server_id=10
innodb_support_xa=1 [mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
3) Bring up mysql
#service mysqld start 4) Verify by running below command that server is now logging bin-logs
# mysql -u root -p
mysql> show master status;
+------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000003 | 5810761 | | |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec) 5) Make sure root user is able to do dump of MYSQL database by running following command
select user,host from mysql.user; - check if user already present mysql>CREATE USER ‘root’@’<serverA.FQDN>’ identified by ‘root’;
mysql>GRANT ALL ON *.* to ‘root’@’<serverA.FQDN>’; 6) Create replication user that will be used in future replications: mysql>GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl'@'<server-B.FQDN>' IDENTIFIED BY 'xxxxx'; 7) Run mysqldump to dump all tables from the master and load them onto the slave as follows: mysqldump --single-transaction --all-databases --master-data=1 --host=<serverA.FQDN> > /home/hadoopMYSQL.out -p 😎 copy this dump file over to the serverB
on server B :
------------- 1) shutdown the mysql server process if its running. 2) Edit my.cnf file with following values: # cat /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# New parameters for Ranger partitioned tables
innodb_file_per_table=1
innodb_buffer_pool_size=4G
innodb_flush_log_at_trx_commit=2
innodb_flush_method=O_DIRECT log_bin = mysql-bin
binlog_format = ROW
server_id = 11
relay_log = mysql-relay-bin
log_slave_updates = 1
read_only = 1 [mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
3) Bring up mysql
#service mysqld start 4) Create user which can load the prepared dump of the mysql database, by running the following command: mysql>create user 'root'@'<server-B.FQDN>' identified by 'root';
mysql>grant all on *.* to 'root'@'<server-B.FQDN>'; 5) Load the dump that was generated on serverA mysql --host=<server-B.FQDN> -p < /home/hadoop/MYSQL.out 6)Verify that all the database was transferred from dump file mysql> show databases 7) Get the File name and position from master server i.e serverA and put in master_log_file and position mysql> show master status;
+------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000003 | 5810761 | | |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec) Run below command on server B mysql> CHANGE MASTER TO MASTER_HOST='<server-A.FQDN>', MASTER_USER='repl',MASTER_PASSWORD='xxxxx', MASTER_LOG_FILE='mysql-bin.000003', MASTER_LOG_POS=5810761; 😎 Restart mysql server 9) check replication is working by verifying Seconds_Behind_Master as being 0.
mysql> SHOW SLAVE STATUS \G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host:
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000003
Read_Master_Log_Pos: 9231663
Relay_Log_File: mysql-relay-bin.000012
Relay_Log_Pos: 6451675
Relay_Master_Log_File: mysql-bin.000003
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 9231663
Relay_Log_Space: 6451830
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0 <------- If replication is working it will show '0' Lag
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
1 row in set (0.00 sec)
10) Now we whave Master - slave replication working from server A--> serverB.
Step 2 - Master-Master Active-passive Replication
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1) Now we need to configure Server-A's master as server-B as we configured server-B's master as server-A in above steps 2) On server B, create replication user mysql> GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'repl'@'<server-A.FQDN>' IDENTIFIED BY 'xxxxx'; mysql> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000007 | 6605279 | | |
+------------------+----------+--------------+------------------+
1 row in set (0.00 sec) 3) on server A, configure master as server B ( provide master log file & positions) mysql> CHANGE MASTER TO MASTER_HOST='<server-B.FQDN>', MASTER_USER='repl',MASTER_PASSWORD='xxxxx',MASTER_LOG_FILE='mysql-bin.000007',MASTER_LOG_POS=6605279; mysql>START SLAVE; 4) on server A, mysql> SHOW SLAVE STATUS \G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host:
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000007
Read_Master_Log_Pos: 6872411
Relay_Log_File: mysqld-relay-bin.000004
Relay_Log_Pos: 251
Relay_Master_Log_File: mysql-bin.000007
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 6872411
Relay_Log_Space: 552
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0 <------- If replication is working it will show '0' Lag
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
1 row in set (0.00 sec)
5) test bidirectional replication is working by creating test tables,databases
Switching Active and Passive Roles in the Master Master Active Passive Setup :
============================================================================== 1) shutdown Hive metastore server so it stops any new writes to the database. 2) switch the "active" master to read-only.
mysql> Flush tables with read lock;
mysql> set global read_only=on; 3) Wait till "passive" master to be caught up to replicating all updates from the old "active" master 4) Now switch the passive master's read only flag to off.
mysql> set global read_only=off;
mysql> Flush tables with read lock; 6) Change the config in AMbari ( hive metastore/Ranger/oozie ) to point to new active mysql server. 7) Point ambari server to new active MYSQL server. 😎 Restart all Application.
... View more
Labels:
11-03-2016
03:22 PM
1 Kudo
The Ranger KMS logs are present cd /usr/hdp/current/ranger-kms/ews
lrwxrwxrwx 1 kms kms 19 sep 23 20:33 logs-> /var/log/ranger/kms We would like to change ranger KMS logs location from /var/log/ ranger/kms to /hadoop/log/ranger/kms. mkdir /hadoop/log/ranger/kms
chmod 755 /hadoop/log/ranger/kms
chown kms:kms /hadoop/log/ranger/kms
rm symbolic link for logs cd /usr/hdp/current/ranger-kms/
vi ranger-kms
search for word "TOMCAT_LOG_DIR" and replace with new location.
TOMCAT_LOG_DIR=/hadoop/log/ranger/kms In Ambari webUI update log directory for ranger KMS kms_log_dir property.
Restart KMS
... View more
09-27-2016
09:19 PM
Labels:
- Labels:
-
Apache Hadoop
-
Apache YARN
09-09-2016
04:51 PM
Thanks @Artem Ervits.It worked! we were using HDP2.2 $hadoop daemonlog -setlevel <resource mgr host :8088> org.apache.hadoop.yarn.server.resourcemanger DEBUG We used above command to debug on and it did worked.
... View more
09-08-2016
09:15 PM
Forgot to mention.we are using Ambari 1.7 version!
... View more
09-08-2016
09:14 PM
2 Kudos
Labels:
- Labels:
-
Apache Hadoop
-
Cloudera Manager
09-07-2016
05:10 PM
Thanks @Yogeshprabhu
... View more
09-06-2016
04:52 PM
Labels:
- Labels:
-
Apache Hadoop
09-05-2016
02:32 PM
Labels:
- Labels:
-
Apache Spark