Member since
04-03-2019
962
Posts
1743
Kudos Received
146
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 14982 | 03-08-2019 06:33 PM | |
| 6167 | 02-15-2019 08:47 PM | |
| 5098 | 09-26-2018 06:02 PM | |
| 12581 | 09-07-2018 10:33 PM | |
| 7437 | 04-25-2018 01:55 AM |
02-07-2017
08:35 PM
@Venkat Ranganathan @peeyush @Abhishek Bafna
... View more
02-07-2017
08:33 PM
Is there any way to skip checking for jobHistory server to avoid Oozie job failures. Sometimes when JobHistory server is down or getting restarted, Oozie job gets failed with an error like "Unknown job id" error ( When it fails to connect to JobHistory server, which is expected ) Any idea how can we workaround on this error? Any timeout parameter?
... View more
Labels:
- Labels:
-
Apache Oozie
02-07-2017
07:08 PM
PROBLEM: Ambari Server won't be able to start because of DB inconsistencies. Sample Error: 2017-02-06 05:08:43,975 ERROR - You have non selected configs: zeppelin-ambari-config for service ZEPPELIN from cluster XXXX!
2017-02-06 05:08:43,976 INFO - ******************************* Check database completed *******************************
2017-02-06 05:10:12,834 INFO - Checking DB store version
2017-02-06 05:10:14,094 INFO - DB store version is compatible
2017-02-07 13:50:31,769 INFO - ******************************* Check database started *******************************
2017-02-07 13:50:41,247 INFO - Checking for configs not mapped to any cluster
2017-02-07 13:50:41,322 INFO - Checking for configs selected more than once
2017-02-07 13:50:41,326 INFO - Checking for hosts without state
2017-02-07 13:50:41,330 INFO - Checking host component states count equals host component desired states count
2017-02-07 13:50:41,334 INFO - Checking services and their configs
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / SQOOP
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / HDFS
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / MAPREDUCE2
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / TEZ
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / SPARK
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / HBASE
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / ZOOKEEPER
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / YARN
2017-02-07 13:50:45,793 INFO - Processing HDP-2.5 / KNOX
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / PIG
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / RANGER
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / HIVE
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / SLIDER
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / AMBARI_INFRA
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / KAFKA
2017-02-07 13:50:45,794 INFO - Processing HDP-2.5 / SMARTSENSE
2017-02-07 13:50:45,809 ERROR - You have non selected configs: zeppelin-ambari-config for service ZEPPELIN from cluster XXXXX!
2017-02-07 13:50:45,810 INFO - ******************************* Check database completed ******************************* . BUSINESS IMPACT: It's not recommended to make any changes in service configurations because backend Database is not consistent. . WORKAROUND: ambari-server start --skip-database-check Note - This is not recommended for production clusters, if you do this, please do not make any modifications in service configurations till you resolves the conflicts. . RESOLUTION: 1. Stop Ambari
server ambari-server stop . 2. Take backup of
Ambari Database For postgres - Use pg_dump command. For MySql - Use mysqldump command. . 3. Run below queries
to resolve conflicts delete from hostcomponentstate where service_name = 'ZEPPELIN';
delete from hostcomponentdesiredstate where service_name = 'ZEPPELIN';
delete from servicecomponentdesiredstate where service_name = 'ZEPPELIN';
delete from servicedesiredstate where service_name = 'ZEPPELIN';
delete from serviceconfighosts where service_config_id in (select service_config_id from serviceconfig where service_name = 'ZEPPELIN');
delete from serviceconfigmapping where service_config_id in (select service_config_id from serviceconfig where service_name = 'ZEPPELIN');
delete from serviceconfig where service_name = 'ZEPPELIN';
delete from requestresourcefilter where service_name = 'ZEPPELIN';
delete from requestoperationlevel where service_name = 'ZEPPELIN';
delete from clusterservices where service_name ='ZEPPELIN';
delete from clusterconfig where type_name like 'zeppelin%';
delete from clusterconfigmapping where type_name like 'zeppelin%'; . 4. Start Ambari Server and it should come up without any inconsistencies. . Please feel free to comment if you need any further help on this. Happy Hadooping!!
... View more
Labels:
02-07-2017
06:42 AM
@Saurabh You can check /var/log/messages to see if installation has started. Also if you want to check how much data has been downloaded, yum keeps package in yum cache while downloading, you can run 'du -sh' in watch command to check the status. Example. Before downloading package [root@prodnode1 ~]# ls -lrt /var/cache/yum//x86_64/6/Updates-ambari-2.4.0.1/packages/
total 0 Download started [root@prodnode1 ~]# /usr/bin/yum -d 0 -e 0 -y install ambari-metrics-hadoop-sink Status of cache directory [root@prodnode1 ~]# ls -lrt /var/cache/yum//x86_64/6/Updates-ambari-2.4.0.1/packages/
total 4552
-rw-r--r--. 1 root root 4660232 Aug 30 20:49 ambari-metrics-hadoop-sink-2.4.0.1-1.x86_64.rpm After installation is complete, package gets removed from cached location. You can run something like below to keep watch over download [root@prodnode1 ~]# watch du -sh /var/cache/yum//x86_64/6/Updates-ambari-2.4.0.1/packages/ambari-metrics-hadoop-sink-2.4.0.1-1.x86_64.rpm Hope this what you were looking for. Please do let us know if you have any further questions! 🙂
... View more
02-07-2017
06:33 AM
@Kargol Meister Is it single node cluster? I see localhost so wondering if you registered ambari agent correctly. You can try below command and see if it helps. ambari-agent stop
ambari-agent reset <ambari-server-hostname>
ambari-agent start If it's a production cluster. Please be careful. This command will reset ambari agent configuration including SSL certs(mostly self signed unless you have configured non default options) used for communication between ambari agent and server. No HDP services will be affected by running this command.
... View more
02-07-2017
06:29 AM
1 Kudo
@PJ Regarding your question - "I understand what you are saying but how can I change this to contact the active RM first? And how come this worked in 2.4.2 and not in 2.5.3, there should be some parameter changes?? Also, everytime it contacts resource manager, it is wasting some time checking which is active.." You can make "rm1" as active using failover command. Please use below command to failover from rm2 to rm1 as 'yarn' user on any yarn-client yarn rmadmin -failover rm2 rm1 If you have enabled automatic failover and for some reason above command fails then use below command( If you are doing it for production, please be very careful or contact Hortonworks Support ) yarn rmadmin -transitionToActive rm1 --forceactive --forcemanual OR If no jobs are running, then simply restart "rm2" from Ambari, rm1 will automatically become Active if automatic failover is enabled.
... View more
02-07-2017
06:24 AM
1 Kudo
@jean rivera In addition to answer given by Predrag. If you are interested to setup Oozie HA with Load balancer in a Kerberized environment, you can refer below article. http://crazyadmins.com/oozie-ha-configuration-with-kerberos/
... View more
02-07-2017
06:22 AM
1 Kudo
@PJ Please have a look at below threads https://community.hortonworks.com/questions/8597/how-to-replicate-all-hive-ddl-and-database-from-on.html https://community.hortonworks.com/questions/4496/how-to-migrate-hive-data-over-to-new-cluster.html Hope this helps!
... View more