Member since
12-11-2015
196
Posts
86
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 13 | 12-20-2016 03:27 PM | |
| 110 | 07-26-2016 06:38 PM |
07-21-2018
02:35 AM
I need to downgrade to hdp 2.6.0 from 2.6.5, I am able to register it but not able to install.
... View more
- Tags:
- Hadoop Core
- hdp-2.6.0
Labels:
07-21-2018
02:31 AM
I need to downgrade from HDP 2.6.5 to 2.6.0 but not able to. When I register and click install on my cluster it takes me back to the same screen where it says "Manage Versions" and installation never starts..
... View more
- Tags:
- Hadoop Core
- hdp-2.6
Labels:
07-16-2018
02:29 AM
Thanks @Jay Kumar SenSharma. Question: On other nodes, do we have to do anything after installing new version of JAVA. Just by putting the JAVA in the same path as Ambari-server, is that going to be ok..?
... View more
07-16-2018
01:26 AM
I need to update Oracle JDK to 8u172. What's the best way of doing it. Is there a documentation
... View more
Labels:
07-16-2018
01:22 AM
Thank you @Jay Kumar SenSharma It did fixed the problem but wondering why it was working before and stopped working after a VM restart
... View more
07-11-2018
01:26 AM
I have few VM in Azure cloud. Upgraded memory on all the VM's and after reboot, looks like all the cluster members are not able to talk to Ambari server. What could be wrong
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
span.s1 {font-variant-ligatures: no-common-ligatures}
INFO 2018-07-11 01:20:43,857 NetUtil.py:70 - Connecting to https://dev1:8440/ca
ERROR 2018-07-11 01:20:43,863 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)
ERROR 2018-07-11 01:20:43,863 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
WARNING 2018-07-11 01:20:43,863 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-07-11 01:20:53,864 main.py:439 - Connecting to Ambari server at https://dev1:8440 (10.0.0.11)
INFO 2018-07-11 01:20:53,864 NetUtil.py:70 - Connecting to https://dev1:8440/ca
ERROR 2018-07-11 01:20:53,869 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)
ERROR 2018-07-11 01:20:53,869 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
WARNING 2018-07-11 01:20:53,869 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-07-11 01:21:03,869 NetUtil.py:70 - Connecting to https://dev1:8440/ca
ERROR 2018-07-11 01:21:03,874 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)
ERROR 2018-07-11 01:21:03,874 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
WARNING 2018-07-11 01:21:03,874 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-07-11 01:21:13,875 NetUtil.py:70 - Connecting to https://dev1:8440/ca
ERROR 2018-07-11 01:21:13,880 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)
ERROR 2018-07-11 01:21:13,880 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
WARNING 2018-07-11 01:21:13,880 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-07-11 01:21:23,881 NetUtil.py:70 - Connecting to https://dev1:8440/ca
ERROR 2018-07-11 01:21:23,886 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)
ERROR 2018-07-11 01:21:23,886 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
WARNING 2018-07-11 01:21:23,886 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
WARNING 2018-07-11 01:17:13,729 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-07-11 01:17:23,730 NetUtil.py:70 - Connecting to https://dev1:8440/ca
ERROR 2018-07-11 01:17:23,734 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579)
ERROR 2018-07-11 01:17:23,734 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions.
Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details.
WARNING 2018-07-11 01:17:23,734 NetUtil.py:124 - Server at https://dev1:8440 is not reachable, sleeping for 10 seconds..
... View more
Labels:
06-27-2018
06:32 PM
Hi ! I am trying to deploy a new cluster on AWS with the following specs: RHEL 7.5, Ambari 2.6.2, HDP 2.6.5. After registering the node, when I add the components it fails. Below are the log from one of the nodes. How can I skip it to install other components if one is giving issue. Please help.. 2018-06-27 18:05:21,063 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=None -> 2.6
2018-06-27 18:05:21,069 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2018-06-27 18:05:21,070 - Group['livy'] {}
2018-06-27 18:05:21,072 - Adding group Group['livy']
2018-06-27 18:05:21,094 - Group['spark'] {}
2018-06-27 18:05:21,094 - Adding group Group['spark']
2018-06-27 18:05:21,109 - Group['hdfs'] {}
2018-06-27 18:05:21,109 - Adding group Group['hdfs']
2018-06-27 18:05:21,124 - Group['hadoop'] {}
2018-06-27 18:05:21,124 - Adding group Group['hadoop']
2018-06-27 18:05:21,138 - Group['users'] {}
2018-06-27 18:05:21,139 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-06-27 18:05:21,139 - Adding user User['hive']
2018-06-27 18:05:21,169 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-06-27 18:05:21,170 - Adding user User['zookeeper']
2018-06-27 18:05:21,193 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-06-27 18:05:21,193 - Adding user User['infra-solr']
2018-06-27 18:05:21,215 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2018-06-27 18:05:21,216 - Adding user User['oozie']
2018-06-27 18:05:21,238 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-06-27 18:05:21,238 - Adding user User['ams']
2018-06-27 18:05:21,262 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2018-06-27 18:05:21,262 - Adding user User['tez']
2018-06-27 18:05:21,285 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-06-27 18:05:21,285 - Adding user User['livy']
2018-06-27 18:05:21,309 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-06-27 18:05:21,309 - Adding user User['spark']
2018-06-27 18:05:21,333 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None}
2018-06-27 18:05:21,333 - Adding user User['ambari-qa']
2018-06-27 18:05:21,365 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-06-27 18:05:21,365 - Adding user User['kafka']
2018-06-27 18:05:21,389 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hdfs'], 'uid': None}
2018-06-27 18:05:21,389 - Adding user User['hdfs']
2018-06-27 18:05:21,412 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-06-27 18:05:21,412 - Adding user User['sqoop']
2018-06-27 18:05:21,435 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-06-27 18:05:21,435 - Adding user User['yarn']
2018-06-27 18:05:21,457 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-06-27 18:05:21,458 - Adding user User['mapred']
2018-06-27 18:05:21,481 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None}
2018-06-27 18:05:21,481 - Adding user User['hcat']
2018-06-27 18:05:21,505 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-06-27 18:05:21,509 - Writing File['/var/lib/ambari-agent/tmp/changeUid.sh'] because it doesn't exist
2018-06-27 18:05:21,509 - Changing permission for /var/lib/ambari-agent/tmp/changeUid.sh from 644 to 555
2018-06-27 18:05:21,510 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-06-27 18:05:21,514 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2018-06-27 18:05:21,515 - Group['hdfs'] {}
2018-06-27 18:05:21,515 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hdfs', u'hdfs']}
2018-06-27 18:05:21,515 - FS Type:
2018-06-27 18:05:21,516 - Directory['/etc/hadoop'] {'mode': 0755}
2018-06-27 18:05:21,516 - Creating directory Directory['/etc/hadoop'] since it doesn't exist.
2018-06-27 18:05:21,516 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2018-06-27 18:05:21,516 - Creating directory Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] since it doesn't exist.
2018-06-27 18:05:21,517 - Changing owner for /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir from 0 to hdfs
2018-06-27 18:05:21,517 - Changing group for /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir from 0 to hadoop
2018-06-27 18:05:21,517 - Changing permission for /var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir from 755 to 1777
2018-06-27 18:05:21,517 - Directory['/var/lib/ambari-agent/tmp/AMBARI-artifacts/'] {'create_parents': True}
2018-06-27 18:05:21,517 - Creating directory Directory['/var/lib/ambari-agent/tmp/AMBARI-artifacts/'] since it doesn't exist.
2018-06-27 18:05:21,518 - File['/var/lib/ambari-agent/tmp/jdk-8u112-linux-x64.tar.gz'] {'content': DownloadSource('http://ip-172-31-7-134.us-east-2.compute.internal:8080/resources/jdk-8u112-linux-x64.tar.gz'), 'not_if': 'test -f /var/lib/ambari-agent/tmp/jdk-8u112-linux-x64.tar.gz'}
2018-06-27 18:05:21,520 - Downloading the file from http://ip-172-31-7-134.us-east-2.compute.internal:8080/resources/jdk-8u112-linux-x64.tar.gz
2018-06-27 18:05:23,877 - File['/var/lib/ambari-agent/tmp/jdk-8u112-linux-x64.tar.gz'] {'mode': 0755}
2018-06-27 18:05:23,877 - Changing permission for /var/lib/ambari-agent/tmp/jdk-8u112-linux-x64.tar.gz from 644 to 755
2018-06-27 18:05:23,878 - Directory['/usr/jdk64'] {}
2018-06-27 18:05:23,878 - Creating directory Directory['/usr/jdk64'] since it doesn't exist.
2018-06-27 18:05:23,878 - Execute[('chmod', 'a+x', u'/usr/jdk64')] {'sudo': True}
2018-06-27 18:05:23,884 - Execute['cd /var/lib/ambari-agent/tmp/jdk_tmp_LrBHXG && tar -xf /var/lib/ambari-agent/tmp/jdk-8u112-linux-x64.tar.gz && ambari-sudo.sh cp -rp /var/lib/ambari-agent/tmp/jdk_tmp_LrBHXG/* /usr/jdk64'] {}
2018-06-27 18:05:27,401 - Directory['/var/lib/ambari-agent/tmp/jdk_tmp_LrBHXG'] {'action': ['delete']}
2018-06-27 18:05:27,402 - Removing directory Directory['/var/lib/ambari-agent/tmp/jdk_tmp_LrBHXG'] and all its content
2018-06-27 18:05:28,056 - File['/usr/jdk64/jdk1.8.0_112/bin/java'] {'mode': 0755, 'cd_access': 'a'}
2018-06-27 18:05:28,057 - Execute[('chmod', '-R', '755', u'/usr/jdk64/jdk1.8.0_112')] {'sudo': True}
2018-06-27 18:05:28,088 - Repository['HDP-2.6-repo-1'] {'append_to_file': False, 'base_url': 'http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None}
2018-06-27 18:05:28,096 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': InlineTemplate(...)}
2018-06-27 18:05:28,096 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because it doesn't exist
2018-06-27 18:05:28,097 - Repository['HDP-2.6-GPL-repo-1'] {'append_to_file': True, 'base_url': 'http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.5.0', 'action': ['create'], 'components': [u'HDP-GPL', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None}
2018-06-27 18:05:28,100 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': '[HDP-2.6-repo-1]\nname=HDP-2.6-repo-1\nbaseurl=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0\n\npath=/\nenabled=1\ngpgcheck=0\n[HDP-2.6-GPL-repo-1]\nname=HDP-2.6-GPL-repo-1\nbaseurl=http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.5.0\n\npath=/\nenabled=1\ngpgcheck=0'}
2018-06-27 18:05:28,100 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because contents don't match
2018-06-27 18:05:28,100 - Repository['HDP-UTILS-1.1.0.22-repo-1'] {'append_to_file': True, 'base_url': 'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'ambari-hdp-1', 'mirror_list': None}
2018-06-27 18:05:28,103 - File['/etc/yum.repos.d/ambari-hdp-1.repo'] {'content': '[HDP-2.6-repo-1]\nname=HDP-2.6-repo-1\nbaseurl=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.5.0\n\npath=/\nenabled=1\ngpgcheck=0\n[HDP-2.6-GPL-repo-1]\nname=HDP-2.6-GPL-repo-1\nbaseurl=http://public-repo-1.hortonworks.com/HDP-GPL/centos7/2.x/updates/2.6.5.0\n\npath=/\nenabled=1\ngpgcheck=0\n[HDP-UTILS-1.1.0.22-repo-1]\nname=HDP-UTILS-1.1.0.22-repo-1\nbaseurl=http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7\n\npath=/\nenabled=1\ngpgcheck=0'}
2018-06-27 18:05:28,103 - Writing File['/etc/yum.repos.d/ambari-hdp-1.repo'] because contents don't match
2018-06-27 18:05:28,104 - Package['unzip'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-06-27 18:05:28,239 - Installing package unzip ('/usr/bin/yum -d 0 -e 0 -y install unzip')
2018-06-27 18:05:30,059 - Package['curl'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-06-27 18:05:30,095 - Skipping installation of existing package curl
2018-06-27 18:05:30,096 - Package['hdp-select'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-06-27 18:05:30,130 - Installing package hdp-select ('/usr/bin/yum -d 0 -e 0 -y install hdp-select')
2018-06-27 18:05:31,377 - call[('ambari-python-wrap', u'/usr/bin/hdp-select', 'versions')] {}
2018-06-27 18:05:31,400 - call returned (1, 'Traceback (most recent call last):\n File "/usr/bin/hdp-select", line 446, in <module>\n printVersions()\n File "/usr/bin/hdp-select", line 286, in printVersions\n for f in os.listdir(root):\nOSError: [Errno 2] No such file or directory: \'/usr/hdp\'')
2018-06-27 18:05:31,615 - Command repositories: HDP-2.6-repo-1, HDP-2.6-GPL-repo-1, HDP-UTILS-1.1.0.22-repo-1
2018-06-27 18:05:31,615 - Applicable repositories: HDP-2.6-repo-1, HDP-2.6-GPL-repo-1, HDP-UTILS-1.1.0.22-repo-1
2018-06-27 18:05:31,617 - Looking for matching packages in the following repositories: HDP-2.6-repo-1, HDP-2.6-GPL-repo-1, HDP-UTILS-1.1.0.22-repo-1
2018-06-27 18:05:34,280 - Package['hadoop_2_6_5_0_292-yarn'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-06-27 18:05:34,412 - Installing package hadoop_2_6_5_0_292-yarn ('/usr/bin/yum -d 0 -e 0 -y install hadoop_2_6_5_0_292-yarn')
2018-06-27 18:06:05,163 - Package['hadoop_2_6_5_0_292-mapreduce'] {'retry_on_repo_unavailability': False, 'retry_count': 5}
2018-06-27 18:06:05,201 - Installing package hadoop_2_6_5_0_292-mapreduce ('/usr/bin/yum -d 0 -e 0 -y install hadoop_2_6_5_0_292-mapreduce')
2018-06-27 18:06:06,663 - Execution of '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_6_5_0_292-mapreduce' returned 1. Error: Package: hadoop_2_6_5_0_292-hdfs-2.7.3.2.6.5.0-292.x86_64 (HDP-2.6-repo-1)
Requires: libtirpc-devel
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
2018-06-27 18:06:06,663 - Failed to install package hadoop_2_6_5_0_292-mapreduce. Executing '/usr/bin/yum clean metadata'
2018-06-27 18:06:06,910 - Retrying to install package hadoop_2_6_5_0_292-mapreduce after 30 seconds
... View more
Labels:
06-26-2018
08:34 PM
@Jay Kumar SenSharma - here is the log. When I restart it it stays up for sometime before going down. [root@D02 yarn]# tail -10 yarn-yarn-nodemanager-D02.asotc.com.log
2018-06-26 16:15:09,830 INFO shuffle.ExternalShuffleBlockResolver (ExternalShuffleBlockResolver.java:application) - Application application_1527390036186_1818 removed, cleanupLocalDirs = false
2018-06-26 16:15:09,854 INFO application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application applic0036186_1818 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2018-06-26 16:15:09,854 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:finishLogAggregationlication just finished : application_1527390036186_1818
2018-06-26 16:15:10,267 INFO zlib.ZlibFactory (ZlibFactory.java:<clinit>(49)) - Successfully loaded & initializeb library
2018-06-26 16:15:10,356 INFO ipc.Server (Server.java:saslProcess(1441)) - Auth successful for appattempt_1527390000001 (auth:SIMPLE)
2018-06-26 16:15:10,377 INFO compress.CodecPool (CodecPool.java:getCompressor(153)) - Got brand-new compressor [
2018-06-26 16:15:10,391 INFO logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:doContainerLogAggreg- Uploading logs for container container_e297_1527390036186_1818_03_000001. Current good log dirs are /data/hadoo
2018-06-26 16:15:10,454 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInte Start request for container_e297_1527390036186_1825_01_000001 by user dr.who
2018-06-26 16:15:10,455 INFO containermanager.ContainerManagerImpl (ContainerManagerImpl.java:startContainerInte Creating a new application reference for app application_1527390036186_1825
2018-06-26 16:15:10,473 INFO application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application applic0036186_1825 transitioned from NEW to INITING
2018-06-26 16:15:10,485 WARN logaggregation.LogAggregationService (LogAggregationService.java:verifyAndCreateRem5)) - Remote Root Log Dir [/app-logs] already exist, but with incorrect permissions. Expected: [rwxrwxrwt], Found]. The cluster may have problems with multiple users.
2018-06-26 16:15:10,486 WARN logaggregation.AppLogAggregatorImpl (AppLogAggregatorImpl.java:<init>(190)) - rollierval is set as -1. The log rolling mornitoring interval is disabled. The logs will be aggregated after this applinished.
2018-06-26 16:15:10,557 INFO nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=dr.who I0 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application86_1825 CONTAINERID=container_e297_1527390036186_1825_01_000001
2018-06-26 16:15:10,574 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(46ng path : /data/hadoop/yarn/log/application_1527390036186_1818/container_e297_1527390036186_1818_03_000001/launchh
2018-06-26 16:15:10,575 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:deleteAsUser(46ng path : /data/hadoop/yarn/log/application_1527390036186_1818/container_e297_1527390036186_1818_03_000001/direct
2018-06-26 16:15:10,679 INFO application.ApplicationImpl (ApplicationImpl.java:transition(304)) - Adding contain390036186_1825_01_000001 to application application_1527390036186_1825
2018-06-26 16:15:10,680 INFO application.ApplicationImpl (ApplicationImpl.java:handle(464)) - Application applic0036186_1825 transitioned from INITING to RUNNING
2018-06-26 16:15:10,680 INFO container.ContainerImpl (ContainerImpl.java:handle(1136)) - Container container_e29186_1825_01_000001 transitioned from NEW to LOCALIZED
2018-06-26 16:15:10,680 INFO containermanager.AuxServices (AuxServices.java:handle(196)) - Got event CONTAINER_Id application_1527390036186_1825
2018-06-26 16:15:10,680 INFO yarn.YarnShuffleService (YarnShuffleService.java:initializeContainer(183)) - Initiainer container_e297_1527390036186_1825_01_000001
2018-06-26 16:15:10,902 INFO container.ContainerImpl (ContainerImpl.java:handle(1136)) - Container container_e29186_1825_01_000001 transitioned from LOCALIZED to RUNNING
2018-06-26 16:15:10,910 INFO nodemanager.DefaultContainerExecutor (DefaultContainerExecutor.java:buildCommandExe- launchContainer: [bash, /data/hadoop/yarn/local/usercache/dr.who/appcache/application_1527390036186_1825/contai7390036186_1825_01_000001/default_container_executor.sh]
... View more
06-15-2018
02:03 AM
I have 3 node manager in my cluster and they are failing soon after restart.. I tried deleting /var/log/hadoop-yarn/nodemanager/recovery-state directory but no help... Please help. Where can I find the log to see why is this failing...
... View more
- Tags:
- Hadoop Core
- YARN
Labels:
06-04-2018
02:06 AM
All of a sudden all my cluster machines lost the hearbeat. What could be wro
INFO 2018-06-03 06:14:50,210 NetUtil.py:70 - Connecting to https://localhost:8440/connection_info
INFO 2018-06-03 06:14:50,351 security.py:55 - Server require two-way SSL authentication. Use it instead of one-way...
INFO 2018-06-03 06:14:50,352 security.py:182 - Server certicate exists, ok
INFO 2018-06-03 06:14:50,352 security.py:190 - Agent key exists, ok
INFO 2018-06-03 06:14:50,352 security.py:198 - Agent certificate exists, ok
INFO 2018-06-03 06:14:50,353 security.py:93 - SSL Connect being called.. connecting to the server
ERROR 2018-06-03 06:14:50,361 security.py:80 - Two-way SSL authentication failed. Ensure that server and agent certificates were signed by the same CA and restart the agent.
In order to receive a new agent certificate, remove existing certificate file from keys directory. As a workaround you can turn off two-way SSL authentication in server configuration(ambari.properties)
Exiting..
... View more
- Tags:
- ambari-server
Labels:
05-09-2018
11:19 AM
@Jay Kumar SenSharma I am trying to understand the host registration process: Ambari Server : in Ambari Properties this is what I have: s ecurity.server.two_way_ssl=true On All the host machine Running Python 2.7.5, I have "force_https_protocol=PROTOCOL_TLSv1_2" in ambari-agent. INI On my EdgeNode, I have Python 2.6 and I have to DISABLE force_https_protocol=PROTOCOL_TLSv1_2 to make it work correctly. PROBLEM: I have One VM running SUSE LINUX 11 SP4. It's running Python 2.6, so I have this "force_https_protocol=PROTOCOL_TLSv1_2" DISABLED but I am not able to register his host using AMBARI. here is the log: INFO 2018-05-08 21:00:39,842 AlertSchedulerHandler.py:185 - [AlertScheduler] Stopped the alert scheduler.
INFO 2018-05-08 21:00:39,842 threadpool.py:120 - Shutting down thread pool
INFO 2018-05-08 21:00:39,842 ExitHelper.py:70 - Cleanup finished, exiting with code:0
INFO 2018-05-08 21:39:06,690 main.py:145 - loglevel=logging.INFO
INFO 2018-05-08 21:39:06,690 main.py:145 - loglevel=logging.INFO
INFO 2018-05-08 21:39:06,690 main.py:145 - loglevel=logging.INFO
INFO 2018-05-08 21:39:06,697 DataCleaner.py:39 - Data cleanup thread started
INFO 2018-05-08 21:39:06,704 DataCleaner.py:120 - Data cleanup started
INFO 2018-05-08 21:39:06,706 DataCleaner.py:122 - Data cleanup finished
INFO 2018-05-08 21:39:06,817 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2018-05-08 21:39:06,818 main.py:181 - Newloglevel=logging.DEBUG
INFO 2018-05-08 21:39:06,820 main.py:437 - Connecting to Ambari server at https://ambari.asotc.com:8440 (172.16.1.15)
DEBUG 2018-05-08 21:39:06,820 NetUtil.py:113 - Trying to connect to https://ambari.asotc.com:8440
INFO 2018-05-08 21:39:06,820 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/ca
DEBUG 2018-05-08 21:39:06,986 NetUtil.py:90 - GET https://ambari.asotc.com:8440/ca -> 200, body:
INFO 2018-05-08 21:39:06,987 main.py:447 - Connected to Ambari server ambari.asotc.com
DEBUG 2018-05-08 21:39:06,987 Controller.py:66 - Initializing Controller RPC thread.
INFO 2018-05-08 21:39:07,027 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname 'vds' using socket.getfqdn().
INFO 2018-05-08 21:39:07,027 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads
WARNING 2018-05-08 21:39:07,028 AlertSchedulerHandler.py:280 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs.
INFO 2018-05-08 21:39:07,028 AlertSchedulerHandler.py:175 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0xf88b50>; currently running: False
DEBUG 2018-05-08 21:39:07,028 scheduler.py:574 - Scheduler started
DEBUG 2018-05-08 21:39:07,029 scheduler.py:579 - Looking for jobs to run
DEBUG 2018-05-08 21:39:07,029 scheduler.py:599 - No jobs; waiting until a job is added
INFO 2018-05-08 21:39:09,064 hostname.py:106 - Read public hostname 'vds' using socket.getfqdn()
INFO 2018-05-08 21:39:09,065 Hardware.py:48 - Initializing host system information.
INFO 2018-05-08 21:39:09,081 Hardware.py:176 - Some mount points were ignored: /dev/shm, /media/SLE-11-SP4-SAP-DVD-x86_6400061
INFO 2018-05-08 21:39:09,131 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname 'vds' using socket.getfqdn().
ERROR 2018-05-08 21:39:09,174 Controller.py:502 - Controller thread failed with exception:
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 489, in run
self.register = Register(self.config)
File "/usr/lib/python2.6/site-packages/ambari_agent/Register.py", line 35, in __init__
self.hardware = Hardware(self.config)
File "/usr/lib/python2.6/site-packages/ambari_agent/Hardware.py", line 53, in __init__
self.hardware.update(Facter(self.config).facterInfo())
File "/usr/lib/python2.6/site-packages/ambari_agent/Facter.py", line 571, in facterInfo
facterInfo = super(FacterLinux, self).facterInfo()
File "/usr/lib/python2.6/site-packages/ambari_agent/Facter.py", line 248, in facterInfo
'ipaddress': self.getIpAddress(),
File "/usr/lib/python2.6/site-packages/ambari_agent/Facter.py", line 81, in getIpAddress
return socket.gethostbyname(self.getFqdn().lower())
gaierror: [Errno -2] Name or service not known
INFO 2018-05-08 21:39:09,233 ExitHelper.py:56 - Performing cleanup before exiting...
INFO 2018-05-08 21:39:09,233 threadpool.py:120 - Shutting down thread pool
INFO 2018-05-08 21:39:09,233 scheduler.py:606 - Scheduler has been shut down
INFO 2018-05-08 21:39:09,234 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads
INFO 2018-05-08 21:39:09,234 AlertSchedulerHandler.py:185 - [AlertScheduler] Stopped the alert scheduler.
INFO 2018-05-08 21:39:09,234 threadpool.py:120 - Shutting down thread pool
INFO 2018-05-08 21:39:09,234 ExitHelper.py:70 - Cleanup finished, exiting with code:0
... View more
Labels:
05-04-2018
02:40 PM
INFO 2018-05-04 10:36:31,383 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/ca
WARNING 2018-05-04 10:36:31,384 NetUtil.py:101 - Failed to connect to https://ambari.asotc.com:8440/ca due to 'module' object has no attribute 'PROTOCOL_TLSv1_2'
WARNING 2018-05-04 10:36:31,384 NetUtil.py:124 - Server at https://ambari.asotc.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-05-04 10:36:41,384 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/ca
WARNING 2018-05-04 10:36:41,385 NetUtil.py:101 - Failed to connect to https://ambari.asotc.com:8440/ca due to 'module' object has no attribute 'PROTOCOL_TLSv1_2'
WARNING 2018-05-04 10:36:41,386 NetUtil.py:124 - Server at https://ambari.asotc.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-05-04 10:36:51,386 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/ca
WARNING 2018-05-04 10:36:51,387 NetUtil.py:101 - Failed to connect to https://ambari.asotc.com:8440/ca due to 'module' object has no attribute 'PROTOCOL_TLSv1_2'
WARNING 2018-05-04 10:36:51,388 NetUtil.py:124 - Server at https://ambari.asotc.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-05-04 10:37:01,388 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/ca
WARNING 2018-05-04 10:37:01,389 NetUtil.py:101 - Failed to connect to https://ambari.asotc.com:8440/ca due to 'module' object has no attribute 'PROTOCOL_TLSv1_2'
WARNING 2018-05-04 10:37:01,390 NetUtil.py:124 - Server at https://ambari.asotc.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-05-04 10:37:11,390 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/ca
WARNING 2018-05-04 10:37:11,391 NetUtil.py:101 - Failed to connect to https://ambari.asotc.com:8440/ca due to 'module' object has no attribute 'PROTOCOL_TLSv1_2'
WARNING 2018-05-04 10:37:11,392 NetUtil.py:124 - Server at https://ambari.asotc.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-05-04 10:37:21,392 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/ca
WARNING 2018-05-04 10:37:21,393 NetUtil.py:101 - Failed to connect to https://ambari.asotc.com:8440/ca due to 'module' object has no attribute 'PROTOCOL_TLSv1_2'
WARNING 2018-05-04 10:37:21,394 NetUtil.py:124 - Server at https://ambari.asotc.com:8440 is not reachable, sleeping for 10 seconds...
INFO 2018-05-04 10:37:31,394 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/ca
WARNING 2018-05-04 10:37:31,395 NetUtil.py:101 - Failed to connect to https://ambari.asotc.com:8440/ca due to 'module' object has no attribute 'PROTOCOL_TLSv1_2'
WARNING 2018-05-04 10:37:31,396 NetUtil.py:124 - Server at https://ambari.asotc.com:8440 is not reachable, sleeping for 10 seconds...
... View more
05-04-2018
02:36 PM
@Jay Kumar SenSharma Please find below ambari-agent INI and versio
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific
[server]
hostname=ambari.asotc.com
url_port=8440
secured_url_port=8441
connect_retry_delay=10
max_reconnect_retry_delay=30
[agent]
logdir=/var/log/ambari-agent
piddir=/var/run/ambari-agent
prefix=/var/lib/ambari-agent/data
;loglevel=(DEBUG/INFO)
loglevel=DEBUG
data_cleanup_interval=86400
data_cleanup_max_age=2592000
data_cleanup_max_size_MB = 100
ping_port=8670
cache_dir=/var/lib/ambari-agent/cache
tolerate_download_failures=true
run_as_user=root
parallel_execution=0
alert_grace_period=5
status_command_timeout=5
alert_kinit_timeout=14400000
system_resource_overrides=/etc/resource_overrides
; memory_threshold_soft_mb=400
; memory_threshold_hard_mb=1000
; ignore_mount_points=/mnt/custom1,/mnt/custom2
[security]
force_https_protocol=PROTOCOL_TLSv1_2
keysdir=/var/lib/ambari-agent/keys
server_crt=ca.crt
passphrase_env_var_name=AMBARI_PASSPHRASE
ssl_verify_cert=0
credential_lib_dir=/var/lib/ambari-agent/cred/lib
credential_conf_dir=/var/lib/ambari-agent/cred/conf
credential_shell_cmd=org.apache.hadoop.security.alias.CredentialShell
[n_protocol=PROTOCOL_TLSv1_2twork]
[services]
pidLookupPath=/var/run/
[heartbeat]
state_interval_seconds=60
dirs=/etc/hadoop,/etc/hadoop/conf,/etc/hbase,/etc/hcatalog,/etc/hive,/etc/oozie,
/etc/sqoop,
/var/run/hadoop,/var/run/zookeeper,/var/run/hbase,/var/run/templeton,/var/run/oozie,
/var/log/hadoop,/var/log/zookeeper,/var/log/hbase,/var/run/templeton,/var/log/hive
; 0 - unlimited
log_lines_count=300
idle_interval_min=1
idle_interval_max=10
[logging]
syslog_enabled=0
vds:~ # ambari-agent --version
2.5.1.0
... View more
05-02-2018
08:59 PM
Hi ! Experts. I am working on a POC for SAP-->Hadoop integration prototype. Basically looking into moving data off of SAP and store into Hadoop. BODS is being used for this purpose. Setup: HIVESERVER2 is setup for "NO Authentication" ODBC connection is used to connect to HIVE Warehouse Connection is successful BODS configuration is looking for HDFS directory which I have created Question: When BODS/Job server copies data to hadoop/hive, does it create tables by default. I assume since hiveserver2 is setup as no authentication, BODS will be able to create tables without any authorization What's the purpose of HDFS directory. I created a folder inside HDFS and made it "777" so that anyone create files in it. Thanks
... View more
Labels:
05-02-2018
02:38 PM
@Jay Kumar SenSharma Trying to register another host ( SUSE LINUX 11, SP4) and getting the following error. OS: SUSE LINUX 11, SP4 Python: 2.6.9 AMbari-Agent: 2.5.1 WARNING 2018-05-02 10:30:45,836 NetUtil.py:101 - Failed to connect to https://ambari.asotc.com:8440/ca due to 'module' object has no attribute 'PROTOCOL_TLSv1_2'
... View more
Labels:
04-24-2018
04:26 PM
@Jay Kumar SenSharma Thank you. Not sure why is this setup like that. security.server.two_way_ssl = true I disabled it and restarted ambari-server. On the host machine I added property "force_https_protocol=PROTOCOL_TLSv1_2" and then I tried registering again BUT this time it did recognized and started registering but then in the review process I got the error below and registration process is stuck at "PREPARING TO DEPLOY: 14 of 14 task COMPLETED" Should I re-image the machine to SUSE LINUX 12 SP1 or SP2 ( per document HDP only supports till SP2) An internal system exception occurred: Trying to map host to cluster where stack does not support host's os type, clusterName=hadoop, clusterStackId=HDP-2.4, hostname=ds.asotc.com, hostOsFamily=suse12
... View more
04-24-2018
02:57 PM
@Jay Kumar SenSharma Looks like it got updated now ds:/var/log/ambari-agent # tail -50 /var/log/ambari-agent/ambari-agent.log
INFO 2018-04-24 10:55:27,940 security.py:55 - Server require two-way SSL authentication. Use it instead of one-way...
INFO 2018-04-24 10:55:27,941 security.py:179 - Server certicate not exists, downloading
INFO 2018-04-24 10:55:27,941 security.py:202 - Downloading server cert from https://ambari.asotc.com:8440/cert/ca/
ERROR 2018-04-24 10:55:27,949 Controller.py:226 - Unable to connect to: https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 175, in registerWithServer
ret = self.sendRequest(self.registerUrl, data)
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 545, in sendRequest
raise IOError('Request to {0} failed due to {1}'.format(url, str(exception)))
IOError: Request to https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com failed due to <urlopen error EOF occurred in violation of protocol (_ssl.c:661)>
ERROR 2018-04-24 10:55:27,949 Controller.py:227 - Error:Request to https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com failed due to <urlopen error EOF occurred in violation of protocol (_ssl.c:661)>
WARNING 2018-04-24 10:55:27,949 Controller.py:228 - Sleeping for 26 seconds and then trying again
DEBUG 2018-04-24 10:55:54,143 HostCheckReportFileHandler.py:126 - Host check report at /var/lib/ambari-agent/data/hostcheck.result
DEBUG 2018-04-24 10:55:54,144 HostCheckReportFileHandler.py:177 - Removing old host check file at /var/lib/ambari-agent/data/hostcheck.result
DEBUG 2018-04-24 10:55:54,145 HostCheckReportFileHandler.py:182 - Creating host check file at /var/lib/ambari-agent/data/hostcheck.result
INFO 2018-04-24 10:55:54,158 Controller.py:170 - Registering with ds.asotc.com (172.16.1.67) (agent='{"hardwareProfile": {"kernel": "Linux", "domain": "asotc.com", "physicalprocessorcount": 1, "kernelrelease": "4.4.73-7-default", "uptime_days": "0", "memorytotal": 8068504, "swapfree": "2.00 GB", "memorysize": 8068504, "osfamily": "suse", "swapsize": "2.00 GB", "processorcount": 1, "netmask": "255.255.255.0", "timezone": "EST", "hardwareisa": "x86_64", "memoryfree": 6838720, "operatingsystem": "sles", "kernelmajversion": "4.4", "kernelversion": "4.4.73", "macaddress": "00:15:5D:01:C8:19", "operatingsystemrelease": "12", "ipaddress": "172.16.1.67", "hostname": "ds", "uptime_hours": "1", "fqdn": "ds.asotc.com", "id": "root", "architecture": "x86_64", "selinux": false, "mounts": [{"available": "4023372", "used": "0", "percent": "0%", "device": "devtmpfs", "mountpoint": "/dev", "type": "devtmpfs", "size": "4023372"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/.snapshots", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/tmp", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/home", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/mysql", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/boot/grub2/x86_64-efi", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/opt", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/tmp", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/srv", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/spool", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/usr/local", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/libvirt/images", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/mailman", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/machines", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/log", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/crash", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/pgsql", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/boot/grub2/i386-pc", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/mariadb", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/named", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/opt", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/cache", "type": "btrfs", "size": "62914560"}], "hardwaremodel": "x86_64", "uptime_seconds": "6573", "interfaces": "eth0,lo"}, "currentPingPort": 8670, "prefix": "/var/lib/ambari-agent/data", "agentVersion": "2.5.1.0", "agentEnv": {"transparentHugePage": "", "hostHealth": {"agentTimeStampAtReporting": 1524581754145, "activeJavaProcs": [], "liveServices": [{"status": "Healthy", "name": "ntpd or ntp", "desc": ""}]}, "reverseLookup": true, "alternatives": [], "hasUnlimitedJcePolicy": null, "umask": "18", "firewallName": "rcSuSEfirewall2", "stackFoldersAndFiles": [], "existingUsers": [], "firewallRunning": false}, "timestamp": 1524581753950, "hostname": "ds.asotc.com", "responseId": -1, "publicHostname": "ds.asotc.com"}')
INFO 2018-04-24 10:55:54,162 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/connection_info
DEBUG 2018-04-24 10:55:54,335 NetUtil.py:90 - GET https://ambari.asotc.com:8440/connection_info -> 200, body: {"security.server.two_way_ssl":"true"}
DEBUG 2018-04-24 10:55:54,336 security.py:52 - Server two-way SSL authentication required: True
INFO 2018-04-24 10:55:54,336 security.py:55 - Server require two-way SSL authentication. Use it instead of one-way...
INFO 2018-04-24 10:55:54,337 security.py:179 - Server certicate not exists, downloading
INFO 2018-04-24 10:55:54,337 security.py:202 - Downloading server cert from https://ambari.asotc.com:8440/cert/ca/
ERROR 2018-04-24 10:55:54,345 Controller.py:226 - Unable to connect to: https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 175, in registerWithServer
ret = self.sendRequest(self.registerUrl, data)
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 545, in sendRequest
raise IOError('Request to {0} failed due to {1}'.format(url, str(exception)))
IOError: Request to https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com failed due to <urlopen error EOF occurred in violation of protocol (_ssl.c:661)>
ERROR 2018-04-24 10:55:54,345 Controller.py:227 - Error:Request to https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com failed due to <urlopen error EOF occurred in violation of protocol (_ssl.c:661)>
WARNING 2018-04-24 10:55:54,345 Controller.py:228 - Sleeping for 25 seconds and then trying again
DEBUG 2018-04-24 10:56:19,552 HostCheckReportFileHandler.py:126 - Host check report at /var/lib/ambari-agent/data/hostcheck.result
DEBUG 2018-04-24 10:56:19,553 HostCheckReportFileHandler.py:177 - Removing old host check file at /var/lib/ambari-agent/data/hostcheck.result
DEBUG 2018-04-24 10:56:19,553 HostCheckReportFileHandler.py:182 - Creating host check file at /var/lib/ambari-agent/data/hostcheck.result
INFO 2018-04-24 10:56:19,558 Controller.py:170 - Registering with ds.asotc.com (172.16.1.67) (agent='{"hardwareProfile": {"kernel": "Linux", "domain": "asotc.com", "physicalprocessorcount": 1, "kernelrelease": "4.4.73-7-default", "uptime_days": "0", "memorytotal": 8068504, "swapfree": "2.00 GB", "memorysize": 8068504, "osfamily": "suse", "swapsize": "2.00 GB", "processorcount": 1, "netmask": "255.255.255.0", "timezone": "EST", "hardwareisa": "x86_64", "memoryfree": 6838720, "operatingsystem": "sles", "kernelmajversion": "4.4", "kernelversion": "4.4.73", "macaddress": "00:15:5D:01:C8:19", "operatingsystemrelease": "12", "ipaddress": "172.16.1.67", "hostname": "ds", "uptime_hours": "1", "fqdn": "ds.asotc.com", "id": "root", "architecture": "x86_64", "selinux": false, "mounts": [{"available": "4023372", "used": "0", "percent": "0%", "device": "devtmpfs", "mountpoint": "/dev", "type": "devtmpfs", "size": "4023372"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/.snapshots", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/tmp", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/home", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/mysql", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/boot/grub2/x86_64-efi", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/opt", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/tmp", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/srv", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/spool", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/usr/local", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/libvirt/images", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/mailman", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/machines", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/log", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/crash", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/pgsql", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/boot/grub2/i386-pc", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/mariadb", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/lib/named", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/opt", "type": "btrfs", "size": "62914560"}, {"available": "55906100", "used": "5387708", "percent": "9%", "device": "/dev/mapper/system-root", "mountpoint": "/var/cache", "type": "btrfs", "size": "62914560"}], "hardwaremodel": "x86_64", "uptime_seconds": "6573", "interfaces": "eth0,lo"}, "currentPingPort": 8670, "prefix": "/var/lib/ambari-agent/data", "agentVersion": "2.5.1.0", "agentEnv": {"transparentHugePage": "", "hostHealth": {"agentTimeStampAtReporting": 1524581779553, "activeJavaProcs": [], "liveServices": [{"status": "Healthy", "name": "ntpd or ntp", "desc": ""}]}, "reverseLookup": true, "alternatives": [], "hasUnlimitedJcePolicy": null, "umask": "18", "firewallName": "rcSuSEfirewall2", "stackFoldersAndFiles": [], "existingUsers": [], "firewallRunning": false}, "timestamp": 1524581779358, "hostname": "ds.asotc.com", "responseId": -1, "publicHostname": "ds.asotc.com"}')
INFO 2018-04-24 10:56:19,561 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/connection_info
DEBUG 2018-04-24 10:56:19,683 NetUtil.py:90 - GET https://ambari.asotc.com:8440/connection_info -> 200, body: {"security.server.two_way_ssl":"true"}
DEBUG 2018-04-24 10:56:19,684 security.py:52 - Server two-way SSL authentication required: True
INFO 2018-04-24 10:56:19,684 security.py:55 - Server require two-way SSL authentication. Use it instead of one-way...
INFO 2018-04-24 10:56:19,684 security.py:179 - Server certicate not exists, downloading
INFO 2018-04-24 10:56:19,684 security.py:202 - Downloading server cert from https://ambari.asotc.com:8440/cert/ca/
ERROR 2018-04-24 10:56:19,693 Controller.py:226 - Unable to connect to: https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 175, in registerWithServer
ret = self.sendRequest(self.registerUrl, data)
File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 545, in sendRequest
raise IOError('Request to {0} failed due to {1}'.format(url, str(exception)))
IOError: Request to https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com failed due to <urlopen error EOF occurred in violation of protocol (_ssl.c:661)>
ERROR 2018-04-24 10:56:19,693 Controller.py:227 - Error:Request to https://ambari.asotc.com:8441/agent/v1/register/ds.asotc.com failed due to <urlopen error EOF occurred in violation of protocol (_ssl.c:661)>
WARNING 2018-04-24 10:56:19,694 Controller.py:228 - Sleeping for 28 seconds and then trying again
... View more
04-24-2018
02:48 PM
@Jay Kumar SenSharma time stamp are changing but file is not getting updated. This is weird: ds:/var/log/ambari-agent # ls -l
total 2112
-rw-r--r-- 1 root root 2158132 Apr 24 10:47 ambari-agent.log
-rw-r--r-- 1 root root 0 Apr 24 10:47 ambari-agent.out
... View more
04-24-2018
02:35 PM
@Jay Kumar SenSharma Changed the INI to Debug and restarted the ambari-agent. Reason of why I am having SUSE LINUX - Our cluster is all Centos 7 but we have a new requirement to move DATA from SAP to hadoop and we need BODS ( SAP tool) to be installed on a machine on the hadoop cluster. So we are deploying a new machine in the cluster with SUSE LINUX as BODS requires SUSE LINUX 12 SP13. That's what SAP group are telling me. Event after changing the loglevel to debug, it's still not showing the detail [server]
hostname=ambari.asotc.com
url_port=8440
secured_url_port=8441
connect_retry_delay=10
max_reconnect_retry_delay=30
[agent]
logdir=/var/log/ambari-agent
piddir=/var/run/ambari-agent
prefix=/var/lib/ambari-agent/data
;loglevel=(DEBUG/INFO)
loglevel=DEBUG
data_cleanup_interval=86400
data_cleanup_max_age=2592000
data_cleanup_max_size_MB = 100
ping_port=8670
cache_dir=/var/lib/ambari-agent/cache
tolerate_download_failures=true
run_as_user=root
LOG OUTPUT:
NFO 2018-04-24 09:23:26,242 main.py:145 - loglevel=logging.INFO
INFO 2018-04-24 09:23:26,242 main.py:145 - loglevel=logging.INFO
INFO 2018-04-24 09:23:26,242 main.py:145 - loglevel=logging.INFO
INFO 2018-04-24 09:23:26,246 DataCleaner.py:39 - Data cleanup thread started
INFO 2018-04-24 09:23:26,257 DataCleaner.py:120 - Data cleanup started
INFO 2018-04-24 09:23:26,258 DataCleaner.py:122 - Data cleanup finished
INFO 2018-04-24 09:23:26,356 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2018-04-24 09:23:26,358 main.py:437 - Connecting to Ambari server at https://ambari.asotc.com:8440 (172.16.1.15)
INFO 2018-04-24 09:23:26,359 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/ca
INFO 2018-04-24 09:23:26,759 main.py:447 - Connected to Ambari server ambari.asotc.com
INFO 2018-04-24 09:23:26,760 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname 'ds.asotc.com' using socket.getfqdn().
INFO 2018-04-24 09:23:26,761 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads
WARNING 2018-04-24 09:23:26,762 AlertSchedulerHandler.py:280 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs.
INFO 2018-04-24 09:23:26,762 AlertSchedulerHandler.py:175 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7f196b693c50>; currently running: False
INFO 2018-04-24 09:23:26,950 hostname.py:106 - Read public hostname 'ds.asotc.com' using socket.getfqdn()
INFO 2018-04-24 09:23:26,951 Hardware.py:48 - Initializing host system information.
INFO 2018-04-24 09:23:26,979 Hardware.py:176 - Some mount points were ignored: /dev/shm, /run, /sys/fs/cgroup, /run/user/0
INFO 2018-04-24 09:23:27,010 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname 'ds.asotc.com' using socket.getfqdn().
INFO 2018-04-24 09:23:27,039 Facter.py:202 - Directory: '/etc/resource_overrides' does not exist - it won't be used for gathering system resources.
... View more
04-24-2018
01:35 PM
I am trying to register a new host running SUSE LINUX 12. Other hosts machines are running Centos 7. I installed same version of Ambari-agent on this hosts but when I add this machine to the cluster, it's failing. . INFO 2018-04-24 09:23:26,242 main.py:145 - loglevel=logging.INFO
INFO 2018-04-24 09:23:26,242 main.py:145 - loglevel=logging.INFO
INFO 2018-04-24 09:23:26,242 main.py:145 - loglevel=logging.INFO
INFO 2018-04-24 09:23:26,246 DataCleaner.py:39 - Data cleanup thread started
INFO 2018-04-24 09:23:26,257 DataCleaner.py:120 - Data cleanup started
INFO 2018-04-24 09:23:26,258 DataCleaner.py:122 - Data cleanup finished
INFO 2018-04-24 09:23:26,356 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2018-04-24 09:23:26,358 main.py:437 - Connecting to Ambari server at https://ambari.asotc.com:8440 (172.16.1.15)
INFO 2018-04-24 09:23:26,359 NetUtil.py:70 - Connecting to https://ambari.asotc.com:8440/ca
INFO 2018-04-24 09:23:26,759 main.py:447 - Connected to Ambari server ambari.asotc.com
INFO 2018-04-24 09:23:26,760 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname 'ds.asotc.com' using socket.getfqdn().
INFO 2018-04-24 09:23:26,761 threadpool.py:58 - Started thread pool with 3 core threads and 20 maximum threads
WARNING 2018-04-24 09:23:26,762 AlertSchedulerHandler.py:280 - [AlertScheduler] /var/lib/ambari-agent/cache/alerts/definitions.json not found or invalid. No alerts will be scheduled until registration occurs.
INFO 2018-04-24 09:23:26,762 AlertSchedulerHandler.py:175 - [AlertScheduler] Starting <ambari_agent.apscheduler.scheduler.Scheduler object at 0x7f196b693c50>; currently running: False
INFO 2018-04-24 09:23:26,950 hostname.py:106 - Read public hostname 'ds.asotc.com' using socket.getfqdn()
INFO 2018-04-24 09:23:26,951 Hardware.py:48 - Initializing host system information.
INFO 2018-04-24 09:23:26,979 Hardware.py:176 - Some mount points were ignored: /dev/shm, /run, /sys/fs/cgroup, /run/user/0
INFO 2018-04-24 09:23:27,010 hostname.py:67 - agent:hostname_script configuration not defined thus read hostname 'ds.asotc.com' using socket.getfqdn().
INFO 2018-04-24 09:23:27,039 Facter.py:202 - Directory: '/etc/resource_overrides' does not exist - it won't be used for gathering system resources.
... View more
Labels:
04-18-2018
02:26 PM
I am working on a project SAP-Hadoop integration. Requirement is I need to add a windows server to hadoop cluster. Wondering if I can install ambari-agent on windows machine and install hadoop clients on that server which will then be used by SAP BODS to connect to hdfs and hive
... View more
Labels:
04-09-2018
02:43 AM
This is bizarre but I discovered that Ambari user cannot change their own password. Is there a workaround of this situation. This is a big security flaw.
... View more
- Tags:
- password
02-20-2018
02:53 AM
I have some basic question about Yarn Memory configuration recommendation. I am following this URL: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/determine-hdp-memory-config.html Questions: In my cluster I have total 7 Nodes, 3 data nodes, SNode, NameNode, Ambari server and one Edge Node. All have different set of memory and vCores and Disk size. When I calculate do I have to consider all the nodes in the cluster regardless of their purpose or only consider DataNodes . How can the script below be used if nodes have different configuration. python hdp-configuration-utils.py -c 16 -m 64 -d 4 -k True
... View more
- Tags:
- Hadoop Core
- YARN
Labels:
02-08-2018
12:23 AM
I am not using solr and I could see the logs in hdfs. It being stored in /ranger/audit folder on HDFS. But why I can't see this in the Ranger Audit page. I would like to see access and denial in Ranger admin UI. I could see all other tabs such as admin, login session and plugins but no access audit..
... View more
02-07-2018
06:26 PM
I don't see access log is being reported in ranger. Need to set this u.
... View more
Labels:
01-05-2018
06:07 PM
Any updated documentation. This document is almost 4 years old..Is it still relevant
... View more
01-05-2018
06:02 PM
1 Kudo
Looking for a good documentation of how to install ELK, LogStash and Kibana on a Ambari managed hadoop platform. My versions are: Centos 7.x Ambari 2.5.1 HDP 2.4.3 There are some documents on GitHub but not sure if that works. Any success stories here ?
... View more
01-02-2018
03:08 AM
Flume and Supervisor are running on same host but they go down immediately after starting, whats wrong
... View more
Labels:
12-30-2017
11:49 PM
I tried upgrading ambari server from 2.2.2. to 2.6. Backed up the databases as per the instructions too. But scheme upgrade failed and then I am not able to start ambari-server. Please help: Ambari Server configured for Embedded Postgres. Confirm you have made a backup of the Ambari Server database [y/n] (y)? y
INFO: Upgrading database schema
INFO: Return code from schema upgrade command, retcode = 1
ERROR: Error executing schema upgrade, please check the server logs.
ERROR: Error output from schema upgrade command:
ERROR: Exception in thread "main" org.apache.ambari.server.AmbariException: Database version does not meet minimal upgrade requirements. Expected version should be not less than 2.4.0, current version is 2.2.2
at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:408)
ERROR: Ambari server upgrade failed. Please look at /var/log/ambari-server/ambari-server.log, for more details.
ERROR: Exiting with exit code 11.
REASON: Schema upgrade failed.
... View more
Labels:
11-08-2017
11:05 PM
I followed the instructions at: https://hortonworks.com/blog/configure-elastic-search-hadoop-hdp-2-0/ It doesn't say how to start kibana, also I am not able to find bin\kibana file. What's wrong
... View more
