Support Questions

cjervis · ‎11-18-2019

Hello I'm new in Big data and I'm trying to create a cluster, but I have an issue with the registration of hosts, I saw a lot of posts about this problem but the problem is persist (You find below the Registration Log for the ambari server):

     ==========================
     Creating target directory...
     ==========================
     Command start time 2019-11-18 10:23:23
     Connection to ambari.server closed.
     SSH command execution finished



 host=ambari.server, exitcode=0
 Command end time 2019-11-18 10:23:23

 ==========================
 Copying ambari sudo script...
 ==========================

 Command start time 2019-11-18 10:23:23

 scp /var/lib/ambari-server/ambari-sudo.sh
 host=ambari.server, exitcode=0
 Command end time 2019-11-18 10:23:24

 ==========================
 Copying common functions script...
 ==========================

 Command start time 2019-11-18 10:23:24

 scp /usr/lib/python2.6/site-packages/ambari_commons
 host=ambari.server, exitcode=0
 Command end time 2019-11-18 10:23:24

 ==========================
 Copying OS type check script...
 ==========================

 Command start time 2019-11-18 10:23:24

 scp /usr/lib/python2.6/site-packages/ambari_server/os_check_type.py
 host=ambari.server, exitcode=0
Command end time 2019-11-18 10:23:24

 ==========================
 Running OS type check...
 ==========================

Command start time 2019-11-18 10:23:24
Cluster primary/cluster OS family is redhat7 and local/current OS family is redhat7

Connection to ambari.server closed.
SSH command execution finished
host=ambari.server, exitcode=0
Command end time 2019-11-18 10:23:24

==========================
Checking 'sudo' package on remote host...
==========================

Command start time 2019-11-18 10:23:24

Connection to ambari.server closed.
SSH command execution finished
host=ambari.server, exitcode=0
Command end time 2019-11-18 10:23:24

==========================
Copying repo file to 'tmp' folder...
==========================

Command start time 2019-11-18 10:23:24

scp /etc/yum.repos.d/ambari.repo
host=ambari.server, exitcode=0
Command end time 2019-11-18 10:23:24

==========================
Moving file to repo dir...
==========================

Command start time 2019-11-18 10:23:24

Connection to ambari.server closed.
SSH command execution finished
host=ambari.server, exitcode=0
Command end time 2019-11-18 10:23:25

==========================
Changing permissions for ambari.repo...
==========================

Command start time 2019-11-18 10:23:25

Connection to ambari.server closed.
SSH command execution finished
host=ambari.server, exitcode=0
Command end time 2019-11-18 10:23:25

==========================
Copying setup script file...
==========================

Command start time 2019-11-18 10:23:25

scp /usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
host=ambari.server, exitcode=0
Command end time 2019-11-18 10:23:25

==========================
Running setup agent script...
==========================

Command start time 2019-11-18 10:23:25
('WARNING 2019-11-18 10:23:34,821 NetUtil.py:116 - Server at https://X:8440 is not reachable, 
sleeping 
for 10 seconds...
INFO 2019-11-18 10:23:34,821 HeartbeatHandlers.py:115 - Stop event received
INFO 2019-11-18 10:23:34,821 NetUtil.py:122 - Stop event received
INFO 2019-11-18 10:23:34,821 ExitHelper.py:53 - Performing cleanup before exiting...
INFO 2019-11-18 10:23:34,821 ExitHelper.py:67 - Cleanup finished, exiting with code:0
INFO 2019-11-18 10:23:37,253 main.py:223 - Agent died gracefully, exiting.
INFO 2019-11-18 10:23:37,254 ExitHelper.py:53 - Performing cleanup before exiting...
INFO 2019-11-18 10:23:37,734 main.py:90 - loglevel=logging.INFO
INFO 2019-11-18 10:23:37,735 main.py:90 - loglevel=logging.INFO
INFO 2019-11-18 10:23:37,735 main.py:90 - loglevel=logging.INFO
INFO 2019-11-18 10:23:37,736 DataCleaner.py:39 - Data cleanup thread started
INFO 2019-11-18 10:23:37,738 DataCleaner.py:120 - Data cleanup started
INFO 2019-11-18 10:23:37,743 DataCleaner.py:122 - Data cleanup finished
INFO 2019-11-18 10:23:37,748 PingPortListener.py:50 - Ping port listener started on port: 8670
INFO 2019-11-18 10:23:37,750 main.py:349 - Connecting to Ambari server at <a href="https://ambari.server:8440" target="_blank">https://ambari.server:8440</a> 
(192.168.253.45)
INFO 2019-11-18 10:23:37,751 NetUtil.py:62 - Connecting to <a href="https://ambari.server:8440/ca" target="_blank">https://ambari.server:8440/ca</a>
ERROR 2019-11-18 10:23:37,754 NetUtil.py:88 - EOF occurred in violation of protocol (_ssl.c:618)
ERROR 2019-11-18 10:23:37,755 NetUtil.py:89 - SSLError: Failed to connect. Please check openssl 
library 
versions. 
Refer to: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1022468" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1022468</a> for more details.
WARNING 2019-11-18 10:23:37,755 NetUtil.py:116 - Server at <a href="https://ambari.server:8440" target="_blank">https://ambari.server:8440</a> is not 
reachable, sleeping for 10 seconds...', None)
Connection to ambari.server closed.
SSH command execution finished
host=ambari.server, exitcode=0
Command end time 2019-11-18 10:23:40
Registering with the server...
Registration with the server failed

Please help me to solve this problem

Shelton · ‎11-25-2019

@Kou_Bou

Biiiiiingo ! I specifically mentioned that see below and once again the vendor's release note is always the first point of reference.
You put in a lot of efforts and frustration I guess but if you had taken some time to read the prerequisites you won't have gone through all that

Having said that please take an accept my answer so the thread is closed and can be referenced by someone encountering the same issue.

Happy hadooping

View solution in original post

Shelton · ‎11-18-2019

@Kou_Bou

That shouldn't be a problem, but before getting to the diagnostics can you confirm you have diligently followed this Prepare the Environment newbies always forget that every step is important and hence things look complicated 🙂

Having said that is it a single/multi node cluster? In the logs I see something like host=ambari.server I hope you it's a pseudonymized value else your ambari should have FQDN the output of Linux command

$ hostname -f

I also see an error NetUtil.py:89 - SSLError: Failed to connect. That is due to python version

To resolve that do this you have to set verify=disable by editing the /etc/python/cert-verification.cfg file.

[https]
verify=platform_default

To
[https]
verify=disable

Can you also share these files

/etc/ambari-server/conf/ambari.properties
/etc/ambari-agent/conf/ambari-agent.ini
/etc/hosts

Please revert

Kou_Bou · ‎11-20-2019

Hello @Shelton thank you for your reply and sorry for my late reply you find below all clarification that you recommended:

[root@ambari ~]# hostname -f
ambari.server

Python version: (I have a lot a python version installed)

[root@ambari ~]# python -V
Python 2.7.5

[root@ambari ~]# vi /etc/python/cert-verification.cfg

# Possible values are:
# 'enable' to ensure HTTPS certificate verification is enabled by default
# 'disable' to ensure HTTPS certificate verification is disabled by default
# 'platform_default' to delegate the decision to the redistributor providing this particular Python version

# For more info refer to <a href="https://www.python.org/dev/peps/pep-0493/" target="_blank">https://www.python.org/dev/peps/pep-0493/</a>
[https]
verify=disable
~

ambari.properties file:

agent.package.install.task.timeout=1800
agent.stack.retry.on_repo_unavailability=false
agent.stack.retry.tries=5
agent.task.timeout=900
agent.threadpool.size.max=120
ambari-server.user=root
ambari.python.wrap=ambari-python-wrap
api.ssl=true
bootstrap.dir=/var/run/ambari-server/bootstrap
bootstrap.script=/usr/lib/python2.6/site-packages/ambari_server/bootstrap.py
bootstrap.setup_agent.script=/usr/lib/python2.6/site-packages/ambari_server/setupAgent.py
check_database_skipped=false
client.api.ssl.cert_name=https.crt
client.api.ssl.key_name=https.key
client.api.ssl.port=8443
client.threadpool.size.max=25
common.services.path=/var/lib/ambari-server/resources/common-services
custom.action.definitions=/var/lib/ambari-server/resources/custom_action_definitions
extensions.path=/var/lib/ambari-server/resources/extensions
http.strict-transport-security=max-age=31536000
http.x-frame-options=DENY
http.x-xss-protection=1; mode=block
java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre
java.releases=jdk1.8,jdk1.7
jce.download.supported=true
jce.name=jce_policy-8.zip
jdk.download.supported=true
jdk.name=jdk-8u77-linux-x64.tar.gz
jdk1.7.desc=Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
jdk1.7.dest-file=jdk-7u67-linux-x64.tar.gz
jdk1.7.home=/usr/jdk64/
jdk1.7.jcpol-file=UnlimitedJCEPolicyJDK7.zip
jdk1.7.jcpol-url=<a href="http://public-repo-1.hortonworks.com/ARTIFACTS/UnlimitedJCEPolicyJDK7.zip" target="_blank">http://public-repo-1.hortonworks.com/ARTIFACTS/UnlimitedJCEPolicyJDK7.zip</a>
jdk1.7.re=(jdk.*)/jre
jdk1.7.url=<a href="http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-7u67-linux-x64.tar.gz" target="_blank">http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-7u67-linux-x64.tar.gz</a>
jdk1.8.desc=Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
jdk1.8.dest-file=java-1.8.0-openjdk.x86_64
jdk1.8.home=/usr/lib/j
jdk1.8.jcpol-file=jce_policy-8.zip
jdk1.8.jcpol-url=<a href="http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip" target="_blank">http://public-repo-1.hortonworks.com/ARTIFACTS/jce_policy-8.zip</a>
jdk1.8.re=(jdk.*)/jre
jdk1.8.url=<a href="http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u77-linux-x64.tar.gz" target="_blank">http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u77-linux-x64.tar.gz</a>
kerberos.keytab.cache.dir=/var/lib/ambari-server/data/cache
metadata.path=/var/lib/ambari-server/resources/stacks
mpacks.staging.path=/var/lib/ambari-server/resources/mpacks
pid.dir=/var/run/ambari-server
recommendations.artifacts.lifetime=1w
recommendations.dir=/var/run/ambari-server/stack-recommendations
recovery.enabled_components=METRICS_COLLECTOR
recovery.type=AUTO_START
resources.dir=/var/lib/ambari-server/resources
rolling.upgrade.skip.packages.prefixes=
security.server.cert_name=https.crt
security.server.crt_pass_file=https.pass.txt
security.server.disabled.ciphers=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384|TLS_RSA_WITH_AES_256_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384|TLS_DHE_RSA_WITH_AES_256_CBC_SHA256|TLS_DHE_DSS_WITH_AES_256_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA|TLS_RSA_WITH_AES_256_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA|TLS_ECDH_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_RSA_WITH_AES_256_CBC_SHA|TLS_DHE_DSS_WITH_AES_256_CBC_SHA|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256|TLS_RSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_RSA_WITH_AES_128_CBC_SHA256|TLS_DHE_DSS_WITH_AES_128_CBC_SHA256|TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA|TLS_RSA_WITH_AES_128_CBC_SHA|TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA|TLS_ECDH_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_RSA_WITH_AES_128_CBC_SHA|TLS_DHE_DSS_WITH_AES_128_CBC_SHA|TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA|TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA|SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA|TLS_EMPTY_RENEGOTIATION_INFO_SCSV|TLS_DH_anon_WITH_AES_256_CBC_SHA256|TLS_ECDH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_256_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA256|TLS_ECDH_anon_WITH_AES_128_CBC_SHA|TLS_DH_anon_WITH_AES_128_CBC_SHA|TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA|SSL_DH_anon_WITH_3DES_EDE_CBC_SHA|SSL_RSA_WITH_DES_CBC_SHA|SSL_DHE_RSA_WITH_DES_CBC_SHA|SSL_DHE_DSS_WITH_DES_CBC_SHA|SSL_DH_anon_WITH_DES_CBC_SHA|SSL_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA|SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA|SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA|TLS_RSA_WITH_NULL_SHA256|TLS_ECDHE_ECDSA_WITH_NULL_SHA|TLS_ECDHE_RSA_WITH_NULL_SHA|SSL_RSA_WITH_NULL_SHA|TLS_ECDH_ECDSA_WITH_NULL_SHA|TLS_ECDH_RSA_WITH_NULL_SHA|TLS_ECDH_anon_WITH_NULL_SHA|SSL_RSA_WITH_NULL_MD5|TLS_KRB5_WITH_3DES_EDE_CBC_SHA|TLS_KRB5_WITH_3DES_EDE_CBC_MD5|TLS_KRB5_WITH_DES_CBC_SHA|TLS_KRB5_WITH_DES_CBC_MD5|TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA|TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5
security.server.key_name=https.key
security.server.keys_dir=/var/lib/ambari-server/keys
security.server.keystore_name=https.keystore.p12
security.server.truststore_name=https.keystore.p12
security.server.two_way_ssl=true
server.connection.max.idle.millis=900000
server.execution.scheduler.isClustered=false
server.execution.scheduler.maxDbConnections=5
server.execution.scheduler.maxThreads=5
server.execution.scheduler.misfire.toleration.minutes=480
server.fqdn.service.url=<a href="http://169.254.169.254/latest/meta-data/public-hostname" target="_blank">http://169.254.169.254/latest/meta-data/public-hostname</a>
server.http.session.inactive_timeout=1800
server.jdbc.connection-pool=internal
server.jdbc.database=postgres
server.jdbc.database_name=datalake
server.jdbc.postgres.schema=datalake
server.jdbc.user.name=root
server.jdbc.user.passwd=/etc/ambari-server/conf/password.dat
server.os_family=redhat7
server.os_type=centos7
server.persistence.type=local
server.stages.parallel=true
server.task.timeout=1200
server.timeline.metrics.https.enabled=true
server.tmp.dir=/var/lib/ambari-server/data/tmp
server.version.file=/var/lib/ambari-server/resources/version
shared.resources.dir=/usr/lib/ambari-server/lib/ambari_commons/resources
skip.service.checks=false
stackadvisor.script=/var/lib/ambari-server/resources/scripts/stack_advisor.py
ulimit.open.files=10000
user.inactivity.timeout.default=0
user.inactivity.timeout.role.readonly.default=0
views.ambari.request.connect.timeout.millis=30000
views.ambari.request.read.timeout.millis=45000
views.http.strict-transport-security=max-age=31536000
views.http.x-frame-options=SAMEORIGIN
views.http.x-xss-protection=1; mode=block
views.request.connect.timeout.millis=5000
views.request.read.timeout.millis=10000
webapp.dir=/usr/lib/ambari-server/web

ambari-agent.ini file

[server]
hostname=ambari.server
url_port=8440
secured_url_port=8441

[agent]
logdir=/var/log/ambari-agent
piddir=/var/run/ambari-agent
prefix=/var/lib/ambari-agent/data
;loglevel=(DEBUG/INFO)
loglevel=INFO
data_cleanup_interval=86400
data_cleanup_max_age=2592000
data_cleanup_max_size_MB = 100
ping_port=8670
cache_dir=/var/lib/ambari-agent/cache
tolerate_download_failures=true
run_as_user=root
parallel_execution=0
alert_grace_period=5
alert_kinit_timeout=14400000
system_resource_overrides=/etc/resource_overrides
; memory_threshold_soft_mb=400
; memory_threshold_hard_mb=1000

[security]
keysdir=/var/lib/ambari-agent/keys
server_crt=ca.crt
passphrase_env_var_name=AMBARI_PASSPHRASE
ssl_verify_cert=0

[services]
pidLookupPath=/var/run/

[heartbeat]
state_interval_seconds=60
dirs=/etc/hadoop,/etc/hadoop/conf,/etc/hbase,/etc/hcatalog,/etc/hive,/etc/oozie,
  /etc/sqoop,/etc/ganglia,
  /var/run/hadoop,/var/run/zookeeper,/var/run/hbase,/var/run/templeton,/var/run/oozie,
  /var/log/hadoop,/var/log/zookeeper,/var/log/hbase,/var/run/templeton,/var/log/hive
; 0 - unlimited
log_lines_count=300
idle_interval_min=1
idle_interval_max=10

[logging]
syslog_enabled=0

/etc/hosts file :

[root@ambari ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.253.46 pocserver2 pocserver2
192.168.253.47 pocserver3 pocserver3
192.168.253.45 ambari.server

Java version:

[root@ambari ~]# java -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

Kou_Bou · ‎11-28-2019

Hello @Shelton I reinstall the environment in Centos 7.5 but I have the same problem what is the issue can be with python version ( when you said : I also see an error NetUtil.py:89 - SSLError: Failed to connect. That is due to python version)

Shelton · ‎11-28-2019

@Kou_Bou

I just deployed a cluster yesterday and never encountered that error

Can you share your documented steps in a document if it's too big to attach use google, I am sure you might be skipping a step but before that can you do the 2 steps below

vi /etc/ambari-agent/conf/ambari-agent.ini
force_https_protocol=PROTOCOL_TLSv1_2

vi /etc/python/cert-verification.cfg 
[https] 
verify=disable

To disable SSL/TLS then retry that should resolve your problem

Kou_Bou · ‎12-02-2019

Hello @Shelton Thank you for your reply, I installed Ambari-2.7.3.0 and the problem is solved

Shelton · ‎12-02-2019

@Kou_Bou

Great !

sagarshimpi · ‎11-20-2019

@Kou_Bou Can you try setting below -

Set below property in ambari agent config.ini
$vi /etc/ambari-agent/conf/ambari-agent.ini
[Note: please add below under Security section -as below]

[security]
force_https_protocol=PROTOCOL_TLSv1_2

Save and exit
Restart ambari-agent

Please check if that works.

Kou_Bou · ‎11-20-2019

Hello @sagarshimpi Thank you for your reply I try this solution but the problem persists.

sagarshimpi · ‎11-20-2019

Hi @Kou_Bou

Can you please check below steps version and paste output here -

Check java version

$java -version
$ls -ltr `which java`
$rpm -wa |grep openssl
Check - If multiple java version are existing then check in case if there is any conflict. You can verify this by using alternatives command -
$/usr/sbin/alternatives --config java [Please do not change here. Let the default java as it is]
Is the issue only with one host registering in cluster or the issue is with multiple hosts?
If you see java used here is openjdk then can you try with using oracle jdk and test
You can change java with below command -
ambari-server setup –j <jdk path>
Please do revert with Ambari server and Ambari agent version

$rpm -qa |grep ambari

From agent node try telnet to master at port 8440 [Check for iptables/selinux rules]

$telnet <ambari-server> 8440

Pass latest ambari agent config.ini file
Do revert with latest error/std log if the issue still exist

Cloudera Community

Support Questions

Ambari server: registration with the server failed