Cloudera Community
prakashpunj
user rank
Expert Contributor
My Accepted Solutions
Show All

Re: Heartbeat lost on AMBARI-SERVER 2.5

by Expert Contributor in Support Questions
‎09-30-2017 10:56 PM
‎09-30-2017 10:56 PM
It's working on. I had to change the hostname from ambari-agent properties file. I restarted ambari-agent and it was OK after that. Thanks.. ... View more

Heartbeat lost on AMBARI-SERVER 2.5

by Expert Contributor in Support Questions
‎09-30-2017 10:07 PM
‎09-30-2017 10:07 PM
I just upgrade Ambari from 2.2.0.0 to 2.5 and I noticed that HearBeat on Amari-server is lost. I also tried running ambari-agent on Ambari-server but still it's not coming back. ... View more
Labels:

Re: AMbari heart beat lost

by Expert Contributor in Support Questions
‎05-26-2017 02:31 PM
‎05-26-2017 02:31 PM
This works. So its gonna happen once every year. Is there a solution for that. ... View more

Re: AMbari heart beat lost

by Expert Contributor in Support Questions
‎05-25-2017 12:54 PM
‎05-25-2017 12:54 PM
Looks like SSL cert is the issue on ambari-server. It expired. How can I renew it ... View more

Re: AMbari heart beat lost

by Expert Contributor in Support Questions
‎05-25-2017 12:53 PM
‎05-25-2017 12:53 PM
--- SSL handshake has read 2303 bytes and written 206 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 4096 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 5926D3A2802AC3DD04F3CD1BA946AFAA8A19EACE5EE04A59AB752ACD63AC55A8 Session-ID-ctx: Master-Key: D205A7AD2A675D7E61B56E0A1A28AC76E5BCCE249CB7A50F4461F5C3EF12D3C9106EAB0B68146BDC5F97849CADDCAFF9 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1495716767 Timeout : 300 (sec) Verify return code: 10 (certificate has expired) ... View more

Re: AMbari heart beat lost

by Expert Contributor in Support Questions
‎05-25-2017 12:49 PM
‎05-25-2017 12:49 PM
Stopping ambari-agent Removing PID file at /var/run/ambari-agent/ambari-agent.pid ambari-agent successfully stopped [root@Namenode ~]# ambari-agent start Verifying Python version compatibility... Using python /usr/bin/python Checking for previously running Ambari Agent... Starting ambari-agent Verifying ambari-agent process status... Ambari Agent successfully started Agent PID at: /var/run/ambari-agent/ambari-agent.pid Agent out at: /var/log/ambari-agent/ambari-agent.out Agent log at: /var/log/ambari-agent/ambari-agent.log [root@Namenode ~]# vi /var/log/ambari-agent/ambari-agent.log INFO 2017-05-25 07:22:07,809 NetUtil.py:60 - Connecting to https://ambari.asotc.com:8440/connection_info INFO 2017-05-25 07:22:07,976 security.py:54 - Server require two-way SSL authentication. Use it instead of one-way... INFO 2017-05-25 07:22:07,976 security.py:188 - Server certicate exists, ok INFO 2017-05-25 07:22:07,977 security.py:196 - Agent key exists, ok INFO 2017-05-25 07:22:07,977 security.py:204 - Agent certificate exists, ok INFO 2017-05-25 07:22:07,977 security.py:99 - SSL Connect being called.. connecting to the server ERROR 2017-05-25 07:22:08,111 security.py:86 - Two-way SSL authentication failed. Ensure that server and agent certificates were signed by the same CA and restart the agent. In order to receive a new agent certificate, remove existing certificate file from keys directory. As a workaround you can turn off two-way SSL authentication in server configuration(ambari.properties) Exiting.. ERROR 2017-05-25 07:22:08,112 Controller.py:350 - Unable to reconnect to https://ambari.asotc.com:8441/agent/v1/heartbeat/namenode.asotc.com (attempts=1699, details=Request to https://ambari.asotc.com:8441/agent/v1/heartbeat/namenode.asotc.com failed due to [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765)) INFO 2017-05-25 07:22:23,014 NetUtil.py:60 - Connecting to https://ambari.asotc.com:8440/connection_info Looks like its connecting to SSL. I have not enabled SSL ... View more

AMbari heart beat lost

by Expert Contributor in Support Questions
‎05-25-2017 11:14 AM
‎05-25-2017 11:14 AM
Not sure what happened but Ambari serveer is showing heart beat lost for all the hosts. I tried restarting the ambari server and agent but no help. ... View more
Labels:

Re: Datanode goes dows after few secs of starting

by Expert Contributor in Support Questions
‎12-20-2016 11:44 PM
1 Kudo
‎12-20-2016 11:44 PM
1 Kudo
Thank you guys specifically @Ward Bekker. After I formatted the namenode, clusterID got mismatch with DataNode and that also preventing other services to start.. ... View more

Re: Datanode goes dows after few secs of starting

by Expert Contributor in Support Questions
‎12-20-2016 06:22 PM
‎12-20-2016 06:22 PM
@Timothy Spann Yes I formatted the namenode as namenode was having issue starting at the beginning. ... View more

Re: Datanode goes dows after few secs of starting

by Expert Contributor in Support Questions
‎12-20-2016 03:27 PM
‎12-20-2016 03:27 PM
Below is what the log says ng DataNode with maxLockedMemory = 0 2016-12-20 09:41:03,533 INFO datanode.DataNode (DataNode.java:initDataXceiver(921)) - Opene d streaming server at /0.0.0.0:50010 2016-12-20 09:41:03,537 INFO datanode.DataNode (DataXceiverServer.java:<init>(76)) - Balanc ing bandwith is 6250000 bytes/s 2016-12-20 09:41:03,537 INFO datanode.DataNode (DataXceiverServer.java:<init>(77)) - Number threads for balancing is 5 2016-12-20 09:41:03,542 INFO datanode.DataNode (DataXceiverServer.java:<init>(76)) - Balanc ing bandwith is 6250000 bytes/s 2016-12-20 09:41:03,542 INFO datanode.DataNode (DataXceiverServer.java:<init>(77)) - Number threads for balancing is 5 2016-12-20 09:41:03,542 INFO datanode.DataNode (DataNode.java:initDataXceiver(936)) - Liste ning on UNIX domain socket: /var/lib/hadoop-hdfs/dn_socket 2016-12-20 09:41:03,740 INFO mortbay.log (Slf4jLog.java:info(67)) - Logging to org.slf4j.im pl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2016-12-20 09:41:03,780 INFO server.AuthenticationFilter (AuthenticationFilter.java:constru ctSecretProvider(294)) - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2016-12-20 09:41:03,791 INFO http.HttpRequestLog (HttpRequestLog.java:getRequestLog(80)) - Http request log for http.requests.datanode is not defined 2016-12-20 09:41:03,799 INFO http.HttpServer2 (HttpServer2.java:addGlobalFilter(710)) - Add ed global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2016-12-20 09:41:03,801 INFO http.HttpServer2 (HttpServer2.java:addFilter(685)) - Added fil ter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilte r) to context datanode 2016-12-20 09:41:03,802 INFO http.HttpServer2 (HttpServer2.java:addFilter(693)) - Added fil ter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilte r) to context static 2016-12-20 09:41:03,802 INFO http.HttpServer2 (HttpServer2.java:addFilter(693)) - Added fil ter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilte r) to context logs 2016-12-20 09:41:03,821 INFO http.HttpServer2 (HttpServer2.java:openListeners(915)) - Jetty bound to port 42822 2016-12-20 09:41:03,822 INFO mortbay.log (Slf4jLog.java:info(67)) - jetty-6.1.26.hwx 2016-12-20 09:41:04,146 INFO mortbay.log (Slf4jLog.java:info(67)) - Started HttpServer2$Sel ectChannelConnectorWithSafeStartup@localhost:42822 2016-12-20 09:41:04,425 INFO web.DatanodeHttpServer (DatanodeHttpServer.java:start(201)) - Listening HTTP traffic on /0.0.0.0:50075 2016-12-20 09:41:04,685 INFO datanode.DataNode (DataNode.java:startDataNode(1144)) - dnUser Name = hdfs 2016-12-20 09:41:04,685 INFO datanode.DataNode (DataNode.java:startDataNode(1145)) - superg roup = hdfs 2016-12-20 09:41:04,770 INFO ipc.CallQueueManager (CallQueueManager.java:<init>(56)) - Usin g callQueue class java.util.concurrent.LinkedBlockingQueue 2016-12-20 09:41:04,804 INFO ipc.Server (Server.java:run(676)) - Starting Socket Reader #1 for port 8010 2016-12-20 09:41:04,887 INFO datanode.DataNode (DataNode.java:initIpcServer(837)) - Opened IPC server at /0.0.0.0:8010 2016-12-20 09:41:04,903 INFO datanode.DataNode (BlockPoolManager.java:refreshNamenodes(152) ) - Refresh request received for nameservices: null 2016-12-20 09:41:04,940 INFO datanode.DataNode (BlockPoolManager.java:doRefreshNamenodes(19 7)) - Starting BPOfferServices for nameservices: <default> 2016-12-20 09:41:04,964 INFO datanode.DataNode (BPServiceActor.java:run(814)) - Block pool <registering> (Datanode Uuid unassigned) service to hdp-m.asotc/10.0.2.23:8020 starting to o ffer service 2016-12-20 09:41:04,989 INFO ipc.Server (Server.java:run(906)) - IPC Server Responder: star ting 2016-12-20 09:41:04,989 INFO ipc.Server (Server.java:run(746)) - IPC Server listener on 801 0: starting 2016-12-20 09:41:05,309 INFO common.Storage (Storage.java:tryLock(715)) - Lock on /hadoop/h dfs/data/in_use.lock acquired by nodename 20341@hdp-m.asotc 2016-12-20 09:41:05,312 WARN common.Storage (DataStorage.java:addStorageLocations(375)) - j ava.io.IOException: Incompatible clusterIDs in /hadoop/hdfs/data: namenode clusterID = CID-3 5394708-aa35-4f25-b43b-0072da288d03; datanode clusterID = CID-d723cf5b-ba4a-43d3-afe1-781149 930f3e 2016-12-20 09:41:05,313 FATAL datanode.DataNode (BPServiceActor.java:run(833)) - Initializat ion failed for Block pool <registering> (Datanode Uuid d0d90f34-c2a9-4e0e-ba5e-237b5820f879) service to hdp-m.asotc/10.0.2.23:8020. Exiting. java.io.IOException: All specified directories are failed to load. at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStor age.java:477) at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1399) at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1364) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(B POfferService.java:317) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPS erviceActor.java:224) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:821 ) at java.lang.Thread.run(Thread.java:745) 2016-12-20 09:41:05,315 WARN datanode.DataNode (BPServiceActor.java:run(854)) - Ending bloc k pool service for: Block pool <registering> (Datanode Uuid d0d90f34-c2a9-4e0e-ba5e-237b5820 f879) service to hdp-m.asotc/10.0.2.23:8020 2016-12-20 09:41:05,420 INFO datanode.DataNode (BlockPoolManager.java:remove(103)) - Remove d Block pool <registering> (Datanode Uuid d0d90f34-c2a9-4e0e-ba5e-237b5820f879) 2016-12-20 09:41:07,421 WARN datanode.DataNode (DataNode.java:secureMain(2540)) - Exiting D atanode 2016-12-20 09:41:07,427 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with st atus 0 2016-12-20 09:41:07,430 INFO datanode.DataNode (LogAdapter.java:info(45)) - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down DataNode at hdp-m.asotc/10.0.2.23 ************************************************************/ ... View more
Contact Me
Online
Offline
Last Visited
‎03-20-2020 09:39 PM
Community Statistics
Member Since ‎12-11-2015 04:02 AM
Last Visited ‎03-20-2020 09:39 PM
Posts 213
Kudos received 87