Member since
02-10-2015
84
Posts
2
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
13057 | 06-04-2015 06:09 PM | |
7174 | 05-22-2015 06:59 AM | |
5757 | 05-13-2015 03:19 PM | |
2311 | 05-11-2015 05:22 AM |
05-10-2015
07:29 PM
Now I am able to execute commands! Here they are: ls / [hive_zookeeper_namespace_hive0, solr, accumulo, hbase, zookeeper, zkdtsm] ls /hbase [meta-region-server, backup-masters, draining, table, region-in-transition, running, table-lock, hbaseid, namespace, online-snapshot, replication, splitWAL, recovering-regions, rs, flush-table-proc] Thank you.
... View more
05-10-2015
12:48 PM
There are 2 ZK servers: 1 Leader and 2 Followers. CM reports all good (green) about ZK. The only hickup reported by CM it was when I restarted ZK, it turned red for a less than 1 minute and the turned green and has stayed up green since then. Here is the error that reported at that moment: << The ZooKeeper service canary failed for an unknown reason. >> Here is some part of the ZK logs: <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> 2015-05-10 09:20:55,205 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /127.0.0.1:37531 2015-05-10 09:20:55,207 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3dfb495f0001 with negotiated timeout 30000 for client /127.0.0.1:37531 2015-05-10 09:21:02,370 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:37531 which had sessionid 0x14d3dfb495f0001 2015-05-10 09:33:13,657 INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig: Reading configuration from: /var/run/cloudera-scm-agent/process/1848-zookeeper-server/zoo.cfg 2015-05-10 09:33:13,668 INFO org.apache.zookeeper.server.quorum.QuorumPeerConfig: Defaulting to majority quorums 2015-05-10 09:33:13,671 INFO org.apache.zookeeper.server.DatadirCleanupManager: autopurge.snapRetainCount set to 5 2015-05-10 09:33:13,671 INFO org.apache.zookeeper.server.DatadirCleanupManager: autopurge.purgeInterval set to 24 2015-05-10 09:33:13,673 INFO org.apache.zookeeper.server.DatadirCleanupManager: Purge task started. 2015-05-10 09:33:13,682 INFO org.apache.zookeeper.server.quorum.QuorumPeerMain: Starting quorum peer 2015-05-10 09:33:13,683 INFO org.apache.zookeeper.server.DatadirCleanupManager: Purge task completed. 2015-05-10 09:33:13,689 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181 2015-05-10 09:33:13,696 INFO org.apache.zookeeper.server.quorum.QuorumPeer: tickTime set to 3000 2015-05-10 09:33:13,696 INFO org.apache.zookeeper.server.quorum.QuorumPeer: minSessionTimeout set to 4000 2015-05-10 09:33:13,696 INFO org.apache.zookeeper.server.quorum.QuorumPeer: maxSessionTimeout set to 60000 2015-05-10 09:33:13,696 INFO org.apache.zookeeper.server.quorum.QuorumPeer: initLimit set to 80 2015-05-10 09:33:13,703 INFO org.apache.zookeeper.server.persistence.FileSnap: Reading snapshot /shared/D12/zookeeper/version-2/snapshot.1400002ac1 2015-05-10 09:33:13,852 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: My election bind port: 0.0.0.0/0.0.0.0:3888 2015-05-10 09:33:13,861 INFO org.apache.zookeeper.server.quorum.QuorumPeer: LOOKING 2015-05-10 09:33:13,862 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: New election. My id = 1, proposed zxid=0x1500000012 2015-05-10 09:33:13,865 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 1 (n.leader), 0x1500000012 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x15 (n.peerEPoch), LOOKING (my state) 2015-05-10 09:33:13,866 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Have smaller server identifier, so dropping the connection: (2, 1) 2015-05-10 09:33:13,866 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Have smaller server identifier, so dropping the connection: (3, 1) 2015-05-10 09:33:13,867 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Received connection request /10.15.230.41:41581 2015-05-10 09:33:13,868 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Received connection request /10.15.230.42:41463 2015-05-10 09:33:13,869 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3 (n.leader), 0x1500000012 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x15 (n.peerEPoch), LOOKING (my state) 2015-05-10 09:33:13,869 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3 (n.leader), 0x1500000012 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x15 (n.peerEPoch), LOOKING (my state) 2015-05-10 09:33:13,870 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3 (n.leader), 0x1500000012 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x15 (n.peerEPoch), LOOKING (my state) 2015-05-10 09:33:14,070 INFO org.apache.zookeeper.server.quorum.QuorumPeer: FOLLOWING 2015-05-10 09:33:14,074 INFO org.apache.zookeeper.server.quorum.Learner: TCP NoDelay set to: true 2015-05-10 09:33:14,078 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:zookeeper.version=3.4.5-cdh5.4.0--1, built on 04/21/2015 19:11 GMT 2015-05-10 09:33:14,078 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:host.name=DAST-master.am.hedani.net 2015-05-10 09:33:14,078 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:java.version=1.7.0_55 2015-05-10 09:33:14,078 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:java.vendor=Oracle Corporation 2015-05-10 09:33:14,078 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:java.home=/usr/java/jdk1.7.0_55-cloudera/jre 2015-05-10 09:33:14,078 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:java.class.path=/var/run/cloudera-scm-agent/process/1848-zookeeper-server:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/lib/log4j.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/build/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/build/lib/*:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/zookeeper-3.4.5-cdh5.4.0.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/zookeeper.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/lib/jline-2.11.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/lib/netty-3.2.2.Final.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/lib/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/lib/log4j-1.2.16.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/lib/slf4j-log4j12.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar:/usr/share/cmf/lib/plugins/event-publish-5.4.0-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.4.0.jar:/usr/share/cmf/lib/plugins/cdh5/audit-plugin-cdh5-2.3.0-shaded.jar 2015-05-10 09:33:14,078 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2015-05-10 09:33:14,078 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:java.io.tmpdir=/tmp 2015-05-10 09:33:14,079 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:java.compiler=<NA> 2015-05-10 09:33:14,079 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:os.name=Linux 2015-05-10 09:33:14,079 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:os.arch=amd64 2015-05-10 09:33:14,079 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:os.version=2.6.32-431.el6.x86_64 2015-05-10 09:33:14,079 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:user.name=zookeeper 2015-05-10 09:33:14,079 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:user.home=/var/lib/zookeeper 2015-05-10 09:33:14,079 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:user.dir=/var/run/cloudera-scm-agent/process/1848-zookeeper-server 2015-05-10 09:33:14,080 INFO org.apache.zookeeper.server.ZooKeeperServer: Created server with tickTime 3000 minSessionTimeout 4000 maxSessionTimeout 60000 datadir /var/log/zookeeper/version-2 snapdir /shared/D12/zookeeper/version-2 2015-05-10 09:33:14,080 INFO org.apache.zookeeper.server.quorum.Learner: FOLLOWING - LEADER ELECTION TOOK - 218 2015-05-10 09:33:14,166 INFO org.apache.zookeeper.server.quorum.Learner: Getting a snapshot from leader 2015-05-10 09:33:14,173 INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 0x1500000012 to /shared/D12/zookeeper/version-2/snapshot.1500000012 2015-05-10 09:35:35,390 WARN org.apache.zookeeper.server.quorum.Learner: Got zxid 0x1600000001 expected 0x1 2015-05-10 09:35:35,391 INFO org.apache.zookeeper.server.persistence.FileTxnLog: Creating new log file: log.1600000001 2015-05-10 09:35:35,448 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:53415 2015-05-10 09:35:35,453 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:53415 2015-05-10 09:35:35,457 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3e07963c0000 with negotiated timeout 30000 for client /10.15.230.22:53415 2015-05-10 09:35:35,517 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:53415 which had sessionid 0x14d3e07963c0000 2015-05-10 09:35:54,650 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:38065 2015-05-10 09:35:54,652 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /127.0.0.1:38065 2015-05-10 09:35:54,654 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3e07963c0001 with negotiated timeout 30000 for client /127.0.0.1:38065 2015-05-10 09:36:22,336 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:38065 which had sessionid 0x14d3e07963c0001 2015-05-10 09:37:05,421 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:53514 2015-05-10 09:37:05,422 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:53514 2015-05-10 09:37:05,423 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3e07963c0002 with negotiated timeout 30000 for client /10.15.230.22:53514 2015-05-10 09:37:05,443 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:53514 which had sessionid 0x14d3e07963c0002 2015-05-10 09:38:35,342 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:53606 2015-05-10 09:38:35,342 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:53606 2015-05-10 09:38:35,345 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3e07963c0003 with negotiated timeout 30000 for client /10.15.230.22:53606 2015-05-10 09:38:35,354 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:53609 2015-05-10 09:38:35,354 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:53609 2015-05-10 09:38:35,356 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3e07963c0004 with negotiated timeout 30000 for client /10.15.230.22:53609 2015-05-10 09:38:35,428 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:53609 which had sessionid 0x14d3e07963c0004 2015-05-10 09:38:35,432 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:53606 which had sessionid 0x14d3e07963c0003 2015-05-10 09:39:35,367 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:53706 2015-05-10 09:39:35,368 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:53706 2015-05-10 09:39:35,369 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3e07963c0005 with negotiated timeout 30000 for client /10.15.230.22:53706 2015-05-10 09:39:35,392 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:53706 which had sessionid 0x14d3e07963c0005 2015-05-10 09:40:35,387 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:53813 2015-05-10 09:40:35,387 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:53813 2015-05-10 09:40:35,389 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3e07963c0006 with negotiated timeout 30000 for client /10.15.230.22:53813 2015-05-10 09:40:35,439 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:53813 which had sessionid 0x14d3e07963c0006 2015-05-10 09:41:40,361 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:53920 2015-05-10 09:41:40,361 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:53920 2015-05-10 09:41:40,362 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3e07963c0007 with negotiated timeout 30000 for client /10.15.230.22:53920 2015-05-10 09:41:40,423 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:53920 which had sessionid 0x14d3e07963c0007
... View more
05-10-2015
06:56 AM
I have restarted all services and agents but NO luck! Here is the latest HBASE-MASTER log: <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> 2015-05-10 09:43:44,744 INFO org.apache.hadoop.hbase.master.ServerManager: Finished waiting for region servers count to settle; checked in 5, slept for 4529 ms, expecting minimum of 1, maximum of 2147483647, master is running 2015-05-10 09:43:44,752 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder hdfs://master:8020/hbase/WALs/node1,60020,1431265418332 belongs to an existing region server 2015-05-10 09:43:44,754 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder hdfs://master:8020/hbase/WALs/node2,60020,1431265418290 belongs to an existing region server 2015-05-10 09:43:44,756 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder hdfs://master:8020/hbase/WALs/node3,60020,1431265418644 belongs to an existing region server 2015-05-10 09:43:44,758 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder hdfs://master:8020/hbase/WALs/node4,60020,1431265418152 belongs to an existing region server 2015-05-10 09:43:44,759 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder hdfs://master:8020/hbase/WALs/node5,60020,1431265418158 belongs to an existing region server 2015-05-10 09:43:44,834 INFO org.apache.hadoop.hbase.zookeeper.MetaTableLocator: Failed verification of hbase:meta,,1 at address=node3,60020,1431091829281, exception=org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on node3,60020,1431265418644 at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2791) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:886) at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1168) at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:20862) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2035) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:107) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745) 2015-05-10 09:43:44,839 INFO org.apache.hadoop.hbase.master.SplitLogManager: dead splitlog workers [node3,60020,1431091829281] 2015-05-10 09:43:44,841 INFO org.apache.hadoop.hbase.master.SplitLogManager: hdfs://master:8020/hbase/WALs/node3,60020,1431091829281-splitting is empty dir, no logs to split 2015-05-10 09:43:44,842 INFO org.apache.hadoop.hbase.master.SplitLogManager: started splitting 0 logs in [hdfs://master:8020/hbase/WALs/node3,60020,1431091829281-splitting] for [node3,60020,1431091829281] 2015-05-10 09:43:44,844 INFO org.apache.hadoop.hbase.master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [hdfs://master:8020/hbase/WALs/node3,60020,1431091829281-splitting] in 2ms 2015-05-10 09:43:44,844 INFO org.apache.hadoop.hbase.zookeeper.MetaTableLocator: Deleting hbase:meta region location in ZooKeeper 2015-05-10 09:43:44,865 INFO org.apache.hadoop.hbase.master.AssignmentManager: Assigning hbase:meta,,1.1588230740 to node2,60020,1431265418290 2015-05-10 09:43:44,865 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {1588230740 state=OFFLINE, ts=1431265424859, server=null} to {1588230740 state=PENDING_OPEN, ts=1431265424865, server=node2,60020,1431265418290} 2015-05-10 09:43:44,907 INFO org.apache.hadoop.hbase.master.ServerManager: AssignmentManager hasn't finished failover cleanup; waiting 2015-05-10 09:43:44,918 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {1588230740 state=PENDING_OPEN, ts=1431265424865, server=node2,60020,1431265418290} to {1588230740 state=OPENING, ts=1431265424918, server=node2,60020,1431265418290} 2015-05-10 09:43:45,126 INFO org.apache.hadoop.hbase.master.RegionStates: Transition {1588230740 state=OPENING, ts=1431265424918, server=node2,60020,1431265418290} to {1588230740 state=OPEN, ts=1431265425126, server=node2,60020,1431265418290} 2015-05-10 09:43:45,128 INFO org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Handling OPENED of 1588230740 from master,60000,1431265418462; deleting unassigned node 2015-05-10 09:43:45,132 INFO org.apache.hadoop.hbase.master.HMaster: hbase:meta assigned=1, rit=false, location=node2,60020,1431265418290 2015-05-10 09:43:45,200 INFO org.apache.hadoop.hbase.MetaMigrationConvertingToPB: hbase:meta doesn't have any entries to update. 2015-05-10 09:43:45,200 INFO org.apache.hadoop.hbase.MetaMigrationConvertingToPB: META already up-to date with PB serialization 2015-05-10 09:43:45,218 INFO org.apache.hadoop.hbase.master.AssignmentManager: Clean cluster startup. Assigning user regions 2015-05-10 09:43:45,223 INFO org.apache.hadoop.hbase.master.AssignmentManager: Joined the cluster in 23ms, failover=false 2015-05-10 09:43:45,233 INFO org.apache.hadoop.hbase.master.TableNamespaceManager: Namespace table not found. Creating... 2015-05-10 09:43:45,256 FATAL org.apache.hadoop.hbase.master.HMaster: Failed to become active master org.apache.hadoop.hbase.TableExistsException: hbase:namespace at org.apache.hadoop.hbase.master.handler.CreateTableHandler.checkAndSetEnablingTable(CreateTableHandler.java:152) at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:125) at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:233) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86) at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:897) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:739) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1469) at java.lang.Thread.run(Thread.java:745) 2015-05-10 09:43:45,262 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: [] 2015-05-10 09:43:45,262 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown. org.apache.hadoop.hbase.TableExistsException: hbase:namespace at org.apache.hadoop.hbase.master.handler.CreateTableHandler.checkAndSetEnablingTable(CreateTableHandler.java:152) at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:125) at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:233) at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86) at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:897) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:739) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:169) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1469) at java.lang.Thread.run(Thread.java:745) 2015-05-10 09:43:45,262 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown.
... View more
05-09-2015
06:52 PM
BTW, even when I execute the zKCli.sh script hangs, but now I am getting "connection refused" error: [root@master ~]# /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/zkCli.sh Connecting to localhost:2181 2015-05-09 21:42:32,601 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.5-cdh5.4.0--1, built o n 04/21/2015 19:11 GMT 2015-05-09 21:42:32,603 [myid:] - INFO [main:Environment@100] - Client environment:host.name=DAST-master.am.hedani.net 2015-05-09 21:42:32,603 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0 2015-05-09 21:42:32,604 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation 2015-05-09 21:42:32,604 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.7.0/jre 2015-05-09 21:42:32,604 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/cloudera/parcels/CDH-5. 4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../build/classes:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../bu ild/lib/*.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/opt/cloudera/parcels/ CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib /zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/netty-3.2.2. Final.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/cloudera/parcels/CDH-5 .4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/jline-2.11.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin /../zookeeper-3.4.5-cdh5.4.0.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../src/java/lib/*.jar:/opt/c loudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../conf: 2015-05-09 21:42:32,604 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd 64:/usr/lib64:/lib64:/lib:/usr/lib 2015-05-09 21:42:32,604 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp 2015-05-09 21:42:32,604 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA> 2015-05-09 21:42:32,605 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux 2015-05-09 21:42:32,605 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64 2015-05-09 21:42:32,605 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.el6.x86_64 2015-05-09 21:42:32,605 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root 2015-05-09 21:42:32,605 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root 2015-05-09 21:42:32,605 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/root 2015-05-09 21:42:32,606 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessio nTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@49b979c0 Welcome to ZooKeeper! JLine support is disabled 2015-05-09 21:42:32,622 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) 2015-05-09 21:42:32,627 [myid:] - WARN [main-SendThread(localhost:2181):ClientCnxn$SendThread@1102] - Session 0x0 for server nu ll, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:701) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) 2015-05-09 21:42:32,730 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2015-05-09 21:42:32,730 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@852] - Socket connection establis hed to localhost/127.0.0.1:2181, initiating session 2015-05-09 21:42:32,737 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1235] - Session establishment com plete on server localhost/127.0.0.1:2181, sessionid = 0x14d3b6fdf7d0012, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null
... View more
05-09-2015
06:49 PM
JM thank you! I have restarted ZK but the issue still persists! [root@DAST-master ~]# zookeeper-client Connecting to localhost:2181 2015-05-09 21:32:09,148 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.5-cdh5.4.0--1, built on 04/21/2015 2015-05-09 21:32:09,150 [myid:] - INFO [main:Environment@100] - Client environment:host.name=master 2015-05-09 21:32:09,151 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.7.0/jre 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.ib/zookeeper/bin/../build/classes:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../build/lib/*.jar:/opt/cloudera/parcels0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/slf4j1.7.5.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh27/lib/zookeeper/bin/../lib/netty-3.2.2.Final.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:dera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/jline-2.11.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeepezookeeper-3.4.5-cdh5.4.0.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/eper/conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/zookeeper/zookeeper-3.4.5-cdh5.4.0.jar:/opt/cloudera/parcels/CDH-5.4.0-1.p0.27/bin/../lib/zookeeper/zookeeper.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/zookeeper/lib/jline-2.11.j ls/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/zookeeper/lib/netty-3.2.2.Final.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bi /slf4j-api-1.7.5.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/zookeeper/lib/log4j-1.2.16.jar:/opt/cloudera/pa .4.0.p0.27/bin/../lib/zookeeper/lib/slf4j-log4j12.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/zookeeper/lib/ r 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd lib:/usr/lib 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA> 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64 2015-05-09 21:32:09,153 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.el6.x86_64 2015-05-09 21:32:09,153 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root 2015-05-09 21:32:09,153 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root 2015-05-09 21:32:09,153 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/root 2015-05-09 21:32:09,154 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessio =org.apache.zookeeper.ZooKeeperMain$MyWatcher@2fe85210 Welcome to ZooKeeper! JLine support is disabled 2015-05-09 21:32:09,171 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@975] - Opening socket connection 7.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2015-05-09 21:32:09,175 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@852] - Socket connection establis .0.1:2181, initiating session 2015-05-09 21:32:09,181 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1235] - Session establishment com ost/127.0.0.1:2181, sessionid = 0x14d3b6fdf7d0005, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null Here is the ZK Log: <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> 2015-05-09 21:32:09,175 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:35338 2015-05-09 21:32:09,178 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /127.0.0.1:35338 2015-05-09 21:32:09,179 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3b6fdf7d0005 with negotiated timeout 30000 for client /127.0.0.1:35338 2015-05-09 21:32:17,657 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:50721 2015-05-09 21:32:17,657 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:50721 2015-05-09 21:32:17,658 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3b6fdf7d0006 with negotiated timeout 30000 for client /10.15.230.22:50721 2015-05-09 21:32:17,674 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:50721 which had sessionid 0x14d3b6fdf7d0006 2015-05-09 21:33:22,661 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:50815 2015-05-09 21:33:22,661 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:50815 2015-05-09 21:33:22,662 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3b6fdf7d0007 with negotiated timeout 30000 for client /10.15.230.22:50815 2015-05-09 21:33:22,680 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:50815 which had sessionid 0x14d3b6fdf7d0007 2015-05-09 21:34:22,663 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:50899 2015-05-09 21:34:22,663 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:50899 2015-05-09 21:34:22,664 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3b6fdf7d0008 with negotiated timeout 30000 for client /10.15.230.22:50899 2015-05-09 21:34:22,720 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:50899 which had sessionid 0x14d3b6fdf7d0008 2015-05-09 21:35:22,666 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:50978 2015-05-09 21:35:22,666 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:50978 2015-05-09 21:35:22,669 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3b6fdf7d0009 with negotiated timeout 30000 for client /10.15.230.22:50978 2015-05-09 21:35:22,685 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:50978 which had sessionid 0x14d3b6fdf7d0009 2015-05-09 21:36:22,667 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:51056 2015-05-09 21:36:22,668 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:51056 2015-05-09 21:36:22,669 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3b6fdf7d000a with negotiated timeout 30000 for client /10.15.230.22:51056 2015-05-09 21:36:22,685 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:51056 which had sessionid 0x14d3b6fdf7d000a
... View more
05-09-2015
06:37 PM
Hi JM, thank you! I have restarted ZK, but still I am unable to get the ZK Client working! [root@master ~]# zookeeper-client Connecting to localhost:2181 2015-05-09 21:32:09,148 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.5-cdh5.4.0--1, built on 04/21/2015 19:11 GMT 2015-05-09 21:32:09,150 [myid:] - INFO [main:Environment@100] - Client environment:host.name=master 2015-05-09 21:32:09,151 [myid:] - INFO [main:Environment@100] - Client environment:java.version=1.7.0 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:java.vendor=Oracle Corporation 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.7.0/jre 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:java.class.path=/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../build/classes:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../build/lib/*.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/netty-3.2.2.Final.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../lib/jline-2.11.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.4.0.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/zookeeper/zookeeper-3.4.5-cdh5.4.0.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/zookeeper/zookeeper.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/zookeeper/lib/jline-2.11.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/zookeeper/lib/netty-3.2.2.Final.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/zookeeper/lib/slf4j-api-1.7.5.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/zookeeper/lib/log4j-1.2.16.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/zookeeper/lib/slf4j-log4j12.jar:/opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/../lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:java.io.tmpdir=/tmp 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:java.compiler=<NA> 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:os.name=Linux 2015-05-09 21:32:09,152 [myid:] - INFO [main:Environment@100] - Client environment:os.arch=amd64 2015-05-09 21:32:09,153 [myid:] - INFO [main:Environment@100] - Client environment:os.version=2.6.32-431.el6.x86_64 2015-05-09 21:32:09,153 [myid:] - INFO [main:Environment@100] - Client environment:user.name=root 2015-05-09 21:32:09,153 [myid:] - INFO [main:Environment@100] - Client environment:user.home=/root 2015-05-09 21:32:09,153 [myid:] - INFO [main:Environment@100] - Client environment:user.dir=/root 2015-05-09 21:32:09,154 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@2fe85210 Welcome to ZooKeeper! JLine support is disabled 2015-05-09 21:32:09,171 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2015-05-09 21:32:09,175 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@852] - Socket connection established to localhost/127.0.0.1:2181, initiating session 2015-05-09 21:32:09,181 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x14d3b6fdf7d0005, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null Here is the ZK Log: <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> 2015-05-09 21:32:09,175 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:35338 2015-05-09 21:32:09,178 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /127.0.0.1:35338 2015-05-09 21:32:09,179 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3b6fdf7d0005 with negotiated timeout 30000 for client /127.0.0.1:35338 2015-05-09 21:32:17,657 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:50721 2015-05-09 21:32:17,657 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:50721 2015-05-09 21:32:17,658 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3b6fdf7d0006 with negotiated timeout 30000 for client /10.15.230.22:50721 2015-05-09 21:32:17,674 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:50721 which had sessionid 0x14d3b6fdf7d0006 2015-05-09 21:33:22,661 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:50815 2015-05-09 21:33:22,661 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:50815 2015-05-09 21:33:22,662 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3b6fdf7d0007 with negotiated timeout 30000 for client /10.15.230.22:50815 2015-05-09 21:33:22,680 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:50815 which had sessionid 0x14d3b6fdf7d0007 2015-05-09 21:34:22,663 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:50899 2015-05-09 21:34:22,663 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:50899 2015-05-09 21:34:22,664 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3b6fdf7d0008 with negotiated timeout 30000 for client /10.15.230.22:50899 2015-05-09 21:34:22,720 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:50899 which had sessionid 0x14d3b6fdf7d0008 2015-05-09 21:35:22,666 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:50978 2015-05-09 21:35:22,666 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:50978 2015-05-09 21:35:22,669 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3b6fdf7d0009 with negotiated timeout 30000 for client /10.15.230.22:50978 2015-05-09 21:35:22,685 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:50978 which had sessionid 0x14d3b6fdf7d0009 2015-05-09 21:36:22,667 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /10.15.230.22:51056 2015-05-09 21:36:22,668 INFO org.apache.zookeeper.server.ZooKeeperServer: Client attempting to establish new session at /10.15.230.22:51056 2015-05-09 21:36:22,669 INFO org.apache.zookeeper.server.ZooKeeperServer: Established session 0x14d3b6fdf7d000a with negotiated timeout 30000 for client /10.15.230.22:51056 2015-05-09 21:36:22,685 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /10.15.230.22:51056 which had sessionid 0x14d3b6fdf7d000a
... View more
05-09-2015
10:01 AM
I have read this similar thread but it is not clear how it got resolved! https://community.cloudera.com/t5/Cloudera-Manager-Installation/Server-Error-The-feature-Operational-Reports-is-not-available-CM/td-p/5111
... View more
05-09-2015
09:58 AM
Why trying to create CMS service via CM API, getting the error below. However, CMS service is created ####################################################################################### scm=> select * from services; service_id | optimistic_lock_version | name | service_type | cluster_id | maintenance_count | display_name | generation ------------+-------------------------+------+--------------+------------+-------------------+----------------------------- +------------ 1 | 1 | mgmt | MGMT | | 0 | Cloudera Management Service | 1 (1 row) ####################################################################################### but I can NOT start CMS service!!! It is not clear to me what roles are allowed under what type of licensing (enterprise vs community). Here are the roles that have been created: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ scm=> select * from roles; role_id | name | host_id | role_type | configured_status | service_id | merged_keytab | maintenance_cou nt | decommission_count | optimistic_lock_version | role_config_group_id | has_ever_started ---------+-------------------+---------+-----------------+-------------------+------------+---------------+---------------- ---+--------------------+-------------------------+----------------------+------------------ 1 | ACTIVITYMONITOR-1 | 1 | ACTIVITYMONITOR | STOPPED | 1 | | 0 | 0 | 0 | 4 | f 2 | ALERTPUBLISHER-1 | 1 | ALERTPUBLISHER | STOPPED | 1 | | 0 | 0 | 0 | 2 | f 3 | EVENTSERVER-1 | 1 | EVENTSERVER | STOPPED | 1 | | 0 | 0 | 0 | 3 | f 4 | HOSTMONITOR-1 | 1 | HOSTMONITOR | STOPPED | 1 | | 0 | 0 | 0 | 7 | f 5 | SERVICEMONITOR-1 | 1 | SERVICEMONITOR | STOPPED | 1 | | 0 | 0 | 0 | 6 | f (5 rows) @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ but again I cannot start them!!!! I have run the same exact CM API script in CM 5.3 and it works without any issues!!!!! Here is the error getting by executing a CM API python script: <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><> Traceback (most recent call last): File "./deploy_cms_service.py", line 238, in <module> main() File "./deploy_cms_service.py", line 222, in main RMAN_ROLE_CONFIG) File "./deploy_cms_service.py", line 151, in deploy_management mgmt.create_role(rman_role_name + "-1", "REPORTSMANAGER", CM_HOST) File "/usr/lib/python2.6/site-packages/cm_api-6.0.2-py2.6.egg/cm_api/endpoints/services.py", line 403, in create_role role_name, host_id, self._get_cluster_name()) File "/usr/lib/python2.6/site-packages/cm_api-6.0.2-py2.6.egg/cm_api/endpoints/roles.py", line 53, in create_role ApiRole, True, data=[apirole])[0] File "/usr/lib/python2.6/site-packages/cm_api-6.0.2-py2.6.egg/cm_api/endpoints/types.py", line 137, in call ret = method(path, data=data, params=params) File "/usr/lib/python2.6/site-packages/cm_api-6.0.2-py2.6.egg/cm_api/resource.py", line 138, in post self._make_headers(contenttype)) File "/usr/lib/python2.6/site-packages/cm_api-6.0.2-py2.6.egg/cm_api/resource.py", line 63, in invoke headers=headers) File "/usr/lib/python2.6/site-packages/cm_api-6.0.2-py2.6.egg/cm_api/http_client.py", line 161, in execute raise self._exc_class(ex) cm_api.api_client.ApiException: The feature Operational Reports is not available. (error 500)
... View more
Labels:
- Labels:
-
Cloudera Manager
05-09-2015
09:13 AM
Anthony, thank you for the clarification! Having said that (JHServer is specific to Map Reduce jobs run on YARN) where other type of jobs will be shown? (You said tha Spark has its own JHS...) BTW: Besides M/R and Spark jobs what other types of jobs can we launch via YARN? How jobs are moved from /tmp/logs/<user-id>/logs fodler to /user/history/done & /user/history/done_intermediate ones? Or they are created simultaneously? Thank you for your assistance, it is much appreciated!
... View more