Support Questions
Find answers, ask questions, and share your expertise

Error when migrating an Apache HBase Table Between Different Clusters.

Explorer

Hi All, I am using CopyTable utility to copy table from one cluster to other. Info 2 cluster following: - Cluster A: HDP-2.2.4.2-2, Hbase 0.98.4.2.2, zk 3.4.6.2.2 - Cluster B: HDP-2.6.0.3, Hbase 1.1.2.2.6,ZK 3.4.6 - Allowed policy firewall between 2 cluster. - Configured file /etc/hosts on together. CopyTable cmd: [hbase@hbaseA hbase]$ ./bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable --peer.adr=mt02.rpt.com,mt03.rpt.com,mt04.rpt.com:2181:/hbase-unsecure tables01 It is not sucessful. So, Everyone trace log and resolve this problem help me! Logs are as given below:

2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=:/usr/hdp/2.2.4.2-2/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.2.4.2-2/hadoop/lib/native
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.el6.x86_64
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=hbase
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/home/hbase
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/usr/hdp/2.2.4.2-2/hbase
2017-05-29 15:39:12,985 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181 sessionTimeout=30000 watcher=hconnection-0x1a80d720, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, baseZNode=/hbase-unsecure
2017-05-29 15:39:12,999 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x1a80d720 connecting to ZooKeeper ensemble=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181
2017-05-29 15:39:13,003 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt03.rpt.com/192.168.100.172:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:39:13,012 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt03.rpt.com/192.168.100.172:2181, initiating session
2017-05-29 15:39:23,024 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10013ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:39:23,141 WARN  [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-05-29 15:39:23,980 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt04.rpt.com/192.168.100.173:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:39:23,982 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt04.rpt.com/192.168.100.173:2181, initiating session
2017-05-29 15:39:33,992 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10009ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:39:34,092 WARN  [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-05-29 15:39:34,944 INFO  [main-SendThread(mt02.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt02.rpt.com/192.168.100.171:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:39:34,947 INFO  [main-SendThread(mt02.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt02.rpt.com/192.168.100.171:2181, initiating session
2017-05-29 15:39:44,956 INFO  [main-SendThread(mt02.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10009ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:39:45,057 WARN  [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-05-29 15:39:46,992 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt03.rpt.com/192.168.100.172:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:39:46,994 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt03.rpt.com/192.168.100.172:2181, initiating session
2017-05-29 15:39:57,002 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10008ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:39:57,102 WARN  [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-05-29 15:39:57,568 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt04.rpt.com/192.168.100.173:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:39:57,569 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt04.rpt.com/192.168.100.173:2181, initiating session
2017-05-29 15:40:07,571 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10002ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:40:07,672 WARN  [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-05-29 15:40:07,672 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 4 attempts
2017-05-29 15:40:07,674 WARN  [main] zookeeper.ZKUtil: hconnection-0x1a80d720, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, baseZNode=/hbase-unsecure Unable to set watcher on znode (/hbase-unsecure/hbaseid)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:199)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:106)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:890)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at org.apache.hadoop.hbase.mapreduce.CopyTable.run(CopyTable.java:275)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.hbase.mapreduce.CopyTable.main(CopyTable.java:266)
2017-05-29 15:40:07,678 ERROR [main] zookeeper.ZooKeeperWatcher: hconnection-0x1a80d720, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, baseZNode=/hbase-unsecure Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:199)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:106)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at org.apache.hadoop.hbase.mapreduce.CopyTable.run(CopyTable.java:275)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.hbase.mapreduce.CopyTable.main(CopyTable.java:266)
2017-05-29 15:40:07,679 WARN  [main] client.ZooKeeperRegistry: Can't retrieve clusterId from Zookeeper
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:199)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:106)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:890)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:667)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:426)
    at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:405)
    at org.apache.hadoop.hbase.client.ConnectionManager.getConnectionInternal(ConnectionManager.java:283)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:207)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:169)
    at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:206)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:559)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at org.apache.hadoop.hbase.mapreduce.CopyTable.run(CopyTable.java:275)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.hbase.mapreduce.CopyTable.main(CopyTable.java:266)
2017-05-29 15:40:07,680 DEBUG [main] client.ConnectionManager$HConnectionImplementation: clusterid came back null, using default default-cluster
2017-05-29 15:40:07,699 DEBUG [main] ipc.RpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@236a277, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null
2017-05-29 15:40:07,708 INFO  [main] mapreduce.TableOutputFormat: Created table instance for orm_backlink_v1
2017-05-29 15:40:07,922 INFO  [main-SendThread(mt02.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt02.rpt.com/192.168.100.171:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:40:07,924 INFO  [main-SendThread(mt02.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt02.rpt.com/192.168.100.171:2181, initiating session
2017-05-29 15:40:08,343 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=zk2.nfw.com:2181,zk1.nfw.com:2181,zk3.nfw.com:2181 sessionTimeout=30000 watcher=hconnection-0x6bac124, quorum=zk2.nfw.com:2181,zk1.nfw.com:2181,zk3.nfw.com:2181, baseZNode=/hbase-unsecure
2017-05-29 15:40:08,345 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6bac124 connecting to ZooKeeper ensemble=zk2.nfw.com:2181,zk1.nfw.com:2181,zk3.nfw.com:2181
2017-05-29 15:40:08,345 INFO  [main-SendThread(zk2.nfw.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server zk2.nfw.com/10.30.154.53:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:40:08,346 INFO  [main-SendThread(zk2.nfw.com:2181)] zookeeper.ClientCnxn: Socket connection established to zk2.nfw.com/10.30.154.53:2181, initiating session
2017-05-29 15:40:08,352 INFO  [main-SendThread(zk2.nfw.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server zk2.nfw.com/10.30.154.53:2181, sessionid = 0x25331211b0c555b, negotiated timeout = 30000
2017-05-29 15:40:08,367 DEBUG [main] ipc.RpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20a19e0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null
2017-05-29 15:40:08,401 INFO  [main] util.RegionSizeCalculator: Calculating region sizes for table "orm_backlink_v1".
2017-05-29 15:40:08,410 DEBUG [main] hbase.HRegionInfo: 1588230740
2017-05-29 15:40:08,733 DEBUG [htable-pool3-t1] ipc.RpcClient: Use SIMPLE authentication for service ClientService, sasl=false
2017-05-29 15:40:08,761 DEBUG [htable-pool3-t1] ipc.RpcClient: Connecting to dn11.nfw.com/10.30.152.11:60020
2017-05-29 15:40:08,863 DEBUG [main] ipc.RpcClient: Use SIMPLE authentication for service MasterService, sasl=false
2017-05-29 15:40:08,864 DEBUG [main] ipc.RpcClient: Connecting to hm2.nfw.com/10.30.154.55:60000
2017-05-29 15:40:09,022 DEBUG [main] util.RegionSizeCalculator: Region orm_backlink_v1,,1438962439939.f59b6aaf686701422f99238880208c2e. has size 150994944
2017-05-29 15:40:09,035 DEBUG [main] util.RegionSizeCalculator: Region orm_backlink_v1,http://shellbrowser_components_delphi_edition.th.downloadastro.com|http://kb,1439701594892.7f3318d1679f4533b9b451837dc94f9d. has size 171966464
2017-05-29 15:40:09,039 DEBUG [main] util.RegionSizeCalculator: Region orm_backlink_v1,http://greatsih,1439701594892.d5f3fff639e9765cfa49e12a18e5d66a. has size 171966464
2017-05-29 15:40:09,043 DEBUG [main] util.RegionSizeCalculator: Region sizes calculated
2017-05-29 15:40:09,123 WARN  [main] mapreduce.TableInputFormatBase: Cannot resolve the host name for dn11.nfw.com/10.30.152.11 because of javax.naming.CommunicationException: DNS error [Root exception is java.net.PortUnreachableException: ICMP Port Unreachable]; remaining name '11.152.30.10.in-addr.arpa'
2017-05-29 15:40:09,126 DEBUG [main] mapreduce.TableInputFormatBase: getSplits: split -> 0 -> dn11.nfw.com:,http://greatsih
2017-05-29 15:40:09,127 WARN  [main] mapreduce.TableInputFormatBase: Cannot resolve the host name for dn27.nfw.com/10.30.153.13 because of javax.naming.CommunicationException: DNS error [Root exception is java.net.PortUnreachableException: ICMP Port Unreachable]; remaining name '13.153.30.10.in-addr.arpa'
2017-05-29 15:40:09,128 DEBUG [main] mapreduce.TableInputFormatBase: getSplits: split -> 1 -> dn27.nfw.com:http://greatsih,http://shellbrowser_components_delphi_edition.th.downloadastro.com|http://kb
2017-05-29 15:40:09,129 WARN  [main] mapreduce.TableInputFormatBase: Cannot resolve the host name for dn24.nfw.com/10.30.153.10 because of javax.naming.CommunicationException: DNS error [Root exception is java.net.PortUnreachableException: ICMP Port Unreachable]; remaining name '10.153.30.10.in-addr.arpa'
2017-05-29 15:40:09,129 DEBUG [main] mapreduce.TableInputFormatBase: getSplits: split -> 2 -> dn24.nfw.com:http://shellbrowser_components_delphi_edition.th.downloadastro.com|http://kb,
2017-05-29 15:40:09,151 INFO  [main] mapreduce.JobSubmitter: number of splits:3
2017-05-29 15:40:09,167 INFO  [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2017-05-29 15:40:09,446 INFO  [main] mapreduce.JobSubmitter: Submitting tokens for job: job_local2084446748_0001
2017-05-29 15:40:10,098 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/hbase-server-0.98.4.2.2.4.2-2-hadoop2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209611/hbase-server-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,247 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/zookeeper/zookeeper-3.4.6.2.2.4.2-2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209612/zookeeper-3.4.6.2.2.4.2-2.jar
2017-05-29 15:40:10,248 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/high-scale-lib-1.1.1.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209613/high-scale-lib-1.1.1.jar
2017-05-29 15:40:10,248 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hadoop/hadoop-common-2.6.0.2.2.4.2-2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209614/hadoop-common-2.6.0.2.2.4.2-2.jar
2017-05-29 15:40:10,248 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/guava-12.0.1.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209615/guava-12.0.1.jar
2017-05-29 15:40:10,248 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/hbase-protocol-0.98.4.2.2.4.2-2-hadoop2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209616/hbase-protocol-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,249 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/htrace-core-2.04.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209617/htrace-core-2.04.jar
2017-05-29 15:40:10,249 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hadoop-mapreduce/hadoop-mapreduce-client-core-2.6.0.2.2.4.2-2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209618/hadoop-mapreduce-client-core-2.6.0.2.2.4.2-2.jar
2017-05-29 15:40:10,249 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/netty-3.6.6.Final.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209619/netty-3.6.6.Final.jar
2017-05-29 15:40:10,328 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/hbase-client-0.98.4.2.2.4.2-2-hadoop2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209620/hbase-client-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,328 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/hbase-hadoop-compat-0.98.4.2.2.4.2-2-hadoop2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209621/hbase-hadoop-compat-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,329 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/protobuf-java-2.5.0.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209622/protobuf-java-2.5.0.jar
2017-05-29 15:40:10,329 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/hbase-common-0.98.4.2.2.4.2-2-hadoop2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209623/hbase-common-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,432 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209611/hbase-server-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209612/zookeeper-3.4.6.2.2.4.2-2.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209613/high-scale-lib-1.1.1.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209614/hadoop-common-2.6.0.2.2.4.2-2.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209615/guava-12.0.1.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209616/hbase-protocol-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209617/htrace-core-2.04.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209618/hadoop-mapreduce-client-core-2.6.0.2.2.4.2-2.jar
2017-05-29 15:40:10,434 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209619/netty-3.6.6.Final.jar
2017-05-29 15:40:10,434 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209620/hbase-client-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,434 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209621/hbase-hadoop-compat-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,434 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209622/protobuf-java-2.5.0.jar
2017-05-29 15:40:10,434 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209623/hbase-common-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,444 INFO  [main] mapreduce.Job: The url to track the job: http://localhost:8080/
2017-05-29 15:40:10,446 INFO  [main] mapreduce.Job: Running job: job_local2084446748_0001
2017-05-29 15:40:10,449 INFO  [Thread-7] mapred.LocalJobRunner: OutputCommitter set in config null
2017-05-29 15:40:10,516 INFO  [Thread-7] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2017-05-29 15:40:10,519 INFO  [Thread-7] mapreduce.TableOutputFormat: Created table instance for orm_backlink_v1
2017-05-29 15:40:10,519 INFO  [Thread-7] mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.hbase.mapreduce.TableOutputCommitter
2017-05-29 15:40:10,586 INFO  [Thread-7] mapred.LocalJobRunner: Waiting for map tasks
2017-05-29 15:40:10,589 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner: Starting task: attempt_local2084446748_0001_m_000000_0
2017-05-29 15:40:10,685 INFO  [LocalJobRunner Map Task Executor #0] mapreduce.TableOutputFormat: Created table instance for orm_backlink_v1
2017-05-29 15:40:10,702 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-05-29 15:40:10,708 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask: Processing split: dn27.nfw.com:http://greatsih,http://shellbrowser_components_delphi_edition.th.downloadastro.com|http://kb
2017-05-29 15:40:10,717 INFO  [LocalJobRunner Map Task Executor #0] mapreduce.TableInputFormatBase: Input split length: 164 M bytes.
2017-05-29 15:40:10,738 DEBUG [htable-pool8-t1] ipc.RpcClient: Use SIMPLE authentication for service ClientService, sasl=false
2017-05-29 15:40:10,738 DEBUG [htable-pool8-t1] ipc.RpcClient: Connecting to dn27.nfw.com/10.30.153.13:60020
2017-05-29 15:40:10,942 DEBUG [LocalJobRunner Map Task Executor #0] mapreduce.Import: No configured filter class, accepting all keyvalues.
2017-05-29 15:40:10,943 INFO  [LocalJobRunner Map Task Executor #0] zookeeper.ZooKeeper: Initiating client connection, connectString=zk2.nfw.com:2181,zk1.nfw.com:2181,zk3.nfw.com:2181 sessionTimeout=30000 watcher=attempt_local2084446748_0001_m_000000_0, quorum=zk2.nfw.com:2181,zk1.nfw.com:2181,zk3.nfw.com:2181, baseZNode=/hbase-unsecure
2017-05-29 15:40:10,945 INFO  [LocalJobRunner Map Task Executor #0] zookeeper.RecoverableZooKeeper: Process identifier=attempt_local2084446748_0001_m_000000_0 connecting to ZooKeeper ensemble=zk2.nfw.com:2181,zk1.nfw.com:2181,zk3.nfw.com:2181
2017-05-29 15:40:10,951 INFO  [LocalJobRunner Map Task Executor #0-SendThread(zk3.nfw.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server zk3.nfw.com/10.30.154.56:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:40:10,952 INFO  [LocalJobRunner Map Task Executor #0-SendThread(zk3.nfw.com:2181)] zookeeper.ClientCnxn: Socket connection established to zk3.nfw.com/10.30.154.56:2181, initiating session
2017-05-29 15:40:10,953 INFO  [LocalJobRunner Map Task Executor #0-SendThread(zk3.nfw.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server zk3.nfw.com/10.30.154.56:2181, sessionid = 0x34f27064a67fb8c, negotiated timeout = 30000
2017-05-29 15:40:10,956 INFO  [LocalJobRunner Map Task Executor #0] zookeeper.ZooKeeper: Session: 0x34f27064a67fb8c closed
2017-05-29 15:40:10,956 INFO  [LocalJobRunner Map Task Executor #0-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-05-29 15:40:11,450 INFO  [main] mapreduce.Job: Job job_local2084446748_0001 running in uber mode : false
2017-05-29 15:40:11,452 INFO  [main] mapreduce.Job:  map 0% reduce 0%
2017-05-29 15:40:16,664 INFO  [communication thread] mapred.LocalJobRunner: map > map
2017-05-29 15:40:17,930 INFO  [main-SendThread(mt02.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10006ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:40:18,031 WARN  [LocalJobRunner Map Task Executor #0] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure
2017-05-29 15:40:19,321 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt03.rpt.com/192.168.100.172:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:40:19,323 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt03.rpt.com/192.168.100.172:2181, initiating session
2017-05-29 15:40:29,333 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10010ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:40:29,434 WARN  [LocalJobRunner Map Task Executor #0] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure
2017-05-29 15:40:29,575 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt04.rpt.com/192.168.100.173:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:40:29,577 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt04.rpt.com/192.168.100.173:2181, initiating session
2017-05-29 15:40:39,585 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10008ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:40:39,686 WARN  [LocalJobRunner Map Task Executor #0] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecur