Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error when migrating an Apache HBase Table Between Different Clusters.

Error when migrating an Apache HBase Table Between Different Clusters.

New Contributor

Hi All, I am using CopyTable utility to copy table from one cluster to other. Info 2 cluster following: - Cluster A: HDP-2.2.4.2-2, Hbase 0.98.4.2.2, zk 3.4.6.2.2 - Cluster B: HDP-2.6.0.3, Hbase 1.1.2.2.6,ZK 3.4.6 - Allowed policy firewall between 2 cluster. - Configured file /etc/hosts on together. CopyTable cmd: [hbase@hbaseA hbase]$ ./bin/hbase org.apache.hadoop.hbase.mapreduce.CopyTable --peer.adr=mt02.rpt.com,mt03.rpt.com,mt04.rpt.com:2181:/hbase-unsecure tables01 It is not sucessful. So, Everyone trace log and resolve this problem help me! Logs are as given below:

2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=:/usr/hdp/2.2.4.2-2/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.2.4.2-2/hadoop/lib/native
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.el6.x86_64
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=hbase
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/home/hbase
2017-05-29 15:39:12,984 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/usr/hdp/2.2.4.2-2/hbase
2017-05-29 15:39:12,985 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181 sessionTimeout=30000 watcher=hconnection-0x1a80d720, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, baseZNode=/hbase-unsecure
2017-05-29 15:39:12,999 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x1a80d720 connecting to ZooKeeper ensemble=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181
2017-05-29 15:39:13,003 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt03.rpt.com/192.168.100.172:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:39:13,012 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt03.rpt.com/192.168.100.172:2181, initiating session
2017-05-29 15:39:23,024 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10013ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:39:23,141 WARN  [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-05-29 15:39:23,980 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt04.rpt.com/192.168.100.173:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:39:23,982 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt04.rpt.com/192.168.100.173:2181, initiating session
2017-05-29 15:39:33,992 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10009ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:39:34,092 WARN  [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-05-29 15:39:34,944 INFO  [main-SendThread(mt02.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt02.rpt.com/192.168.100.171:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:39:34,947 INFO  [main-SendThread(mt02.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt02.rpt.com/192.168.100.171:2181, initiating session
2017-05-29 15:39:44,956 INFO  [main-SendThread(mt02.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10009ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:39:45,057 WARN  [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-05-29 15:39:46,992 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt03.rpt.com/192.168.100.172:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:39:46,994 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt03.rpt.com/192.168.100.172:2181, initiating session
2017-05-29 15:39:57,002 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10008ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:39:57,102 WARN  [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-05-29 15:39:57,568 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt04.rpt.com/192.168.100.173:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:39:57,569 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt04.rpt.com/192.168.100.173:2181, initiating session
2017-05-29 15:40:07,571 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10002ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:40:07,672 WARN  [main] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
2017-05-29 15:40:07,672 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 4 attempts
2017-05-29 15:40:07,674 WARN  [main] zookeeper.ZKUtil: hconnection-0x1a80d720, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, baseZNode=/hbase-unsecure Unable to set watcher on znode (/hbase-unsecure/hbaseid)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:199)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:106)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:890)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at org.apache.hadoop.hbase.mapreduce.CopyTable.run(CopyTable.java:275)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.hbase.mapreduce.CopyTable.main(CopyTable.java:266)
2017-05-29 15:40:07,678 ERROR [main] zookeeper.ZooKeeperWatcher: hconnection-0x1a80d720, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, baseZNode=/hbase-unsecure Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:199)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:106)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at org.apache.hadoop.hbase.mapreduce.CopyTable.run(CopyTable.java:275)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.hbase.mapreduce.CopyTable.main(CopyTable.java:266)
2017-05-29 15:40:07,679 WARN  [main] client.ZooKeeperRegistry: Can't retrieve clusterId from Zookeeper
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure/hbaseid
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:199)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:106)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:890)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:667)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:426)
    at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:405)
    at org.apache.hadoop.hbase.client.ConnectionManager.getConnectionInternal(ConnectionManager.java:283)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:207)
    at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:169)
    at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:206)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:559)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at org.apache.hadoop.hbase.mapreduce.CopyTable.run(CopyTable.java:275)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.hbase.mapreduce.CopyTable.main(CopyTable.java:266)
2017-05-29 15:40:07,680 DEBUG [main] client.ConnectionManager$HConnectionImplementation: clusterid came back null, using default default-cluster
2017-05-29 15:40:07,699 DEBUG [main] ipc.RpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@236a277, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null
2017-05-29 15:40:07,708 INFO  [main] mapreduce.TableOutputFormat: Created table instance for orm_backlink_v1
2017-05-29 15:40:07,922 INFO  [main-SendThread(mt02.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt02.rpt.com/192.168.100.171:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:40:07,924 INFO  [main-SendThread(mt02.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt02.rpt.com/192.168.100.171:2181, initiating session
2017-05-29 15:40:08,343 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=zk2.nfw.com:2181,zk1.nfw.com:2181,zk3.nfw.com:2181 sessionTimeout=30000 watcher=hconnection-0x6bac124, quorum=zk2.nfw.com:2181,zk1.nfw.com:2181,zk3.nfw.com:2181, baseZNode=/hbase-unsecure
2017-05-29 15:40:08,345 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x6bac124 connecting to ZooKeeper ensemble=zk2.nfw.com:2181,zk1.nfw.com:2181,zk3.nfw.com:2181
2017-05-29 15:40:08,345 INFO  [main-SendThread(zk2.nfw.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server zk2.nfw.com/10.30.154.53:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:40:08,346 INFO  [main-SendThread(zk2.nfw.com:2181)] zookeeper.ClientCnxn: Socket connection established to zk2.nfw.com/10.30.154.53:2181, initiating session
2017-05-29 15:40:08,352 INFO  [main-SendThread(zk2.nfw.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server zk2.nfw.com/10.30.154.53:2181, sessionid = 0x25331211b0c555b, negotiated timeout = 30000
2017-05-29 15:40:08,367 DEBUG [main] ipc.RpcClient: Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20a19e0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null
2017-05-29 15:40:08,401 INFO  [main] util.RegionSizeCalculator: Calculating region sizes for table "orm_backlink_v1".
2017-05-29 15:40:08,410 DEBUG [main] hbase.HRegionInfo: 1588230740
2017-05-29 15:40:08,733 DEBUG [htable-pool3-t1] ipc.RpcClient: Use SIMPLE authentication for service ClientService, sasl=false
2017-05-29 15:40:08,761 DEBUG [htable-pool3-t1] ipc.RpcClient: Connecting to dn11.nfw.com/10.30.152.11:60020
2017-05-29 15:40:08,863 DEBUG [main] ipc.RpcClient: Use SIMPLE authentication for service MasterService, sasl=false
2017-05-29 15:40:08,864 DEBUG [main] ipc.RpcClient: Connecting to hm2.nfw.com/10.30.154.55:60000
2017-05-29 15:40:09,022 DEBUG [main] util.RegionSizeCalculator: Region orm_backlink_v1,,1438962439939.f59b6aaf686701422f99238880208c2e. has size 150994944
2017-05-29 15:40:09,035 DEBUG [main] util.RegionSizeCalculator: Region orm_backlink_v1,http://shellbrowser_components_delphi_edition.th.downloadastro.com|http://kb,1439701594892.7f3318d1679f4533b9b451837dc94f9d. has size 171966464
2017-05-29 15:40:09,039 DEBUG [main] util.RegionSizeCalculator: Region orm_backlink_v1,http://greatsih,1439701594892.d5f3fff639e9765cfa49e12a18e5d66a. has size 171966464
2017-05-29 15:40:09,043 DEBUG [main] util.RegionSizeCalculator: Region sizes calculated
2017-05-29 15:40:09,123 WARN  [main] mapreduce.TableInputFormatBase: Cannot resolve the host name for dn11.nfw.com/10.30.152.11 because of javax.naming.CommunicationException: DNS error [Root exception is java.net.PortUnreachableException: ICMP Port Unreachable]; remaining name '11.152.30.10.in-addr.arpa'
2017-05-29 15:40:09,126 DEBUG [main] mapreduce.TableInputFormatBase: getSplits: split -> 0 -> dn11.nfw.com:,http://greatsih
2017-05-29 15:40:09,127 WARN  [main] mapreduce.TableInputFormatBase: Cannot resolve the host name for dn27.nfw.com/10.30.153.13 because of javax.naming.CommunicationException: DNS error [Root exception is java.net.PortUnreachableException: ICMP Port Unreachable]; remaining name '13.153.30.10.in-addr.arpa'
2017-05-29 15:40:09,128 DEBUG [main] mapreduce.TableInputFormatBase: getSplits: split -> 1 -> dn27.nfw.com:http://greatsih,http://shellbrowser_components_delphi_edition.th.downloadastro.com|http://kb
2017-05-29 15:40:09,129 WARN  [main] mapreduce.TableInputFormatBase: Cannot resolve the host name for dn24.nfw.com/10.30.153.10 because of javax.naming.CommunicationException: DNS error [Root exception is java.net.PortUnreachableException: ICMP Port Unreachable]; remaining name '10.153.30.10.in-addr.arpa'
2017-05-29 15:40:09,129 DEBUG [main] mapreduce.TableInputFormatBase: getSplits: split -> 2 -> dn24.nfw.com:http://shellbrowser_components_delphi_edition.th.downloadastro.com|http://kb,
2017-05-29 15:40:09,151 INFO  [main] mapreduce.JobSubmitter: number of splits:3
2017-05-29 15:40:09,167 INFO  [main] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2017-05-29 15:40:09,446 INFO  [main] mapreduce.JobSubmitter: Submitting tokens for job: job_local2084446748_0001
2017-05-29 15:40:10,098 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/hbase-server-0.98.4.2.2.4.2-2-hadoop2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209611/hbase-server-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,247 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/zookeeper/zookeeper-3.4.6.2.2.4.2-2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209612/zookeeper-3.4.6.2.2.4.2-2.jar
2017-05-29 15:40:10,248 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/high-scale-lib-1.1.1.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209613/high-scale-lib-1.1.1.jar
2017-05-29 15:40:10,248 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hadoop/hadoop-common-2.6.0.2.2.4.2-2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209614/hadoop-common-2.6.0.2.2.4.2-2.jar
2017-05-29 15:40:10,248 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/guava-12.0.1.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209615/guava-12.0.1.jar
2017-05-29 15:40:10,248 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/hbase-protocol-0.98.4.2.2.4.2-2-hadoop2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209616/hbase-protocol-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,249 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/htrace-core-2.04.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209617/htrace-core-2.04.jar
2017-05-29 15:40:10,249 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hadoop-mapreduce/hadoop-mapreduce-client-core-2.6.0.2.2.4.2-2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209618/hadoop-mapreduce-client-core-2.6.0.2.2.4.2-2.jar
2017-05-29 15:40:10,249 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/netty-3.6.6.Final.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209619/netty-3.6.6.Final.jar
2017-05-29 15:40:10,328 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/hbase-client-0.98.4.2.2.4.2-2-hadoop2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209620/hbase-client-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,328 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/hbase-hadoop-compat-0.98.4.2.2.4.2-2-hadoop2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209621/hbase-hadoop-compat-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,329 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/protobuf-java-2.5.0.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209622/protobuf-java-2.5.0.jar
2017-05-29 15:40:10,329 INFO  [main] mapred.LocalDistributedCacheManager: Localized hdfs://nfwcluster/usr/hdp/2.2.4.2-2/hbase/lib/hbase-common-0.98.4.2.2.4.2-2-hadoop2.jar as file:/tmp/hadoop-hbase/mapred/local/1496047209623/hbase-common-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,432 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209611/hbase-server-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209612/zookeeper-3.4.6.2.2.4.2-2.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209613/high-scale-lib-1.1.1.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209614/hadoop-common-2.6.0.2.2.4.2-2.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209615/guava-12.0.1.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209616/hbase-protocol-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209617/htrace-core-2.04.jar
2017-05-29 15:40:10,433 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209618/hadoop-mapreduce-client-core-2.6.0.2.2.4.2-2.jar
2017-05-29 15:40:10,434 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209619/netty-3.6.6.Final.jar
2017-05-29 15:40:10,434 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209620/hbase-client-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,434 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209621/hbase-hadoop-compat-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,434 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209622/protobuf-java-2.5.0.jar
2017-05-29 15:40:10,434 INFO  [main] mapred.LocalDistributedCacheManager: file:/tmp/hadoop-hbase/mapred/local/1496047209623/hbase-common-0.98.4.2.2.4.2-2-hadoop2.jar
2017-05-29 15:40:10,444 INFO  [main] mapreduce.Job: The url to track the job: http://localhost:8080/
2017-05-29 15:40:10,446 INFO  [main] mapreduce.Job: Running job: job_local2084446748_0001
2017-05-29 15:40:10,449 INFO  [Thread-7] mapred.LocalJobRunner: OutputCommitter set in config null
2017-05-29 15:40:10,516 INFO  [Thread-7] Configuration.deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2017-05-29 15:40:10,519 INFO  [Thread-7] mapreduce.TableOutputFormat: Created table instance for orm_backlink_v1
2017-05-29 15:40:10,519 INFO  [Thread-7] mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.hbase.mapreduce.TableOutputCommitter
2017-05-29 15:40:10,586 INFO  [Thread-7] mapred.LocalJobRunner: Waiting for map tasks
2017-05-29 15:40:10,589 INFO  [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner: Starting task: attempt_local2084446748_0001_m_000000_0
2017-05-29 15:40:10,685 INFO  [LocalJobRunner Map Task Executor #0] mapreduce.TableOutputFormat: Created table instance for orm_backlink_v1
2017-05-29 15:40:10,702 INFO  [LocalJobRunner Map Task Executor #0] mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2017-05-29 15:40:10,708 INFO  [LocalJobRunner Map Task Executor #0] mapred.MapTask: Processing split: dn27.nfw.com:http://greatsih,http://shellbrowser_components_delphi_edition.th.downloadastro.com|http://kb
2017-05-29 15:40:10,717 INFO  [LocalJobRunner Map Task Executor #0] mapreduce.TableInputFormatBase: Input split length: 164 M bytes.
2017-05-29 15:40:10,738 DEBUG [htable-pool8-t1] ipc.RpcClient: Use SIMPLE authentication for service ClientService, sasl=false
2017-05-29 15:40:10,738 DEBUG [htable-pool8-t1] ipc.RpcClient: Connecting to dn27.nfw.com/10.30.153.13:60020
2017-05-29 15:40:10,942 DEBUG [LocalJobRunner Map Task Executor #0] mapreduce.Import: No configured filter class, accepting all keyvalues.
2017-05-29 15:40:10,943 INFO  [LocalJobRunner Map Task Executor #0] zookeeper.ZooKeeper: Initiating client connection, connectString=zk2.nfw.com:2181,zk1.nfw.com:2181,zk3.nfw.com:2181 sessionTimeout=30000 watcher=attempt_local2084446748_0001_m_000000_0, quorum=zk2.nfw.com:2181,zk1.nfw.com:2181,zk3.nfw.com:2181, baseZNode=/hbase-unsecure
2017-05-29 15:40:10,945 INFO  [LocalJobRunner Map Task Executor #0] zookeeper.RecoverableZooKeeper: Process identifier=attempt_local2084446748_0001_m_000000_0 connecting to ZooKeeper ensemble=zk2.nfw.com:2181,zk1.nfw.com:2181,zk3.nfw.com:2181
2017-05-29 15:40:10,951 INFO  [LocalJobRunner Map Task Executor #0-SendThread(zk3.nfw.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server zk3.nfw.com/10.30.154.56:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:40:10,952 INFO  [LocalJobRunner Map Task Executor #0-SendThread(zk3.nfw.com:2181)] zookeeper.ClientCnxn: Socket connection established to zk3.nfw.com/10.30.154.56:2181, initiating session
2017-05-29 15:40:10,953 INFO  [LocalJobRunner Map Task Executor #0-SendThread(zk3.nfw.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server zk3.nfw.com/10.30.154.56:2181, sessionid = 0x34f27064a67fb8c, negotiated timeout = 30000
2017-05-29 15:40:10,956 INFO  [LocalJobRunner Map Task Executor #0] zookeeper.ZooKeeper: Session: 0x34f27064a67fb8c closed
2017-05-29 15:40:10,956 INFO  [LocalJobRunner Map Task Executor #0-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-05-29 15:40:11,450 INFO  [main] mapreduce.Job: Job job_local2084446748_0001 running in uber mode : false
2017-05-29 15:40:11,452 INFO  [main] mapreduce.Job:  map 0% reduce 0%
2017-05-29 15:40:16,664 INFO  [communication thread] mapred.LocalJobRunner: map > map
2017-05-29 15:40:17,930 INFO  [main-SendThread(mt02.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10006ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:40:18,031 WARN  [LocalJobRunner Map Task Executor #0] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure
2017-05-29 15:40:19,321 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt03.rpt.com/192.168.100.172:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:40:19,323 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt03.rpt.com/192.168.100.172:2181, initiating session
2017-05-29 15:40:29,333 INFO  [main-SendThread(mt03.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10010ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:40:29,434 WARN  [LocalJobRunner Map Task Executor #0] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecure
2017-05-29 15:40:29,575 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mt04.rpt.com/192.168.100.173:2181. Will not attempt to authenticate using SASL (unknown error)
2017-05-29 15:40:29,577 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Socket connection established to mt04.rpt.com/192.168.100.173:2181, initiating session
2017-05-29 15:40:39,585 INFO  [main-SendThread(mt04.rpt.com:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 10008ms for sessionid 0x0, closing socket connection and attempting reconnect
2017-05-29 15:40:39,686 WARN  [LocalJobRunner Map Task Executor #0] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=mt03.rpt.com:2181,mt02.rpt.com:2181,mt04.rpt.com:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-unsecur