hbase client error when using kerberos

[#1]

I have encounter following error:

Config name: C:\Windows\krb5.ini
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:host.name=9.78.211.112
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:java.version=1.6.0_33
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc.
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:java.home=C:\Java\jdk1.6\jre
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:java.class.path=C:\project\github\Big_data_example\me.huiwen.example.hbase.kerberos\target\classes;C:\Users\sfb_ADMIN\.m2\repository\junit\junit\4.11\junit-4.11.jar;C:\Users\sfb_ADMIN\.m2\repository\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar;C:\Java\jdk1.7.0_21\lib\tools.jar;C:\Users\sfb_ADMIN\.m2\repository\log4j\log4j\1.2.15\log4j-1.2.15.jar;C:\Users\sfb_ADMIN\.m2\repository\org\apache\hbase\hbase-client\0.95.0\hbase-client-0.95.0.jar;C:\Users\sfb_ADMIN\.m2\repository\org\apache\hbase\hbase-common\0.95.0\hbase-common-0.95.0.jar;C:\Users\sfb_ADMIN\.m2\repository\com\google\guava\guava\12.0.1\guava-12.0.1.jar;C:\Users\sfb_ADMIN\.m2\repository\com\google\code\findbugs\jsr305\1.3.9\jsr305-1.3.9.jar;C:\Users\sfb_ADMIN\.m2\repository\commons-logging\commons-logging\1.1.1\commons-logging-1.1.1.jar;C:\Users\sfb_ADMIN\.m2\repository\org\slf4j\slf4j-api\1.4.3\slf4j-api-1.4.3.jar;C:\Users\sfb_ADMIN\.m2\repository\org\slf4j\slf4j-log4j12\1.4.3\slf4j-log4j12-1.4.3.jar;C:\Users\sfb_ADMIN\.m2\repository\org\apache\hbase\hbase-protocol\0.95.0\hbase-protocol-0.95.0.jar;C:\Users\sfb_ADMIN\.m2\repository\com\google\protobuf\protobuf-java\2.4.1\protobuf-java-2.4.1.jar;C:\Users\sfb_ADMIN\.m2\repository\org\apache\zookeeper\zookeeper\3.4.5\zookeeper-3.4.5.jar;C:\Users\sfb_ADMIN\.m2\repository\org\cloudera\htrace\htrace\1.50\htrace-1.50.jar;C:\Users\sfb_ADMIN\.m2\repository\org\mortbay\jetty\jetty-util\6.1.26\jetty-util-6.1.26.jar;C:\Users\sfb_ADMIN\.m2\repository\io\netty\netty\3.5.9.Final\netty-3.5.9.Final.jar;C:\Users\sfb_ADMIN\.m2\repository\org\apache\hadoop\hadoop-core\1.1.2\hadoop-core-1.1.2.jar;C:\Users\sfb_ADMIN\.m2\repository\commons-cli\commons-cli\1.2\commons-cli-1.2.jar;C:\Users\sfb_ADMIN\.m2\repository\xmlenc\xmlenc\0.52\xmlenc-0.52.jar;C:\Users\sfb_ADMIN\.m2\repository\com\sun\jersey\jersey-core\1.8\jersey-core-1.8.jar;C:\Users\sfb_ADMIN\.m2\repository\com\sun\jersey\jersey-json\1.8\jersey-json-1.8.jar;C:\Users\sfb_ADMIN\.m2\repository\org\codehaus\jettison\jettison\1.1\jettison-1.1.jar;C:\Users\sfb_ADMIN\.m2\repository\stax\stax-api\1.0.1\stax-api-1.0.1.jar;C:\Users\sfb_ADMIN\.m2\repository\com\sun\xml\bind\jaxb-impl\2.2.3-1\jaxb-impl-2.2.3-1.jar;C:\Users\sfb_ADMIN\.m2\repository\javax\xml\bind\jaxb-api\2.2.2\jaxb-api-2.2.2.jar;C:\Users\sfb_ADMIN\.m2\repository\javax\xml\stream\stax-api\1.0-2\stax-api-1.0-2.jar;C:\Users\sfb_ADMIN\.m2\repository\javax\activation\activation\1.1\activation-1.1.jar;C:\Users\sfb_ADMIN\.m2\repository\org\codehaus\jackson\jackson-xc\1.7.1\jackson-xc-1.7.1.jar;C:\Users\sfb_ADMIN\.m2\repository\com\sun\jersey\jersey-server\1.8\jersey-server-1.8.jar;C:\Users\sfb_ADMIN\.m2\repository\asm\asm\3.1\asm-3.1.jar;C:\Users\sfb_ADMIN\.m2\repository\commons-io\commons-io\2.1\commons-io-2.1.jar;C:\Users\sfb_ADMIN\.m2\repository\commons-httpclient\commons-httpclient\3.0.1\commons-httpclient-3.0.1.jar;C:\Users\sfb_ADMIN\.m2\repository\commons-codec\commons-codec\1.4\commons-codec-1.4.jar;C:\Users\sfb_ADMIN\.m2\repository\org\apache\commons\commons-math\2.1\commons-math-2.1.jar;C:\Users\sfb_ADMIN\.m2\repository\commons-configuration\commons-configuration\1.6\commons-configuration-1.6.jar;C:\Users\sfb_ADMIN\.m2\repository\commons-collections\commons-collections\3.2.1\commons-collections-3.2.1.jar;C:\Users\sfb_ADMIN\.m2\repository\commons-lang\commons-lang\2.4\commons-lang-2.4.jar;C:\Users\sfb_ADMIN\.m2\repository\commons-digester\commons-digester\1.8\commons-digester-1.8.jar;C:\Users\sfb_ADMIN\.m2\repository\commons-beanutils\commons-beanutils\1.7.0\commons-beanutils-1.7.0.jar;C:\Users\sfb_ADMIN\.m2\repository\commons-beanutils\commons-beanutils-core\1.8.0\commons-beanutils-core-1.8.0.jar;C:\Users\sfb_ADMIN\.m2\repository\commons-net\commons-net\1.4.1\commons-net-1.4.1.jar;C:\Users\sfb_ADMIN\.m2\repository\org\mortbay\jetty\jetty\6.1.26\jetty-6.1.26.jar;C:\Users\sfb_ADMIN\.m2\repository\org\mortbay\jetty\servlet-api\2.5-20081211\servlet-api-2.5-20081211.jar;C:\Users\sfb_ADMIN\.m2\repository\tomcat\jasper-runtime\5.5.12\jasper-runtime-5.5.12.jar;C:\Users\sfb_ADMIN\.m2\repository\tomcat\jasper-compiler\5.5.12\jasper-compiler-5.5.12.jar;C:\Users\sfb_ADMIN\.m2\repository\org\mortbay\jetty\jsp-api-2.1\6.1.14\jsp-api-2.1-6.1.14.jar;C:\Users\sfb_ADMIN\.m2\repository\org\mortbay\jetty\servlet-api-2.5\6.1.14\servlet-api-2.5-6.1.14.jar;C:\Users\sfb_ADMIN\.m2\repository\org\mortbay\jetty\jsp-2.1\6.1.14\jsp-2.1-6.1.14.jar;C:\Users\sfb_ADMIN\.m2\repository\ant\ant\1.6.5\ant-1.6.5.jar;C:\Users\sfb_ADMIN\.m2\repository\commons-el\commons-el\1.0\commons-el-1.0.jar;C:\Users\sfb_ADMIN\.m2\repository\com\github\stephenc\findbugs\findbugs-annotations\1.3.9-1\findbugs-annotations-1.3.9-1.jar
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:java.library.path=C:\Java\jdk1.6\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:/Java/jdk1.7.0_21/bin/../jre/bin/server;C:/Java/jdk1.7.0_21/bin/../jre/bin;C:/Java/jdk1.7.0_21/bin/../jre/lib/amd64;C:\develop\Ruby193\bin;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Program Files (x86)\Quest Software\PuTTY\;C:\develop\apache-maven-3.0.4\bin;C:\Java\jdk1.7.0_21\bin;C:\develop\mingw64\bin;C:\wsa\deepqa\blade\latest_build\lib\x86_64\xsg;C:\develop\mingw64\x86_64-w64-mingw32\lib;C:\develop\mingw64\lib;C:\develop\mingw64\x86_64-w64-mingw32\lib;C:\develop\play-2.0.3;C:\Program Files\Mercurial;c:\notes;C:\app\Git\bin;C:\develop\Python27;C:\develop\Python27\Scripts;C:\develop\eclipse-jee-galileo-SR2-win32\eclipse;;.
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=C:\Users\sfb_AD~1\AppData\Local\Temp\
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:os.name=Windows 7
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:os.version=6.1
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:user.name=henry
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:user.home=C:\Users\sfb_ADMIN
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Client environment:user.dir=C:\project\github\Big_data_example\me.huiwen.example.hbase.kerberos
13/08/09 15:53:18 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=dpev209.innovate.sfb.com:2181 sessionTimeout=600000 watcher=hconnection-0x42552c
13/08/09 15:53:18 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is hconnection-0x42552c
>>> KeyTabInputStream, readName(): sfb.COM
>>> KeyTabInputStream, readName(): zkcli
>>> KeyTab: load() entry length: 63; type: 18
>>> KeyTabInputStream, readName(): sfb.COM
>>> KeyTabInputStream, readName(): zkcli
>>> KeyTab: load() entry length: 47; type: 17
>>> KeyTabInputStream, readName(): sfb.COM
>>> KeyTabInputStream, readName(): zkcli
>>> KeyTab: load() entry length: 55; type: 16
>>> KeyTabInputStream, readName(): sfb.COM
>>> KeyTabInputStream, readName(): zkcli
>>> KeyTab: load() entry length: 47; type: 23
>>> KeyTabInputStream, readName(): sfb.COM
>>> KeyTabInputStream, readName(): zkcli
>>> KeyTab: load() entry length: 39; type: 8
>>> KeyTabInputStream, readName(): sfb.COM
>>> KeyTabInputStream, readName(): zkcli
>>> KeyTab: load() entry length: 39; type: 3
>>> KeyTabInputStream, readName(): sfb.COM
>>> KeyTabInputStream, readName(): zkcli
>>> KeyTab: load() entry length: 63; type: 18
>>> KeyTabInputStream, readName(): sfb.COM
>>> KeyTabInputStream, readName(): zkcli
>>> KeyTab: load() entry length: 47; type: 17
>>> KeyTabInputStream, readName(): sfb.COM
>>> KeyTabInputStream, readName(): zkcli
>>> KeyTab: load() entry length: 55; type: 16
>>> KeyTabInputStream, readName(): sfb.COM
>>> KeyTabInputStream, readName(): zkcli
>>> KeyTab: load() entry length: 47; type: 23
>>> KeyTabInputStream, readName(): sfb.COM
>>> KeyTabInputStream, readName(): zkcli
>>> KeyTab: load() entry length: 39; type: 8
>>> KeyTabInputStream, readName(): sfb.COM
>>> KeyTabInputStream, readName(): zkcli
>>> KeyTab: load() entry length: 39; type: 3
Added key: 3version: 8
Found unsupported keytype (8) for zkcli@sfb.COM
Added key: 23version: 8
Added key: 16version: 8
Added key: 17version: 8
Added key: 18version: 8
Added key: 3version: 7
Found unsupported keytype (8) for zkcli@sfb.COM
Added key: 23version: 7
Added key: 16version: 7
Added key: 17version: 7
Added key: 18version: 7
Ordering keys wrt default_tkt_enctypes list
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 3 1 23 16 17 18.
>>> KdcAccessibility: reset
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 3 1 23 16 17 18.
>>> KrbAsReq calling createMessage
>>> KrbAsReq in createMessage
>>> KrbKdcReq send: kdc=dpev210.innovate.sfb.com UDP:88, timeout=30000, number of retries =3, #bytes=139
>>> KDCCommunication: kdc=dpev210.innovate.sfb.com UDP:88, timeout=30000,Attempt =1, #bytes=139
>>> KrbKdcReq send: #bytes read=495
>>> KrbKdcReq send: #bytes read=495
>>> KdcAccessibility: remove dpev210.innovate.sfb.com
>>> EType: sun.security.krb5.internal.crypto.DesCbcMd5EType
>>> KrbAsRep cons in KrbAsReq.getReply zkcli
13/08/09 15:53:19 INFO zookeeper.Login: successfully logged in.
13/08/09 15:53:19 INFO zookeeper.Login: TGT refresh thread started.
13/08/09 15:53:19 INFO client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
13/08/09 15:53:19 INFO zookeeper.Login: TGT valid starting at: Fri Aug 09 15:53:39 GMT 2013
13/08/09 15:53:19 INFO zookeeper.Login: TGT expires: Sat Aug 10 03:53:39 GMT 2013
13/08/09 15:53:19 INFO zookeeper.Login: TGT refresh sleeping until: Sat Aug 10 01:29:42 GMT 2013
13/08/09 15:53:19 INFO zookeeper.ClientCnxn: Opening socket connection to server dpev209.innovate.sfb.com/9.45.92.37:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
13/08/09 15:53:20 INFO zookeeper.ClientCnxn: Socket connection established to dpev209.innovate.sfb.com/9.45.92.37:2181, initiating session
13/08/09 15:53:20 INFO zookeeper.ClientCnxn: Session establishment complete on server dpev209.innovate.sfb.com/9.45.92.37:2181, sessionid = 0x1406363f3110107, negotiated timeout = 600000
Found ticket for zkcli@sfb.COM to go to krbtgt/sfb.COM@sfb.COM expiring on Sat Aug 10 03:53:39 GMT 2013
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for zkcli@sfb.COM to go to krbtgt/sfb.COM@sfb.COM expiring on Sat Aug 10 03:53:39 GMT 2013
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 3 1 23 16 17 18.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.DesCbcCrcEType
>>>crc32: c65aeb1
>>>crc32: 1100011001011010111010110001
>>> KrbKdcReq send: kdc=dpev210.innovate.sfb.com UDP:88, timeout=30000, number of retries =3, #bytes=539
>>> KDCCommunication: kdc=dpev210.innovate.sfb.com UDP:88, timeout=30000,Attempt =1, #bytes=539
>>> KrbKdcReq send: #bytes read=538
>>> KrbKdcReq send: #bytes read=538
>>> KdcAccessibility: remove dpev210.innovate.sfb.com
>>> EType: sun.security.krb5.internal.crypto.DesCbcCrcEType
>>>crc32: 1963d0dd
>>>crc32: 11001011000111101000011011101
>>> KrbApReq: APOptions are 00000000 00000000 00000000 00000000
>>> EType: sun.security.krb5.internal.crypto.DesCbcCrcEType
>>>crc32: 5812045a
>>>crc32: 1011000000100100000010001011010
Krb5Context setting mySeqNumber to: 137426464
Krb5Context setting peerSeqNumber to: 0
Created InitSecContextToken:
0000: 01 00 6E 82 01 C8 30 82 01 C4 A0 03 02 01 05 A1 ..n...0.........
0010: 03 02 01 0E A2 07 03 05 00 00 00 00 00 A3 82 01 ................
0020: 04 61 82 01 00 30 81 FD A0 03 02 01 05 A1 09 1B .a...0..........
0030: 07 49 42 4D 2E 43 4F 4D A2 30 30 2E A0 03 02 01 .sfb.COM.00.....
0040: 00 A1 27 30 25 1B 09 7A 6F 6F 6B 65 65 70 65 72 ..'0%..zookeeper
0050: 1B 18 64 70 65 76 32 30 39 2E 69 6E 6E 6F 76 61 ..dpev209.innova
0060: 74 65 2E 69 62 6D 2E 63 6F 6D A3 81 B8 30 81 B5 te.sfb.com...0..
0070: A0 03 02 01 12 A1 03 02 01 04 A2 81 A8 04 81 A5 ................
0080: E8 B6 34 EB 89 EB 15 54 2A 80 BF BA FB FD 6A 5A ..4....T*.....jZ
0090: F7 2F CE A9 D2 36 FB B8 A0 AA 15 09 C8 37 50 15 ./...6.......7P.
00A0: 37 6E A4 93 D6 B6 08 BD 0A C2 46 84 F4 BC 7C AF 7n........F.....
00B0: F4 42 73 7B AA E4 9C A2 0E F6 E9 EA 67 CE A0 7D .Bs.........g...
00C0: D9 81 D9 F0 56 1B 70 FA 4F D1 92 EA 0D 13 59 A0 ....V.p.O.....Y.
00D0: 20 9D AA EF 18 77 F1 56 F4 7A DA 5E 8D CC 26 AC ....w.V.z.^..&.
00E0: 30 C0 E4 67 F6 65 2A F5 D1 EC 98 F2 6C 7E 61 2D 0..g.e*.....l.a-
00F0: 1A E5 C6 50 D1 35 21 50 95 3C 76 2F 08 57 30 FB ...P.5!P.<v/.W0.
0100: 77 84 F1 BF 8C DA EA AA 61 CE 85 FA 11 36 43 B8 w.......a....6C.
0110: 8D BF 56 06 D0 E9 D6 BD F6 7A DC A1 2A D6 80 F8 ..V......z..*...
0120: A9 47 1A 3F 2E A4 81 A6 30 81 A3 A0 03 02 01 01 .G.?....0.......
0130: A2 81 9B 04 81 98 96 08 25 CA 86 98 86 5E 1B 47 ........%....^.G
0140: 88 FB 57 FC 25 63 F7 91 8E 37 A2 5A 27 AF B0 C8 ..W.%c...7.Z'...
0150: E1 C3 F6 BE B5 87 D7 43 57 60 7D 92 19 A6 85 3B .......CW`.....;
0160: 08 6C 01 9F 24 C7 EA E4 16 D1 1A B7 53 85 F8 56 .l..$.......S..V
0170: 52 A0 1A 24 C4 3C EA 9B 29 15 0A E1 54 1E 6C 37 R..$.<..)...T.l7
0180: A0 A9 BA 73 62 37 3B 55 24 42 DE D4 CF 6E 5F F9 ...sb7;U$B...n_.
0190: 7E E9 65 D2 D4 76 9C DE 1B 27 21 41 C2 5A D9 A4 ..e..v...'!A.Z..
01A0: 12 48 62 55 22 B0 D1 59 1B B7 6E CC DD 2B 61 7E .HbU"..Y..n..+a.
01B0: 7B FC CA 4C E6 C2 99 23 63 BC 4A 21 1D E6 C8 91 ...L...#c.J!....
01C0: FB 4E 20 74 37 C4 42 F6 4F 3E FC 7B F8 76 .N t7.B.O>...v

Krb5Context.unwrap: token=[60 33 06 09 2a 86 48 86 f7 12 01 02 02 02 01 00 00 ff ff ff ff 36 35 72 7c 19 9e 17 bc d7 17 c4 1f 9c f7 e5 c4 95 a6 97 2b 8e ae f0 e7 01 01 00 00 04 04 04 04 ]
Krb5Context.unwrap: data=[01 01 00 00 ]
Krb5Context.wrap: data=[01 01 00 00 7a 6b 63 6c 69 40 49 42 4d 2e 43 4f 4d ]
Krb5Context.wrap: token=[60 43 06 09 2a 86 48 86 f7 12 01 02 02 02 01 00 00 ff ff ff ff b7 a7 7d 39 a5 2e a3 82 24 50 6b 54 b1 92 ce 19 7d 01 ae 93 8b a9 60 1c 01 01 00 00 7a 6b 63 6c 69 40 49 42 4d 2e 43 4f 4d 07 07 07 07 07 07 07 ]
13/08/09 15:53:21 ERROR zookeeper.ClientCnxn: Error while calling watcher
java.lang.IllegalStateException: Received event is not valid.
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:372)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:283)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)

Seems it a bug https://issues.apache.org/jira/browse/HBASE-8757

I am not sure cloudera 4.3.0 using the fixed version of Hbase 

[#2]

I am not sure cloudera 4.3.0 using the fixed version of Hbase 

CDH 4.3 includes Apache HBase hbase-0.94.6+96

"The following lists all Apache HBase Jiras included in CDH 4 that are not included in the Apache HBase base version 0.94.6."

http://archive.cloudera.com/cdh4/cdh/4/hbase-0.94.6-cdh4.3.0.releasenotes.html

HBASE-8757 is not listed.

"The hbase-0.94.6-cdh4.3.0.CHANGES.txt file lists all changes included in CDH 4. "

HBASE-8757 is not listed.

Which implies HBASE-8757 is not fixed in CDH 4.3.

[#3]

HBASE-8757 is fixed in HBase 0.98.0, 0.95.2

Install HBase 0.98.0, or  0.95.2 

[#4]

HBASE-8757 is a nice-to-have so it is not fixed in CDH4.3.0.  It's a warning and doesn't cause any function loss.