Unable to upload new files to encrypted zone in HDFS

[#1]

We have an encryption zone set up around a file structure for an application. We are trying to load the files into hdfs in this encrypted zone but we continue to get the following error:

GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

I have a valid Kerberos ticket and we opened up the KMS acls to allow all users to DECRYPT_EEK just to make sure we did not have an ACL setting wrong

Does anybody else have experience with this issue? 

[#2]

What version of CDH do you use? Can you share the full stack trace around the exception?

Depending on your version and the stack trace you're most likely hitting the https://issues.apache.org/jira/browse/HADOOP-12559 described failure. This has been addressed in CDH 5.5.4 onwards for the 5.5.x line, and is also in all 5.6.x and 5.7.x and any future releases since then.

An ACL setting failure would give you a different error, such as a 403 from a KMS.

[#3]

Hi,

Thanks for responsing. We are using CDH 5.5.2 and Cloudera Manager 5.5.3 (if that matters). I'm sure how to get a full stack trace on a command that is running so quickly but the command I'm entering and the following error are:

hadoop fs -put testkb.txt /data/fi
put: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)

We are definitely willing to upgrade if that Jira is the cause of the issue. If you could direct me how to get a full stacktrace of the hadoop fs -put command and the kerberos error I would be happy to provide it.

Thanks,

Matt Rice

[#4]

Retry your command this way:

~> HADOOP_ROOT_LOGGER=DEBUG,console hadoop fs -put testkb.txt /data/fi

Among other outputs, it should produce the fuller exception trace before it aborts with the same message.

[#5]

This is the full output of the stack trace. Thanks

[mrice@hare8 ~]$ HADOOP_ROOT_LOGGER=DEBUG,console hadoop fs -put testkb.txt /data/fi
16/07/27 10:28:53 DEBUG util.Shell: setsid exited with exit code 0
16/07/27 10:28:53 DEBUG conf.Configuration: parsing URL jar:file:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/ja rs/hadoop-common-2.6.0-cdh5.5.2.jar!/core-default.xml
16/07/27 10:28:53 DEBUG conf.Configuration: parsing input stream sun.net.www.protocol.jar.JarURLConnection$JarURLIn putStream@15427c33
16/07/27 10:28:53 DEBUG conf.Configuration: parsing URL file:/etc/hadoop/conf.cloudera.yarn/core-site.xml
16/07/27 10:28:53 DEBUG conf.Configuration: parsing input stream java.io.BufferedInputStream@726b37ad
16/07/27 10:28:53 DEBUG core.Tracer: sampler.classes = ; loaded no samplers
16/07/27 10:28:53 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers
16/07/27 10:28:54 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hado op.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Met ric(valueName=Time, about=, value=[Rate of successful kerberos logins and latency (milliseconds)], always=false, ty pe=DEFAULT, sampleName=Ops)
16/07/27 10:28:54 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hado op.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Met ric(valueName=Time, about=, value=[Rate of failed kerberos logins and latency (milliseconds)], always=false, type=D EFAULT, sampleName=Ops)
16/07/27 10:28:54 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hado op.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric (valueName=Time, about=, value=[GetGroups], always=false, type=DEFAULT, sampleName=Ops)
16/07/27 10:28:54 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
16/07/27 10:28:54 DEBUG security.Groups: Creating new Groups object
16/07/27 10:28:54 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000
16/07/27 10:28:54 DEBUG security.UserGroupInformation: hadoop login
16/07/27 10:28:54 DEBUG security.UserGroupInformation: hadoop login commit
16/07/27 10:28:54 DEBUG security.UserGroupInformation: using kerberos user:mrice@PUTNAMINV.COM
16/07/27 10:28:54 DEBUG security.UserGroupInformation: Using user: "mrice@PUTNAMINV.COM" with name mrice@PUTNAMINV. COM
16/07/27 10:28:54 DEBUG security.UserGroupInformation: User entry: "mrice@PUTNAMINV.COM"
16/07/27 10:28:54 DEBUG security.UserGroupInformation: UGI loginUser:mrice@PUTNAMINV.COM (auth:KERBEROS)
16/07/27 10:28:54 DEBUG core.Tracer: sampler.classes = ; loaded no samplers
16/07/27 10:28:54 DEBUG core.Tracer: span.receiver.classes = ; loaded no span receivers
16/07/27 10:28:54 DEBUG security.UserGroupInformation: Found tgt Ticket (hex) =
0000: 61 82 05 23 30 82 05 1F A0 03 02 01 05 A1 0F 1B a..#0...........
0010: 0D 50 55 54 4E 41 4D 49 4E 56 2E 43 4F 4D A2 22 .PUTNAMINV.COM."
0020: 30 20 A0 03 02 01 02 A1 19 30 17 1B 06 6B 72 62 0 .......0...krb
0030: 74 67 74 1B 0D 50 55 54 4E 41 4D 49 4E 56 2E 43 tgt..PUTNAMINV.C
0040: 4F 4D A3 82 04 E1 30 82 04 DD A0 03 02 01 17 A1 OM....0.........
0050: 03 02 01 02 A2 82 04 CF 04 82 04 CB C3 65 D3 C9 .............e..
0060: 71 75 0C 09 0C 7E E9 C0 CF 98 4F 6C AE B3 D2 A9 qu........Ol....
0070: 27 C6 70 C9 0A 84 5B 18 80 0B D3 94 2A 44 17 A9 '.p...[.....*D..
0080: BE F4 70 E2 67 DD AB 0D 27 A6 58 FD 80 B0 4A FB ..p.g...'.X...J.
0090: FA 5D B7 34 75 FE C0 82 E3 78 51 ED 5D E1 F0 6F .].4u....xQ.]..o
00A0: F8 65 9F B2 31 76 A0 03 F1 8C F1 4B 9F 45 03 63 .e..1v.....K.E.c
00B0: 90 67 55 D8 90 41 A3 26 61 A2 9A A4 13 50 B9 33 .gU..A.&a....P.3
00C0: 58 47 6B 9E D9 AF EB E3 07 54 83 10 C4 DB D4 2D XGk......T.....-
00D0: F9 C4 48 FB 6D BB B9 C8 0C B6 01 DC 3B 84 49 6C ..H.m.......;.Il
00E0: 04 B9 87 AA BB 75 A3 BB 96 14 16 13 74 B8 EF 12 .....u......t...
00F0: 05 B1 1A 3C D4 D8 02 D5 2A CF D2 3F B4 80 DE 75 ...<....*..?...u
0100: 3E 22 78 70 B4 68 B9 D7 1A C8 6B 4A 94 98 C4 FA >"xp.h....kJ....
0110: AF A3 93 31 CC 33 07 37 9F 95 B2 4B 77 3F 56 A1 ...1.3.7...Kw?V.
0120: 5B 14 20 25 E5 82 57 91 9B 62 A9 44 44 42 CE 03 [. %..W..b.DDB..
0130: 59 C3 03 E0 E1 6E 42 F5 2F EC B8 B2 B8 1C 7B 0E Y....nB./.......
0140: 96 6C 09 3B 3E F6 0C B0 2E 25 DA 5A ED CC 64 A9 .l.;>....%.Z..d.
0150: C5 13 98 B2 F8 BC 7B 8B 65 27 69 1D 68 D6 3A E9 ........e'i.h.:.
0160: 9C 6D 93 35 22 17 F8 6C 5F DC 8E A3 62 94 52 F0 .m.5"..l_...b.R.
0170: 14 E9 8B F4 34 DF CD 7C 6F 65 7A 41 44 32 9D 64 ....4...oezAD2.d
0180: 86 0C B2 2F 3C 06 EC 5D F2 38 9F 79 6A 58 20 27 .../<..].8.yjX '
0190: DA B3 0E 5F 67 45 B4 60 6D 0A 88 BC 60 36 F6 42 ..._gE.`m...`6.B
01A0: DB 16 38 3C A6 3F 24 4E 2E 02 DA 8D 70 05 F4 AF ..8<.?$N....p...
01B0: 03 64 68 76 12 10 21 21 13 E9 89 9B 92 4B 37 69 .dhv..!!.....K7i
01C0: 25 23 21 AE C1 C3 F5 1A 32 B1 C1 BC 29 CA 37 EB %#!.....2...).7.
01D0: 15 66 4A CA 3E 22 A5 65 76 95 DC F8 AB E5 19 28 .fJ.>".ev......(
01E0: 61 9C 22 15 80 55 9C 9F 5C AD 6A B0 A9 BA 83 C5 a."..U..\.j.....
01F0: 08 0D 13 15 71 14 5B B4 AF 8A D9 99 46 6E 74 8E ....q.[.....Fnt.
0200: 58 73 E1 1C 66 D2 AF 80 65 C0 AE 1D 20 E8 43 84 Xs..f...e... .C.
0210: F3 0F 4D 7F 4A BA C6 BD 7A 47 44 5B 48 5E A4 7B ..M.J...zGD[H^..
0220: A4 7A E3 7D FB 33 5E D4 09 0A 72 5D 2D 03 85 C7 .z...3^...r]-...
0230: B3 CD D9 A2 B5 99 88 E3 96 6B E9 A2 88 AA FD E9 .........k......
0240: 1C C7 E9 AA BE E3 69 8A E8 15 84 10 8B 68 E9 A2 ......i......h..
0250: BB 50 DC D7 DF 82 F4 46 54 9C FB D6 F8 59 A3 10 .P.....FT....Y..
0260: 5E 2B 7D E2 98 D8 FA 05 58 D4 25 CD 9C 47 A7 FC ^+......X.%..G..
0270: DE 78 07 58 FB 2B B3 4A E9 AB ED E7 24 77 9B 53 .x.X.+.J....$w.S
0280: BD 2E 40 F9 F9 A2 64 5C E4 E5 AA C5 DB C8 C3 1E ..@...d\........
0290: 61 B1 4F E2 B2 83 3D 81 C2 3F 3C BB 21 83 91 0D a.O...=..?<.!...
02A0: 9D B0 F0 C4 1D 09 15 82 33 16 D0 D9 A6 6E 36 79 ........3....n6y
02B0: 1B 75 34 A3 8C F6 DB 1E 69 91 C2 08 18 E6 6B 0B .u4.....i.....k.
02C0: 79 C5 C6 BD 12 56 A8 C4 63 48 53 32 CA 7A C5 95 y....V..cHS2.z..
02D0: 41 1E FD 5C 8D D9 CC C5 16 1B C3 EA 55 D4 15 6A A..\........U..j
02E0: B2 E2 E5 AF 9C 54 4D 11 09 FA 89 96 8E 9C A6 BE .....TM.........
02F0: 6A 70 08 AD F3 88 C3 F0 69 8A EB B7 3B 3F DB B6 jp......i...;?..
0300: E4 BC D6 FB B3 2C 23 59 90 2B 79 F7 A4 A3 75 71 .....,#Y.+y...uq
0310: 78 D2 5E CF B3 05 14 28 D0 D1 48 EB 4F 97 B9 2D x.^....(..H.O..-
0320: D3 73 DF 59 62 AB DB 52 CE 64 88 4B 7F 12 CF C5 .s.Yb..R.d.K....
0330: 84 AD 8E 89 0B 57 FA AE 0C B5 DB 65 D5 F9 FD 42 .....W.....e...B
0340: 70 58 31 B8 07 D4 8F 38 76 FA 7E 8B B6 19 71 EE pX1....8v.....q.
0350: 7D 3C 1D 37 96 2E 6D 41 E6 C5 7B 37 CD 52 2B 81 .<.7..mA...7.R+.
0360: B3 81 CD A9 AE A5 5C E2 32 3D 0F C8 A9 98 5A D9 ......\.2=....Z.
0370: 96 FD 9A FF 6A C6 9E F4 4E 8D 6F 71 9B BC B2 F4 ....j...N.oq....
0380: CD B8 84 B1 CF A0 ED 06 CF 58 67 30 10 51 0D 87 .........Xg0.Q..
0390: 8D 18 44 BF 0A 6B 09 66 C3 9E 31 9C 77 69 77 32 ..D..k.f..1.wiw2
03A0: A0 3C 9B 4F 5F 9C 16 6E 7E 00 D2 2A 05 70 4D 67 .<.O_..n...*.pMg
03B0: C5 06 EC 44 62 D4 50 0A 86 12 3E F1 E9 66 78 E8 ...Db.P...>..fx.
03C0: 85 2F D9 CE A4 D8 A0 87 97 0C E7 8F 06 B7 5F 01 ./............_.
03D0: 83 DB 40 79 08 5E EF 87 83 C2 7A 8F CD C9 48 21 ..@y.^....z...H!
03E0: 17 9D 3A 25 81 22 D1 2F E7 C4 0C 3A 0A 18 1E 8D ..:%."./...:....
03F0: FF B7 69 2F 0E D5 7F BA 67 AE 75 97 6E 48 DE 00 ..i/....g.u.nH..
0400: 4D 2D 4A 87 64 B2 17 0D 4A D8 A1 EB 3D F0 DF 2B M-J.d...J...=..+
0410: D8 AD 17 0F D6 AA 13 4F 42 24 16 59 BA 03 F7 C2 .......OB$.Y....
0420: A6 2A 78 05 F1 C2 3D 1A 7B 0B 9C D6 8D 11 20 DA .*x...=....... .
0430: D3 8C EF 4A 23 73 35 55 30 0E 95 04 C8 90 22 F6 ...J#s5U0.....".
0440: 8D C0 90 48 41 AB FE 43 1D 99 61 2B C1 47 BA 95 ...HA..C..a+.G..
0450: 01 F7 BB 37 54 49 DE CE F2 2F 3F DE AB A8 04 D3 ...7TI.../?.....
0460: A9 F5 7A 2E 42 36 CA 04 48 0A 5F 3B BF BB F2 99 ..z.B6..H._;....
0470: F5 D6 56 9F 78 77 6F 0E 01 66 F8 C3 F5 13 10 76 ..V.xwo..f.....v
0480: D5 E7 1F 3D FA 8C C8 03 BB 16 A6 78 35 67 37 B7 ...=.......x5g7.
0490: 5E 0A A7 4A F1 A9 42 B6 EA 96 94 1D 72 61 75 E6 ^..J..B.....rau.
04A0: 10 97 BD AB 1F 40 62 AF 3D B4 0E BE D4 C6 92 36 .....@b.=......6
04B0: B4 78 2E C8 63 B0 AC FD 68 83 45 DF 09 EA 8B 0F .x..c...h.E.....
04C0: 7A 0D 6A 40 53 6B 3C B0 78 F4 7B 6E 61 1E DD B5 z.j@Sk<.x..na...
04D0: 9A 40 D0 FE B0 7B B0 F1 48 86 74 EF 03 02 65 F6 .@......H.t...e.
04E0: 8D 98 3B B3 53 7C 22 B9 AB 51 E1 FA 05 7D 05 E1 ..;.S."..Q......
04F0: 27 AF 19 C4 39 FE 87 21 14 A0 95 5B F9 89 DB BE '...9..!...[....
0500: E7 D8 FC FA 8C D4 B4 5A 54 53 93 B4 1E 47 80 46 .......ZTS...G.F
0510: 1A 3E AD D6 66 A6 68 8E E8 29 64 DF A8 8F 48 0F .>..f.h..)d...H.
0520: A3 E7 33 52 67 34 05 ..3Rg4.

Client Principal = mrice@PUTNAMINV.COM
Server Principal = krbtgt/PUTNAMINV.COM@PUTNAMINV.COM
Session Key = EncryptionKey: keyType=23 keyBytes (hex dump)=
0000: A5 E3 07 E9 7F 5D E2 9E B1 BA 12 1B A0 8E 93 37 .....].........7

Forwardable Ticket true
Forwarded Ticket false
Proxiable Ticket false
Proxy Ticket false
Postdated Ticket false
Renewable Ticket true
Initial Ticket true
Auth Time = Wed Jul 27 08:37:02 EDT 2016
Start Time = Wed Jul 27 08:36:55 EDT 2016
End Time = Wed Jul 27 18:37:02 EDT 2016
Renew Till = Thu Jul 28 08:36:55 EDT 2016
Client Addresses Null
16/07/27 10:28:54 DEBUG security.UserGroupInformation: Current time is 1469629734340
16/07/27 10:28:54 DEBUG security.UserGroupInformation: Next refresh is 1469651820600
16/07/27 10:28:54 DEBUG hdfs.BlockReaderLocal: dfs.client.use.legacy.blockreader.local = false
16/07/27 10:28:54 DEBUG hdfs.BlockReaderLocal: dfs.client.read.shortcircuit = false
16/07/27 10:28:54 DEBUG hdfs.BlockReaderLocal: dfs.client.domain.socket.data.traffic = false
16/07/27 10:28:54 DEBUG hdfs.BlockReaderLocal: dfs.domain.socket.path = /var/run/hdfs-sockets/dn
16/07/27 10:28:54 DEBUG retry.RetryUtils: multipleLinearRandomRetry = null
16/07/27 10:28:54 DEBUG ipc.Server: rpcKind=RPC_PROTOCOL_BUFFER, rpcRequestWrapperClass=class org.apache.hadoop.ipc .ProtobufRpcEngine$RpcRequestWrapper, rpcInvoker=org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker@ 7fe6640c
16/07/27 10:28:54 DEBUG ipc.Client: getting client out of cache: org.apache.hadoop.ipc.Client@f67264e
16/07/27 10:28:54 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...
16/07/27 10:28:54 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
16/07/27 10:28:54 DEBUG unix.DomainSocketWatcher: org.apache.hadoop.net.unix.DomainSocketWatcher$2@7a85273: startin g with interruptCheckPeriodMs = 60000
16/07/27 10:28:54 DEBUG util.PerformanceAdvisory: Both short-circuit local reads and UNIX domain socket are disable d.
16/07/27 10:28:54 DEBUG sasl.DataTransferSaslUtil: DataTransferProtocol using SaslPropertiesResolver, configured QO P dfs.data.transfer.protection = privacy, configured class dfs.data.transfer.saslproperties.resolver.class = class org.apache.hadoop.security.SaslPropertiesResolver
16/07/27 10:28:54 DEBUG ipc.Client: The ping interval is 60000 ms.
16/07/27 10:28:54 DEBUG ipc.Client: Connecting to antelope1.putnaminv.com/172.20.164.89:8020
16/07/27 10:28:54 DEBUG security.UserGroupInformation: PrivilegedAction as:mrice@PUTNAMINV.COM (auth:KERBEROS) from :org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:720)
16/07/27 10:28:54 DEBUG security.SaslRpcClient: Sending sasl message state: NEGOTIATE

16/07/27 10:28:54 DEBUG security.SaslRpcClient: Received SASL message state: NEGOTIATE
auths {
method: "TOKEN"
mechanism: "DIGEST-MD5"
protocol: ""
serverId: "default"
challenge: "realm=\"default\",nonce=\"2vqGs0+R6dAlI0Z3dwGfXeyk3jM+rF6zLYy76xIU\",qop=\"auth-conf\",charset=utf-8, cipher=\"3des,rc4,des,rc4-56,rc4-40\",algorithm=md5-sess"
}
auths {
method: "KERBEROS"
mechanism: "GSSAPI"
protocol: "hdfs"
serverId: "antelope1.putnaminv.com"
}

16/07/27 10:28:54 DEBUG security.SaslRpcClient: Get token info proto:interface org.apache.hadoop.hdfs.protocolPB.Cl ientNamenodeProtocolPB info:@org.apache.hadoop.security.token.TokenInfo(value=class org.apache.hadoop.hdfs.security .token.delegation.DelegationTokenSelector)
16/07/27 10:28:54 DEBUG security.SaslRpcClient: Get kerberos info proto:interface org.apache.hadoop.hdfs.protocolPB .ClientNamenodeProtocolPB info:@org.apache.hadoop.security.KerberosInfo(clientPrincipal=, serverPrincipal=dfs.namen ode.kerberos.principal)
16/07/27 10:28:54 DEBUG security.SaslRpcClient: RPC Server's Kerberos principal name for protocol=org.apache.hadoop .hdfs.protocolPB.ClientNamenodeProtocolPB is hdfs/antelope1.putnaminv.com@PUTNAMINV.COM
16/07/27 10:28:54 DEBUG security.SaslRpcClient: Creating SASL GSSAPI(KERBEROS) client to authenticate to service a t antelope1.putnaminv.com
16/07/27 10:28:54 DEBUG security.SaslRpcClient: Use KERBEROS authentication for protocol ClientNamenodeProtocolPB
16/07/27 10:28:54 DEBUG security.SaslRpcClient: Sending sasl message state: INITIATE
token: "`\202\006\a\006\t*\206H\206\367\022\001\002\002\001\000n\202\005\3660\202\005\362\240\003\002\001\005\241\0 03\002\001\016\242\a\003\005\000 \000\000\000\243\202\005\034a\202\005\0300\202\005\024\240\003\002\001\005\241\017 \033\rPUTNAMINV.COM\242*0(\240\003\002\001\000\241!0\037\033\004hdfs\033\027antelope1.putnaminv.com\243\202\004\316 0\202\004\312\240\003\002\001\027\241\003\002\001\001\242\202\004\274\004\202\004\270\006\030\310\032\274\316P \373 \232\245|\206\275\270\371\307\333\235@\001$hMZ\216u\f\307u\207\365\224u\265\347}\004zt0\366\005\317\241\234Ti\021\2 14\274+\234\\|\0010\314U\313\002}\232\017\231;\333\356\034\000\332\031\301o\344\236\357])\276\253)\354z\333\360g\b= \003\375\'\020\330y>\260@\342\005\213\253\"\204\367\226\br\3637\034\262u\a\335\335\2407\217\3616\027%A\346\006\247\ 372|\256\300\032\371A\t\357\223Iw\250(\236\366z\244\242\n&#$\344\aF\\@\203\351\327%\025\271)\301\261 \035\201\261Z\ 223\320\337\362\346\027\'-5\322\201\3765\305?\267\224n\037F\"\037G\332\275\231\262\327[\271i\300\331\223\210\331\25 23\241\305t\274\214\270\252\027\251\221e\027\342\316\236\361\016\030>\313\204\253\2077\256Iw\025r\375$\317N}\230\01 6&p\326H\377\357N\346\r\204\v\351\275]\214A>\371\222\267\273\245\005\212}\253\206L\246J\315\006\254\202\335\277k[i\ 305+8\242X`\030+\3575\204\306i\234}\247K<c\024\234n\377\274\201\345\221\260\365b\307G\353)\241K\206S\002\002\206#i\ 356\202\232\313\273U\000\334B\332\306\255\2639\207\312\361O:\372Z\326M\211\320\255\370L\237\362\371\302\255\211\244 X\225\271\275\263\355\224*\236\\0:W8\341\020\353\275\201_83D\355\266w|@5iO\253\250\3312\371\225d\267 \2360\031f\367 \0348\214L\ts\367\314\005\330\265\335R\325\365,|\376\\\307\347X\2771\302e\354w\216o\367\001\216M{\002\344\aPO4\315\ 257\367\233\251\365\\\024$\370\r5\353X.\231iyl\037\365n\247f\357\301K\333\351 \306\027\033\271>QD\300\v3.\244F\330K \256\005\374\265\335\340db\230\307FI\025e\237H\023\210\320\000\270)pi2\325\246)ID\206\235\266[\275\016T\260T\277d{\ 220T\374\271\342A\301\333:\326\344\226\352M\344\327U>\324^)\232^q\370h\337\302\376\314\275\344\321_\267\332!\376\36 1\247\212\023\261Y\372\314\231\027\331\374\217x\312z\356\255\360$.\331\245\265Qcx\317\\\240_\344\322\264\204D\344}< \252\'\211\233\373\017\303(\316\332\352\0279`\a\333\275\334n\230\372\266\314=\235\324&\213\272\330-C\247\"\323\v\36 23-\"v\327\333h\240\336\215be\262\266\224\002\336\035\276$\346\016\245i\233\033V!.\017\331sd\004\267\204Ri\36\037\3 56\260\310\002\310\022%>8_# \257\276\212\000a\372F\224\326G\221cz\222\247\277\3743\227\241\2708\347\337\314M\002\25 2Ou\266\251o*\027\256+\224\350\261\327c\253hoh\232y\227vT\2031\031\350j\213J\224\346\315\344\234\aI\264;M\304+S\347 \265\316\026Z\332C\264U\2053e1\224\271\370ux\335\234\367P\232\367AMdf\271\"#\240^\"{\037\327di\225\275\325\2656\312 \263\215\211J\240\374\370\260\004|q\216\264\f\253\340\005j\217\215\376f\224R\241I\242\036\034\bK\254H\216]L\235\r\3 56\332\275\357\276\271\242\020\327A\'\223S\260\233s\264T\360*\030\254\375\246\374|\333e(|~\360\255\205XS\261C\310\2 67\223\352\202\a\325\274h\261\253D\227\213[\024V\254\327G\204\ba\306\266\234\221\257\265IL\255c\214\231\363{f\205\3 76\022\253\347\230\214r\210\244\225G\0063\310\313;\3016\254\266\b!y\032 0\353\2377\2549\017\254\r\227\205\242\341O\ 2741_\366\375P\212\276\r\257\323\234g\236\321\0035\236\036\f\312\037\033\025\241\236\254t\f\016V\3415\331|\f\224\34 6\"\0000\032p\324*|`\023p\321\016\226Z\266\314oQ\211I\344B\205a\207\266\246\321I|S\\T!\024\353:\032U\346\274\234M\3 46lV\274\361_\022\331,\v_\355?w\271\232\373:\372\002\372\215\246\352\325\313Yi\206/9\312\365\335\031\004d\360\005j\ 230k\346\261\253\233A\302$:\356\035\3378\337\035\334\344\350\362\321\276\246\331\331\376\321\243\277\016\3374\020\3 42E\316\002&7,\345c\341`\034\032\213~.\263\246\341Z\254\255q\350\240\205}\345R\354/\314\r\330\314\242k\205\247\257\ 002\034\376\001\336\354d\020]\'\030\237\365\2518\203\375\310\247$2\355]\211s\313\220\354\354\t\363\2027@/>\243\302\ 330\244\201\2740\201\271\240\003\002\001\027\242\201\261\004\201\256:\2658nM\350\333t\304F\264\361}\026\033\332A\22 7TD-0\262\233\f\246\034`B\v\310\021h\"t\2673\345\2411\230=P\275\"{\327\035\240\032B\252\003d\365N\333\324\364\342\2 765\001\026\225\005\022JAO\312\201\302\264^\312K\225\3520]\241-9n\amb\374\333-KV\027\240a\353\366&%\304\250X\254\31 7k\201\231\025\215\ns\252\004(\324^\036\no\034\365~\217\261\025\274M:M|\276\376!\245\317&\353M)/>\376\250b:\r?\262a \276Z\004\000\345I\330p\374B}\207\aB\366 \223{<\331\305\017`B"
auths {
method: "KERBEROS"
mechanism: "GSSAPI"
protocol: "hdfs"
serverId: "antelope1.putnaminv.com"
}

16/07/27 10:28:54 DEBUG security.SaslRpcClient: Received SASL message state: CHALLENGE
token: "`f\006\t*\206H\206\367\022\001\002\002\002\000oW0U\240\003\002\001\005\241\003\002\001\017\242I0G\240\003\0 02\001\027\242@\004>T\330\005\220a\022\321\257R\200\026\273\\\3772k?#\230\361>^\236|7\203\025Kcj\344\253Aa \005:\31 6\017\027\020\205W\355\300\3339\202\234\030\344\324e\205\215`n\306^\036x\337"

16/07/27 10:28:54 DEBUG security.SaslRpcClient: Sending sasl message state: RESPONSE
token: ""

16/07/27 10:28:54 DEBUG security.SaslRpcClient: Received SASL message state: CHALLENGE
token: "`0\006\t*\206H\206\367\022\001\002\002\002\001\021\000\377\377\377\377\244\\\367Iq\ba\341v\212\324\356\031\ 215\207\322]\250\200\333\200\374t\357\004\001\000\000\001"

16/07/27 10:28:54 DEBUG security.SaslRpcClient: Sending sasl message state: RESPONSE
token: "`0\006\t*\206H\206\367\022\001\002\002\002\001\021\000\377\377\377\377G\302\366)\373\350\"\265\2743\n\266d? \214\255\001\021\234\360\264&\200\333\004\001\000\000\001"

16/07/27 10:28:54 DEBUG security.SaslRpcClient: Received SASL message state: SUCCESS

16/07/27 10:28:54 DEBUG ipc.Client: Negotiated QOP is :auth-conf
16/07/27 10:28:54 DEBUG ipc.Client: IPC Client (899473761) connection to antelope1.putnaminv.com/172.20.164.89:8020 from mrice@PUTNAMINV.COM: starting, having connections 1
16/07/27 10:28:54 DEBUG security.SaslRpcClient: reading next wrapped RPC packet
16/07/27 10:28:54 DEBUG ipc.Client: IPC Client (899473761) connection to antelope1.putnaminv.com/172.20.164.89:8020 from mrice@PUTNAMINV.COM sending #0
16/07/27 10:28:54 DEBUG security.SaslRpcClient: wrapping token of length:209
16/07/27 10:28:54 DEBUG security.SaslRpcClient: unwrapping token of length:132
16/07/27 10:28:54 DEBUG ipc.Client: IPC Client (899473761) connection to antelope1.putnaminv.com/172.20.164.89:8020 from mrice@PUTNAMINV.COM got value #0
16/07/27 10:28:54 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 159ms
16/07/27 10:28:54 DEBUG security.SaslRpcClient: reading next wrapped RPC packet
16/07/27 10:28:54 DEBUG ipc.Client: IPC Client (899473761) connection to antelope1.putnaminv.com/172.20.164.89:8020 from mrice@PUTNAMINV.COM sending #1
16/07/27 10:28:54 DEBUG security.SaslRpcClient: wrapping token of length:117
16/07/27 10:28:54 DEBUG security.SaslRpcClient: unwrapping token of length:78
16/07/27 10:28:54 DEBUG ipc.Client: IPC Client (899473761) connection to antelope1.putnaminv.com/172.20.164.89:8020 from mrice@PUTNAMINV.COM got value #1
16/07/27 10:28:54 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 8ms
16/07/27 10:28:54 DEBUG security.SaslRpcClient: reading next wrapped RPC packet
16/07/27 10:28:54 DEBUG ipc.Client: IPC Client (899473761) connection to antelope1.putnaminv.com/172.20.164.89:8020 from mrice@PUTNAMINV.COM sending #2
16/07/27 10:28:54 DEBUG security.SaslRpcClient: wrapping token of length:127
16/07/27 10:28:54 DEBUG security.SaslRpcClient: unwrapping token of length:78
16/07/27 10:28:54 DEBUG ipc.Client: IPC Client (899473761) connection to antelope1.putnaminv.com/172.20.164.89:8020 from mrice@PUTNAMINV.COM got value #2
16/07/27 10:28:54 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 6ms
16/07/27 10:28:54 DEBUG hdfs.DFSClient: /data/fi/testkb.txt._COPYING_: masked=rw-r--r--
16/07/27 10:28:55 DEBUG security.SaslRpcClient: reading next wrapped RPC packet
16/07/27 10:28:55 DEBUG ipc.Client: IPC Client (899473761) connection to antelope1.putnaminv.com/172.20.164.89:8020 from mrice@PUTNAMINV.COM sending #3
16/07/27 10:28:55 DEBUG security.SaslRpcClient: wrapping token of length:177
16/07/27 10:28:55 DEBUG security.SaslRpcClient: unwrapping token of length:7093
16/07/27 10:28:55 DEBUG ipc.Client: IPC Client (899473761) connection to antelope1.putnaminv.com/172.20.164.89:8020 from mrice@PUTNAMINV.COM got value #3
16/07/27 10:28:55 DEBUG security.SaslRpcClient: reading next wrapped RPC packet
16/07/27 10:28:55 DEBUG ipc.Client: IPC Client (899473761) connection to antelope1.putnaminv.com/172.20.164.89:8020 from mrice@PUTNAMINV.COM sending #4
16/07/27 10:28:55 DEBUG security.SaslRpcClient: wrapping token of length:127
16/07/27 10:28:55 DEBUG security.SaslRpcClient: unwrapping token of length:78
16/07/27 10:28:55 DEBUG ipc.Client: IPC Client (899473761) connection to antelope1.putnaminv.com/172.20.164.89:8020 from mrice@PUTNAMINV.COM got value #4
16/07/27 10:28:55 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 6ms
put: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.hadoop.security.authentication.client .AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
16/07/27 10:28:55 DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@f67264e
16/07/27 10:28:55 DEBUG ipc.Client: removing client from cache: org.apache.hadoop.ipc.Client@f67264e
16/07/27 10:28:55 DEBUG ipc.Client: stopping actual client because no more references remain: org.apache.hadoop.ipc .Client@f67264e
16/07/27 10:28:55 DEBUG ipc.Client: Stopping client
16/07/27 10:28:55 DEBUG ipc.Client: IPC Client (899473761) connection to antelope1.putnaminv.com/172.20.164.89:8020 from mrice@PUTNAMINV.COM: closed
16/07/27 10:28:55 DEBUG ipc.Client: IPC Client (899473761) connection to antelope1.putnaminv.com/172.20.164.89:8020 from mrice@PUTNAMINV.COM: stopped, remaining connections 0

[#6 · Accepted solution]

Thanks I'm certain you're hitting the same error as HADOOP-12559, given the
AuthenticationException is coming at write-time, and from the client
package that's used for HTTP work - indicating that the NN is unable to
contact the KMS.

You'll also likely observe this error only much after a NameNode restart
period (but that it works immediately after NN restart), and that it may go
away after one day or so, only to return again, which is inline with
HADOOP-12559's
behaviour within the NameNode.

The bug-fix update of 5.5.x or any minor upgrade to the newer releases
should solve this up.

[#7]

Thank you for all your help. We will attempt to upgrade this afternoon. If it solves our problem I will come back and mark your answer as correct.

Thank you so much

[#8]

Thank you for the update, please keep us posted.

[#9]

Just wanted to come back and confirm that updating to 5.5.4 did in fact solve the issue. Thank you for your help!

[#10]

I'm running into a similar problem, but only in regards to Data At Rest Encryption (DARE).  All other HDFS operations work perpetually and tickets renew as needed.

With DARE, everything seems to be set up correctly and works transparently through our app for about an hour, then all we get are "Execution of 'abc.csv' failed. Error details: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)" errors.

I thought this might be related to HADOOP-12559 and/or HADOOP-10786 but we upgraded our test environment to CDH 5.8.5 and the problem persists.

Manual kinit does not seem to help (and I see valid tickets for our app and for hdfs).

Restarting our app seems to reset everything, but I can find no explicit kerberos login that would account for that.

My best guess is that there is some principal (possibly HTTP/ourserver.com@REALM.com ?) that needs to renew so that it can validate against the KMS, but doesn't.  I tried manually kinit-ing the HTTP principal on the cm server, but to no avail.

An alternate possibility is that something else is failing and the tgt error is a red herring, but the timeout aspect inclines me to think it's a kerberos issue.

Any help appreciated!!