Member since
03-15-2024
16
Posts
8
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2180 | 05-10-2024 02:13 AM | |
| 4206 | 03-29-2024 10:29 PM |
08-19-2025
07:10 AM
@RAGHUY Thank you! I figured that later but Router starting to fail with below error.I have the jaas.conf in place. Any help on this is appreciated. ERROR client.ZooKeeperSaslClient - SASL authentication failed using login context 'ZKDelegationTokenSecretManagerClient' with exception: {} javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null. at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312) at org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275) at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:101) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) 2025-08-19 20:45:37,097 ERROR curator.ConnectionState - Authentication failed 2025-08-19 20:45:37,098 INFO zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x1088d05c6550015, likely server has closed socket, closing socket connection and attempting reconnect 2025-08-19 20:45:37,098 INFO zookeeper.ClientCnxn - EventThread shut down for session: 0x1088d05c6550015 2025-08-19 20:45:37,212 ERROR imps.CuratorFrameworkImpl - Ensure path threw exception org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hdfs-router-tokens
... View more
05-16-2025
09:07 AM
Hello, I'm trying to insert data into a hive table configured with hdfs router based federation as table location Location: | hdfs://router_host:8888/router/router.db/router_test_table Cluster is kebrerized and all components including Hive and RBF are working as expected except this specific use case. The hive table insert job is failing with kerberos error when using RBF as the table location Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: DestHost:destPort router_host:8888 , LocalHost:localPort datanode_host. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:639) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:563) ... 17 more Caused by: java.io.IOException: DestHost:destPort router_host:8888 , LocalHost:localPort datanode_host. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:913) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:888) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1616) at org.apache.hadoop.ipc.Client.call(Client.java:1558) at org.apache.hadoop.ipc.Client.call(Client.java:1455) at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:242)
... View more
Labels:
- Labels:
-
Apache Hive
-
HDFS
-
Kerberos
04-11-2025
06:38 PM
Unable to start dfsrouter daemon. Cluster is kerberized and using hadoop 3.3.4. Trying to implement hdfs router based federation. Configured hdfs-rbf-site.xml with required properties. It is a working cluster. Zookeeper quorum is already configured in core-site.xml. Below is the error from log. "Zookeeper connection string cannot be null". ERROR router.FederationUtil - Could not instantiate: ZKDelegationTokenSecretManagerImpl java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hdfs.server.federation.router.FederationUtil.newInstance(FederationUtil.java:164) at org.apache.hadoop.hdfs.server.federation.router.FederationUtil.newSecretManager(FederationUtil.java:224) at org.apache.hadoop.hdfs.server.federation.router.security.RouterSecurityManager.<init>(RouterSecurityManager.java:60) at org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.<init>(RouterRpcServer.java:293) at org.apache.hadoop.hdfs.server.federation.router.Router.createRpcServer(Router.java:391) at org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:188) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69) Caused by: java.lang.NullPointerException: Zookeeper connection string cannot be null at org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:899) at org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.<init>(ZKDelegationTokenSecretManager.java:168) at org.apache.hadoop.hdfs.server.federation.router.security.token.ZKDelegationTokenSecretManagerImpl.<init>(ZKDelegationTokenSecretManagerImpl.java:42) ... 12 more Any help is appreciated!
... View more
Labels:
- Labels:
-
Apache Zookeeper
-
HDFS
-
Kerberos
-
Security
03-31-2025
07:12 PM
@upadhyayk04 Thank you! I tried with Kerberos enabled on Ranger and Kafka but still policies are downloading fine but not becoming active. I could see below error in Kafka log. DEBUG Failed to get groups for user ANONYMOUS (org.apache.hadoop.security.UserGroupInformation) java.io.IOException: No groups found for user ANONYMOUS at org.apache.hadoop.security.Groups.noGroupsForUser(Groups.java:200)
... View more
03-30-2025
02:17 PM
Kafka policies created in Ranger are getting downloaded but not becoming active. Using Apache Ranger 2.6 and Apache Kafka 3.6. Couldn't find any specific errors related to this issue. Ranger and Kafka are configured with LDAP and no kerberos. What could be the possible issue? Any help is appreciated! Ranger policies for HDFS and Hive works fine. Below are the ldap and ranger configs in Kafka authorizer.class.name=org.apache.ranger.authorization.kafka.authorizer.RangerKafkaAuthorizer sasl.enabled.mechanisms=PLAIN listener.name.sasl_plaintext.sasl.enabled.mechanisms=PLAIN listener.name.sasl_plaintext.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required; listener.name.sasl_plaintext.plain.sasl.server.callback.handler.class=io.confluent.security.auth.provider.ldap.LdapAuthenticateCallbackHandler ldap.java.naming.provider.url=ldap://<ldap_host>:389 ldap.java.naming.security.authentication=simple ldap.java.naming.security.principal=CN=<bind_user>,OU=Service_Accounts,DC=hadoop,DC=hdp,DC=com ldap.java.naming.security.credentials= ldap.user.name.attribute=sAMAccountName ldap.user.object.class=user ldap.user.search.base=OU=User_Accounts,DC=hadoop,DC=hdp,DC=com;OU=Service_Accounts,DC=hadoop,DC=hdp,DC=com #server properties ldap.java.naming.provider.url=ldap://<ldap_host>:389 ldap.java.naming.security.authentication=simple ldap.java.naming.security.principal=CN=<bind_dn>,OU=Service_Accounts,DC=hadoop,DC=hdp,DC=com ldap.java.naming.security.credentials= ldap.search.mode=GROUPS ldap.user.search.base=OU=User_Accounts,DC=hadoop,DC=hdp,DC=com;OU=Service_Accounts,DC=hadoop,DC=hdp,DC=com ldap.user.object.class=user ldap.user.name.attribute=sAMAccountName ldap.group.search.base=OU=Groups,DC=hadoop,DC=hdp,DC=com ldap.group.object.class=group ldap.group.name.attribute=cn ldap.group.member.attribute=member
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Ranger
07-16-2024
09:29 AM
I have configure knox in host knox111.abc.hadoop.dc.corp.com and namenodes in nn01.abc.hadoop.dc.corp.com, nn02.abc.hadoop.dc.corp.com. The knoxsso redirection works fine when directly configuring NN with individual Knox host but when using VIP that is configured as knox-hadoop-def.corp.com the redirection to NN doesn't work. I have configured below whitelist setting but still redirection fails. VIP is configured with CA certs and individual knox hosts are configured with self-signed. gateway.dispatch.whitelist = .* knoxsso.redirect.whitelist.regex = .* knoxsso.redirect.whitelist.regex = corp.com Any help is appreciated!
... View more
Labels:
- Labels:
-
Apache Knox
05-10-2024
11:37 PM
1 Kudo
Hello @Scharan From the debug log I think the issue is when Knoxsso is redirecting to NN UI, it is sending user as anonymous. AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(133)) - PrimaryPrincipal: anonymous Do you know what configs at hdfs or Knox could help here?
... View more
05-10-2024
07:36 AM
1 Kudo
Hello @Scharan Below are the entries populated when reaching HDFSUI via knoxsso 2024-05-10 14:32:54,143 INFO knox.gateway (AclsAuthorizationFilter.java:init(72)) - Initializing AclsAuthz Provider for: knoxauth 2024-05-10 14:32:54,143 INFO knox.gateway (AclParser.java:parseAcls(50)) - ACLs found for: knoxauth 2024-05-10 14:33:04,139 INFO knox.gateway (KnoxLdapRealm.java:getUserDn(721)) - Computed userDn: CN=lastname\, firstname,OU=XXXX,OU=XXXXX,DC=XXX,DC=XXX,DC=com using ldapSearch for principal: userid 2024-05-10 14:33:04,790 INFO knox.gateway (AclsAuthorizationFilter.java:init(72)) - Initializing AclsAuthz Provider for: KNOXSSO 2024-05-10 14:33:04,790 INFO knox.gateway (AclParser.java:parseAcls(50)) - ACLs found for: KNOXSSO 2024-05-10 14:33:06,030 INFO knox.gateway (CookieUtils.java:getCookiesForName(46)) - Unable to find cookie with name: original-url 2024-05-10 14:33:06,095 INFO service.knoxsso (WebSSOResource.java:addJWTHadoopCookie(386)) - JWT cookie successfully added. 2024-05-10 14:33:06,095 INFO service.knoxsso (WebSSOResource.java:getAuthenticationToken(278)) - About to redirect to original URL: http://NN_host50070/index.html
... View more
05-10-2024
02:13 AM
I was able to resolve the "Invalid redirect" by adding knoxsso.redirect.whitelist.regex but when I enter AD credentials in the KnoxSSO page it keeps redirecting to the same login page. I could see below msgs in gateway.log 2024-05-10 09:04:16,722 DEBUG knox.gateway (AclsAuthorizationFilter.java:doFilter(105)) - Access Granted: true 2024-05-10 09:04:16,760 DEBUG knox.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /images/loading.gif 2024-05-10 09:04:16,761 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(133)) - PrimaryPrincipal: anonymous 2024-05-10 09:04:16,761 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(142)) - PrimaryPrincipal has access: true 2024-05-10 09:04:16,761 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(158)) - Remote IP Address: 2024-05-10 09:04:16,761 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(160)) - Remote IP Address has access: true 2024-05-10 09:04:16,762 DEBUG knox.gateway (AclsAuthorizationFilter.java:doFilter(105)) - Access Granted: true 2024-05-10 09:04:16,795 DEBUG knox.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /redirecting.jsp 2024-05-10 09:04:16,796 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(133)) - PrimaryPrincipal: anonymous 2024-05-10 09:04:16,796 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(142)) - PrimaryPrincipal has access: true 2024-05-10 09:04:16,797 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(158)) - Remote IP Address: 2024-05-10 09:04:16,797 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(160)) - Remote IP Address has access: true 2024-05-10 09:04:16,797 DEBUG knox.gateway (AclsAuthorizationFilter.java:doFilter(105)) - Access Granted: true 2024-05-10 09:04:20,773 DEBUG knox.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /images/loading.gif 2024-05-10 09:04:20,774 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(133)) - PrimaryPrincipal: anonymous 2024-05-10 09:04:20,775 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(142)) - PrimaryPrincipal has access: true 2024-05-10 09:04:20,775 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(158)) - Remote IP Address: 2024-05-10 09:04:20,775 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(160)) - Remote IP Address has access: true 2024-05-10 09:04:20,775 DEBUG knox.gateway (AclsAuthorizationFilter.java:doFilter(105)) - Access Granted: true 2024-05-10 09:04:20,916 DEBUG knox.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /api/v1/websso 2024-05-10 09:04:20,943 DEBUG knox.gateway (GatewayFilter.java:doFilter(116)) - Received request: GET /login.html 2024-05-10 09:04:20,944 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(133)) - PrimaryPrincipal: anonymous 2024-05-10 09:04:20,944 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(142)) - PrimaryPrincipal has access: true 2024-05-10 09:04:20,945 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(158)) - Remote IP Address: 2024-05-10 09:04:20,945 DEBUG knox.gateway (AclsAuthorizationFilter.java:enforceAclAuthorizationPolicy(160)) - Remote IP Address has access: true 2024-05-10 09:04:20,945 DEBUG knox.gateway (AclsAuthorizationFilter.java:doFilter(105)) - Access Granted: true
... View more
05-09-2024
02:30 PM
1 Kudo
Configured Knoxsso without Ambari. Knoxsso is configured with Shiro provider and updated the core-site.xml with below configs hadoop.http.authentication.type hadoop.http.authentication.authentication.provider.url hadoop.http.authentication.public.key.pem Followed: https://knox.apache.org/books/knox-1-6-0/user-guide.html#KnoxSSO+Setup+and+Configuration After restart, NN UI is redirecting to KnoxSSO and after entering the AD credentials it is throwing below error in the UI. The redirect to originalUrl looks valid from Knoxsso url. ERROR Invalid Redirect: Possible Phishing Attempt Any help is appreciated!
... View more
Labels:
- Labels:
-
Apache Knox
-
HDFS