question HDFS router based federation daemon failing to start in Support Questions

HDFS router based federation daemon failing to start

Hadoop16 — Sat, 12 Apr 2025 01:38:26 GMT

Unable to start dfsrouter daemon. Cluster is kerberized and using hadoop 3.3.4. Trying to implement hdfs router based federation. Configured hdfs-rbf-site.xml with required properties.

It is a working cluster. Zookeeper quorum is already configured in core-site.xml.

Below is the error from log. "Zookeeper connection string cannot be null".

ERROR router.FederationUtil - Could not instantiate: ZKDelegationTokenSecretManagerImpl
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hdfs.server.federation.router.FederationUtil.newInstance(FederationUtil.java:164)
at org.apache.hadoop.hdfs.server.federation.router.FederationUtil.newSecretManager(FederationUtil.java:224)
at org.apache.hadoop.hdfs.server.federation.router.security.RouterSecurityManager.<init>(RouterSecurityManager.java:60)
at org.apache.hadoop.hdfs.server.federation.router.RouterRpcServer.<init>(RouterRpcServer.java:293)
at org.apache.hadoop.hdfs.server.federation.router.Router.createRpcServer(Router.java:391)
at org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:188)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69)
Caused by: java.lang.NullPointerException: Zookeeper connection string cannot be null
at org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:899)
at org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.<init>(ZKDelegationTokenSecretManager.java:168)
at org.apache.hadoop.hdfs.server.federation.router.security.token.ZKDelegationTokenSecretManagerImpl.<init>(ZKDelegationTokenSecretManagerImpl.java:42)
... 12 more

Any help is appreciated!

Re: HDFS router based federation daemon failing to start

RAGHUY — Tue, 19 Aug 2025 08:56:59 GMT

Hi, @Hadoop16

The error Zookeeper connection string cannot be null means the Router process is expecting ZooKeeper configs for token management but isn’t finding them. Even if you already have a ZooKeeper quorum set in core-site.xml, the Router Federation requires its own configs in hdfs-rbf-site.xml. Specifically, you need to set hadoop.kms.authentication.zk-dt-secret-manager.zkConnectionString (or in some versions hadoop.security.token.service.use_ip + hadoop.zk.address) depending on your setup. Please double-check that hdfs-rbf-site.xml contains the federation and router related properties, including ZooKeeper connection string and Kerberos settings. Ensure the file is deployed to all Router nodes and included in the classpath. Also verify that the Router service user has Kerberos credentials and permissions to connect to ZooKeeper. Once the ZK connection string is set properly, the Router daemon should start without this error.

Re: HDFS router based federation daemon failing to start

Hadoop16 — Tue, 19 Aug 2025 20:48:18 GMT

@RAGHUY Thank you! I figured that later but Router starting to fail with below error.I have the jaas.conf in place. Any help on this is appreciated.

ERROR client.ZooKeeperSaslClient - SASL authentication failed using login context 'ZKDelegationTokenSecretManagerClient' with exception: {}

javax.security.sasl.SaslException: Error in authenticating with a Zookeeper Quorum member: the quorum member's saslToken is null.

at org.apache.zookeeper.client.ZooKeeperSaslClient.createSaslToken(ZooKeeperSaslClient.java:312)

at org.apache.zookeeper.client.ZooKeeperSaslClient.respondToServer(ZooKeeperSaslClient.java:275)

at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:882)

at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:101)

at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)

at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)

2025-08-19 20:45:37,097 ERROR curator.ConnectionState - Authentication failed

2025-08-19 20:45:37,098 INFO zookeeper.ClientCnxn - Unable to read additional data from server sessionid 0x1088d05c6550015, likely server has closed socket, closing socket connection and attempting reconnect

2025-08-19 20:45:37,098 INFO zookeeper.ClientCnxn - EventThread shut down for session: 0x1088d05c6550015

2025-08-19 20:45:37,212 ERROR imps.CuratorFrameworkImpl - Ensure path threw exception

org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hdfs-router-tokens