@Hadoop16 FYI
➤ This error occurs because of a token delegation gap between Hive and the HDFS Router.
In a Kerberized cluster, when Hive (running on a DataNode/Compute node via Tez or MapReduce) attempts to write to HDFS, it needs a Delegation Token. When you use an HDFS Router address, Hive must be explicitly told to obtain a token specifically for the Router's service principal, which may be different from the backend NameNodes.
➤ The Root Cause
The error Client cannot authenticate via:[TOKEN, KERBEROS] at the FileSinkOperator stage indicates that the tasks running on your worker nodes do not have a valid token to "speak" to the Router at router_host:8888.
When Hive plans the job, it usually fetches tokens for the default filesystem. If your fs.defaultFS is set to a regular NameNode but your table location is an RBF address, Hive might not be fetching the secondary token required for the Router.
➤ The Fix: Configure Token Requirements
You need to ensure Hive and the underlying MapReduce/Tez framework know to fetch tokens for the Router's URI.
1. Add the Router URI to Hive's Token List
In your Hive session (or globally in hive-site.xml), you must define the Router as a "known" filesystem that requires tokens.
SET hive.metastore.token.signature=hdfs://router_host:8888;
SET mapreduce.job.hdfs-servers=hdfs://router_host:8888,hdfs://nameservice-backend;
2. Configure HDFS Client to "Trust" the Router for Tokens
In core-site.xml or hdfs-site.xml, you need to enable the Router to act as a proxy for the backend NameNodes so it can pass the tokens correctly.
<property>
<name>dfs.federation.router.delegation.token.enable</name>
<value>true</value>
</property>
➤ Critical Kerberos Configuration
Because the Router is an intermediary, it must be allowed to impersonate the user (Hive) when talking to the backend. Ensure your ProxyUser settings in core-site.xml include the Router's service principal.
Assuming your Router runs as the hdfs or router user:
<property>
<name>hadoop.proxyuser.router.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.router.hosts</name>
<value>*</value>
</property>
➤ Diagnostic Verification
To prove if the token is missing, run this command from the datanode_host mentioned in your error logs using the same user running the Hive job:
# Check if you can manually get a token for the router
hdfs fetchdt --renewer hdfs hdfs://router_host:8888 router.token
# Check the contents of your current credentials cache
klist -f
If fetchdt fails, the issue is with the Router's ability to issue tokens. If it succeeds but Hive fails, the issue is with Hive's Job Submission not including the Router URI in the mapreduce.job.hdfs-servers list.
