Storm Topology not submitting with Secure HDFS [Internal error processing submitTopologyWithOpts]

[#1]

Hi ,

I am using cloudera CDP Public cloud with CDP Runtime version 7.2.17.

As Apache storm is not a part of cloudera stack, i downloaded from apache website and configured kerberos manually with the help of storm documentation

https://storm.apache.org/releases/2.6.1/SECURITY.html

I am using kafka spout and hdfs bolt , 
I was able to connect to kafka spout with kafka Spout configuration properties,

But i faced SIMPLE authentication is not enabled. Available:[TOKEN] error for hdfs bolt
by going through storm documentation related to
hdfs bolt( https://storm.apache.org/releases/2.1.0/storm-hdfs.html ) i configured some of properties in yaml file

storm.yaml

storm.zookeeper.servers:
- "hari-cluster-test1-master0.avinash.ceje-5ray.a5.cloudera.site"
storm.local.dir: "/home/venkataavinash_kesineni/storm/apache-storm-2.6.1/data"
nimbus.seeds: ["hari-cluster-test1-master0.avinash.ceje-5ray.a5.cloudera.site"]
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
- 6704
- 6705
storm.thrift.transport: "org.apache.storm.security.auth.kerberos.KerberosSaslTransportPlugin"
nimbus.childpots: "-Xmx1024m -Djava.security.auth.login.config=/home/venkataavinash_kesineni/storm/apache-storm-2.6.1/conf/jaas.conf"
ui.childpots: "-Xmx1024m -Djava.security.auth.login.config=/home/venkataavinash_kesineni/storm/apache-storm-2.6.1/conf/jaas.conf"
supervisor.childopts: "-Xmx256m -Djava.security.auth.login.config=/home/venkataavinash_kesineni/storm/apache-storm-2.6.1/conf/jaas.conf"
storm.zookeeper.superACL: "sasl:storm"
java.security.auth.login.config: "/home/venkataavinash_kesineni/storm/apache-storm-2.6.1/conf/jaas.conf"
nimbus.autocredential.plugins.classes: ["org.apache.storm.hdfs.security.AutoHDFSNimbus"]
nimbus.credential.renewers.classes: ["org.apache.storm.hdfs.security.AutoHDFSNimbus"]
hdfs.keytab.file: "/root/hdfs.keytab"
hdfs.kerberos.principal: "hdfs/hari-cluster-test1-master0.avinash.ceje-5ray.a5.cloudera.site@AVINASH.CEJE-5RAY.A5.CLOUDERA.SITE"
nimbus.credential.renewers.freq.secs: 82800
topology.auto-credentials: ["org.apache.storm.hdfs.security.AutoHDFS"]

When i am submitting topology i am getting the following error
Error

java.lang.RuntimeException: org.apache.storm.thrift.TApplicationException: Internal error processing submitTopologyWithOpts
        at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:304) ~[storm-client-2.6.1.jar:2.6.1]
        at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:212) ~[storm-client-2.6.1.jar:2.6.1]
        at org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:175) ~[storm-client-2.6.1.jar:2.6.1]
        at com.voltage.securedata.stream.storm.StormTopology.main(StormTopology.java:220) [vs-kafka-storm-1.0.jar:?]
Caused by: org.apache.storm.thrift.TApplicationException: Internal error processing submitTopologyWithOpts
        at org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:81) ~[storm-shaded-deps-2.6.1.jar:2.6.1]
        at org.apache.storm.generated.Nimbus$Client.recv_submitTopologyWithOpts(Nimbus.java:376) ~[storm-client-2.6.1.jar:2.6.1]
        at org.apache.storm.generated.Nimbus$Client.submitTopologyWithOpts(Nimbus.java:359) ~[storm-client-2.6.1.jar:2.6.1]
        at org.apache.storm.StormSubmitter.submitTopologyInDistributeMode(StormSubmitter.java:354) ~[storm-client-2.6.1.jar:2.6.1]
        at org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:292) ~[storm-client-2.6.1.jar:2.6.1]
        ... 3 more
​

Please let me know if I need to do any other steps/config related changes for storm-hdfs bolt to work with Kerberos.

[#2]

@VenkataAvinash 

The error you're encountering (java.lang.RuntimeException: org.apache.storm.thrift.TApplicationException: Internal error processing submitTopologyWithOpts) indicates that there's an issue with submitting the Storm topology, but it doesn't directly point to the specific cause.
However, based on your configuration and the error message, it seems like there might be an issue with the Kerberos authentication setup or configuration for the Storm Nimbus service.
=>Review Kerberos Configuration: Double-check the Kerberos configuration for Storm Nimbus and ensure that it matches the settings in your storm.yaml file. Verify that the Kerberos principal (hdfs/hari-cluster-test1-master0.avinash.ceje-5ray.a5.cloudera.site@AVINASH.CEJE-5RAY.A5.CLOUDERA.SITE) and keytab file (/root/hdfs.keytab) are correctly specified.
=>Check Keytab Permissions: Ensure that the keytab file /root/hdfs.keytab has the correct permissions set and is accessible by the Storm Nimbus service.
=>Verify Service Principals: Confirm that the Kerberos principal (hdfs/hari-cluster-test1-master0.avinash.ceje-5ray.a5.cloudera.site@AVINASH.CEJE-5RAY.A5.CLOUDERA.SITE) is correctly configured for the Storm Nimbus service and that it has the necessary permissions to access HDFS.
=>Check Nimbus Logs: Check the Nimbus logs (nimbus.log) for any additional error messages or stack traces that might provide more insight into the issue.
=>Classpath Issues:Confirm that the versions of Storm, HDFS, and Kerberos libraries on your cluster are compatible with each other. Refer to the documentation for each component for known compatibility issues.
=>Try submitting a simpler topology without the HDFS bolt initially to see if the basic Kerberos configuration works. This can help isolate the issue further.
=>Consider using a tool like klist to verify if your user has successfully obtained a Kerberos ticket before submitting the topology.