Support Questions

Find answers, ask questions, and share your expertise

Storm HDFS Bolt in Kerberos Enabled HDP 2.6 cluster failing


I have enabled Kerberos Security in my HDP 2.6 cluster, on which services- Kafka and Storm are installed prior to enabling Kerberos.

The topology that I am running has kafka-spout followed by hdfs-bolt. So, CSV data from a specific Kafka topic is ingested using in-built Kafka Spout, and then transferred to HDFS directory using in-built HDFS Bolt.


storm, storm-kafka, storm-hdfs ->

Kafka ->

When I submit this topology, I get the following error for HDFS Bolt -->

Caused by: org.apache.hadoop.ipc.RemoteException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at org.apache.hadoop.ipc.Client.getRpcResponse( ~[stormjar.jar:?] at ~[stormjar.jar:?] at ~[stormjar.jar:?]

I checked in all the config settings in Ambari UI. Everywhere security is set to Kerberos.

Solutions already tried in this case are:

1) Modify the topology code to add ->

List<String> auto_tgts = new ArrayList<String>();

auto_tgts.add(""); auto_tgts.add("");

Config conf = new Config(); conf.put(Config.TOPOLOGY_AUTO_CREDENTIALS, auto_tgts);

2) Modify the topology code to add ->

Map<String, Object> map = new HashMap<String,Object>(); map.put("hdfs.keytab.file","/etc/security/keytabs/storm.headless.keytab");


map.put("", "kerberos");

Config conf = new Config(); conf.put("hdfs.config", map);

HdfsBolt hdfsbolt = new HdfsBolt() .withFsUrl(hdfsUrl) .withFileNameFormat(fileNameFormat).withRecordFormat(recordFormat) .withRotationPolicy(rotationPolicy).withSyncPolicy(syncPolicy).withConfigKey("hdfs.config");

Please let me know if I need to do any other steps/config related changes for storm-hdfs connector to work with Kerberos.

Thanks in advance!




I am also facing same issues.

I have referred below URl but still no luck:

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.