Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Storm HDFS Bolt in Kerberos Enabled HDP 2.6 cluster failing


Storm HDFS Bolt in Kerberos Enabled HDP 2.6 cluster failing


I have enabled Kerberos Security in my HDP 2.6 cluster, on which services- Kafka and Storm are installed prior to enabling Kerberos.

The topology that I am running has kafka-spout followed by hdfs-bolt. So, CSV data from a specific Kafka topic is ingested using in-built Kafka Spout, and then transferred to HDFS directory using in-built HDFS Bolt.


storm, storm-kafka, storm-hdfs ->

Kafka ->

When I submit this topology, I get the following error for HDFS Bolt -->

Caused by: org.apache.hadoop.ipc.RemoteException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at org.apache.hadoop.ipc.Client.getRpcResponse( ~[stormjar.jar:?] at ~[stormjar.jar:?] at ~[stormjar.jar:?]

I checked in all the config settings in Ambari UI. Everywhere security is set to Kerberos.

Solutions already tried in this case are:

1) Modify the topology code to add ->

List<String> auto_tgts = new ArrayList<String>();

auto_tgts.add(""); auto_tgts.add("");

Config conf = new Config(); conf.put(Config.TOPOLOGY_AUTO_CREDENTIALS, auto_tgts);

2) Modify the topology code to add ->

Map<String, Object> map = new HashMap<String,Object>(); map.put("hdfs.keytab.file","/etc/security/keytabs/storm.headless.keytab");


map.put("", "kerberos");

Config conf = new Config(); conf.put("hdfs.config", map);

HdfsBolt hdfsbolt = new HdfsBolt() .withFsUrl(hdfsUrl) .withFileNameFormat(fileNameFormat).withRecordFormat(recordFormat) .withRotationPolicy(rotationPolicy).withSyncPolicy(syncPolicy).withConfigKey("hdfs.config");

Please let me know if I need to do any other steps/config related changes for storm-hdfs connector to work with Kerberos.

Thanks in advance!


Re: Storm HDFS Bolt in Kerberos Enabled HDP 2.6 cluster failing



I am also facing same issues.

I have referred below URl but still no luck:

Don't have an account?
Coming from Hortonworks? Activate your account here