Support Questions

Find answers, ask questions, and share your expertise

Storm HDFS Bolt in Kerberos Enabled HDP 2.6 cluster failing

Hi,

I have enabled Kerberos Security in my HDP 2.6 cluster, on which services- Kafka and Storm are installed prior to enabling Kerberos.

The topology that I am running has kafka-spout followed by hdfs-bolt. So, CSV data from a specific Kafka topic is ingested using in-built Kafka Spout, and then transferred to HDFS directory using in-built HDFS Bolt.

Versions:

storm, storm-kafka, storm-hdfs -> 1.1.0.2.6.0.3-8

Kafka -> 0.10.1.2.6.0.3-8

When I submit this topology, I get the following error for HDFS Bolt -->

Caused by: org.apache.hadoop.ipc.RemoteException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554) ~[stormjar.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1498) ~[stormjar.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1398) ~[stormjar.jar:?]

I checked in all the config settings in Ambari UI. Everywhere security is set to Kerberos.

Solutions already tried in this case are:

1) Modify the topology code to add ->

List<String> auto_tgts = new ArrayList<String>();

auto_tgts.add("org.apache.storm.security.auth.kerberos.AutoTGT"); auto_tgts.add("org.apache.storm.hdfs.common.security.AutoHDFS");

Config conf = new Config(); conf.put(Config.TOPOLOGY_AUTO_CREDENTIALS, auto_tgts);

2) Modify the topology code to add ->

Map<String, Object> map = new HashMap<String,Object>(); map.put("hdfs.keytab.file","/etc/security/keytabs/storm.headless.keytab");

map.put("hdfs.kerberos.principal","storm-hdp26ks@XYZ.COM");

map.put("hadoop.security.authentication", "kerberos");

Config conf = new Config(); conf.put("hdfs.config", map);

HdfsBolt hdfsbolt = new HdfsBolt() .withFsUrl(hdfsUrl) .withFileNameFormat(fileNameFormat).withRecordFormat(recordFormat) .withRotationPolicy(rotationPolicy).withSyncPolicy(syncPolicy).withConfigKey("hdfs.config");

Please let me know if I need to do any other steps/config related changes for storm-hdfs connector to work with Kerberos.

Thanks in advance!

1 REPLY 1

Explorer

Hi,

I am also facing same issues.

I have referred below URl but still no luck:

https://github.com/apache/storm/tree/master/external/storm-hdfs#using-hdfs-delegation-tokens

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.