Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Storm HDFS Bolt in Kerberos Enabled HDP 2.6 cluster failing

Highlighted

Storm HDFS Bolt in Kerberos Enabled HDP 2.6 cluster failing

Hi,

I have enabled Kerberos Security in my HDP 2.6 cluster, on which services- Kafka and Storm are installed prior to enabling Kerberos.

The topology that I am running has kafka-spout followed by hdfs-bolt. So, CSV data from a specific Kafka topic is ingested using in-built Kafka Spout, and then transferred to HDFS directory using in-built HDFS Bolt.

Versions:

storm, storm-kafka, storm-hdfs -> 1.1.0.2.6.0.3-8

Kafka -> 0.10.1.2.6.0.3-8

When I submit this topology, I get the following error for HDFS Bolt -->

Caused by: org.apache.hadoop.ipc.RemoteException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554) ~[stormjar.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1498) ~[stormjar.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1398) ~[stormjar.jar:?]

I checked in all the config settings in Ambari UI. Everywhere security is set to Kerberos.

Solutions already tried in this case are:

1) Modify the topology code to add ->

List<String> auto_tgts = new ArrayList<String>();

auto_tgts.add("org.apache.storm.security.auth.kerberos.AutoTGT"); auto_tgts.add("org.apache.storm.hdfs.common.security.AutoHDFS");

Config conf = new Config(); conf.put(Config.TOPOLOGY_AUTO_CREDENTIALS, auto_tgts);

2) Modify the topology code to add ->

Map<String, Object> map = new HashMap<String,Object>(); map.put("hdfs.keytab.file","/etc/security/keytabs/storm.headless.keytab");

map.put("hdfs.kerberos.principal","storm-hdp26ks@XYZ.COM");

map.put("hadoop.security.authentication", "kerberos");

Config conf = new Config(); conf.put("hdfs.config", map);

HdfsBolt hdfsbolt = new HdfsBolt() .withFsUrl(hdfsUrl) .withFileNameFormat(fileNameFormat).withRecordFormat(recordFormat) .withRotationPolicy(rotationPolicy).withSyncPolicy(syncPolicy).withConfigKey("hdfs.config");

Please let me know if I need to do any other steps/config related changes for storm-hdfs connector to work with Kerberos.

Thanks in advance!

1 REPLY 1
Highlighted

Re: Storm HDFS Bolt in Kerberos Enabled HDP 2.6 cluster failing

Explorer

Hi,

I am also facing same issues.

I have referred below URl but still no luck:

https://github.com/apache/storm/tree/master/external/storm-hdfs#using-hdfs-delegation-tokens

Don't have an account?
Coming from Hortonworks? Activate your account here