Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hadoop cluster setup on Azure with ADLS storage

Highlighted

Hadoop cluster setup on Azure with ADLS storage

I am trying to deploy HDP cluster on Azure which uses Azure Data Lake storage. I followed the instructions here for Ambari and cluster setup. I followed the instruction here for ADLS access.

When is try to start Hive I get the following error.

<code>Exception in thread "main" java.lang.RuntimeException: java.io.IOException: The ownership on the staging directory adl://home/tmp/hive/root/_tez_session_dir/c96088dd-444e-42e3-9293-251656b01b17 is not as expected. It is owned by <ADLS Service Principal>. The directory must be owned by the submitter root or by root
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
I am able to access ADLS via the HDFS commands
Permission on /tmp folder

> hdfs dfs -ls /tmp
Found 1 items
drwxrwxrwt+  - <ADLS Service Principal>          0 2018-01-20 21:29 /tmp/hive

TIA
<br>
2 REPLIES 2

Re: Hadoop cluster setup on Azure with ADLS storage

Explorer

Hi Siddharth, were you able to get this to work?

Highlighted

Re: Hadoop cluster setup on Azure with ADLS storage

Give this a try.

In the linux box from where you start Hive CLI create a user with the same name as the <ADLS ServicePrincipal> name.

Start Hive CLI from this new user account.

Don't have an account?
Coming from Hortonworks? Activate your account here