Unable to run Query on Hive-3.0 with ADLS

Rarjun — Tue, 13 Nov 2018 02:07:53 GMT

18/11/12 10:37:25 [main]: INFO jdbc.HiveConnection: Connected to server1.test.com:10000 Connected to: Apache Hive (version 3.0.0.3.0.0.0-1334) Driver: Hive JDBC (version 3.0.0.3.0.0.0-1334) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 3.0.0.3.0.0.0-1334 by Apache Hive 0: jdbc:hive2://server1.test.com:> select count(*) from lnd_edw4x_qn1useh1.redemption_fact_t;

INFO : Compiling command(queryId=hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216): select count(*) from lnd_edw4x_qn1useh1.redemption_fact_t

INFO : Semantic Analysis Completed

INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null) INFO : Completed compiling command(queryId=hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216); Time taken: 1.218 seconds

INFO : Executing command(queryId=hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216): select count(*) from lnd_edw4x_qn1useh1.redemption_fact_t

INFO : Query ID = hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216 INFO : Total jobs = 1 INFO : Launching Job 1 out of 1 INFO : Starting task [Stage-1:MAPRED] in serial mode

INFO : Subscribed to counters: [] for queryId: hive_20181112103728_c02d0353-978c-457c-bde7-9ae45fcb1216

INFO : Tez session hasn't been created yet. Opening session

INFO : Dag name: select count(*) from ...h1.redemption_fact_t (Stage-1)

ERROR : Status: Failed ERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1542008498326_0022_1_00, diagnostics=[Vertex vertex_1542008498326_0022_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: redemption_fact_t initializer failed, vertex=vertex_1542008498326_0022_1_00 [Map 1], java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.adl.AdlFileSystem not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2595)

at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3320)

I am already having below jars in Hadoop/hive libs still it's not working:

hadoop-azure-3.0.0.3.0.0.0-1334.jar, hadoop-azure-datalake-3.0.0.3.0.0.0-1334.jar, azure-data-lake-store-sdk-2.2.7.jar

Re: Unable to run Query on Hive-3.0 with ADLS

quadoss — Thu, 22 Nov 2018 02:57:41 GMT

Hi @arjun more

Maybe that the below jars are not there for the yarn containers for it to make use of when hive launches the job. You can try adding in these jars with 'add jar <path_to_jar>/jarfilename.jar; and see if this helps.

Re: Unable to run Query on Hive-3.0 with ADLS

pmohan — Thu, 22 Nov 2018 12:07:51 GMT

What version of HDP are you using ?

Re: Unable to run Query on Hive-3.0 with ADLS

Rarjun — Sun, 18 Aug 2019 01:07:27 GMT

@Prabhu M @Mahesh Balakrishnan : Issue has been resolved.

It's a bug with Tez version 0.9.0 (HDP-3.0.0.0) which they have fixed in Tez 0.9.1 version(HDP-3.0.1.0)

question Re: Unable to run Query on Hive-3.0 with ADLS in Archives of Support Questions (Read Only)

Unable to run Query on Hive-3.0 with ADLS

Re: Unable to run Query on Hive-3.0 with ADLS

Re: Unable to run Query on Hive-3.0 with ADLS

Re: Unable to run Query on Hive-3.0 with ADLS