question Re: Issue when using parquet org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist in Archives of Support Questions (Read Only)

Issue when using parquet org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist

ClouderaVMissue — Wed, 08 Jun 2016 20:11:45 GMT

I am getting this issue when using sqoop with parquet

Re: Issue when using parquet org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist

ssubhas — Wed, 08 Jun 2016 20:49:00 GMT

Can you please share the complete error along with sqoop command being used?

The issue might be when HIVE_HOME/HCAT_HOME is not set as Sqoop will use HIVE_HOME/HCAT_HOME to find hive libs, which are needed in hive import as Parquet file.

Thanks and Regards,

Sindhu

Re: Issue when using parquet org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist

ClouderaVMissue — Thu, 09 Jun 2016 17:34:01 GMT

sqoop import --connect jdbc:oracle:thin:@XXX:XXXX/YYYY --username YYYYY --password YYYYY --table A.BBBB --hive-import --hive-database default --hive-table test15 --as-parquetfile -m 1

Job job_1465371735536_0055 failed with state FAILED due to: Job commit failed: java.lang.IllegalArgumentException: Wrong FS: ____file:/tmp/default/.temp/job_1465371735536_0055/mr/job_1465371735536_0055/b402d4ba-1a16-46bc-92c6-91fe141070d2.parquet, expected: hdfs://lxapp5524.dc.corp.telstra.com:8020 at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:646)

Re: Issue when using parquet org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist

ClouderaVMissue — Thu, 09 Jun 2016 18:04:52 GMT

We are only getting the above error if we are using parquet otherwise my table get pulled in hive easily .Please keep in mind that we have installed the HDP without internet so it's highly possible that we missed some thing.

Re: Issue when using parquet org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist

ClouderaVMissue — Fri, 10 Jun 2016 07:41:35 GMT

We are facing the above error while we were using the below query

sqoop import --connect jdbc:oracle:thin:@XX:1521/DATABASENAME --username USER --password PWD --table SCHEMANAME.TABLENAME --hive-import --hive-table TABLENAME --hive-overwrite --num-mappers 1 --as-parquetfile

It's an issue when we are using Parquet and trying to ingest data in hive only because if we do the ingestion in hdfs with parquet , it gets completed.

Re: Issue when using parquet org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist

Rabin — Fri, 10 Jun 2016 10:57:19 GMT

I am also getting same wrror on hdp 2.4 , while doing sqoop hive-import with parquet . Without parquet it is working fine .

16/06/09 21:12:11 INFO mapreduce.Job: Job job_1465467652802_0011 failed with state FAILED due to: Job commit failed: java.lang.IllegalArgumentException: Wrong FS: _______file:/tmp/default/.temp/job_1465467652802_0011/mr/job_1465467652802_0011/dc944213-b925-4e5b-ac2c-736e5fa8610f.parquet, expected: hdfs://lxapp5524.dc.corp.hdp.com:8020 at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:646) at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194) at org.apache.hadoop.hdfs.DistributedFileSystem.rename(DistributedFileSystem.java:636) at org.kitesdk.data.spi.filesystem.FileSystemDataset.merge(FileSystemDataset.java:327) at org.kitesdk.data.spi.filesystem.FileSystemDataset.merge(FileSystemDataset.java:56) at org.kitesdk.data.mapreduce.DatasetKeyOutputFormat$MergeOutputCommitter.commitJob(DatasetKeyOutputFormat.java:370) at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:285) at org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Re: Issue when using parquet org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist

Rabin — Sun, 12 Jun 2016 00:37:41 GMT

Then tried changing the dependency version for kite-sdk from 1.0.0 to 1.1.0 , and the issue gone . It worked !!! Issue resolved.

Re: Issue when using parquet org.kitesdk.data.DatasetNotFoundException: Descriptor location does not exist

sandeep1 — Thu, 12 Apr 2018 14:07:39 GMT

We were able to resolve it here:

https://discuss.cloudxlab.com/t/sqoop-import-to-hive-as-parquet-file-is-failing/1089/6