Skip to main content
analytics-shek
analytics-shek
Gold ⭐️
September 24, 2020
Solved

Error: Invalid path when uploading from MariaDB to Hive Database using Sqoop

I am running the command from hdfs user hdfs@master>

sqoop import-all-tables --connect jdbc:mysql://10.11.11.15:6306/siki_asmet?serverTimezone=Asia/Jakarta --username micronics -P --hive-import --warehouse-dir /warehouse/siki --hive-database siki_ods --exclude-tables "Sheet1$" --m 1;

 

When i run the above command, i get the following error: 

 

FAILED: SemanticException Line 1:17 Invalid path ''hdfs://master.lpjk.com:8020/warehouse/siki/_asdamkindo_personal_ska_pendidikan'': No files matching path hdfs://master.lpjk.com:8020/warehouse/siki/_asdamkindo_personal_ska_pendidikan (state=42000,code=40000)

 

But when i try to run the command again, it said file already exists. 

analytics-shek
analytics-shekAnswer
Gold ⭐️
September 24, 2020

I solved my problem. In my case one of the talbes' name was starting with a character underscore "_" because of which there was an issue where 2 single quotes were added automatically in the path of the hdfs directory where the copy of the file was stored.

I changed the name of the column by removing the underscore character and now i can import the table into Hive database. I think special characters like that are not easily parsed in Hive or HDFS.

View in thread

3 replies

stevenmatison
stevenmatison
Platinum ⭐️⭐️
September 24, 2020

Check that the user executing sqoop is able to write to the target directory.  You may need to create a service user and directory with proper permissions since the hdfs user cannot write to the intended directory.

 

 

If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  

 

Thanks,


Steven

Translate
analytics-shek
analytics-shekAuthor
Gold ⭐️
September 24, 2020

Since i am executing sqoop from the user hdfs (hdfs@master>) it should have access to all the directories inside it right? 

if i just use the command without specifying the --warehouse-dir, then also i get the same error. 

 

Also, i disabled permission in hdfs-site.xml from true to false in hdfs.permission.enabled

Translate
analytics-shek
analytics-shekAuthorAnswer
Gold ⭐️
September 24, 2020

I solved my problem. In my case one of the talbes' name was starting with a character underscore "_" because of which there was an issue where 2 single quotes were added automatically in the path of the hdfs directory where the copy of the file was stored.

I changed the name of the column by removing the underscore character and now i can import the table into Hive database. I think special characters like that are not easily parsed in Hive or HDFS.

Translate
    Terms & ConditionsAccessibility statement