how to copy file from mount point to hdfs (WebShellClient)

[#1]

Hi everyone ,

This is the Sandbox env.,

M trying to load data into Hive table called: dwstg.dummy; here are the steps i followed inside WebShellClient

Step1)after connecting to JDBC Hive , below is the command i issued

Input)

LOAD DATA INPATH '/mnt/tmp/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;

Output) I get following error

Error: Error while compiling statement: FAILED: SemanticException Line 1:17 Invalid path ''/mnt/tmp/testme.csv'':

No files matching path hdfs://sandbox-hdp.hortonworks.com:8020/mnt/tmp/testme.csv (state=42000,code=40000)

Step2)based on the above error , i researched & realized JDBC is not seeing the mount point as it should exist only in HDFS directory for it to see?

Step2.1)So i ran below command -->

Input) hadoop fs -cp /mnt/tmp/testme.csv hdfs:/

Output: cp: `/mnt/tmp/testme.csv': No such file or directory

So what m trying to figure out is , how to copy file from Mount point to hdfs directory structure?

Regards:

Bhasker.V

Jsglp

[#2]

you should do 1)hdfs dfs -put ${dir_home}/testme.csv /mnt/tmp/ 2)LOAD DATA INPATH '/mnt/tmp/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;

[#3 · Accepted solution]

@shuaili gong,

Thanks for your reply, I went through below steps and finally managed upload the file in the HDFS directory so Hive can access it and this was sucessful.

Now m having permission issues from source directory to /warehouse directory. I manually changed to 777 on /warehouse so everybody have full access and yet m still getting same permission issue. i have also checked via Ambari on this /warehouse directory and i can see it has full access to everyone but still same issue. Any suggestions highly appreciated

Below are the steps i followed on how i uploaded file etc and gave permissions as well.

[root@sandbox-hdp ~]# cp /mnt/tmp/testme.csv testme.csv

[root@sandbox-hdp ~]# ls -ltr

[root@sandbox-hdp ~]# su hdfs

[hdfs@sandbox-hdp root]$ cd

[hdfs@sandbox-hdp ~]$ hdfs dfs -chmod 777 /user

[hdfs@sandbox-hdp ~]$ exit

exit

[root@sandbox-hdp ~]#

[root@sandbox-hdp ~]# su hdfs

[hdfs@sandbox-hdp root]$ cd

[hdfs@sandbox-hdp ~]$ ls -ltr

[hdfs@sandbox-hdp ~]$ exit

exit

[root@sandbox-hdp ~]# hdfs dfs -mkdir /user/hadoop/

[root@sandbox-hdp ~]# hdfs dfs -mkdir /user/hadoop/silverPop

[root@sandbox-hdp ~]# hdfs dfs -put testme.csv /user/hadoop/silverPop

[root@sandbox-hdp ~]# hdfs dfs -ls /user/hadoop/silverPop

Found 1 items

-rw-r--r-- 1 root hdfs 12967 2019-04-08 20:41 /user/hadoop/silverPop/testme.csv

[root@sandbox-hdp ~]# hdfs dfs -du /user/hadoop/silverPop/testme.csv

12967 12967 /user/hadoop/testme.csv

[root@sandbox-hdp ~]# hadoop fs -chmod g+w /warehouse

0: jdbc:hive2://10.88.79.112:10000> LOAD DATA INPATH '/user/hadoop/silverPop/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;

ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. org.apache.hadoop.hive.ql.metadata.HiveException: Access denied: Unable to move source hdfs://sandbox-hdp.hortonworks.com:8020/user/hadoop/silverPop/testme.csv to

destination hdfs://sandbox-hdp.hortonworks.com:8020/warehouse/tablespace/managed/hive/dwstg.db/temp_silverpop/base_0000012: Permission denied: user=hive, access=WRITE, inode="/user/hadoop/silverPop":root:hdfs:drwxr-xr-x

INFO : Completed executing command(queryId=hive_20190409152723_ee9c9ad2-9213-407d-8edc-16df7e13a2e0); Time taken: 0.112 seconds

[#4]

i figured the issues was not with target directory , it was source DIR where permissions were lacking, hence I fixed using below commands (Please note 777 is not a good practice , you will need adjust per your production scenario)

[hdfs@sandbox-hdp ~]$ hdfs dfs -chmod 777 /user/hadoop

[hdfs@sandbox-hdp ~]$ hdfs dfs -chmod 777 /user/hadoop/silverPop

[#5]

I figured out the issue was not with target directory, it was with source directory where the files were coming from , hence i fixed the permissions on those directories and now all got succeeded

here is further commands i wrote to grand permissions (Please becarefull with 777 as it is not a good practice)

[hdfs@sandbox-hdp ~]$ hdfs dfs -chmod 777 /user/hadoop

[hdfs@sandbox-hdp ~]$ hdfs dfs -chmod 777 /user/hadoop/silverPop

0: jdbc:hive2://10.88.79.112:10000> LOAD DATA INPATH '/user/hadoop/silverPop/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;

successful