Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how to copy file from mount point to hdfs (WebShellClient)

SOLVED Go to solution
Highlighted

how to copy file from mount point to hdfs (WebShellClient)

New Contributor

Hi everyone ,

This is the Sandbox env.,


M trying to load data into Hive table called: dwstg.dummy; here are the steps i followed inside WebShellClient

Step1)after connecting to JDBC Hive , below is the command i issued

Input)

LOAD DATA INPATH '/mnt/tmp/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;


Output) I get following error

Error: Error while compiling statement: FAILED: SemanticException Line 1:17 Invalid path ''/mnt/tmp/testme.csv'':

No files matching path hdfs://sandbox-hdp.hortonworks.com:8020/mnt/tmp/testme.csv (state=42000,code=40000)


Step2)based on the above error , i researched & realized JDBC is not seeing the mount point as it should exist only in HDFS directory for it to see?


Step2.1)So i ran below command -->

Input) hadoop fs -cp /mnt/tmp/testme.csv hdfs:/

Output: cp: `/mnt/tmp/testme.csv': No such file or directory


So what m trying to figure out is , how to copy file from Mount point to hdfs directory structure?




Regards:

Bhasker.V

Jsglp


1 ACCEPTED SOLUTION

Accepted Solutions

Re: how to copy file from mount point to hdfs (WebShellClient)

New Contributor

@shuaili gong,

Thanks for your reply, I went through below steps and finally managed upload the file in the HDFS directory so Hive can access it and this was sucessful.


Now m having permission issues from source directory to /warehouse directory. I manually changed to 777 on /warehouse so everybody have full access and yet m still getting same permission issue. i have also checked via Ambari on this /warehouse directory and i can see it has full access to everyone but still same issue. Any suggestions highly appreciated


Below are the steps i followed on how i uploaded file etc and gave permissions as well.


[root@sandbox-hdp ~]# cp /mnt/tmp/testme.csv testme.csv

[root@sandbox-hdp ~]# ls -ltr

[root@sandbox-hdp ~]# su hdfs

[hdfs@sandbox-hdp root]$ cd

[hdfs@sandbox-hdp ~]$ hdfs dfs -chmod 777 /user

[hdfs@sandbox-hdp ~]$ exit

exit

[root@sandbox-hdp ~]#

[root@sandbox-hdp ~]# su hdfs

[hdfs@sandbox-hdp root]$ cd

[hdfs@sandbox-hdp ~]$ ls -ltr

[hdfs@sandbox-hdp ~]$ exit

exit

[root@sandbox-hdp ~]# hdfs dfs -mkdir /user/hadoop/

[root@sandbox-hdp ~]# hdfs dfs -mkdir /user/hadoop/silverPop

[root@sandbox-hdp ~]# hdfs dfs -put testme.csv /user/hadoop/silverPop

[root@sandbox-hdp ~]# hdfs dfs -ls /user/hadoop/silverPop

Found 1 items

-rw-r--r-- 1 root hdfs 12967 2019-04-08 20:41 /user/hadoop/silverPop/testme.csv

[root@sandbox-hdp ~]# hdfs dfs -du /user/hadoop/silverPop/testme.csv

12967 12967 /user/hadoop/testme.csv

[root@sandbox-hdp ~]# hadoop fs -chmod g+w /warehouse





0: jdbc:hive2://10.88.79.112:10000> LOAD DATA INPATH '/user/hadoop/silverPop/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;


ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. org.apache.hadoop.hive.ql.metadata.HiveException: Access denied: Unable to move source hdfs://sandbox-hdp.hortonworks.com:8020/user/hadoop/silverPop/testme.csv to

destination hdfs://sandbox-hdp.hortonworks.com:8020/warehouse/tablespace/managed/hive/dwstg.db/temp_silverpop/base_0000012: Permission denied: user=hive, access=WRITE, inode="/user/hadoop/silverPop":root:hdfs:drwxr-xr-x

INFO : Completed executing command(queryId=hive_20190409152723_ee9c9ad2-9213-407d-8edc-16df7e13a2e0); Time taken: 0.112 seconds

4 REPLIES 4

Re: how to copy file from mount point to hdfs (WebShellClient)

New Contributor

you should do 1)hdfs dfs -put ${dir_home}/testme.csv /mnt/tmp/ 2)LOAD DATA INPATH '/mnt/tmp/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;

Re: how to copy file from mount point to hdfs (WebShellClient)

New Contributor

@shuaili gong,

Thanks for your reply, I went through below steps and finally managed upload the file in the HDFS directory so Hive can access it and this was sucessful.


Now m having permission issues from source directory to /warehouse directory. I manually changed to 777 on /warehouse so everybody have full access and yet m still getting same permission issue. i have also checked via Ambari on this /warehouse directory and i can see it has full access to everyone but still same issue. Any suggestions highly appreciated


Below are the steps i followed on how i uploaded file etc and gave permissions as well.


[root@sandbox-hdp ~]# cp /mnt/tmp/testme.csv testme.csv

[root@sandbox-hdp ~]# ls -ltr

[root@sandbox-hdp ~]# su hdfs

[hdfs@sandbox-hdp root]$ cd

[hdfs@sandbox-hdp ~]$ hdfs dfs -chmod 777 /user

[hdfs@sandbox-hdp ~]$ exit

exit

[root@sandbox-hdp ~]#

[root@sandbox-hdp ~]# su hdfs

[hdfs@sandbox-hdp root]$ cd

[hdfs@sandbox-hdp ~]$ ls -ltr

[hdfs@sandbox-hdp ~]$ exit

exit

[root@sandbox-hdp ~]# hdfs dfs -mkdir /user/hadoop/

[root@sandbox-hdp ~]# hdfs dfs -mkdir /user/hadoop/silverPop

[root@sandbox-hdp ~]# hdfs dfs -put testme.csv /user/hadoop/silverPop

[root@sandbox-hdp ~]# hdfs dfs -ls /user/hadoop/silverPop

Found 1 items

-rw-r--r-- 1 root hdfs 12967 2019-04-08 20:41 /user/hadoop/silverPop/testme.csv

[root@sandbox-hdp ~]# hdfs dfs -du /user/hadoop/silverPop/testme.csv

12967 12967 /user/hadoop/testme.csv

[root@sandbox-hdp ~]# hadoop fs -chmod g+w /warehouse





0: jdbc:hive2://10.88.79.112:10000> LOAD DATA INPATH '/user/hadoop/silverPop/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;


ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. org.apache.hadoop.hive.ql.metadata.HiveException: Access denied: Unable to move source hdfs://sandbox-hdp.hortonworks.com:8020/user/hadoop/silverPop/testme.csv to

destination hdfs://sandbox-hdp.hortonworks.com:8020/warehouse/tablespace/managed/hive/dwstg.db/temp_silverpop/base_0000012: Permission denied: user=hive, access=WRITE, inode="/user/hadoop/silverPop":root:hdfs:drwxr-xr-x

INFO : Completed executing command(queryId=hive_20190409152723_ee9c9ad2-9213-407d-8edc-16df7e13a2e0); Time taken: 0.112 seconds

Re: how to copy file from mount point to hdfs (WebShellClient)

New Contributor

i figured the issues was not with target directory , it was source DIR where permissions were lacking, hence I fixed using below commands (Please note 777 is not a good practice , you will need adjust per your production scenario)


[hdfs@sandbox-hdp ~]$ hdfs dfs -chmod 777 /user/hadoop

[hdfs@sandbox-hdp ~]$ hdfs dfs -chmod 777 /user/hadoop/silverPop

Re: how to copy file from mount point to hdfs (WebShellClient)

New Contributor

I figured out the issue was not with target directory, it was with source directory where the files were coming from , hence i fixed the permissions on those directories and now all got succeeded

here is further commands i wrote to grand permissions (Please becarefull with 777 as it is not a good practice)

[hdfs@sandbox-hdp ~]$ hdfs dfs -chmod 777 /user/hadoop

[hdfs@sandbox-hdp ~]$ hdfs dfs -chmod 777 /user/hadoop/silverPop


0: jdbc:hive2://10.88.79.112:10000> LOAD DATA INPATH '/user/hadoop/silverPop/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;

successful