how to copy file from mount point to hdfs (WebShellClient)

Question

Hi everyone , This is the Sandbox env.,M trying to load data into Hive table called: dwstg.dummy; here are the steps i followed inside WebShellClientStep1)after connecting to JDBC Hive , below is the command i issuedInput) LOAD DATA INPATH '/mnt/tmp/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;Output) I get following errorError: Error while compiling statement: FAILED: SemanticException Line 1:17 Invalid path ''/mnt/tmp/testme.csv'': No files matching path hdfs://sandbox-hdp.hortonworks.com:8020/mnt/tmp/testme.csv (state=42000,code=40000) Step2)based on the above error , i researched & realized JDBC is not seeing the mount point as it should exist only in HDFS directory for it to see?Step2.1)So i ran below command -->Input)  hadoop fs -cp /mnt/tmp/testme.csv hdfs:/    Output:   cp: `/mnt/tmp/testme.csv': No such file or directory                                                                                                                                                                                        So what m trying to figure out is , how to copy file from Mount point to hdfs directory structure?Regards:Bhasker.VJsglp

bhaskerbobby · Accepted Answer

@shuaili gong,Thanks for your reply, I went through below steps and finally managed upload the file in the HDFS directory so Hive can access it and this was sucessful.Now m having permission issues from source directory to /warehouse directory. I manually changed to 777 on /warehouse so everybody have full access and yet m still getting same permission issue. i have also checked via Ambari on this /warehouse directory and i can see it has full access to everyone but still same issue. Any suggestions highly appreciatedBelow are the steps i followed on how i uploaded file etc and gave permissions as well.[root@sandbox-hdp ~]# cp /mnt/tmp/testme.csv testme.csv[root@sandbox-hdp ~]# ls -ltr[root@sandbox-hdp ~]# su hdfs[hdfs@sandbox-hdp root]$ cd[hdfs@sandbox-hdp ~]$ hdfs dfs -chmod 777 /user[hdfs@sandbox-hdp ~]$ exitexit[root@sandbox-hdp ~]#[root@sandbox-hdp ~]# su hdfs[hdfs@sandbox-hdp root]$ cd[hdfs@sandbox-hdp ~]$ ls -ltr[hdfs@sandbox-hdp ~]$ exitexit[root@sandbox-hdp ~]# hdfs dfs -mkdir /user/hadoop/[root@sandbox-hdp ~]# hdfs dfs -mkdir /user/hadoop/silverPop[root@sandbox-hdp ~]# hdfs dfs -put testme.csv /user/hadoop/silverPop[root@sandbox-hdp ~]# hdfs dfs -ls /user/hadoop/silverPopFound 1 items-rw-r--r-- 1 root hdfs 12967 2019-04-08 20:41 /user/hadoop/silverPop/testme.csv[root@sandbox-hdp ~]# hdfs dfs -du /user/hadoop/silverPop/testme.csv12967 12967 /user/hadoop/testme.csv[root@sandbox-hdp ~]# hadoop fs -chmod g+w /warehouse0: jdbc:hive2://10.88.79.112:10000> LOAD DATA INPATH '/user/hadoop/silverPop/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask. org.apache.hadoop.hive.ql.metadata.HiveException: Access denied: Unable to move source hdfs://sandbox-hdp.hortonworks.com:8020/user/hadoop/silverPop/testme.csv todestination hdfs://sandbox-hdp.hortonworks.com:8020/warehouse/tablespace/managed/hive/dwstg.db/temp_silverpop/base_0000012: Permission denied: user=hive, access=WRITE, inode="/user/hadoop/silverPop":root:hdfs:drwxr-xr-xINFO : Completed executing command(queryId=hive_20190409152723_ee9c9ad2-9213-407d-8edc-16df7e13a2e0); Time taken: 0.112 seconds

user13526464656 · Answer

you should do  1)hdfs dfs -put ${dir_home}/testme.csv /mnt/tmp/    2)LOAD DATA INPATH '/mnt/tmp/testme.csv' OVERWRITE INTO TABLE dwstg.dummy;

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.