How to join multiple csv files in folder to one output file?

[#1]

How can I now join all the files in one folder to one single csv file?

I have a folder called Folder1 and I want to combine them all to a file called "output.csv".

I tried:

hadoop fs -getmerge Folder1 /user/maria_dev/output.csv

But I get the error:

getmerge: Mkdirs failed to create file:/user/maria_dev (exists=false, cwd=file:/home/maria_dev)

I also tried:

hadoop fs -cat Folder1 /output.csv 

But receive error: No such file or directory.

Thanks

[#2]

@ Jay Kumar SenSharma

thank you for your help with this. I am still not quite there.

The Folder1 definitely exists as shown in the attached (pic4) with two files.

I then tried:

chmod 777 -R Folder1

but then when I ran the getmerge command again, it produced the same error?

pic4.jpg

[#3]

@Matt

Your attached image "pic4.jpg" says that "maria_dev" user does not have WRITE permission on the "Folder1" contents. Only "root" user has WRITE permission on that directory.

-rw-r--r--

.

So you will need to make sure that when you do "ls -l" then you see the WRITE permission.

[root@sandbox ~]# mkdir -p /Folder1
[root@sandbox ~]# chmod 777 -R /Folder1/

(OR)

[root@sandbox ~]# chown maria_dev:hadoop -R /Folder1/

.

For testing try creating a simple file inside the folder where you are getting the error (this is just to see if "maria_dev" user has permission to create files inside that folder or not?)

[root@sandbox ~]# su - maria_dev

[maria_dev@sandbox ~]$ cd /Folder1/

[maria_dev@sandbox Folder1]$ echo "Hello"  > test.txt

.

[#4]

@ Jay Kumar SenSharma

I changed the "ls-l" to see WRITE permission as shown in the attached pic.

I also created tespic5.jpgt.txt but where should I see this now? It is not showing in list, also not showing in my directory for WinSCP?

[#5 · Accepted solution]

@Matt

Are you looking at the correct directory?

Can you please share the complete PATH of the directory in the One screenshot with all the commands that you are trying from Directory.

.

[#6]

@ Jay Kumar SenSharma

do you mean from WinSCP? I am not sure. I have included pics in my previous post. If you could take a look, that would be most appreciated. When I create a Folder1 directory, it does not show up under root - home - maria_dev.

[#7]

I was looking at the wrong directory. This appears to have resolved the issue. Thank you.

[#8]

@ Jay Kumar SenSharma

are you referring to WinSCP directory? If so, please see screen shot attached.pic6.jpg

There is no Folder1 here so this is why I am still confused.

Pic7 shows directory under root (pic 6 was as maria_dev).pic7.jpg The Folder1 shows here but I am still unsure as to how to progress.

Should I be logged into WinSCP as root or maria_dev?