Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How to copy files from one location(/tmp/files folder) to another location(to hdfs://user/username directory) using java application

avatar

I'm trying to move files from one location to another location while executing java application on execution server

Example:From one location(/tmp/files folder) to another location(hdfs://user/username directory) using java application.

4 REPLIES 4

avatar

Currently I'm doing it using hdfs dfs -cp -f file:"/tmp/files" hdfs:/user/username in script.

I want to do it in java now

avatar
Expert Contributor

@Amithesh Merugu

Use this method: copyFromLocalFile(Path src, Path dst). The first parameter is a path on your local disk (in your example /tmp/files) and the second is the HDFS path (hdfs://user/username). The documentation doesn't make it clear, but the source can be a dictionary and then the whole content is copied to the HDFS.

FileSystem fs = FileSystem.get(hdfsUri, conf);
fs.copyFromLocalFile(new Path("/tmp/files"), new Path("/user/username"));

avatar

@gnovak

Thanks for quick response.my hdfs uri looks like "hdfs://demo"

I'm getting below exception

java.net.UnknownHostException: demo

java.lang.IllegalArgumentException: java.net.UnknownHostException: demo
	at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:438)
	at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311)
	at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:688)
	at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:629)
	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:159)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2761)
	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2795)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2777)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:386)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:179)
	at com.mckinsey.hcva.aaas.execute.ReportsGeneration$1.run(ReportsGeneration.java:157)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: demo

avatar
Expert Contributor

@Amithesh Merugu Try to use the IP address of the NameNode. And also add the port (default is 8020).