Created 12-12-2017 09:40 AM
I'm trying to move files from one location to another location while executing java application on execution server
Example:From one location(/tmp/files folder) to another location(hdfs://user/username directory) using java application.
Created 12-12-2017 09:47 AM
Currently I'm doing it using hdfs dfs -cp -f file:"/tmp/files" hdfs:/user/username in script.
I want to do it in java now
Created 12-12-2017 10:35 AM
Use this method: copyFromLocalFile(Path src, Path dst). The first parameter is a path on your local disk (in your example /tmp/files) and the second is the HDFS path (hdfs://user/username). The documentation doesn't make it clear, but the source can be a dictionary and then the whole content is copied to the HDFS.
FileSystem fs = FileSystem.get(hdfsUri, conf); fs.copyFromLocalFile(new Path("/tmp/files"), new Path("/user/username"));
Created 12-12-2017 04:32 PM
Thanks for quick response.my hdfs uri looks like "hdfs://demo"
I'm getting below exception
java.net.UnknownHostException: demo
java.lang.IllegalArgumentException: java.net.UnknownHostException: demo at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:438) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:688) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:629) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:159) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2761) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2795) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2777) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:386) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:179) at com.mckinsey.hcva.aaas.execute.ReportsGeneration$1.run(ReportsGeneration.java:157) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.UnknownHostException: demo
Created 12-12-2017 04:48 PM
@Amithesh Merugu Try to use the IP address of the NameNode. And also add the port (default is 8020).