Member since
09-19-2022
4
Posts
0
Kudos Received
0
Solutions
04-09-2023
10:00 PM
@Shelton Please note that the directory on edge server to which I am trying to copy the file is already present. So ideally it should not try to perform mkdir operation. as I mentioned in my first post, it is looking for the directory in cwd of the node where the code is being executed and since it is not able to find it , it is trying to create one. So basically it should look for the directory in edge server instead of the directory mentioned in the cwd of the error posted.
... View more
04-07-2023
04:59 AM
@Shelton Below is the hadoop fs function being used copyToLocalFile(new Path(src_HDFSPath), new Path(dest_edgePath)) Please note that my goal is to copy the file from HDFS to edge server local file system when i run the spark job in cluster mode
... View more
04-06-2023
07:28 PM
Hello, I have a scenario where I need to copy the file from HDFS to local file system on edge server.
When i run the code in cluster mode with spark submit , I am getting below kind of error:
User class threw exception: java.io.IOException: Mkdirs failed to create /some_specific/path/in_edge_server (exists=false, cwd=file:/some/path/yarn/nm/usercache/1234/appcache/application_123456744/container_t1234)
It seems like it is trying to find the destination path where the execution takes place and since it is not finding the specific path it is trying to create a directory and failing?
Below is the function being used: fs.copyToLocalFile(new Path(src_HDFSPath), new Path(dest_edgePath))
Do we have a solution to copy the file from HDFS to specific path of edge server when we run the program in cluster mode?
... View more
Labels:
09-19-2022
02:05 AM
Hello, I wanted to understand what happens to the process and flow files which is currently running when the nifi gets restarted? Please share the details for the better understanding of nifi.
... View more
Labels:
- Labels:
-
Apache NiFi