This might be caused by NIFI-5525. Check for double quotes in your CSV. Either remove them or update NiFi to >=1.8.0. ... View more

@RAUI The answer is no. Renaming is the way to move files on HDFS: FileSystem.rename(). Actually, this is exactly what the HDFS shell command "-mv" does as well, you can check it in the source code. If you think about it, it's pretty logical, since when you move a file on the distributed file system, you don't really move any blocks of the file, you just update the "path" metadata of the file in the NameNode. ... View more

@Manikandan Jeyabal Your question is not quite clear to me. If you really want to fetch data from the YARN Resource Manager REST API in Java, all you need to do is open an HttpURLConnection and get the data from any endpoint. E.g.: URL url = new URL("http://" + rmHost + ":8088/ws/v1/cluster/apps"); HttpURLConnection conn = (HttpURLConnection) url.openConnection(); BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream())); ... // read and process your data conn.disconnect(); But there is a much easier solution to get data from the RM in Java: YarnClient, which is basically a Java API for YARN. YarnClient yarnClient = YarnClient.createYarnClient(); Configuration conf = new YarnConfiguration(); conf.set("yarn.resourcemanager.hostname", "your RM hostname"); yarnClient.init(conf); yarnClient.start(); for (ApplicationReport applicationReport : yarnClient.getApplications()) { System.out.println(applicationReport.getApplicationId()); } ... View more

@Anton P You are doing everything just fine, this is by design. The "Ordering Policy" can indeed only be set for leaf queues, because it defines the ordering policy between applications in the same queue. So it has nothing to do with your use case. "I try to run two Yarn queues where if only one queue is active it will consume all the resources and once a job will arrive to the second queue Yarn will preempt some of the resources of the first queue to start the second job." To achieve this, you need to configure your queues like this (I think, you already did this): yarn.scheduler.capacity.root.queues=test1,test2 yarn.scheduler.capacity.root.test1.capacity=50 yarn.scheduler.capacity.root.test1.maximum-capacity=100 yarn.scheduler.capacity.root.test2.capacity=50 yarn.scheduler.capacity.root.test2.maximum-capacity=100 ... and enable preemption (as described in the article you attached). This will let the first application in the first queue to use all the resources, until the second job arrives to the second queue, then the resources will be devided equally between the two queues. Hope this makes everything clear, give it a try 🙂 ... View more