@RAUI Yes there is another way of achieving this. You can use the method copy() from the FileUtil class and pass your FileSystem object to it to effectively copy your files from the source HDFS location to the target. As with using rename() you will need to ensure you target directory is created before calling copy. FileUtil.copy() has a signature where you provide a source and destination FS and in this case you would provide the same FS object since you are looking to copy files to a different location on the same HDFS. There is also a boolean option to delete the source file after the copy if that fits your use case. Here is a link to the FileUtil API: http://hadoop.apache.org/docs/r2.8.0/api/org/apache/hadoop/fs/FileUtil.html ... View more

@Anton P I'm glad it works. I'm not sure how exactly the "fair" ordering policy works inside one queue, but preemption is only for between queues. I assume, that it will try to give resources to the applications/users in the same queue equally, but once a container is running it will not preempt it. If you would like to achieve that, you should consider creating sub-queues. ... View more

@gnovak Thank you so much. I was really confused about which version of Python that requires parentheses after print. I thought it was Python2. ... View more

how can we identify a job taking more resources than allocation? ... View more

Hi, jq can be found in most Linux distributions. If you want to use basic unix commands, maybe try date -d $((1497691710912 / 1000)) Or maybe you can use python, it's also part of every distributions. import json from datetime import datetime def timestamp_to_str(timestamp): return datetime.fromtimestamp(timestamp / 1000).strftime('%Y-%m-%d') def search(timestamp): with open('a') as f: data = json.loads(f.read()) for cluster in data: cluster['original_timestamp'] = timestamp_to_str(cluster['original_timestamp']) if cluster['original_timestamp'] == timestamp: yield cluster ... View more

Ok,thank you very much.I thought that in the fact there is a way to remove it,just I don't know。 ... View more

Unable to update .jar file using command jar uf hadoop-yarn-server-nodemanager-*.jar container-log4j.properties Getting the following error. java.io.IOException: Error in writing existing jar file at sun.tools.jar.Main.run(Main.java:286) at sun.tools.jar.Main.main(Main.java:1288) ... View more

You can check if the values you configured are actually in use with this command: hdfs getconf -confKey yarn.nodemanager.resource.memory-mb or from the ResourceManager UI: <rm.host>:8088/conf If the values are in sync, you can also check the ResourceManager logs for further information. ... View more