FIFO was the default scheduler in Hadoop1, when you deploy Apache Hadoop vanilla version. It is not used in production, as only 1 job can run. But 1 job can have many containers and each node in the cluster can have few of them running. Size of the container has nothing to do with how many jobs are running on a cluster. It is decided by the map/reduce memory which asks for a container from YARN, which should be a multiple of yarn minimum allocation. There are lot of details here, but keeping it simple the size of container is not related to the number of containers. Yes, for mathematical calculations and to find how many containers can run on a node, we say that the number is equal to total memory avail for yarn/container memory. Size of a container is decided based on the request, whether it is map or reduce or spark task container etc. ... View more

You can do: #!/bin/bash usage="Usage: dir_diff.sh [days]" if [ ! "$1" ] then echo $usage exit 1 fi now=$(date +%s) hadoop fs -ls -R /tmp/ | grep "^d" | while read f; do dir_date=`echo $f | awk '{print $6}'` difference=$(( ( $now - $(date -d "$dir_date" +%s) ) / (24 * 60 * 60 ) )) if [ $difference -gt $1 ]; then hadoop fs -rm -r `echo $f | awk '{ print $8 }'`; fi done Replace the directories or files you need to clean up appropriately. ... View more