The request for n raises a NameError. This is since n is a variable local to func and we cannot access it elsewhere. It is also true that Python only evaluates default parameter values once; every invocation shares the default value. If one invocation modifies it, that is what another gets. This means you should only ever use primitives, strings, and tuples as default parameters, not mutable objects. ... View more

The number of instance of record readers will be equal to the number of input splits( the number of mappers). In the parallel processing, mappers will run parallelly and each mapper will create an instance of the mapper. ... View more

The cluster consists of one or more computers/machines working together to provide high availability, reliability, and scalability towards service being provided to clients. If one server/machine fails then work/resources get distributed among other machines in the same cluster. Single node cluster or pseudo-distributed cluster is the one in which all daemon like NameNode, data node, Jobtracker, and tasktracker runs on the single machine. Default replication factor is 1. Multinode cluster is a cluster which is basically used in master-slave fashion where master and slaves runs on different machines and master node/machine runs Namenode and TaskTracker daemons and slave machine runs Datanode and JobTracker daemons. Note: - YARN is a cluster resource management. ... View more

The main work of speculative execution is to reduce the job execution time; however, the clustering efficiency is affected due to duplicate tasks. Since in speculative execution redundant tasks are being executed, thus this can reduce overall throughput. For this reason, some cluster administrators prefer to turn off the speculative execution in Hadoop. ... View more

Small file means files which are considerably smaller than block size(64 MB or 128 MB) from Hadoop perspective. Since Hadoop is used for processing huge amount of data, if we are using small files, a number of files would be obviously large. Hadoop is actually designed for a large amount of data ie a small number of large files. Following are the issues with small file 1. Each file, directory, and a block in HDFS is represented as an object in name node’s memory (ie Metadata), and each of which occupies approx. 150 bytes. Scaling these much amounts of memory in the name node for each of these objects is not feasible. In short, if a number of files increases, the memory required to store metadata will be more. 2. HDFS is not designed for efficient access of small files. Handling a large number of small files causes a lot of seeks and a lot of hopping from the data node to the data node to retrieve small files. This is an inefficient data access pattern. 3. Mapper node usually takes a block of input at a time. If the file is very small(ie less than typical block size), the number of mapper task would increase and each task process very little input. This would create a lot of task in queue and overhead would be high. This decreases the overall speed and efficiency of map jobs. Solution: 1. Hadoop archive Files (HAR): HAR command creates a HAR file, which runs a map reduce job to prevent HDFS data to get archived into small files. HAR ensures file size is large and the number is low. 2. Sequence files: By this method, data is stored in such a way that file name will be kay and file name will be valued. MapReduce programs can be created to make a lot of small files into a single sequence file. MapReduce divides sequence files into parts and works on each part independently. ... View more

Speculative execution is a MapReduce job optimization technique in Hadoop that is enabled by default. You can disable speculative execution for mappers and reducers in mapped-site.xml as shown below: <property> <name>mapred.map.tasks.speculative.execution</name> <value>false</value> </property> <property> <name>mapred.reduce.tasks.speculative.execution</name> <value>false</value> </property> ... View more