Do I need to put the namenode in safe mode to execute this command? or I can execute this on live cluster?   hadoop fs –setrep –w 3 -R / ... View more

Hadoop supports two kinds of joins to join two or more data sets based on some column. The Map side join and the reduce side join. Map side join is usually used when one data set is large and the other data set is small. Whereas the Reduce side join can join both the large data sets. The Map side join is faster as it does not have to wait for all mappers to complete as in case of reducer. Hence reduce side join is slower. Map Side Join: · Sorted by the same key. · Equal number of partition. · All the records of the same key should be in same partition. Reduce Side Join: · Much flexible to implement. · There has to be custom WritableComparable with necessary function over ridden. · We need a custom partitioner. · Custom group comparator is required. ... View more

Normal SSH gateway requires the password to be entered each time when a service tries to connect to any node. This will slow down the process to a great extent. Usually, Passwordless ssh is getting set up in distributed technology, where always node to node communication must be faster. As we know that Hadoop is fully distributed the technology. All data getting a store in multiple commodity hardware, so there must be faster communication with each other. Actually, Hadoop is working on the Master-Slave architecture. When a client needs to store or access the data from HDFS, then its submit request to Master node then the master node distributing the request to multiple Slave nodes. Mean if we are not doing Passwordless ssh setup then for every client request Master will need to login slaves via credentials. Is this really feasible for faster data processing? Of course No. That’s why we need the Passwordless ssh setup in Hadoop to make it feasible. Master will not need to log in the slaves, it will directly go to the slave address and will fetch or store the required data. ... View more

The users can be created using below steps: a)Get the information from user as to which machine is he working from. b)create the user in in OS first. c)Create the user in Hadoop by creating his home folder /user/username in Hadoop d)make sure that we have 777 permission for temp directory in HDFS e)using chown command change ownership from Hadoop to user for only his home directory so that he can write into only his directory and not other users. f)add his name into name node hdfs dfsadmin -refreshUserToGroupMappings G)If needed set a space limit for the user to limit the amount of data stored by him.hdfs dfsadmin -setSpaceQuota 50g /user/username ... View more

Once a map reduce program is built a driver class has to be created that will be submitted to the cluster. For this, we create the object of the JobConf class. One of the properties of this object is setMapperClass. Conf.setMapperClass sets the mapper class in the driver. It helps the driver class to get the details like reading data and generating key-Value pairs out of the mapper. ... View more

@Dukool SHarma Any updates? ... View more

RAM.Because metadata information will be needing in every 3 seconds after each Heartbeat. So fast processing will require to process the metadata information. To fasten this kinetic momentum of metadata, Name Node used to stores it into RAM. How we can change Replication factor when Data is already stored in HDFS hdfs-site.xml is used to configure HDFS . Changing the dfs.replication property in hdfs-site.xml will change the default replication for all files placed in HDFS. or using hadoop fs shell "hadoop fs –setrep –w 3 ... View more

Sorting is carried out at the Map side. When all the map outputs have been copied, the reduce task moves into the sort phase i.e.maerging phase. which merges the map outputs, maintaining their sort ordering. This is done in rounds. For example, if there were 60 map outputs and the merge factor was 15 (the default, controlled by the mapreduce.task.io.sort.factor property, just like in the map’s merge), there would be four rounds. Each round would merge 15 files into 1, so at the end, there would be 4 intermediate files to be processed. This is done using a key-value pair. ... View more

There is no specific rule in Hadoop on how many times a combiner should be called. Sometimes it may not be called at all, while Sometimes it may be used once, twice or more depending on the number and size of the output file generated by the mapper. ... View more

When the mapper starts producing the intermediate output it does not directly write the data on the local disk. Rather it writers the data in memory and some sorting of the data (Quick Sort) happens for performance reasons. Each map task has a circular memory buffer which it writes the output to. By default, this circular buffer is of 100 MB. It can be modified by the parameter mapreduce.task.io.sort.mb. When the contents of the buffer reach a certain threshold size (MapReduce.map.sort.spill.percent, which has the default value 0.80, or 80%), a background thread will start to spill the contents to disk. Map outputs will continue to be written to the buffer while the spill takes place, but if the buffer fills up during this time, the map will block until the spill is complete. ... View more

There are two phases normally in a MapReduce job, Map phase and Reduce phase. As the name Map, only job itself depicts that the Map only job contains only one phase, Map phase. So hence there’s no sorting and shuffling of intermediate key-value pairs involved in the process, no need of partitioner and combiner, aggregation or summation of key-value pairs is not required, so the output of mapper is directly written to HDFS . Not all jobs can be processed using map only jobs rather jobs like data parsing can be done. Therefore, map only jobs performance is better than MapReduce jobs. ... View more

yes,If u want to build hadoop cluster quickly, u can use docker. ... View more

hdfs-site.xml – This file contains the configuration setting for HDFS daemons. hdfs-site.xml also specify default block replication and permission checking on HDFS. The three main hdfs-site.xml properties are: dfs.name.dir gives you the location where NameNode stores the metadata (FsImage and edit logs). And also specify where DFS should locate – on the disk or in the remote directory. dfs.data.dir gives the location of DataNodes where it stores the data. fs.checkpoint.dir is the directory on the file system. On which secondary NameNode stores the temporary images of edit logs. Then this EditLogs and FsImage will merge for backup. ... View more

Let's start with Hive and then HCatalog. Hive Layer for analyzing, querying and managing large datasets that reside in Hadoop various file systems ⇢ uses HiveQL (HQL) as processing engine ⇢ uses SerDes for serialization and deserialization ⇢ works best with huge volumes of data HCatalog Table and storage management layer for Hadoop ⇢ basically, the EDW system for Hadoop (it supports several file formats such as RCFile, CSV, JSON, SequenceFile, ORC) ⇢ is a sub-component of Hive, which enables ETL processes ⇢ tool for accessing metadata that reside in Hive Metastore ⇢ acts as an API to expose the metastore as REST interface to external tools such as Pig ⇢ uses WebHcat, a web server for engaging with the Hive Metastore I think the focus has to be made on how they complement each other rather than focusing on their differences. Documentation (3) This answer from @Scott Shaw is worth checking This slideshare presents the use cases and features of Hive and Hcatalog This direct graph from IBM shows how they use both layers in a batch job I hope this helps! 🙂 ... View more

It is a Java class for reading Hadoop SequenceFIles http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/lib/input/SequenceFileInputFormat.html ... View more

There are many use cases to leverage DS. Typically when you require reference/lookup data to be available across the nifi cluster, DS is a good fit. Ie enriching dataflow. ... View more

Btw, they spam us with granular & top-notch resources. I think it worths the spam. ^.^ ... View more

Yes, as long as the appropriate clients are installed on the slave node, if you also have the /etc/config populated with the correct details for connecting to your instance, then no connection parameters need to be specified for the clients (this is automatically populated if the slave node is deployed/configured by ambari). in that case you submit the job exactly as you would on any other node ... View more

The number of instance of record readers will be equal to the number of input splits( the number of mappers). In the parallel processing, mappers will run parallelly and each mapper will create an instance of the mapper. ... View more

Both Flume & Kafka are used for real-time event processing but they are quite different from each other as per below mentioned points: 1. Kafka is a general purpose publish-subscribe model messaging system. It is not specifically designed for Hadoop as hadoop ecosystem just acts as one of its possible consumer. On the other hand flume is a part of Hadoop ecosystem , which is used for efficiently collecting, aggregating, and moving large amounts of data from many different sources to a centralized data store, such as HDFS or HBase. It is more tightly integrated with Hadoop ecosystem. Ex, the flume HDFS sink integrates with the HDFS security very well. So its common use case is to act as a data pipeline to ingest data into Hadoop. 2. It is very easy to increase the number of consumers in kafka without affecting its performance & without any downtime. Also it does not keep any track of messages in the topic delivered to consumers. Although it is the consumer’s responsibility to do the tracking of data through offset. Hence it is very scalable contrary to flume as adding more consumers in the flume means changing the topology of Flume pipeline design, which requires some downtime also. 3. Kafka is basically working as a pull model. kafka different consumers can pull data from their respective topic at same time as consumer can process their data in real-time as well as batch mode. On the contrary flume supports push model as there may be a chances of getting data loss if consumer does not recover their data expeditly. 4. Kafka supports both synchronous and asynchronous replication based on your durability requirement and it uses commodity hard drive. Flume supports both ephemeral memory-based channel and durable file-based channel. Even when you use a durable file-based channel, any event stored in a channel not yet written to a sink will be unavailable until the agent is recovered. Moreover, the file-based channel does not replicate event data to a different node. It totally depends on the durability of the storage it writes upon. 5. For Kafka we need to write our own producer and consumer but in case of flume, it uses built-in sources and sinks, which can be used out of box. That’s why if flume agent failure occurs then we lose events in the channel. 6. Kafka always needs to integrate with other event processing framework, that’s why it does not provide native support for message processing In contrast, Flume supports different data flow models and interceptors chaining, which makes event filtering and transforming very easy. For example, you can filter out messages that you are not interested in the pipeline first before sending it through the network for obvious performance reason. However, It is not suitable for complex event processing. ... View more

HDFS Block- Block is a continuous location on the hard drive where data is stored. In general, FileSystem stores data as a collection of blocks. In the same way, HDFS stores each file as blocks. The Hadoop application is responsible for distributing the data block across multiple nodes. Input Split in Hadoop- The data to be processed by an individual Mapper is represented by InputSplit. The split is divided into records and each record (which is a key-value pair) is processed by the map. The number of map tasks is equal to the number of InputSplits. Initially, the data for MapReduce task is stored in input files and input files typically reside in HDFS. InputFormat is used to define how these input files are split and read. InputFormat is responsible for creating InputSplit. MapReduce InputSplit vs Blocks in Hadoop InputSplit vs Block Size in Hadoop- • Block – The default size of the HDFS block is 128 MB which we can configure as per our requirement. All blocks of the file are of the same size except the last block, which can be of same size or smaller. The files are split into 128 MB blocks and then stored into Hadoop FileSystem. • InputSplit – By default, split size is approximately equal to block size. InputSplit is user defined and the user can control split size based on the size of data in MapReduce program. Data Representation in Hadoop Blocks vs InputSplit- • Block – It is the physical representation of data. It contains a minimum amount of data that can be read or write. • InputSplit – It is the logical representation of data present in the block. It is used during data processing in MapReduce program or other processing techniques. InputSplit doesn’t contain actual data, but a reference to the data ... View more

Small file means files which are considerably smaller than block size(64 MB or 128 MB) from Hadoop perspective. Since Hadoop is used for processing huge amount of data, if we are using small files, a number of files would be obviously large. Hadoop is actually designed for a large amount of data ie a small number of large files. Following are the issues with small file 1. Each file, directory, and a block in HDFS is represented as an object in name node’s memory (ie Metadata), and each of which occupies approx. 150 bytes. Scaling these much amounts of memory in the name node for each of these objects is not feasible. In short, if a number of files increases, the memory required to store metadata will be more. 2. HDFS is not designed for efficient access of small files. Handling a large number of small files causes a lot of seeks and a lot of hopping from the data node to the data node to retrieve small files. This is an inefficient data access pattern. 3. Mapper node usually takes a block of input at a time. If the file is very small(ie less than typical block size), the number of mapper task would increase and each task process very little input. This would create a lot of task in queue and overhead would be high. This decreases the overall speed and efficiency of map jobs. Solution: 1. Hadoop archive Files (HAR): HAR command creates a HAR file, which runs a map reduce job to prevent HDFS data to get archived into small files. HAR ensures file size is large and the number is low. 2. Sequence files: By this method, data is stored in such a way that file name will be kay and file name will be valued. MapReduce programs can be created to make a lot of small files into a single sequence file. MapReduce divides sequence files into parts and works on each part independently. ... View more

Hi @Harshali Patel , regarding _HDFS_ there is no need to use RAID at all. In addition to @Aayush Kasliwal 's answer, I'd highly recommend to configure Namenode-HA to avoid any single-point-of-failure for HDFS. This also ensures that the Namenode Metadata will be written in multiple copies throughout the JournalNodes (e.g. you can configure multiple directories and you should use e.g. 3 JournalNodes). Where I see RAID as benefitial are the partitions for OS, logs, ... but of course, this is "below" HDFS HTH, Gerd ... View more

@Harshali Patel Any updates? If you found this answer addressed your question, please take a moment to log in and click the "accept" link on the answer. ... View more

a) Export the eclipse project as .JAR file b) Copy the .JAR file in Ubuntu desktop c) Move the file from ubuntu desktop to home path using mv command d) Write few lines for word count code execution through nano file name e) Create directory in hdfs dfs using –mkdir f) Put the code in that directory using –put g) Execute the job using YARN jar ../jarname packagename.drivername /inputfilename /outputfilename h) Check output status using hdfs dfs –ls /outputfilename i) It will show you actual output file name j) Check the output : hdfs dfs *cat /output / outputfilename(generated in step i) ... View more

A Small file typically means file size less than the default block size(64 MB). Small file can affect the performance of complete Hadoop clusters. Problems caused by Small files to HDFS are as follows: ------------------------------------------------------------------------- If we store a lot of small blocks in the datanode, namenode should store all the metadata in its memory. Namenode typicaly stores files, directories,blocks etc in its memory and each of these will be represented as an object of 150 bytes approx. So, a lot of metadata in namenode memory can slow down the system and affect the performance. Secondly, if a client comes and looks for the data in the block present in a datanode, and if the block is small and a part of client’s data is present in another node, client needs to communicate again with namenode to get the location of the new block for reading rest of data. This can slow down the system and affect the performance. Problems caused by Small files to Mapreduce are as follows: ------------------------------------------------------------------------ In Mapreduce design, a map process a single block at a time. For this a JVM starts. Consider there are millions of small blocks, so millions of map tasks have to be initiated and this can take time to process all the data. But at the same time if a single map can process an ideal sized(64MB) block, it can result in optimum performance in Mapreduce processing. ... View more

Recordwriter writes the output the reducer phase (output can be zero 1 or more key value pairs. ) to the output files output format determines how recordwrite writes these key value pairs in output files files output format subclass will create output files , even if they empty but some application prefer not to create empty files , lazy output format used in this scenario . ... View more