About shivanageshchow

shivanageshchow · ‎09-24-2016

Merge is something we do to get records together, in joins we append columns and build a record.... how can we say merge in this case. If merge? what is the meaning merging records in one table with records in another??

shivanageshchow · ‎09-19-2016

I have tested it that we can run the jobs on nodes where there is no data node daemon running and is configured as a edge node. correct me if i am wrong.

shivanageshchow · ‎09-19-2016

Junping Du what is the word "places" in "flexible for AM to pickup places to store temporary results/progress" exactly refering to?/ RMstatestore?? or any other place ?? if any other place what is it and can it be configurable??

shivanageshchow · ‎09-19-2016

if i configure my edge node and not as data node i cannot store data in that datanode . But can i configure node manager on edge node and can i bring the data to the edge node and run the task if all other nodes are busy??

shivanageshchow · ‎09-19-2016

in SMB join.

shivanageshchow · ‎09-19-2016

@Rushikesh Deshmukh What is the purpose of merging the tables used in joins ?? can you please explain??

shivanageshchow · ‎09-16-2016

1)Why Secondary namenode is explicitly copying Fsimage from Primary name node when secondary name node is having the same copy of FS image as primary has? There is no guaranty that the fs image in secondary namenode will be exactly same as that in Primary namenode. During checkpoint period of time , there may happen any corruption of data or any crashes and data loss. Its better to get the latest available data from Primary namenode and then merge the editlogs. 2) Initially when cluster is setup will it be having any fsimage at primary node if yes will it contains any data. Yes, When a new namenode is setup in a new cluster it will have a FSimage with no data in it with file name like Fsimage_000000000 representing no transactions. 3) Looks like both primary name node and secondary name node are maintaining all the transaction logs? Is it required to maintain same logs in both locations? if yes, How many old transactions that we have to keep in cluster? is there any configuration for this By default HDFS stores till the transactions count reaches 1 million. Files which are storing transaction logs greater than 1 million are removed from HDFS.

shivanageshchow · ‎09-14-2016

@Eugene Koifman in which version the bug has been fixed?? i am currently using 0.14.0.2.2 .

shivanageshchow · ‎09-13-2016

@Eugene Koifman when i tried to set the txn.manager to db and run an update statement in Hive, it didnt work. But when i open a new session and updated the txn.manager to db it worked. Is there a specific reason to specify a new session for txns in Hive.

shivanageshchow · ‎09-13-2016

@Rajkumar Singh The state-store of RM is fine , will be in levelDB or zkRMstatestore or FSRMstatestore. My doubt is , about Appmaster. As per my knowledge Appmaster will only notify RM about the status of the application, but not at the each task level. I have read that Appmaster will store its state in a persisted storage. What is this persisted storage exactly.

shivanageshchow · ‎09-13-2016

shivanageshchow · ‎09-11-2016

shivanageshchow · ‎09-11-2016

shivanageshchow · ‎09-07-2016

@Lester Martin In this case, as sequence files stores many small files in it.. how does a mapper know in which sequence file the needed file is?? it cant go through all the sequence files right??

shivanageshchow · ‎08-26-2016

Zookeeper does that. But how and where the "state" of the machine is stored?? can i see that??

shivanageshchow · ‎08-10-2016

According to the definition sequence is a container of the small files and increases the efficiency of the mapper by reading complete block rather than small parts. And also decrease load on Namenode by decreasing the Metadata for each file. We can store files of any format in a sequence file. Correct me if i am wrong Coming to my question, Does Namenode stores the metadata of a sequence file differently compared to normal files?? if yes how it is exactly?? And while processing how does Mapper knows which sequence file contains the data it should process?? and if mapper is to process one file which is contained in a sequence file again it reads only that part. Where is the improvement??

shivanageshchow · ‎08-04-2016

Am not familiar with spark. But looks like it has some functions to meet your requirement. http://stackoverflow.com/questions/36436020/converting-csv-to-orc-with-spark

shivanageshchow · ‎08-04-2016

The -m or --num-mappers is just a hint to the engine to maintain that degree of parallelism. But its not mandatory to launch those number of tasks always. The mappers count may vary based on you input data. Sqoop client serializes the data, generates the deserializer and sets the inputformat and submits the job to be run. Here, the inputformat controls the number of mappers like it happens in the normal text file processing. This also answers your second question where some mappers launched may not find the start() of the data in the split and will not be run.

shivanageshchow · ‎08-04-2016

@Josh Persinger The -m or --num-mappers is just a hint to the engine to maintain that degree of parallelism. But its not mandatory to launch those number of tasks always. The mappers count may vary based on you input data. Sqoop client serializes the data, generates the deserializer and sets the inputformat and submits the job to be run. Maybe the inputformat controls the number of mappers like it happens in the normal text file processing. This also answers your second question where some mappers launched may not find the start() of the data in the split and will not be run.

shivanageshchow · ‎08-04-2016

@Benjamin Leonhardi As per YARN appMaster is a mere code. So am unable to figure out how the new DAG can be submitted to existing AppMaster written to handle some other DAG.

shivanageshchow · ‎08-04-2016

Thank you @Shiv kumar

shivanageshchow · ‎08-04-2016

So, the handshake between client and AppMaster in YARN(which decommissions once job is done) is continued here in a Tez session. and client submits new DAGs directly to AppMaster and resource manager thinks its still the same application running , so the DAGs run with same application id. Correct me if i am wrong.

shivanageshchow · ‎08-03-2016

Hi @ARUN The main reason might be the data blocks needed for the MapReduce job to run are located in those two nodes itself. Can you please check the data blocks of the file you are processing and verify that the data is distributed in 3 nodes. Speculative execution( case when your nodes are too busy running the tasks then the data can be moved temporarily to the third node and run the task.) also not be happening.

shivanageshchow · ‎08-03-2016

As per Tez sessions, DAGs submitted within a session are handled by the same AppMaster. Unable to understand how the new application (DAG) is mapped to the already running AppMaster?? Who does it and how?? As per YARN the resource manager is responsible for launching appmasters. How this functionality is eclipsed by Tez?? Thanks in advance.

shivanageshchow · ‎08-03-2016

Am not feeling good to say this. But am not satisfied with you answer. It is fine that application master doing the job of calling inputformat() adn calcuating the input splits and goes on. But am asking what is the meaning of the sentence quoted in the Definitive guide that client places computed inputsplits in HDFS. Am sorry if i am unable to explain my doubt properly.

shivanageshchow · ‎08-01-2016

@Shiv kumar That is what am saying. So " where this happens? " is my question.

shivanageshchow · ‎07-26-2016

Very neatly explained.!

shivanageshchow · ‎07-26-2016

so is it like it will read all the data 1GB and then split the data into logical splits and assign map task to it?? Then what are the computed input splits placed in HDFS while job being submitted... at that AppMaster will not be even launched. and how come 1 GB file will be divided into 10 splits if the block size is 256?? the division is based on splitsize which can be configurable (as of my knowledge).

shivanageshchow · ‎07-26-2016

If it runs in the Appmaster, what exactly are "the computed input splits" that jobclient stores into HDFS while submitting the Job ?? "Copies the resources needed to run the job, including the job JAR file, the configuration file, and the computed input splits, to the shared filesystem in a directory named after the job ID (step 3).". Above is the line form Hadoop Definitive guide. And how map works if the split spans over data blocks in two different data nodes??

shivanageshchow · ‎07-19-2016

Can MaintenanceMode be the answer?? if yes what happens when a node is kept in maintenance mode.?? How replication works for the data kept in maintenance mode node.?? what happens when i decomission a data node?? and what happens when i delete a datanode???

Online	Offline
Last Visited	‎11-08-2017 01:54 PM

Date Registered	‎07-04-2016 05:55 AM
Last Visited	‎11-08-2017 01:54 PM
Total Messages Posted	38
Total Kudos Received	5

Re: Why Secondary namenode is explicitly copying F...

Re: What is the purpose of merging the tables used...

Re: Can i use edge nodes for mapreduce??

Re: Where is the state of the application master i...

Can i use edge nodes for mapreduce??

What is the purpose of merging the tables used in ...

Re: What is Sort Merge Bucket (SMB) Join in Hive? ...

Re: Why Secondary namenode is explicitly copying F...

Re: why to set the transaction properties in hive ...

Re: why to set the transaction properties in hive ...

Re: Where is the state of the application master i...

Where is the state of the application master is pe...

why to set the transaction properties in hive in t...

what is the usecase of Strict mode in Dynamic part...

Re: How Map reduce works on Sequence Files?

How is the state of the cluster saved and replica...

How Map reduce works on Sequence Files?

Re: How to load CSV file directly into Hive ORC ta...

Re: Sqoop Importing Files with no data in them

Re: Sqoop ignoring number of mappers

Re: How new DAGs are submitted to existing Tez App...

Re: How new DAGs are submitted to existing Tez App...

Re: How new DAGs are submitted to existing Tez App...

Re: Load not distributed in the cluster

How new DAGs are submitted to existing Tez Applica...

Re: where an when does the fileinputformat() runs...

Re: where an when does the fileinputformat() runs...

Re: On what basis Application Master decides that ...

Re: where an when does the fileinputformat() runs...

where an when does the fileinputformat() runs.?

Is there a way to take the data -node willing down...