question Re: How indexing is done in HDFS? in Archives of Support Questions (Read Only)

How indexing is done in HDFS?

rushikeshdeshmu — Fri, 19 Feb 2016 14:26:18 GMT

Hi,

Can anyone explains me how indexing is done in HDFS?

Re: How indexing is done in HDFS?

rahulpathak109 — Fri, 19 Feb 2016 14:37:07 GMT

@Rushikesh Deshmukh

See this http://stackoverflow.com/questions/11954904/indexing-and-searching-in-hadoop

Re: How indexing is done in HDFS?

rushikeshdeshmu — Fri, 19 Feb 2016 14:42:38 GMT

I got below answer:

Hadoop has its own way of indexing. Depending upon the block size, once the data is stored, HDFS will keep on storing the last part of the data which will say where the next part of the data will be. In fact, this is the base of HDFS.

Is this answer correct?

Re: How indexing is done in HDFS?

bleonhardi — Fri, 19 Feb 2016 17:38:41 GMT

Hello Rukishek,

unless I misunderstand something this is not correct. HDFS doesn't store in the data where the next block is. Instead the Namenode knows which blocks make up a file and also the order of the blocks. Using this the HDFS client knows which block to load at any time if you seek in the file.

HDFS blocks are stupid, simple 128MB cut blocks of the data. Datanodes are stupid and only know which blocks they have. The Namenode pieces it all together using an in-memory image of all files and blocks that make these files and where they are stored. The clients get this information from the namenode.

Now if you mean full-text indexing then you should look at Solr like Rahul said.

Re: How indexing is done in HDFS?

nsabharwal — Fri, 19 Feb 2016 19:31:00 GMT

@Rushikesh Deshmukh Look at @Benjamin Leonhardi answer.

No, it's not correct.