How indexing is done in HDFS?

[#1]

Hi,

Can anyone explains me how indexing is done in HDFS?

[#2]

@Rushikesh Deshmukh

See this http://stackoverflow.com/questions/11954904/indexing-and-searching-in-hadoop

[#3]

I got below answer:

Hadoop has its own way of indexing. Depending upon the block size, once the data is stored, HDFS will keep on storing the last part of the data which will say where the next part of the data will be. In fact, this is the base of HDFS.

-

Is this answer correct?

[#4]

@Rushikesh Deshmukh Look at @Benjamin Leonhardi answer.

No, it's not correct.

[#5 · Accepted solution]

Hello Rukishek,

unless I misunderstand something this is not correct. HDFS doesn't store in the data where the next block is. Instead the Namenode knows which blocks make up a file and also the order of the blocks. Using this the HDFS client knows which block to load at any time if you seek in the file.

HDFS blocks are stupid, simple 128MB cut blocks of the data. Datanodes are stupid and only know which blocks they have. The Namenode pieces it all together using an in-memory image of all files and blocks that make these files and where they are stored. The clients get this information from the namenode.

Now if you mean full-text indexing then you should look at Solr like Rahul said.