question Re: How to find no of lines in all the files in a hadoop directory? in Archives of Support Questions (Read Only)

How to find no of lines in all the files in a hadoop directory?

balavignesh_nag — Thu, 11 Aug 2016 14:47:01 GMT

Re: How to find no of lines in all the files in a hadoop directory?

ssubhas — Thu, 11 Aug 2016 14:53:01 GMT

@Bala Vignesh N V

You can use below command to check the number of lines in a HDFS file:

[hdfs@ssnode1 root]$ hdfs dfs -cat /tmp/test.txt |wc -l

Re: How to find no of lines in all the files in a hadoop directory?

balavignesh_nag — Thu, 11 Aug 2016 14:56:01 GMT

Sindhu I need to know the count for each file in a directory not for a single file in a directory.

Re: How to find no of lines in all the files in a hadoop directory?

ssharma — Thu, 11 Aug 2016 18:41:28 GMT

@Bala Vignesh N V

You can try below command :

for i in `hdfs dfs -ls -R <DIRECTORY_PATH> | awk '{print $8}'`; do echo $i ; hdfs dfs -cat $i | wc -l; done

It will recursively list the files in <DIRECTORY_PATH> and then print the number of lines in each file.

Re: How to find no of lines in all the files in a hadoop directory?

balavignesh_nag — Thu, 11 Aug 2016 19:36:33 GMT

Thanks ssharma. It helps. Is there any commands available to check the no of lines in each file in a directory or even just to find in a single file?

Re: How to find no of lines in all the files in a hadoop directory?

ssharma — Thu, 11 Aug 2016 22:58:30 GMT

@Bala Vignesh N V

I dont think there's any single command to achieve this. Not only in HDFS but also in regular linux. So its better to use multiple commands with pipes or create a simple script which will provide you the desired output.

Please accept the answer if it was helpful 🙂

Re: How to find no of lines in all the files in a hadoop directory?

jyadav — Fri, 12 Aug 2016 00:49:00 GMT

Hi @Bala Vignesh N V

The above approach is pretty good and work very well when you having small number of files but what if you have thousands or millions of files in directories? In that case its better to use Hadoop Mapreduce framework to do same job on large files but in less time. Below is an example to count lines using mapreduce.

https://sites.google.com/site/hadoopandhive/home/hadoop-how-to-count-number-of-lines-in-a-file-using-map-reduce-framework

Re: How to find no of lines in all the files in a hadoop directory?

gvkmdkra — Fri, 21 Sep 2018 21:53:31 GMT

To get the sum of count all rows in a directory, you can follow the below.

a=0

for i in `hdfs dfs -ls -R <DIRECTORY_PATH> | awk '{print $8}'`;

echo $i ;

b="`hdfs dfs -cat $i | wc -l`";

a=`expr $a + $b`

echo $a;

done

Re: How to find no of lines in all the files in a hadoop directory?

Ram303 — Mon, 20 Apr 2020 16:47:45 GMT

hdfs dfs -ls -R <directory> |grep part-r* |awk '{print $8}' |xargs hdfs dfs -cat | wc -l