Support Questions
Find answers, ask questions, and share your expertise

how can we read multiple files from one directory in Blob storage for MapReduce Job?

Explorer

I'm using Azure HDInsight.I have written MapReduce code in .Net SDK. The following code i have written to read multiple files: HadoopJobConfiguration hadoopConfiguration = new HadoopJobConfiguration(); hadoopConfiguration.InputPath ="wasb:///demo/inputTwinkle/*.txt"; hadoopConfiguration.OutputFolder = "wasb:///demo/outputTwinkle"; but it did't work. can please provide a way to read multiple files using .NET with hdinsight.

1 REPLY 1

Super Guru
@priyal patel

I am not familiar with .Net SDK, so I'll let you find out an equivalent class but in Hadoop there is a MultipleInput class which lets you read multiple files in parallel.

https://hadoop.apache.org/docs/r2.6.3/api/org/apache/hadoop/mapreduce/lib/input/MultipleInputs.html

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.