Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

how can we read multiple files from one directory in Blob storage for MapReduce Job?

Highlighted

how can we read multiple files from one directory in Blob storage for MapReduce Job?

New Contributor

I'm using Azure HDInsight.I have written MapReduce code in .Net SDK. The following code i have written to read multiple files: HadoopJobConfiguration hadoopConfiguration = new HadoopJobConfiguration(); hadoopConfiguration.InputPath ="wasb:///demo/inputTwinkle/*.txt"; hadoopConfiguration.OutputFolder = "wasb:///demo/outputTwinkle"; but it did't work. can please provide a way to read multiple files using .NET with hdinsight.

1 REPLY 1

Re: how can we read multiple files from one directory in Blob storage for MapReduce Job?

Super Guru
@priyal patel

I am not familiar with .Net SDK, so I'll let you find out an equivalent class but in Hadoop there is a MultipleInput class which lets you read multiple files in parallel.

https://hadoop.apache.org/docs/r2.6.3/api/org/apache/hadoop/mapreduce/lib/input/MultipleInputs.html

Don't have an account?
Coming from Hortonworks? Activate your account here