Regarding combining. You would have to extend the existing class, modify the linerecordreader etc. This would only work for text files (not sequence or ORC files ) because they need their own InputFormats. But you could have any kind of content in the text file since the deserialization is handled by the Serde ( which we would not touch ) . Doing the same for sequence files should be possible, doing the same for ORC files is most likely very difficult. In that case InputFormat und Serde are much more interconnected. ... View more

In my opinion you would need to extend the TextInputformat. Then change the linerecordreader class to read the xattributes when reading the filesplit and prefixing each line with the attributes you want before it is been given to the Serde ( you don't want to change that if you can avoid it ) You could also filter out files completely here. You can then do more advanced filters in SQL. http://grepcode.com/file/repo1.maven.org/maven2/org.jvnet.hudson.hadoop/hadoop-core/0.19.1-hudson-2/org/apache/hadoop/mapred/LineRecordReader.java#LineRecordReader ... View more

I think your only choice would be to build a custom InputFormat that reads these properties and then adds them as columns to the data set. I don't know of any other ways to propagate metadata with the data set. ... View more

I suppose you mean HDFS blocks. And the answer is it depends: Normally MapReduce will start one Map task for each block. However that is not always the case. MapReduce provides CombinedFileInputFormat that merge small files into single tasks. In pig you can for example enable/disable this with pig.splitCombination. Tez which is an alternative framework to MapReduce and which is used under Hive and optionally pig also merges small files into map tasks, this is controlled by tez.grouping.min-size and tez.grouping.max-size. For Reducers it is more complicated since they work on map output and not on files. In MapReduce the number of reducers needs to be specified by the user. -Dmapred.reduce.tasks=x However frameworks like hive predict the number of reducers that is optimal and set it depending on data sizes. For example hive ( on tez ) uses hive.exec.reducers.bytes.per.reducer to predict an optimal number of reducers. Pig does something similar. Here the parameter is pig.exec.reducers.bytes.per.reducer. Hope that helps. ... View more

The KeyedMessage by the way can be confusing if called KeyedMessage("topic", "value") your kafka topic does not have a key. The first string is the topic the second the value of the message. This is good for most situations. IF you need a message key ( db primary key for example ) to do log compaction etc. you can do KeyedMessage("topic","key","message") But for you I would assume that you simply serialize the SOAP output into a String and push it into Kafka, then you need a consumer on the other side that can parse it again. ( for example in spark or storm ) ... View more

So you need to pull your data from the soap service. Like you normally would. Which will give you some data. Lets assume a JSON String? There are different options to push it into Kafka, the Example I used directly takes a String, ( there is one taking a byte array as well ) After that you can simple connect a consumer to kafka and you get the message out as you put them in, your consumer then needs to parse the string/byte array to whatever format you want. ... View more

Just be wary of potential load issues. We reached the connection limits of our consolidated postgresql database because all services were pointing to the same db, This essentially stopped oozie and hive randomly. The biggest culprit seems to have been ranger. If auditing to db is switched on it puts quite a load on the database. ... View more