twitter4j jars included to Flume libs by default. However, twitter source from cloudera is built with another version of twitter4j framework. I'd recommend to remove all *twitter4j* jars from flume_home/libs folder and add proper version (mentioned in cloudera's source pom) to aux_lib instead (along with custom source) ... View more

I suppose is the issue with loading data. Try to create external table instead.. create EXTERNAL table tweets .... row format serde 'org.openx.data.jsonserde.JsonSerDe' LOCATION '/tmp/tweets_staging/'; ... View more

As i recall is smth related to nested arrays. We're using another JSON serde lib and it does work with any complexity of jsons. Here i posted an example of twitter table ddl which is tested well. Regards, Michael ... View more

It would be great to see the log of the agent ... View more

Can you explain a bit the issue with MQ? Im not an expert in WebSphere, but seems MQ is supposed to deliver each event only once. So, there should be no duplicates by design. Is it correct? ... View more

I'd say (in general) whenever you need to parallelize your algorithm, and i suppose TF-IDF is a good candidate for it, you need to submit this job to the cluster in any way. It can be a streaming mentioned by @Lester Martin, or Pyspark mentioned by @Artem Ervits (just note - spark is not map-reduce, so if you want to learn map-reduce first, then streaming option is the best for you). And in case you have some lite algorithm to implement and it can be done on client machine/your laptop/application server etc - you can just submit to Hadoop cluster some Hive query and process the results locally then. ... View more

hadoop-annotations-2.7.1.2.3.4.0-3485.jar hadoop-auth-2.7.1.2.3.4.0-3485.jar hadoop-aws-2.7.1.2.3.4.0-3485.jar hadoop-azure-2.7.1.2.3.4.0-3485.jar hadoop-common-2.7.1.2.3.4.0-3485-tests.jar hadoop-common-2.7.1.2.3.4.0-3485.jar hadoop-nfs-2.7.1.2.3.4.0-3485.jar Double check it's a classes from Azure.. also you need to add hadoop-hdfs.jar and core-site.xml ... View more