Using NiFi, Storm and Kafka to analyze twitter data

[#1]

Repo Description

This Demo is built for Hortonworks HDP 2.3 Sandbox.

This is based on the Hortonworks Twitter Demo

Purpose: Monitor Twitter stream for the procided Hastags & act on unexpected increases in tweet volume

• Ingest: Listen for Twitter streams related to Hashtags input in NiFi Garden Hose (GetHTTP) processor
• Processing:
  • Monitor tweets for unexpected volume
  • Volume thresholds managed in HBASE
• Persistence:
  • HDFS (for future batch processing)
  • Hive (for interactive query)
  • HBase (for realtime alerts)
  • Solr/Banana (for search and reports/dashboards)
• Refine:
  • Update threshold values based on historical analysis of tweet volumes
• Demo setup:
  • Either download and start prebuilt VM
  • Start HDP 2.3 sandbox and run provided scripts to setup demo

Repo Info
Github Repo URL https://github.com/vedantja/hdp_nifi_twitter_demo
Github account name vedantja
Repo name hdp_nifi_twitter_demo