@tuxnet Sure you can use any IDE with PySpark. Here is short instructions for Eclipse and PyDev: - set HADOOP_HOME variable referncing location of winutils.exe - set SPARK_HOME variable referencing your local spark folder - set SPARK_CONF_DIR to the folder where you have actual cluster config copied (spark-defaults and log4j) - add %SPARK_HOME%/python/lib/pyspark.zip and %SPARK_HOME%/python/lib/py4j-xx.x.zip to a PYTHONPATH of the interpreter For the testing purposes i'm adding code like: spark = SparkSession.builder.set_master("my-cluster-master-node:7077").. but with proper configuration file in SPARK_CONF_DIR it should work with just SparkSession.builder.getOrCreate() Alternatively you can setup your run configurations to use spark-submit directly. Hope it helps ... View more

@vamsi valiveti you can trigger Flume from the oozie shell action. However pay attention that action will be executed on random cluster node, so all your nodes should have Flume installed. Also you will need to somehow control the agents after that, and if you have >10 nodes it became a problem.. That's why is not common scenario of flume usage. I'd say the good approach is to keep Flume running all the time. And schedule oozie jobs to process the data whenever you need. ... View more

Grant write permissions to /var/log/flume directory. Also you can specify alternative log file for specific agent: -Dflume.log.file=my_path/my_file.log ... View more

High availability in flume is just a matter of agents configuration regardless if you're using Ambari or not. Here few links you can check: https://flume.apache.org/FlumeUserGuide.html#flow-reliability-in-flume https://flume.apache.org/FlumeUserGuide.html#failover-sink-processor ... View more

@azza messaoudi, check the following Twitter doc: https://dev.twitter.com/streaming/reference/post/statuses/filter And here is the custom Flume source implementation with support of all twitter streaming parameters: http://www.dataprocessingtips.com/2016/04/24/custom-twitter-source-for-apache-flume/ (including "follow" parameter which you're interested in actually) ... View more

The easiest way in hortonworks hadoop is to use Ambari to run flume. It will show you some basic metrics and status of the agents. If you dont want to use Ambari or you have some custom flume installation, i'd recommend to read this doc: http://flume.apache.org/FlumeUserGuide.html#monitoring In any linux env you can install atleast ganglia. It will cover most of your needs in terms of agents monitoring ... View more

Well, based on what we know so far, i'd say 2 flume agents with the file or jdbc channel should work for you. There will be no overlap in data because is controlled by MQ itself, so it not a matter of flume. From flume processing side we ensure that no data loss happens by using file or jdbc channel. ... View more

I can propose much easier steps: 1. Download flume binaries - http://flume.apache.org/download.html and extract it somewhere (this is going to be a FLUME_HOME) 2. Download winutils and put it somwhere (f.e. C:/winutils/bin, in this case C:/winutils is going to be a HADOOP_HOME) 3. Copy all missed hdfs libs to your FLUME_HOME/lib (you can find them in your hadoop cluster, is always preferable to have exact the same versions as in /usr/hdp/current/hadoop or /usr/hdp/current/hadoop-hdfs) 4. Run flume agent with the following command: bin\flume-ng agent -name MyAgent -f conf/MyAgent.properties -property "flume.root.logger=INFO,LOGFILE,console;flume.log.file=MyLog.log;hadoop.home.dir=C:/winutils" ... View more

Just put everything to single config. Like Agent1.sources.. Agent1.sinks.. Agent1.channels.. Agent2.sources.. Agent2.sinks.. Agent2.channels.. Note, is possible to manage those agent separately (ambari will split them) and each one will run in a separate process ... View more

Replace <ok to="get_run_date" /> With <ok to="join-fork-actions" /> In general each of the "subflows" should end up with the join node. ... View more