About bluesmix

bluesmix · ‎04-05-2017

@tuxnet it should work with spark 1.6 as well. You can check master url in spark-defaults.config file in your cluster. If you setup SPARK_CONF_DIR variable and copy spark-defaults config from your cluster to it there is no need to specify master explicitly.

bluesmix · ‎04-03-2017

@tuxnet Sure you can use any IDE with PySpark. Here is short instructions for Eclipse and PyDev: - set HADOOP_HOME variable referncing location of winutils.exe - set SPARK_HOME variable referencing your local spark folder - set SPARK_CONF_DIR to the folder where you have actual cluster config copied (spark-defaults and log4j) - add %SPARK_HOME%/python/lib/pyspark.zip and %SPARK_HOME%/python/lib/py4j-xx.x.zip to a PYTHONPATH of the interpreter For the testing purposes i'm adding code like: spark = SparkSession.builder.set_master("my-cluster-master-node:7077").. but with proper configuration file in SPARK_CONF_DIR it should work with just SparkSession.builder.getOrCreate() Alternatively you can setup your run configurations to use spark-submit directly. Hope it helps

bluesmix · ‎02-15-2017

@Cord thomas Turn on debug logging and check the log file first

bluesmix · ‎01-20-2017

@shilpa kumar i suspect it's not a debug mode. Try to edit log4j settings in flume config directory. -c argument should reference configuration directory with log4j property. In your case it's /opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/flume-ng/conf

bluesmix · ‎01-20-2017

@shilpa kumar the only suggestion in this case is turn on debug logging and analyze what happens. Good luck with that:)

bluesmix · ‎01-20-2017

@shilpa kumar try to set rollInterval to 0 or greater than 1 hr.

bluesmix · ‎01-20-2017

with this setup: agent.sinks.hdfs-sink.hdfs.path = hdfs://40.122.210.251:8020/user/hdfs/flume/%y-%m-%d/%H%M/%S agent.sinks.hdfs-sink.useLocalTimeStamp = true flume has to roll the files each 1 second, just set the mask to smth like %y-%m-%d/%H

bluesmix · ‎12-29-2016

I'd recommend https://github.com/stickfigure/batchfb over the restfb because of nice implementation of fb batch API. It fits very well for any facebook data consuming task.

bluesmix · ‎12-29-2016

@vamsi valiveti it could be the option, right. But for production usage i'd think additionally about how to stop the agents and how to monitor the agent. From my experience init.d service script + ganglia monitoring is a best option. It allows you to run/stop agents easily with the commands like: /etc/init.d/flume "agent" stop/start. And ganglia provides a nice web interface for the monitoring.

bluesmix · ‎12-29-2016

@vamsi valiveti the easiest way is to detach shell from the command using nohup: nohup <my_command> & Another option is to create flume init.d service script. I've posted some example script here (search for "Setup flume agent auto startup" on the page), and run the flume as a service. And third option is to use Ambari to control the agents.

bluesmix · ‎12-29-2016

@vamsi valiveti you can trigger Flume from the oozie shell action. However pay attention that action will be executed on random cluster node, so all your nodes should have Flume installed. Also you will need to somehow control the agents after that, and if you have >10 nodes it became a problem.. That's why is not common scenario of flume usage. I'd say the good approach is to keep Flume running all the time. And schedule oozie jobs to process the data whenever you need.

bluesmix · ‎12-01-2016

@Magesh Kumar, is hard to say what's happening without the flume logs.. However 2 comments about your config: - you're using file_roll sink, not HDFS - from what i understand that source consumes root folder of ftp server. .folder and .name parameters have another purpose.

bluesmix · ‎12-01-2016

Hi @Avijeet Dash, unfortunately i have no experience with rsyslog, but if i understand correctly is fully compatible with syslog. Flume has some integration with it - https://flume.apache.org/FlumeUserGuide.html#syslog-sources As for network sockets, i'd say there is 2 options - use exec source with some bash command like here - http://stackoverflow.com/questions/4283209/bash-command-to-read-from-network-socket or, write custom source. Benefit of custom source here is more control over the process (i.e. choose between pool and stream types of the source..)

bluesmix · ‎12-01-2016

@Magesh Kumar, 3 Options here: 1) you can mount any remote ftp/share to local linux folder, f.e. - https://linuxconfig.org/mount-remote-ftp-directory-host-locally-into-linux-filesystem and then use flume exec source with tail command or spool directory source. 2) Install 2 flume agents: - on remote host with exec/spool source and avro sink - on hdfs/hadoop host with avro source and hdfs sink 3) Write custom source to serve your needs for any custom protocol or non-standard requirements

bluesmix · ‎11-28-2016

You can try to find particular file by the command itself, instead of just "tail" command use smth like: for i in `find <your_folder> -xtype f`;do tail -n 10 $i; break; done You can put some regexp here in order to find exact file you need. As for writing everything to a single file - i'm not sure it's possible. You can route input file to some particular output folder in hdfs, but not the same file (at least you can not count on that, even if its possible by setting some large rollIntervals) Routing can be done with custom headers/channel selectors and/or interceptors

bluesmix · ‎11-25-2016

Not sure i understand it correctly, but why don't use spooling directory source instead - https://flume.apache.org/FlumeUserGuide.html#spooling-directory-source If you need some complex approach like tail on the latest file in the directory there is no easy way to do it without custom source development

bluesmix · ‎08-17-2016

Usually such issues are result of some mess with twitter4j files in flume installation. I'd recommend to remove all twitter4j*.jar files from flume_home/lib, build twitter source from scratch on your environment and put new jar file along with twitter4j dependencies to a separate folder. And, for sure, check the logs first. Also you might consider to write your own source, or customize cloudera's one. Here i did some comparison of built-in twitter sources - http://www.dataprocessingtips.com/2016/04/24/custom-twitter-source-for-apache-flume/

bluesmix · ‎07-26-2016

Default transactionCapacity for file channel is 10 000. For memory channel - 100 Thats why it works for you. Add transactionCapacity property to your file channel or increase memory available for flume process (like -Xmx1024m)

bluesmix · ‎07-16-2016

Is it a real config? Since i see some typos in it: agent1.channels.hdfs2-chn.transactionCapacity=500000 agent1.channels.hdfs1-chn.checkpointDir = /tmp/flume/check2 agent1.channels.hdfs1-chn.dataDirs = /tmp/flume/channel2 .... agent1.sinks.hdfs1-sink.hdfs.path = /data/hdfs1 agent1.sinks.hdfs2-sink.writeFormat = Text

bluesmix · ‎07-11-2016

@Benny Bazumnik Have you considered custom flume source? You can use Syslog UDP source as an example..

bluesmix · ‎07-03-2016

Kind of this: Where Spool Directory Source can be defined with "ignorePattern" property (check here). If it's not enough you can always adjust existing code of spooling directory source to add your patterns and properties. Is not too complex even without good java knowledge

bluesmix · ‎06-27-2016

Grant write permissions to /var/log/flume directory. Also you can specify alternative log file for specific agent: -Dflume.log.file=my_path/my_file.log

bluesmix · ‎06-24-2016

Hard to say where the issue is.. Usually im following next steps to understand the bottlenecks: 1. Check flume metrics. You can use Ambari metrics or Ganglia monitoring (better option from my POV) 2. Check host status for each agent node (ram consumed by the agent, disk usage if you're using file channel etc) F.e. here might be 2 potential things to try: - If the bottleneck is a source - parallelize sources (create more spooling directories or using some regex to consume files by 2 or more agents) - If the bottleneck is HDFS (too many writers f.e.) - create an "aggregation" tier before actual writing to HDFS

bluesmix · ‎06-23-2016

https://flume.apache.org/FlumeUserGuide.html#flume-topology-design https://flume.apache.org/FlumeUserGuide.html#sizing-a-flume-deployment

bluesmix · ‎06-21-2016

High availability in flume is just a matter of agents configuration regardless if you're using Ambari or not. Here few links you can check: https://flume.apache.org/FlumeUserGuide.html#flow-reliability-in-flume https://flume.apache.org/FlumeUserGuide.html#failover-sink-processor

bluesmix · ‎06-20-2016

I'd say whenever you need some Spark specific features like ML, GraphX or Streaming - use spark as ETL engine since it provides All-in-one solution for most usecases. If you have no such requirements - use Hive on TEZ If you have no TEZ - use Hive on MR In any case Hive acts just like a metastore..

bluesmix · ‎06-20-2016

Try to reproduce it as mentioned here - https://qnalist.com/questions/5206657/invalid-file-with-avro-tools-random-and-tojson Alternate approach is to use another flume source to save data in raw json instead.

bluesmix · ‎06-16-2016

It was there from Ambari 2.0, not sure about earlier versions. There is still not fixed issue preventing us to use it normally:(

bluesmix · ‎06-15-2016

it will be different agent/jvm. And you will be able to control them separately (start/stop) from ambari. Whenever you're restarting ambari flume service it will recreate the agents (config and .sh script) on local FS

bluesmix · ‎06-15-2016

@Benjamin Leonhardi, just a small comment: You CAN run multiple agents on the same host through ambari (at least as of hdp2.3). Just put all the agents config together, like agent1.sources.. agent1.sinks.. agent1.channels ... agent2.sources. agent2.sinks. agent2.channels It will create separate folder for each agent on the host FS

Online	Offline
Last Visited	‎09-05-2017 12:42 PM

Date Registered	‎01-15-2016 07:31 PM
Last Visited	‎09-05-2017 12:42 PM
Total Messages Posted	82
Total Kudos Received	29

Re: Python IDE for HDP Spark cluster

Re: Flume with oozie

Re: Is it possible to run flume agent in multiple ...

Re: Using ambari for high availability setup for f...

Re: Does anyone know how can we stream Twitter dat...

Re: Python IDE for HDP Spark cluster

Re: Python IDE for HDP Spark cluster

Re: Flume is leaving .tmp files,Flume leaving .tmp...

Re: Flume hdfs sink is creating multiple tiny file...

Re: Flume hdfs sink is creating multiple tiny file...

Re: Flume hdfs sink is creating multiple tiny file...

Re: Flume hdfs sink is creating multiple tiny file...

Re: RestFB (Facebook Graph API) in Java

Re: Flume with oozie

Re: Flume with oozie

Re: Flume with oozie

Re: How to load the Logs data from Remote System ...

Re: How to load the Logs data from Remote System ...

Re: How to load the Logs data from Remote System ...

Re: Whether flume reads timestamp from the particu...

Re: Whether flume reads timestamp from the particu...

Re: I am ingesting Twitter data using Flume but do...

Re: java.lang.OutOfMemoryError: Java heap space wi...

Re: Output of Replicating Flume agent does not mat...

Re: Flume UDP source

Re: what are the guidelines to improve performance...

Re: Is it possible to run flume agent in multiple ...

Re: what are the guidelines to improve performance...

Re: what are the guidelines to improve performance...

Re: Using ambari for high availability setup for f...

Re: When to go with ETL on Hive using Tez VS When...

Re: Twitter data ingestion to Hive giving error.

Re: How to configue multiple different flume agent...

Re: How to configue multiple different flume agent...

Re: How to configue multiple different flume agent...