Member since
01-15-2016
82
Posts
29
Kudos Received
10
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
6336 | 04-03-2017 09:35 PM | |
3963 | 12-29-2016 02:22 PM | |
1185 | 06-27-2016 11:18 AM | |
973 | 06-21-2016 10:08 AM | |
994 | 05-26-2016 01:43 PM |
03-17-2016
05:27 PM
Use jar files from your azure cluster, not sandbox. You need exactly same versions of libs used on azure cluster. Also copy core-site.xml to flume classpath (FLUME_HOME/conf should be fine) Regards
... View more
03-16-2016
11:52 AM
Seems a common issue for single node cluster and sandboxes:)
https://www.google.pl/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=hadoop+file+could+only+be+replicated+to+0+nodes+instead+of+minReplication+(%3D1)
... View more
03-16-2016
11:45 AM
This is actually steps for windows. And i tested it locally - it works
... View more
03-15-2016
08:38 PM
2 Kudos
I can propose much easier steps: 1. Download flume binaries - http://flume.apache.org/download.html and extract it somewhere (this is going to be a FLUME_HOME) 2. Download winutils and put it somwhere (f.e. C:/winutils/bin, in this case C:/winutils is going to be a HADOOP_HOME) 3. Copy all missed hdfs libs to your FLUME_HOME/lib (you can find them in your hadoop cluster, is always preferable to have exact the same versions as in /usr/hdp/current/hadoop or /usr/hdp/current/hadoop-hdfs) 4. Run flume agent with the following command: bin\flume-ng agent -name MyAgent -f conf/MyAgent.properties -property "flume.root.logger=INFO,LOGFILE,console;flume.log.file=MyLog.log;hadoop.home.dir=C:/winutils"
... View more
03-15-2016
08:30 PM
2 Kudos
That should be commons-configuration, commons-io and htrace-core from /usr/hdp/current/hadoop/lib
... View more
03-14-2016
03:36 PM
2 Kudos
I've never tried that scenario, but it should be possible. All you need is to install flume on windows machine (just extract zip file) and add jars needed to connect to azure (if any). You can use hdfs.kerberosPrincipal, hdfs.kerberosKeytab properties if you have secure hdfs Regards
... View more
03-08-2016
06:59 PM
1 Kudo
Just put everything to single config. Like Agent1.sources..
Agent1.sinks..
Agent1.channels..
Agent2.sources..
Agent2.sinks..
Agent2.channels..
Note, is possible to manage those agent separately (ambari will split them) and each one will run in a separate process
... View more
02-03-2016
06:44 PM
3 Kudos
Why dont use ambari rest api to manage Flume configs outside of main ambari screen.
Personally i didnt tested it, but it should be a valid approach. Also one issue is still there (it prevents us to use ambari for Flume managing): https://issues.apache.org/jira/browse/AMBARI-9421
And the workaround mentioned doesn't works with more than 1 agent
... View more
02-03-2016
06:35 PM
Replace <ok to="get_run_date" /> With <ok to="join-fork-actions" /> In general each of the "subflows" should end up with the join node.
... View more
01-16-2016
11:40 PM
I'd recommend to test connectivity first. You can do it with beeline tool. Once you connected to jdbc url from beeline u can continue experiments with eclipse.
... View more
- « Previous
- Next »