Member since
09-13-2015
13
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
23399 | 09-14-2015 12:32 AM |
10-07-2017
04:23 AM
Hi Team, I am also getting same error.I am able t oconnect remote db with scm,rman,hive,oozie and where as for hue it is throwing error :Oracle client libraries cannot be found on host Any specific steps are required for hue: [root@xxxxx java]# pwd /usr/share/java [root@xxxx java]# ll total 3320 -rwxr-xr-x. 1 cloudera-scm cloudera-scm 3397734 Oct 5 09:53 oracle-connector-java.jar Thanks, Sridhar
... View more
09-14-2015
12:32 AM
Hi Harish, We cleared all things and sink is working now. Sridhar
... View more
09-14-2015
12:03 AM
Hi Harish, Please see modified .conf: # example.conf: A single-node Flume configuration # Name the components on this agent a1.sources = r1 r2 a1.sinks = k1 k2 a1.channels = c1 c2 # Describe/configure the source r1 a1.sources.r1.type = exec a1.sources.r1.command = tail -F /home/hadoop/hadoop-2.7.1/logs/yarn-hadoop-resourcemanager-sri.test.com.log a1.sources.r1.channels = c1 # Describe/configure the source r2 a1.sources.r2.type = exec a1.sources.r2.command = tail -F /var/log/messages a1.sources.r2.channels = c2 # Describe the sink a1.sinks.k1.type = logger a1.sinks.k2.type = logger # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Use a channel which buffers events in memory a1.channels.c2.type = memory a1.channels.c2.capacity = 1000 a1.channels.c2.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sources.r2.channels = c2 a1.sinks.k1.channel = c1 a1.sinks.k2.channel = c2 # Define a sink that outputs to hdfs dir. a1.sink.k1.type = hdfs a1.sink.k1.hdfs.path = /yarn a1.sink.k1.fileType = TEXT # Define a sink that outputs to hdfs dir. a1.sink.k2.type = hdfs a1.sink.k2.hdfs.path = /yarn/test a1.sink.k2.fileType = TEXT Please see log file: 15/09/14 12:08:34 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 15/09/14 12:08:34 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/home/hadoop/flume-1.6.0/conf/yarn.conf 15/09/14 12:08:34 WARN conf.FlumeConfiguration: Invalid property specified: sink.k2.hdfs.path 15/09/14 12:08:34 WARN conf.FlumeConfiguration: Configuration property ignored: a1.sink.k2.hdfs.path = /yarn/test 15/09/14 12:08:34 WARN conf.FlumeConfiguration: Invalid property specified: sink.k2.fileType 15/09/14 12:08:34 WARN conf.FlumeConfiguration: Configuration property ignored: a1.sink.k2.fileType = TEXT 15/09/14 12:08:34 INFO conf.FlumeConfiguration: Processing:k1 15/09/14 12:08:34 WARN conf.FlumeConfiguration: Invalid property specified: sink.k1.type 15/09/14 12:08:34 WARN conf.FlumeConfiguration: Configuration property ignored: a1.sink.k1.type = hdfs 15/09/14 12:08:34 WARN conf.FlumeConfiguration: Invalid property specified: sink.k1.fileType 15/09/14 12:08:34 WARN conf.FlumeConfiguration: Configuration property ignored: a1.sink.k1.fileType = TEXT 15/09/14 12:08:34 WARN conf.FlumeConfiguration: Invalid property specified: sink.k1.hdfs.path 15/09/14 12:08:34 WARN conf.FlumeConfiguration: Configuration property ignored: a1.sink.k1.hdfs.path = /yarn 15/09/14 12:08:34 WARN conf.FlumeConfiguration: Invalid property specified: sink.k2.type 15/09/14 12:08:34 WARN conf.FlumeConfiguration: Configuration property ignored: a1.sink.k2.type = hdfs 15/09/14 12:08:34 INFO conf.FlumeConfiguration: Processing:k2 15/09/14 12:08:34 INFO conf.FlumeConfiguration: Added sinks: k1 k2 Agent: a1 15/09/14 12:08:34 INFO conf.FlumeConfiguration: Processing:k1 15/09/14 12:08:34 INFO conf.FlumeConfiguration: Processing:k2 15/09/14 12:08:34 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [a1] 15/09/14 12:08:34 INFO node.AbstractConfigurationProvider: Creating channels Please suggest any modification is requird for .conf file Sri
... View more
09-13-2015
11:18 PM
Harish, I saw the url you (am using same) suggested.Can you make sample .conf where we can use multiple sources and multiple channels and multple sinks in same host. Thanks in advance Sridhar
... View more
09-13-2015
10:54 PM
Hi Team, I need to put log info from system,hadoop logs in hdfs in same machine Do we specify multiple sources of flume agent in same machine. The sample conf file i created is : # list the sources, sinks and channels in the agent agent_foo.sources = avro-AppSrv-source1 exec-tail-source2 agent_foo.sinks = hdfs-Cluster1-sink1 avro-forward-sink2 agent_foo.channels = mem-channel-1 mem-channel-2 # flow #1 configuration agent_foo.sources.avro-AppSrv-source1.channels = mem-channel-1 agent_foo.sources.avro-collection-source1.type = memory agent_foo.sources.avro-collection-source1.command = tail -F /home/hadoop/hadoop-2.7.1/logs/yarn-hadoop-resourcemanager-sri.test.com.log # flow #2 configuration agent_foo.sources.exec-tail-source2.channels = mem-channel-2 agent_foo.sources.exec-tail-source2.type = memory agent_foo.sources.exec-tail-source2.command = tail -F /var/log/messages # Define a sink that outputs to hdfs dir. agent_foo.sources.hdfs-sink.channel = mem-channel-1 mem-channel-2 agent_foo.sources.hdfs-sink.type = hdfs agent_foo.sources.hdfs-sink.hdfs.path = /yarn It is not working for me.Flume agent is struck with these: 15/09/14 11:08:07 INFO conf.FlumeConfiguration: Added sinks: hdfs-Cluster1-sink1 avro-forward-sink2 Agent: agent_foo 15/09/14 11:08:07 WARN conf.FlumeConfiguration: Agent configuration for 'agent_foo' does not contain any valid channels. Marking it as invalid. 15/09/14 11:08:07 WARN conf.FlumeConfiguration: Agent configuration invalid for agent 'agent_foo'. It will be removed. 15/09/14 11:08:07 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [] 15/09/14 11:08:07 WARN node.AbstractConfigurationProvider: No configuration found for this host:agent 15/09/14 11:08:07 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} } Please suggest a solution. ...Sri agent_foo.sources.hdfs-sink.hdfs.fileType = TEXT
... View more
Labels:
- Labels:
-
Apache Flume
-
Apache Hadoop
-
Apache YARN
-
HDFS