About PentaReddy

PentaReddy · ‎12-24-2016

Thanks after making correction in the conf file below is what i got it. However, i could not see the data into Hdfs 16/12/24 11:48:34 INFO conf.FlumeConfiguration: Processing:HDFS 16/12/24 11:48:34 INFO conf.FlumeConfiguration: Processing:HDFS 16/12/24 11:48:35 WARN conf.FlumeConfiguration: Could not configure source Twitter due to: No Channels configured for Twitter org.apache.flume.conf.ConfigurationException: No Channels configured for Twitter at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSources(FlumeConfiguration.java:574) at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:346) at org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.access$000(FlumeConfiguration.java:213) at org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:127) at org.apache.flume.conf.FlumeConfiguration.<init>(FlumeConfiguration.java:109) at org.apache.flume.node.PropertiesFileConfigurationProvider.getFlumeConfiguration(PropertiesFileConfigurationProvider.java:189) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:89) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 16/12/24 11:48:35 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [TwitterAgent] 16/12/24 11:48:35 INFO node.AbstractConfigurationProvider: Creating channels 16/12/24 11:48:35 INFO channel.DefaultChannelFactory: Creating instance of channel Memchannel type memory 16/12/24 11:48:35 INFO node.AbstractConfigurationProvider: Created channel Memchannel 16/12/24 11:48:35 INFO sink.DefaultSinkFactory: Creating instance of sink: HDFS, type: hdfs 16/12/24 11:48:35 INFO node.AbstractConfigurationProvider: Channel Memchannel connected to [HDFS] 16/12/24 11:48:35 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{HDFS=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@1f8dbaa6 counterGroup:{ name:null counters:{} } }} channels:{Memchannel=org.apache.flume.channel.MemoryChannel{name: Memchannel}} } 16/12/24 11:48:35 INFO node.Application: Starting Channel Memchannel 16/12/24 11:48:36 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: Memchannel: Successfully registered new MBean. 16/12/24 11:48:36 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: Memchannel started 16/12/24 11:48:36 INFO node.Application: Starting Sink HDFS 16/12/24 11:48:36 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: HDFS: Successfully registered new MBean. 16/12/24 11:48:36 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: HDFS started

PentaReddy · ‎12-24-2016

Hi I am trying to fetch the data from twitter to my hdfs and while running my flume-ng agent, i am getting below logs. could you please assist me 16/12/24 00:06:22 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting 16/12/24 00:06:22 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:/etc/flume-ng/conf/flume.conf 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Invalid property specified: sink.HDFS.hdfs.path 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Configuration property ignored: TwitterAgent.sink.HDFS.hdfs.path = hdfs://master:8020/user/cloudera/tweets/ 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Invalid property specified: channel.Memchannel.capacity 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Configuration property ignored: TwitterAgent.channel.Memchannel.capacity = 10000 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Invalid property specified: sink.HDFS.channel 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Configuration property ignored: TwitterAgent.sink.HDFS.channel = Memchannel 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Invalid property specified: sink.HDFS.hdfs.writeFormat 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Configuration property ignored: TwitterAgent.sink.HDFS.hdfs.writeFormat = Text 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Invalid property specified: channel.Memchannel.type 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Configuration property ignored: TwitterAgent.channel.Memchannel.type = memory 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Invalid property specified: sink.HDFS.hdfs.rollCount 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Configuration property ignored: TwitterAgent.sink.HDFS.hdfs.rollCount = 10000 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Invalid property specified: channel.Memchannel.transactionalCapacity 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Configuration property ignored: TwitterAgent.channel.Memchannel.transactionalCapacity = 100 16/12/24 00:06:22 INFO conf.FlumeConfiguration: Added sinks: HDFS Agent: TwitterAgent 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Invalid property specified: sink.HDFS.hdfs.batchsize 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Configuration property ignored: TwitterAgent.sink.HDFS.hdfs.batchsize = 1000 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Invalid property specified: sink.HDFS.hdfs.rollSize 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Configuration property ignored: TwitterAgent.sink.HDFS.hdfs.rollSize = 0 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Invalid property specified: sink.HDFS.hdfs.filetype 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Configuration property ignored: TwitterAgent.sink.HDFS.hdfs.filetype = DataStream 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Invalid property specified: sink.HDFS.type 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Configuration property ignored: TwitterAgent.sink.HDFS.type = hdfs 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Agent configuration for 'TwitterAgent' does not contain any valid channels. Marking it as invalid. 16/12/24 00:06:22 WARN conf.FlumeConfiguration: Agent configuration invalid for agent 'TwitterAgent'. It will be removed. 16/12/24 00:06:22 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [] 16/12/24 00:06:22 WARN node.AbstractConfigurationProvider: No configuration found for this host:TwitterAgent 16/12/24 00:06:22 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }

PentaReddy · ‎12-21-2016

I am trying to establish the connection between Twitter and My local Hdfs (vm). I have installed all the required jars file and flume corresponding file and below are the steps which i followed. However i am getting error as Error: Could not find or load main class org.apache.flume.tools.GetJavaProperty 1. Create application in twitter 2. Created flume.conf with the consumer secret keys and so on.. 3. placed the appropriate jar file in the flume respective lib directory. When run below command i am getting the error. bin/flume-ng agent -n TwitterAgent -c conf -f conf /usr/lib/apache-flume-1.6.0-bin/conf/flume.conf Any suggestion or help is appreciated.

PentaReddy · ‎12-20-2016

https://issues.apache.org/jira/browse/PIG-5079

PentaReddy · ‎12-19-2016

Thanks, i have raised an enhancement Jira.

PentaReddy · ‎12-19-2016

Yes, This is testing only and i am doing it out of my curiosity. Thanks you very much for your valuable responses. Could you let me know how to open an enhancement Jira request ?

PentaReddy · ‎12-18-2016

I think that's the only solution i think is the best at the moment using temporary table. Does Overwrite exists in Pig? Like Hive Overwrite table data.

PentaReddy · ‎12-16-2016

Any Help highly appreciated.

PentaReddy · ‎12-16-2016

Per your last conversation it looks like HcatStorer is not overwriting the data, rather it is appending the data. In your last test, you were having the record count of 823 before the transformation and after the transformation you stored back the results and the result count is 1646.(which is nothing but appending the data). My expectations are below. Read the records of 823 and do transformation on it and update back the result and once updated the result count should be 823(overwriting the data with new transformation data) and not 1643. Is this kind of overwriting the data is possible through HCatStorer ? if not is there any other alternatives ways of overwriting the data in pig ( Eg :- PigStorage or any other functions).

PentaReddy · ‎12-16-2016

Yaah.. i have another version and let me know test on it and will get back with you with the result.

Online	Offline
Last Visited	‎12-20-2020 08:08 PM

Member Since	‎12-21-2016 06:37 PM
Last Visited	‎12-20-2020 08:08 PM
Posts	83
Kudos received	5

Cloudera Community

Re: How to delete/drop a partition of an external ...

Re: Flume Authentication credentials are missing

Re: Flume Twitter - Could not find any valid chann...

Flume Twitter - Could not find any valid channels....

Flume - Error: Could not find or load main class o...

Re: HCatStorer is not overwriting a Hive table. or...

Re: HCatStorer is not overwriting a Hive table. or...

Re: HCatStorer is not overwriting a Hive table. or...

Re: HCatStorer is not overwriting a Hive table. or...

Re: HCatStorer is not overwriting a Hive table. or...

Re: HCatStorer is not overwriting a Hive table. or...

Re: HCatStorer is not overwriting a Hive table. or...