Read flume twitter files with HIVE

AndresUrrego — Sun, 18 Aug 2019 00:41:26 GMT

Hello guys, I've used flume to catch a few twitts. My agent and whole flume configuration run pretty well, my key point is when I need to read the outcome file with HIVE.

I create an avro schema file to reuse it in hive to create the table to store flume data. (flume outcome file comes in avro format) Once the table in hive is ready I tried to check it out and confirm the right format and that looks good as you can see in the attached file tweettable.jpg.

Then, I perform the command to load the flume data into this table and according to result message in hive that's also performed as expected. Even if the numRows is marked as 0. (attached file load.jpg)

Now, finally when I try to read the data is when I got an error message saying that is not possible read this data. Unfortunately, I don't understand why guys. Please if someone can give me a hand with that I really appreciate. (attached file result.jpg)

If you need more details about the scripts and everything I have used in this test I've posted all info in a GitHub repository https://github.com/AndresUrregoAngel/Flume_Twitter

thanks so much

Re: Read flume twitter files with HIVE

AndresUrrego — Fri, 01 Sep 2017 03:20:23 GMT

Hi guys I want to posted the solution , finally I have added in my flume file the options below:

TwitterAgent.sources.Twitter.maxBatchSize = 50000

TwitterAgent.sources.Twitter.maxBatchDurationMillis = 100000

thanks

question Re: Read flume twitter files with HIVE in Archives of Support Questions (Read Only)

Read flume twitter files with HIVE

Re: Read flume twitter files with HIVE