Support Questions
Find answers, ask questions, and share your expertise

Continuous parsing of sql statements in nifi logs

Continuous parsing of sql statements in nifi logs

Explorer

I have the following settings in the schedule tab of nifi selectHiveQl Processor

settings.png

schedule.png

And I am using "select * from tablename" query in my selectHiveQlProcessor with fetchsize of 20000 and max number of rows in flow files as 20000. As soon as i schedule the selectHiveQl processor and check the app.log I can see continuous parse statements being printed in the log like below :

2018-05-15 06:29:54,919 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:54,919 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,063 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,063 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,213 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,213 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,356 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,356 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,501 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,502 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,634 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,634 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,764 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,764 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:55,914 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:55,914 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:56,046 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:56,046 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:56,195 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:56,195 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed 2018-05-15 06:29:56,334 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parsing command: select * from random 2018-05-15 06:29:56,335 INFO [Timer-Driven Process Thread-1] hive.ql.parse.ParseDriver Parse Completed

It should be parsed one time only , multiple parsing might be using CPU more than what is expected . Can some one help me understand how to prevent this parsing multiple times .

@Matt Burgess@Bryan Bende