Hi @Andrew Lim Thanks for the article. I tried to execute this using NiFi 1.5 but I get an error "already in use for an active callback or an InputStream created by ProcessSession.read(FlowFile) has not been closed" in QueryRecord processor. PFb excerpt from logs - ErrorHandlingInputStream[FlowFile=StandardFlowFileRecord[uuid=9688750b-0f83-4c64-91c6-80499ef63c24,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1519378477705-1, container=default, section=1], offset=532, length=179],offset=0,name=input.csv,size=179]] for StandardFlowFileRecord[uuid=9688750b-0f83-4c64-91c6-80499ef63c24,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1519378477705-1, container=default, section=1], offset=532, length=179],offset=0,name=input.csv,size=179] because the session was rolled back without the input stream being closed. 2018-02-23 15:10:17,160 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.QueryRecord QueryRecord[id=aff8e1fe-ff3c-3d77-9fcf-dcd3c1cf6ed9] QueryRecord[id=aff8e1fe-ff3c-3d77-9fcf-dcd3c1cf6ed9] failed to process session due to java.lang.IllegalStateException: StandardFlowFileRecord[uuid=9688750b-0f83-4c64-91c6-80499ef63c24,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1519378477705-1, container=default, section=1], offset=532, length=179],offset=0,name=input.csv,size=179] already in use for an active callback or an InputStream created by ProcessSession.read(FlowFile) has not been closed: {} java.lang.IllegalStateException: StandardFlowFileRecord[uuid=9688750b-0f83-4c64-91c6-80499ef63c24,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1519378477705-1, container=default, section=1], offset=532, length=179],offset=0,name=input.csv,size=179] already in use for an active callback or an InputStream created by ProcessSession.read(FlowFile) has not been closed<br> Can you please tell what's wrong? ... View more

Thanks for replying @Abdelkrim Hadjidj @Matt Burgess. Now I have two different schemas for input and output CSV for LookupRecord. I modified Result RecordPath to /COMPANY. When I start LookupRecord it takes in 6 flow files but doesn't return anything. ... View more

Hi Andrew, Is there any way to automate clearing state from ListS3 processor when the flow is scheduled? I cannot clear the state every time I run the flow. ... View more

Hi slachterman thanks for the post. I get below error while running PutHDFS processor. I didn't find the SNAPSHOT jars in HDInsight, May I know the HDI version you used? Can you help me fixing this error? Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.security.ProviderUtils.excludeIncompatibleCredentialProviders(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/Class;)Lorg/apache/hadoop/conf/Configuration; Thanks Vinay ... View more

Thanks @Timothy Spann Can you please share more information on this. Steps or link to any tutorial would be helpful. ... View more

Thanks @Edgar Orendain ... View more

Hello @Binu Mathew Thanks for the tutorial. I need your help with Kite API. I get "Descriptor location not found" when I try to use below dataset dataset:hdfs://ip-172-31-2-101.us-west-2.compute.internal:8020/tmp/sample_data/parquet . And I get an exception "java.lang.noclassdeffounderror org/apache/hadoop/hive/conf/HiveConf" when I execute below similar command in hadoop server. ./kite-dataset create dataset:hdfs://ip-172-31-2-101.us-west-2.compute.internal:8020/tmp/sample_data/parquet --schema sample.avsc --format parquet Can you please suggest the configuration I am missing here? Please suggest Thanks ... View more

Yes Raj, I went through all related questions here in HCC. I implemented tail file with nifi-app.log file but we need to process each row from this log as a flowfile and route it based on error/warn/exception keywords in it. And parse it for pushing it to each column of the Postgres table. Moreover in Ambari enabled NiFi nifi-app.log doesn't contain error messages but only heartbeat log, I couldn't figure out which log file have the errors as of now. To avoid the above-mentioned way I wanted to make use of Rest-API using which we can pick up the bulletins. This seems much easier way. ... View more

@Sunile Manjee Yes Sunile I have created a scenario to get an error and I see the error bulletin. PFB. When I start the above invoke-http processor I get an HTML file without error log, I couldn't infer anything from it. PFB. And I don't see the error in Ambari solr infra webpage which I see in bulletin OR in /var/log/ambari-infra-solr . Can you please suggest how to grab errors/warnings from ambar-infra? Thanks. ... View more

@Constantin Stanca The queue was piling up without pushing data to Postgres so I stopped Setup SQL Query and the queue stopped at 9900. I see 10000 in the queue if I don't stop it. I don't get any error bulletin. Earlier I could see error/warn logs in nifi-app.log file. Now I have Ambari installed NiFi, I could only see heartbeat messages in nifi-app.log. I also didn't see any error in Ambari LogSearch. ... View more

@Constantin Stanca Thank you for your response for my another question. I want to understand why the data is being queued when the PutSQL processor is running? Even though all other processors are stopped the flow files are still in the queue. I don't think what you said would solve my issue. ... View more

Even though nifi-app.log contains every log, the bulletin message is small and precise. The idea is to capture the error message from bulletin and store it in table/file for auditing purposes. We have new versions since a year, hoping we have some way for this. Please share if there is any way to capture the error messages from the bulletin. ... View more