In Nifi is there anyway to consume kafka messages from beginning like we do from kafka console consumer? Apart from consuming messages when Nifi has started it will receive current messages which are produced after the application has started but if i need to consume messages while my application was offline or from next offset which last consumed? Thank you ... View more

Just wondering can we achieve this requirement using Nifi processor. Is it designed for such historical data or bulk upload from 250 tables in sql server to s3. Iam aware that querydatabasetable processor support incremental updates when a new record is inserted. How to configure this processor to sync with existing record updates and deletes i.e. do we give unique column so that nifi can identify when ever an existing record is updated then it needs to sync it to s3. How to achieve this with nifi. I beleive nifi is not designed for bulk upload. I have another usecase for bulk upload to s3 from mongodb. Please let me know if nifi can be the right tool for these type of usecases. If so i believe i need to write a shell script to copy 250 sql server tables to each file in s3 in json format? Plz confirm? whats the best practice to query 250 tables and export to s3? Thank you. ... View more

We have created lot of poc's for nifi like KafkatoS3, SQLServertoS3, KafkatoCassandra etc. We are planning to use nifi for few more usecases. Currently in poc we are using single instance of nifi in EC2 instance but if we go live we have around terabytes of SQLServer db data and also we will be having multiple data sources like Kafka, SQL Server. 1. Can a single instance of nifi handle huge volumes of data like exporting terabytes of sqlserver data to s3 as batches. 2. I think nifi will be sitting on edge node rather than inside the cluster plz confirm? 3. Is any organization currently using nifi in production? 4. What are the best practices to productionize nifi? 5. To handle large data and data from multiple sources do i need to have multiple instances of nifi? Thank you. ... View more

Hi ALL, Just wondering if we have any example templates to see the configuration of site to site i think using Nifi? Or if we have any instructions on how to configure site to site it will be helpful for me to create a new one. Thank you? ... View more

Looking for suggestions to insert json into cassandra. I need to filter the json event from kafka topic based on EventName and need to pipe the whole json file lets say if i have an eventname="LoginSucceeded" then i need to pipe it to next processor the whole file not just the single property of the whole json file. I looked at EvaluateJson, RouteOnAttribute & AttributesToJson but they are piping out only subset of properties not the whole json event when there is a match to eventName. Also i want to know If i use RouteOnAttribute then how to pass some subset of json properties to PutCassandraQL. I used below example from github https://github.com/hortonworks-gallery/nifi-templates/blob/master/templates/CassandraProcessors.xml which uses ExecuteScript to insert hardcoded values into cassandra but im looking to insert values from my json event. In PutCassandraQL doesnt have a in built property to write CQL. So i used one of the example templates from github which uses ExecuteScript processor and the values inserted through executescript processor are hardcoded and i want to use my json values i.e. insert values from my Json event not hardcoded values. Any suggestions plz. Currently iam using this flow to put json in cassandra. GetKafka->EvaluateJson->RouteOnAttribute->ExecuteScript->PutCassandraSQL. This flow is broken as iam stuck on how to pipe my json properties from RouteOnAttribute to ExecuteScript? Thank you ... View more