@ehehtenandayo You should be looking at nifi's record based processors.   NiFi includes the following RecordReader and RecordWriter Processors: ConsumeKafkaRecord_0_10 1.2.0 ConvertRecord PublishKafkaRecord_0_10 PutDatabaseRecord QueryRecord SplitRecord   These processor use record reader and record writer, which can define or infer the schema of the json.   Then, you are able to write a SQL statement for the results you want.   Additionally this will help with taking those flowfiles of json and feeding them to a putHive processor. Here is an older post that will help with some of the fundamentals: https://community.cloudera.com/t5/Community-Articles/Using-HiveQL-Processors-in-Apache-NiFi-1-2/ta-p/246803         ... View more

@ZorteK This is certainly an interesting scenario.   First thing i would wonder is why there is 1 out 10000 that is different.  If that is not something you can remove, then you will not able able to apply the same schema to all parts.   Its possible this is a bug with the CHOICE, but im not sure or familiar with that.   Few solutions here:  1 check for and handle the exception in a separate flow branch able to parse that second schema.   Or you can run the entire flow where the GEO value is a full string.  Do not try to parse it down to finite schema within that string.   ... View more

@Alfies Thanks for posting.   I cant see anything that jumps as far as why this would not work in 1.21.   I would wonder if it works in 1.24.    If it does, then for sure that compatiblity was added since 1.21.   If you are required to use 1.21,  i would recommend opening a Jira to report the bug you are describing.  ... View more

Ofcourse we can help.  Just make a new post.  Try to include as much detail as possible, screenshots, how you have invokeHttp configured, errors, etc.   In your new post tag me using the @ with my username "steven-matison". ... View more

@EbieCAS First, i would recommend NOT using that older hortonworks docker.   Check out this article with more recent method to run nifi in docker from apache nifi community: https://community.cloudera.com/t5/Community-Articles/NiFi-cluster-sandbox-on-Docker/ta-p/346271 Second, jump into the community search bar above, search for "nifi docker" and have a look at quite a few posts with various solutions.     There is a lot to learn here with examples and feedback from some of our best community members. Third, be mindful of docker basics for networking, port forwarding, etc as you approach connecting the nifi flow to external MSSQL. Last but not least,  once you have progress on your flow, and/or if you have more issues or specific questions make a new post for those.  Be sure to include details about your flow, what you have tried, what works, what does not, etc.  Take screen shots so configurations, details on errors, etc.     ... View more

This is awesome!   ... View more

@enam   I cant see the entire configuration of the second InvokeHTTP so I am not sure if you are passing those new attributes into the HTTP URL.   If you are not doing that, I think the solution you are looking for is to have the AttributesToJson set Destination to FlowFile Content.  Then when you post that to invokeHttp,  those values are seen as the post content.     ... View more

@edim2525   I have never had a concern to replicate kafka with nifi keeping exact order.    Perhaps others can respond with more details in a nifi context. That said,  my customers are using Streams Replication Manager (SRM) for heavy duty kafka replications.     https://docs.cloudera.com/cdf-datahub/7.2.16/howto-streams-replication-manager.html https://docs.cloudera.com/runtime/7.2.16/srm-overview/topics/srm-replication-overview.html There are several different form factors of SRM across private and public cloud but the doc links above should give you a good idea of what it is.   ... View more

@SAMSAL Thank you.     For future reference.  If you ever need to figure out what the request is,  simply use NIFI UI to do the action you want while monitoring your Developer Tools.   This will expose the full request/response on any given UI action. I have filed a JIRA for this. https://issues.apache.org/jira/browse/NIFI-12503 ... View more