Member since
07-29-2020
574
Posts
323
Kudos Received
176
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2496 | 12-20-2024 05:49 AM | |
| 2878 | 12-19-2024 08:33 PM | |
| 2542 | 12-19-2024 06:48 AM | |
| 1710 | 12-17-2024 12:56 PM | |
| 2379 | 12-16-2024 04:38 AM |
07-19-2023
10:20 AM
hmmm, I dont see any issue with what you have sent. What is your JsonTreeReader & JsonRecordSetWriter looks like. Mine looks like the following: Also what version of Nifi are you using? Thanks
... View more
07-19-2023
06:29 AM
You are welcome. Have you changed anything in the jolt transformation spec, and if so how does the output looks like? Can you also provide the QueryRecord processor configuration.
... View more
07-18-2023
04:32 PM
Hi @kanchanDesai , You have couple of options : 1 - You can just dump the data from CSV into its own table in the target DB (or staging) , and then dump the data from the sql source into the same target DB in case its different from the source. Once you have both sources in SQL you can join both tables by GlCode and do the need calculation and store the result in the destination table. 2- If you want to do the join and calculation in Nifi , then you can take advantage of the ForkEnrichment\JoinEnrichment processors to join data together. If you are using SQL Strategy to join then you can do your calculation there as well and get the desired result that will feed into the destination table, otherwise you can use UpdateRecord, QueryRecord or JoltTransformationJSON to do the calculation. For more information on the Fork\Join Enrichment processors see : https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.JoinEnrichment/additionalDetails.html If that helps please accept solution. Thanks
... View more
07-18-2023
04:13 PM
1 Kudo
Hi @Paulito , The easiest way I can think of is to do this in two Processors: 1- JoltTransformJson: Allows you to transform your json by simplifying it into an array or records where each record has list of fieldname:fieldvalue. To achieve this you need to provide the following jolt spec in the "Jolt Specification" property of the processor: [
{
"operation": "shift",
"spec": {
"Result": {
"ResultData": {
"DATARECORD": {
"*": {
"DATAFIELD": {
"*": {
"FIELDVALUE": "[&3].@(1,FIELDNAME)"
}
}
}
}
}
}
}
}
] Basically the spec above will give you the following json based on the provided input: [ {
"name" : "JSMITH",
"namedesc" : "John Smith",
"hireddate" : "01-JAN-2010"
}, {
"name" : "RSTONE",
"namedesc" : "Robert Stone",
"hireddate" : "01-JAN-2011"
} ] 2- QueryRecord Processor: to allow you to select the fields you are interested in for the given API as follows. The query is just like sql query and you can either specify wildcard (*) for all fields or just list particular fields as follows: The out put of the QueryRecord will look like this: [
{
"name": "JSMITH",
"namedesc": "John Smith"
},
{
"name": "RSTONE",
"namedesc": "Robert Stone"
}
] Of course you can make this dynamic for each API by providing both the Jolt spec and the Query as flowfile attributes since both allow expression language (EL) in the value field. You can also achieve the same result by using just the Jolt Transformation processor but the spec would be more complex and I dont want to overwhelm you in case you are new to it, but if you are interested in that let me know and I will be happy to provide you the spec. If this helps please accept solution. Thanks
... View more
07-17-2023
12:06 PM
Hi @Madhav_VD , In addition to what @steven-matison has mentioned which will work, there is another option where you can capture errors through setting up "SiteToSiteBulletinReportingTask" which will channel all errors to an input port where you can capture the error information in json format from all processors and all nodes in cluster and then store that error in sql or send notification email. This option might require additional steps to set up but its worth it specially if you have cluster. For more info on how to set up the "SiteToSiteBulletinReportingTask" please refer to : https://pierrevillard.com/2017/05/13/monitoring-nifi-site2site-reporting-tasks/ Hope that helps
... View more
07-17-2023
11:54 AM
Hi @Paulito , How do you like to get the output all the names and namesdesc? Would like to get them as list in json, csv or any other format, or you want them to be store in flowfile attributes ?
... View more
07-13-2023
10:09 AM
Hi, It seems you are only providing the output. Can you please provide the input json and the desired output? Thanks
... View more
06-16-2023
11:26 AM
You have to create a separate case statement for each column you are trying to update similar to what is done for the shipment number.
... View more
06-16-2023
11:14 AM
You either have to select each column one by one to have the same column name you are trying to update in the case statement, or select * but create different column for the case statement and then use jolt transformation to write the value back to the original column.
... View more
06-16-2023
07:19 AM
Hi @Ray82 Its hard to predict how this is going to work without understanding the full data flow you have, so I had to come up with simple flow to test the query which worked for me as expected. Please see the my flow below and each processor configuration to get the expected result: Data Flow: 1) Generate Input Josn: GenerateFlowFile processor to produce the Input json as you provided and provide the query attributes. 2) QueryRecord: Run the Query provided above to generate expected result: Notice that you need to connect the QueryRelationship "QueryRel" to downstream processor to get the expected output and not the original relationship: After running the DataFlow above and getting the output in the QueryRel relationship queue , the output looks like as follows: Notice to get the out put as an array the JsonRecordWrite in the QueryRecord RecordWriter is configured as the following: Like I said in the beginning everything depends at the end on your overall dataflow and how you are getting the data , attributes and in what format. The details above hopefully can guide you in case you are missing something or figure out what is making this not working in your case. Also be aware that Im using version 1.20.0 so also depending what version you are using the behavior can be different. Hope that helps.
... View more