Member since
09-27-2018
138
Posts
23
Kudos Received
10
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 12791 | 02-28-2021 10:23 PM | |
| 2886 | 02-08-2021 11:53 PM | |
| 38702 | 12-16-2020 11:31 PM | |
| 10477 | 12-14-2020 11:02 PM | |
| 7953 | 12-14-2020 12:18 AM |
12-14-2020
03:46 AM
@adhishankarit I don't know wether I get your requirement right. You want to log EACH single NiFi-processer regardless wether it succeeds successful or not (like error, retry, etc.). Right? Evaluation of data of nifi-app.log (haven't done this by now) or by REST-API (I suppose something in the area "Flow" https://nifi.apache.org/docs/nifi-docs/rest-api/index.html) might be possible. But why don't you log this data direct from the flow by writing to a database logtable after each processor? Concerns about performance?
... View more
12-14-2020
12:18 AM
2 Kudos
@stephane_davy Sorry I couldn't work on this further last week. Now my solution looks like this. JSON from GenerateFlowFile [
{
"myfield": "JustForHavingJson",
"myfield1": "A"
},
{
"myfield": "JustForHavingJson",
"myfield1": "B"
},
{
"myfield": "JustForHavingJson",
"myfield1": "C"
},
{
"myfield": "JustForHavingJson",
"myfield1": "D"
},
{
"myfield": "JustForHavingJson",
"myfield1": "E"
},
{
"myfield": "JustForHavingJson",
"myfield1": "X"
},
{
"myfield": "JustForHavingJson",
"myfield1": ""
}
] Definitions of ControllerServices and AvroSchemaRegistry ControllerServices and AvroSchemaRegistry Flow Flow Details of RecordProcessing-Processors Details RecordProcessing-Processors Result FF-Content after MergeRecord: {"myfield":"JustForHavingJson","myfield1":"A","myfield2":"C"}
{"myfield":"JustForHavingJson","myfield1":"B","myfield2":"C"}
{"myfield":"JustForHavingJson","myfield1":"C","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"E","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"X","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"D","myfield2":"G"} Personally I wouldn't do a MergeRecord at the end but go on with the three single connections of the UpdateRecord processors. Do you think this could be a possible solution or have you found a better way to do this?
... View more
12-09-2020
04:03 AM
@stephane_davy I'm working in the same direction because I couldn't believe that there is no possibility. But I'm still struggling with problems of the correct MergeRecord. So we both go the same way.
... View more
12-07-2020
11:54 PM
Hello@stephane_davy I'm sorry, I haven't seen that record-processing is a "must". Because I haven't much experience with this kind of syntax I sadly will not be able to help you here. But if you find a solution just be so kind and show it all of us here. Thank you!
... View more
12-07-2020
11:39 PM
1 Kudo
@ang_coder The AvroSchemaRegistry is to set in the ControllerServices. If it should be available in your whole NiFi just go to the canvas and click anywhere right -> choose "Configure". It is the same place where you define your database connections. Configure -> ControllerServices If you already have JSON-format just ignore the first two steps of my example flow where data were retrieved and converted to JSON. Start with your JSON at UpdateAttribute which doesn't care about the kind of flowfile content. The ConvertRecord in my example uses an JSON-reader and JSON-writer where the writer points to the AvroSchemaRegistry with the schema.name set in UpdateAttribute. Therefore the JsonRecordSetWriter_IJ_TestMoreColumnsAtTarget (just to take a short name) has the specification "Schema Access Strategy" with option "Use Schema.Name Property". Have a look at the configuration details in my post before. Hope this helps.
... View more
12-06-2020
10:50 PM
1 Kudo
@dzbeda Try it with: /*:Event/*:System/*:Channel
... View more
12-04-2020
02:39 AM
2 Kudos
@ang_coder Tried it myself and have to correct my idea a little bit. Have a look at my example. Here are my tables. The target table has two columns more than the source. Tables I defined following ControllerServices: ControllerServices And created this flow: Flow Here the information of the single processors: Processor information In my example it works. Sure there will be a more elegant solution but I don't know how.
... View more
12-04-2020
01:03 AM
@ang_coder I'm not sure if i get you right and haven't done such a thing myselfe until now. But if you use PutDatabaseRecord instead of PutSQL you have to specify a RecordReader. This will be configured as ControllerService and you can add a schema in AvroSchemaRegistry (also in ControllerServices). In this schema provide the full table definition with default values. f.e. { "name" : "zeichen", "type" : ["string","null"] },... I suppose that all columns which are not included by your flowfile should be taken from the schema considering the default values.
... View more
12-03-2020
03:01 AM
@stephane_davy Sorry, my above picture is a kind of vision test... Flow and Step1 and 2 Step 3 Rule 1 Step 3 Rule 2 Step 4
... View more
12-03-2020
02:52 AM
@stephane_davy Maybe you want to give this solution a try? Hope the explanation is clear. If not feel free to ask again. <Picture removed>
... View more