Member since
09-27-2018
135
Posts
21
Kudos Received
9
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
39 | 02-28-2021 10:23 PM | |
635 | 12-16-2020 11:31 PM | |
317 | 12-14-2020 11:02 PM | |
341 | 12-14-2020 12:18 AM | |
225 | 12-07-2020 11:39 PM |
02-28-2021
10:23 PM
1 Kudo
@murali2425 Here are two possible solutions. At solution 1 the value is set into an array. At solution 2 I took the flowfile-content (json) and set it into an attribute. Then you can work with expression language to get the value. For testing the syntax I recommend this site: http://jsonpath.herokuapp.com/
... View more
02-18-2021
07:47 AM
@totti1 I'm sorry, I'm obviously confused. :-(( Just to correct the csv-output for the solution you don't need (but maybe someone else...) I add this picture: correct csv output format
... View more
02-18-2021
07:34 AM
@totti1 I can't confirm that ConvertExcelToCSVProcessor processor does not work with xlsx. It works, only thing I haven't achieved until now is to format the csv-output in the right manner. Maybe this will help you to get it? My test xlsx data: id title first last
1 miss marlene shaw
2 ms letitia jordan
3 mr todd graham
4 mr seth martinez
5 mr guy mckinney
6 ms anna smith
7 mr johnny johnson
8 mrs robin white
9 miss allison williams
10 mrs erika king Processors Flow My not correct formatet temporary csv-output: temporary csv-output
... View more
02-18-2021
05:45 AM
@totti1 Oh, that's a shame and would have been to easy... 😉 Can you tell what other options you have tried? Can you provide information on what the data looks like?
... View more
02-18-2021
05:40 AM
@totti1 Haven't done this until now, but there is a ConvertExcelToCSVProcessor Processor. Have you tried this?
... View more
02-17-2021
05:55 AM
@memad Have a look here at the processor: To create or check cron syntax look here: https://www.freeformatter.com/cron-expression-generator-quartz.html Hope this helps! Bye
... View more
02-16-2021
09:40 AM
1 Kudo
@BerniHacker Come on work can wait... Here my description with NiFi 1.12.1. on a windows standalone installation. This is my test csv. id;title;first;last
1;miss;marlene;shaw
2;ms;letitia;jordan
3;mr;todd;graham
4;mr;seth;martinez
5;mr;guy;mckinney
6;ms;anna;smith
7;mr;johnny;johnson
8;mrs;robin;white
9;miss;allison;williams
10;mrs;erika;king This is my flow: Flow This is my sql table. SQL table This is the configuration of the csv-reader. Note the green bordered information! Controller Service - dsv reader The schema has to be defined in a controller service of the type " AvroSchemaRegistry ". AvroSchemaRegistry With the UpdateAttribute you set the NiFi-attribute schema.name to the concrete schema you use. UpdateAttribute schema.name And last but not least configure the PutDatabaseRecord. PutDatabaseRecord Here is the result in my sql table. SQL Result Hope this helps and works for you. Bye!
... View more
02-16-2021
09:13 AM
@BerniHacker The whole day I couldn't achive to work on this further. Now I found a solution but at the moment I can't document this detailed because "work is calling". In small words: GetFile -> UpdateAttribute (to set NiFi-attribute schema.name) -> PutDatabaseRecord (with CSV-Record Reader) BUT the reader has to use a schema which is defined in the controller services as AvroSchemaRegistry and there has to exist a avro schema for the structure. Maybe this will help you further. Have a look here https://community.cloudera.com/t5/Support-Questions/Loading-a-simple-CSV-file-using-nifi-PutDatabaseRecord-and/td-p/222678 Tomorrow I will describe my solution detailed! Sorry...
... View more
02-15-2021
11:11 PM
@BerniHacker Sorry, my reply was not good. Did not see that you are working with "Use String Fields From Header" and not a separate schema. I will try to get it work on my local installation. If you found a solution by so kind to let us know. Thanks.
... View more
02-15-2021
10:42 PM
@BerniHacker As far as I see the cause could be different column names and in the PutDatabaseRecord the option "Unmatched Column Behavior" is set to "Fail on Unmatched Columns". I suppose the options concerning "Unmatched Field Behavior" or "Unmatched Column Behavior" in connection with "Translate Field Names" causing your problem. Perhaps you like to post your schema-definition? But at the moment I can't give you a concreter answer.
... View more
02-08-2021
11:53 PM
Hi @Jarinek I think you have to set -pv in front of the new value: nifi set-param -u http://localhost:8088 -pcid 85b8b5bf-0177-1000-3881-8ddd8d31fed8 -pn sample.URL -pv newvalue This one worked on my local NiFi. Remark: Haven't seen that you found it yourself. Great!
... View more
02-08-2021
07:30 AM
Hi @ashwinrk maybe this information will help you further: https://community.cloudera.com/t5/Support-Questions/NiFi-InvokeHTTP-POST-JSON/m-p/307990#M223438 It depends on the concrete target whether the "Authorization" has to be base64encoded or you can use basic authentication.
... View more
02-01-2021
01:20 AM
1 Kudo
@believe7 One possibility is shown here. Maybe there is a more comfortable way, but this would be my solution. Flow Details processors Result response If you have further questions please let me know.
... View more
01-27-2021
03:42 AM
@VidyaSargur i have to change my email. Would you be so kind and get intouch with me? Thank you!
... View more
12-16-2020
11:44 PM
@stephane_davy Have only found the correct avro-schema for the JsonTreeReader (see above), unfortunately I failed at the output array definition. When trying to use an online generated output schema, errors occurred continuously. Hope that an avro-guru can come forward and help. I am looking forward to the solution, which I will find after my vacation in January. I'm outta here now, all the best!
... View more
12-16-2020
11:31 PM
1 Kudo
@GMAN Here is my try to help you further. flow Look at the picture of the InvokeHTTP. InvokeHTTP From my experience most options can mostly be left to default values. Your Json-body should be fine and will be send with the request. Remarks: - Authenticaton/Authorization: You have to try the two possibilities I know. For the second you have to add custom-property "Authorization" with the plus symbol on the right above. - Accept: Concerns the response format not sure wether you will need this. Maybe you can first build your request in Postman and after successfull execution then click to "code" -> "HTTP". There you can see the details needed for the request. In the "Settings" of the InvokeHTTP you can autoterminate the "original" flowfile. For "Retry" you can use the shown processor with default values. All "Failure" or "No retry" go to some error-handling. With "Response" you can go further in your flow. Hope this helps you to come forward.
... View more
12-16-2020
12:31 AM
@stephane_davy Found this https://stackoverflow.com/questions/54981540/nifi-recordreader-recordwriter-serialization-error-illegaltypeconversionexcep When I use your above JOLT-spec within JoltTransformJSON it works fine. No need of readers, writers, schemas. Output: [ {
"myfield" : "JustForHavingJson",
"myfield1" : "A",
"myfield2" : "C"
}, {
"myfield" : "JustForHavingJson",
"myfield1" : "B",
"myfield2" : "C"
}, {
"myfield" : "JustForHavingJson",
"myfield1" : "C",
"myfield2" : ""
}, {
"myfield" : "JustForHavingJson",
"myfield1" : "E",
"myfield2" : ""
}, {
"myfield" : "JustForHavingJson",
"myfield1" : "X",
"myfield2" : ""
}, {
"myfield" : "JustForHavingJson",
"myfield1" : "",
"myfield2" : ""
}, {
"myfield" : "JustForHavingJson",
"myfield1" : "D",
"myfield2" : "G"
} ]
... View more
12-15-2020
11:31 PM
@stephane_davy Now I came one step further. I defined this schema for the JsonTreeReader in JoltTransformRecord. {
"name": "HCC_JOLTTRANSFORMRECORD_IN",
"type": "record",
"namespace": "HCC_JOLTTRANSFORMRECORD_IN",
"fields": [
{
"name": "myJSON",
"type": {
"type": "array",
"items": {
"name": "myJSON_record",
"type": "record",
"fields": [
{
"name": "myfield",
"type": "string"
},
{
"name": "myfield1",
"type": "string"
},
{
"name": "myfield2",
"type": "string"
}
]
}
}
}
]
} So the error Error transforming the first record: is gone! Now get another error concerning the Writer-schema: JoltTransformRecord[id=65b2b5fd-0176-1000-ffff-ffffd0f23bd9] Unable to transform StandardFlowFileRecord[uuid=7e4fe006-1eb0-44cd-9e16-f4c8a8c533df,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1608074221236-4, container=default, section=4], offset=584493, length=780],offset=0,name=d92eab41-fa79-441f-a5d9-c6e7f6be10c0,size=780] due to org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert value [[Ljava.lang.Object;@1b321d9e] of type class [Ljava.lang.Object; to Record for field r: Cannot convert value [[Ljava.lang.Object;@1b321d9e] of type class [Ljava.lang.Object; to Record for field r Working on it...
... View more
12-15-2020
10:42 PM
@stephane_davy I assume that this question follows the previous one https://community.cloudera.com/t5/Support-Questions/Nifi-Multiple-predicate-in-recordpath-filter/m-p/307645#M223278 So I tried with my test-JSON to do the same like you. {
"myJSON": [
{
"myfield": "JustForHavingJson",
"myfield1": "A",
"myfield2": "C"
},
{
"myfield": "JustForHavingJson",
"myfield1": "B",
"myfield2": "C"
},
{
"myfield": "JustForHavingJson",
"myfield1": "C",
"myfield2": ""
},
{
"myfield": "JustForHavingJson",
"myfield1": "E",
"myfield2": ""
},
{
"myfield": "JustForHavingJson",
"myfield1": "X",
"myfield2": ""
},
{
"myfield": "JustForHavingJson",
"myfield1": "",
"myfield2": ""
},
{
"myfield": "JustForHavingJson",
"myfield1": "D",
"myfield2": "G"
}
]
} But neither without nor with schema the JoltTransformRecord works. Getting the same error as you (NiFi 1.11.1). One possibility and quick solution I found is to: EvaluateJsonPath --> setting the flowfile-content to an attribute UpdateAttribute --> ${FF_CONTENT:jsonPath('$.myJSON')} ReplaceText --> bring the attribute back to flowfile-content If you get the JoltTransformRecord work I would like to know how. Thanks.
... View more
12-14-2020
11:02 PM
@adhishankarit Sadly I have no experience doing such kind of logging in own files to a NFS. But what about logging to nifi-app.log? I don't understand the problem you mentioned in your first post concerning this. Using the LogAttribute-processor gives you some options (attributes, content) what and how to log there. With the option "Log prefix" you can evaluate data by this value later. IMHO Logging all attributes would be advantage because not all processors deliver the same attributes after they ran. Maybe you can do something with scripts like mentioned here: https://community.cloudera.com/t5/Support-Questions/Is-possible-to-write-an-attribute-into-a-file-and-also-keep/td-p/184414 Sorry but im neither fit with scripting nor with this kind of logging so I'm not able to help you further. But maybe some other guys here will help you.
... View more
12-14-2020
08:21 AM
@opalo54 Thanks for posting your solution! I was just trying too but I think your input data has to be set to an array for this. Am I getting right?
... View more
12-14-2020
03:53 AM
@jainN Maybe you can describe your problem a bit more? So it maybe will be possible to give you some help. f.e. Does all files (with different formats) come from the same processor? What is your "previous processor"? Do you get data from InvokeHTTP? "When we get the json file" - Do you get responses without content?
... View more
12-14-2020
03:46 AM
@adhishankarit I don't know wether I get your requirement right. You want to log EACH single NiFi-processer regardless wether it succeeds successful or not (like error, retry, etc.). Right? Evaluation of data of nifi-app.log (haven't done this by now) or by REST-API (I suppose something in the area "Flow" https://nifi.apache.org/docs/nifi-docs/rest-api/index.html) might be possible. But why don't you log this data direct from the flow by writing to a database logtable after each processor? Concerns about performance?
... View more
12-14-2020
12:18 AM
2 Kudos
@stephane_davy Sorry I couldn't work on this further last week. Now my solution looks like this. JSON from GenerateFlowFile [
{
"myfield": "JustForHavingJson",
"myfield1": "A"
},
{
"myfield": "JustForHavingJson",
"myfield1": "B"
},
{
"myfield": "JustForHavingJson",
"myfield1": "C"
},
{
"myfield": "JustForHavingJson",
"myfield1": "D"
},
{
"myfield": "JustForHavingJson",
"myfield1": "E"
},
{
"myfield": "JustForHavingJson",
"myfield1": "X"
},
{
"myfield": "JustForHavingJson",
"myfield1": ""
}
] Definitions of ControllerServices and AvroSchemaRegistry ControllerServices and AvroSchemaRegistry Flow Flow Details of RecordProcessing-Processors Details RecordProcessing-Processors Result FF-Content after MergeRecord: {"myfield":"JustForHavingJson","myfield1":"A","myfield2":"C"}
{"myfield":"JustForHavingJson","myfield1":"B","myfield2":"C"}
{"myfield":"JustForHavingJson","myfield1":"C","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"E","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"X","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"","myfield2":""}
{"myfield":"JustForHavingJson","myfield1":"D","myfield2":"G"} Personally I wouldn't do a MergeRecord at the end but go on with the three single connections of the UpdateRecord processors. Do you think this could be a possible solution or have you found a better way to do this?
... View more
12-13-2020
10:58 PM
@GMAN You can do this with UpdateAttribute. In my example: Date_Time ==> ${YourUnixTimestamp:toDate():format('YYYY-MM-dd hh:mm:ss.SSS', 'UTC')} YourDate_YYYY-MM-dd ==> ${YourUnixTimestamp:toDate():format('YYYY-MM-dd')} You can check the result here: https://www.unixtimestamp.com/index.php But you have to remove the last three digits of the value. Hope this helps!
... View more
12-09-2020
04:03 AM
@stephane_davy I'm working in the same direction because I couldn't believe that there is no possibility. But I'm still struggling with problems of the correct MergeRecord. So we both go the same way.
... View more
12-07-2020
11:54 PM
Hello@stephane_davy I'm sorry, I haven't seen that record-processing is a "must". Because I haven't much experience with this kind of syntax I sadly will not be able to help you here. But if you find a solution just be so kind and show it all of us here. Thank you!
... View more
12-07-2020
11:39 PM
1 Kudo
@ang_coder The AvroSchemaRegistry is to set in the ControllerServices. If it should be available in your whole NiFi just go to the canvas and click anywhere right -> choose "Configure". It is the same place where you define your database connections. Configure -> ControllerServices If you already have JSON-format just ignore the first two steps of my example flow where data were retrieved and converted to JSON. Start with your JSON at UpdateAttribute which doesn't care about the kind of flowfile content. The ConvertRecord in my example uses an JSON-reader and JSON-writer where the writer points to the AvroSchemaRegistry with the schema.name set in UpdateAttribute. Therefore the JsonRecordSetWriter_IJ_TestMoreColumnsAtTarget (just to take a short name) has the specification "Schema Access Strategy" with option "Use Schema.Name Property". Have a look at the configuration details in my post before. Hope this helps.
... View more
12-06-2020
10:50 PM
1 Kudo
@dzbeda Try it with: /*:Event/*:System/*:Channel
... View more
12-04-2020
02:39 AM
2 Kudos
@ang_coder Tried it myself and have to correct my idea a little bit. Have a look at my example. Here are my tables. The target table has two columns more than the source. Tables I defined following ControllerServices: ControllerServices And created this flow: Flow Here the information of the single processors: Processor information In my example it works. Sure there will be a more elegant solution but I don't know how.
... View more