Member since
07-10-2023
25
Posts
1
Kudos Received
0
Solutions
01-17-2024
03:38 AM
1 Kudo
Hi guys! I have some problem with joltTransformation.I have json like that: { "system": "TGG", "income": "1000.0", "costs": "700.0", "id": "1", "dateOfEvent": 1705446000000, "name": "JAMES", "surname": "SMITH", "mail": "jsmith@gmail.com", "dateOfBirth": 1705446000000, "country": "US", "city": "ORLANDO", "eventId": "22", "eventName": "EVENT", "personalData": "true" } And I need output like this: { "system" : "TGG", "income" : "1000.0", "costs" : 700.0, "id" : "1", "dateOfEvent" : 1705446000000, "person" : { "name" : "JAMES", "surname" : "SMITH", "mail" : "jsmith@gmail.com", "dateOfBirth" : 1705446000000, "address" : { "country" : "US", "city" : "ORLANDO" } }, "eventData" : { "personalData" : true, "eventName" : "EVENT", "eventId" : "22" } } I have JoltSpec like this: [ { "operation": "shift", "spec": { "system": "system", "income": { "": { "": "income" }, "*": { "$": "income" } }, "costs": { "": { "": "costs" }, "*": { "$": "costs" } }, "id": "id", "dateOfEvent": { "": { "": "dateOfEvent" }, "*": { "$": "dateOfEvent" } }, "name": "person.name", "surname": "person.surname", "mail": "person.mail", "dateOfBirth": { "": { "": "person.dateOfBirth" }, "*": { "$": "person.dateOfBirth" } }, "country": "person.address.country", "city": "person.address.city" } }, { "operation": "modify-overwrite-beta", "spec": { "dateOfEvent": "=toLong(@(1,dateOfEvent))", "costsArray": "=split('[,]',@(1,costs))", "costsJoin": "=join('.',@(1,costsArray))", "costs": "=toDouble(@(1,costsJoin))", "incomeArray": "=split('[,]',@(1,income))", "incomeJoin": "=join('.',@(1,incoemArray))", "income": "=toDouble(@(1,incomeJoin))", "eventData": { "personalData": "=toBoolean(@(1,personalData))" }, "person": { "dateOfBirth": "=toLong(@(1,dateOfBirth))" } } }, { "operation": "default", "spec": { "income": null, "costs": null, "dateOfEvent": null, "person": { "dateOfBirth": null }, "eventData": { "eventName": "${eventName}", "eventId": "${eventId}", "personalData": "${personalData}" } } }, { "operation": "remove", "spec": { "costsArray": "", "costsJoin": "", "incomeArray": "", "incomeJoin": "" } } ] Everything is fine when I have data like dateOfBirth or income or costs. But when this data are null my json changes its forms and this data ends up at the and and look like this: { "system" : "TGG", "costs" : 700.0, "id" : "1", "person" : { "name" : "JAMES", "surname" : "SMITH", "mail" : "jsmith@gmail.com", "address" : { "country" : "US", "city" : "ORLANDO" }, "dateOfBirth" : null }, "eventData" : { "eventName" : "${eventName}", "eventId" : "${eventId}", "personalData" : "${personalData}" }, "income" : null, "dateOfEvent" : null } I need the json to always look the same, fields to be in the same order, regardless of whether they have values or are null-How can I improve this in my jolt spec?
... View more
Labels:
- Labels:
-
Apache NiFi
12-29-2023
06:02 AM
Hi, I have the following problem - I have a flow in which I have a json file, then via updateAttribute I give this flow some id and then I divide this flow into two success paths. The json file stays on one of them, on the other I perform some operations, I clear the contest and I have nothing there except the attributes I need. Now I would like to combine these paths using MergeRecord and use the id I gave at the beginning as the correlation attribute. Is it possible to connect json file that I have in one path with the empty content that I have in the other? Because I only care about the attributes from the second path and I noticed that in the case of MergeRecord, it is merged only in the case of the same content when selecting Bin-Packing Algorithm as the merge strategy. Is there any way around this?
... View more
Labels:
- Labels:
-
Apache NiFi
12-29-2023
05:25 AM
Hi @SAMSAL your solution is workink, thank you so much!
... View more
12-28-2023
06:13 AM
Hi guys! I have json like that: { "id" : "20", "user" : { "name" : "John", "date" : "" } } I transform this date to Long using nifi langugage ${date:isEmpty():ifElse('',${date:toDate('yyyy-MM-dd HH:mm:ss'):toNumber()})} and after this i do in jolttrasformation something like [ { "operation": "shift", "spec": { "idTransakcji" : "id", "date": "user.date", } }, { "operation": "modify-overwrite-beta", "spec": { "user": { "date": "=toLong(@(1,date))" } } } ] and it works. But I have problem beacause when I have emptyString in this date i need to transfer it in null(not null in String) and i dont know how to do it using JoltTransformation or something else so as not to lose what I already have and have null only when there is an empty string.
... View more
Labels:
- Labels:
-
Apache NiFi
12-18-2023
05:49 AM
Hi @SAMSAL , Yes i can convert negative value back but i need to send this date as long to other system and this is my problem with converting this date to long in nifi using toNumber() because my value is negative
... View more
12-18-2023
05:01 AM
@SAMSAL Thank you for help, everything works unfortunately except the date, but as you said, it is more complicated
... View more
12-18-2023
02:57 AM
Hi guys, I have some problems with transform date from string to long in NIFI. I have date like this: "02-01-1968|" and I need to transform this to long. i am using UpdateAttribte Processor to format it to number like this: ${my_date:toDate("dd-MM-yyyy","GMT"):toNumber()})} but I have result like this: "my_date" : -63072000000 .Does anyone know how to solve this better?
... View more
Labels:
- Labels:
-
Apache NiFi
12-15-2023
05:22 AM
Hi guys! i have some problems with JoltTransformation in NIFI. I have plain json like that: { "id": "c47f79f5-77b1-45d1-91df-968c2fa3def8", "system" : "OP", "sourceSystem" : "DKG", "name" : "Agnes", "lastName" : "Smith", "mail" : "a.smith@gmail.com", "country" : "Australia", "city" : "Sydney", "street" : "Colorado", "building" : "23", "local" : "1', "paycheck" : "3456,98', "joiningDate" : "12-09-2023", "isOfAge": "true", } And i need to have output like this: { "paycheck":3456.98, "joiningDate":1702048486451, "person":{ "name":"Agnes", "lastName":"Smith", "mail":"a.smith@gmail.com", "isOfAge":true, "adress":{ "country":"Australia", "city":"Sydney", "street":"Colorado", "building":"23", "local":"1" } }, "system_data":{ "id":"c47f79f5-77b1-45d1-91df-968c2fa3def8", "system":"OP", "sourceSystem":"DKG" } } In addition to nesting, I need to convert date to long, paycheck to double and isOfAge to boolean.I tried use JoltTransformation and Jolt Specification like this: [ { "operation":"shift", "spec":{ "paycheck":"paycheck", "joiningDate":"joiningDate", "name" : "person.name", "lastName" : "person.lastName", "mail" : "person.mail", "isOfAge" : "person.isOfAge", "country" : "person.adress.country", "city" : "person.adress.city", "street" : "person.adress.street", "building" : "person.adress.building", "local" : "person.adress.local", "id" : "system_data.id", "system" : "system_data.system", "sourceSystem" : "system_data.sourceSystem" } }, { "operation": "modify-overwrite-beta", "spec": { "paycheck": "=toDouble", "joiningDate": "=toNumber", "isOfAge": "=toBoolean" } } ] Nesting works but type conversion not. What am I doing wrong because i admit that jolt is little bit complicated for me
... View more
Labels:
- Labels:
-
Apache NiFi
10-18-2023
12:58 AM
@cotopaul it turned pout that this type in databyse is RAW not vairchar
... View more
10-17-2023
07:30 AM
@cotopaul when i change strategy to RecordPathValue i have something like this: "UUID_1" : "����}ZM�]����!p",
... View more