Member since
09-27-2018
138
Posts
23
Kudos Received
10
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
9744 | 02-28-2021 10:23 PM | |
1941 | 02-08-2021 11:53 PM | |
31781 | 12-16-2020 11:31 PM | |
7611 | 12-14-2020 11:02 PM | |
4844 | 12-14-2020 12:18 AM |
07-09-2019
02:44 PM
Hi @Rosa Negra thanks for your answer! I worked with ff-attributes as long as the JSON-file contained only one entry at a time and it worked fine. Now with the complex structure I couldn't find a way to manipulate the fields of the positions. Unfortunately I have no experience with record processing. If I understand you correctly it would be possible to update the JSON doing record processing. So I am "afraid" I have to explore it... Maybe there are further solutions?
... View more
07-08-2019
04:24 PM
1 Kudo
Hi all, as far as I investigated this (https://community.hortonworks.com/questions/247330/nifi-need-help-with-jolt-syntax-jolttransformjson-1.html) it is not possible to do this with JOLT. Now I want to manipulate the JSON-content BEFORE JOLT does simple transformation. I'm getting used to do some things with groovy, viewed some examples but couldn't find the right syntax. There are two things I need to to: 1. All timestamps from JSON have to be converted and formated from CET yyyyMMddHHmmss ==> to UTC yyyy-MM-dd HH:mm:ss.SSS 2. Some string manipulation with some kind of expression language doing this: Expression Language to build "dnr_group" ${DNR:substringAfter('('):substringBefore(')'):toNumber()
:lt( 10 )
:ifElse(${DNR:substringAfter('('):substringBefore(')'):prepend('0')},
${DNR:substringAfter('('):substringBefore(')')})
:append('-'):append(${GROUP:toNumber()
:lt( 10 )
:ifElse(${GROUP:prepend('0')},
${GROUP})})
:prepend('V-')} Input-JSON {
"user": "Justen",
"stand": "20190702121621",
"table": [
{
"zn": 1,
"elem_user": "Meier",
"elem_stand": "20190705081410",
"dnr": "(0)",
"group": "1"
},
{
"elem_stand": "20190706201918",
"zn": 2,
"elem_user": "Schmidt",
"dnr": "(0)",
"group": "2"
}
]
} Desired Output-JSON {
"user": "Justen",
"stand": "2019-07-02 10:16:21",
"table": [
{
"zn": 1,
"elem_user": "Meier",
"elem_stand": "2019-07-05 06:14:10",
"dnr_group": "V-00-01"
},
{
"elem_stand": "2019-07-06 18:19:18",
"zn": 2,
"elem_user": "Schmidt",
"dnr_group": "V-00-02"
}
]
} Is it possible to do this with groovy or does anybody know other ways manipulating JSON-content in NiFi? Thanks for any help!
... View more
Labels:
- Labels:
-
Apache NiFi
07-01-2019
05:52 AM
@Matt Burgess It works! Thank you so much not only for the solution but also for the explanation!
... View more
06-26-2019
11:07 AM
Hi, I need some help with JOLT-syntax. JSON-Input {
"nummer": "22",
"id": "177",
"table": [
{
"zn": 1,
"stfflbisart": 10
},
{
"zn": 2,
"stfflbisart": 50
}
]
} JOLT-Specification [{
"operation": "shift",
"spec": {
"id": "ID",
"nummer": "Nummer",
"table": {
"*": {
"zn": "ArtikelPreise_Pos.[#2].ZeileNr",
"stfflbisart": "ArtikelPreise_Pos.[#2].StaffelBis"
}
}
}
}, {
"operation": "default",
"spec": {
"Default_Kopf": "${VAR_KD}"
}
}
]
JSON-Output {
"ID" : "177",
"Nummer" : "22",
"ArtikelPreise_Pos" : [ {
"ZeileNr" : 1,
"StaffelBis" : 10
}, {
"ZeileNr" : 2,
"StaffelBis" : 50
} ],
"Default_Kopf" : "${DFT_KOPF}"
} Like in "Default_Kopf" I need to set a default value to each element of the array "ArtikelPreise_Pos" coming from flowfile attribute "DFT_POS". The result should look like this: WANTED JSON-Output {
"ID" : "177",
"Nummer" : "22",
"ArtikelPreise_Pos" : [ {
"ZeileNr" : 1,
"StaffelBis" : 10,
"Default_Pos" : "${DFT_POS}"
}, {
"ZeileNr" : 2,
"StaffelBis" : 50,
"Default_Pos" : "${DFT_POS}"
} ],
"Default_Kopf" : "${VAR_KD}"
} Tried things like this, but it doesn't work: [{
"operation": "shift",
"spec": {
"id": "ID",
"nummer": "Nummer",
"table": {
"*": {
"zn": "ArtikelPreise_Pos.[#2].ZeileNr",
"stfflbisart": "ArtikelPreise_Pos.[#2].StaffelBis"
}
}
}
}, {
"operation": "default",
"spec": {
"Default_Kopf": "${DFT_KOPF}",
"ArtikelPreise_Pos": {
"Default_Pos": "${DFT_POS}"
}
}
}
] Any help? Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
06-26-2019
08:03 AM
Hi @Matt Burgess following your above solution for returning the max-value in an attribute my script looks like this: def flowFile = session.get()
if(!flowFile) return
try {
def inputStream = session.read(flowFile)
def objList = new groovy.json.JsonSlurper().parse(inputStream)
def max = objList.max {Date.parse("yyyyMMddHHmmss",it.elem_stand)}
inputStream.close()
flowFile = session.putAttribute(flowFile, 'MAX_elem_stand', max.elem_stand.toString())
session.transfer(flowFile, REL_SUCCESS)
} catch(e) {
log.error("Error while determining max", e)
session.transfer(flowFile, REL_FAILURE)
} It worked fine with Json-format: [{
"zn": 1,
"elem_stand": "20190611140623",
"elem_id": "1086"
},
{
"zn": 2,
"elem_stand": "20190624170807",
"elem_id": "1089"
} ] But now the input format is changed to: {
"name": "Belegart",
"id": "269",
"table": [
{
"zn": 1,
"elem_stand": "20190611140623",
"elem_id": "1086"
},
{
"zn": 2,
"elem_stand": "20190624170807",
"elem_id": "1089"
}
]
} And I can't find the right syntax to access the elem_stand within the array for checking max-value. Please could you tell me how this is possible? Thanks!
... View more
06-19-2019
08:59 AM
@Archon Gum Embarrassing... OK, now this happens. Attributes of nonzero status FF:
... View more
06-19-2019
08:38 AM
@Archon Gum I'm sorry, but the bucket definitely exists! Your solution came so close to my target... Do you have another idea? By the way. If I add your supplement this error occurs: ExecuteStreamCommand[id=e0bc9898-7dfb-3992-9e35-46fd7e983658] Transferring flow file StandardFlowFileRecord[uuid=6a70ebcc-13a0-431b-aa85-014bf8fb825f,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1560791493717-1, container=default, section=1], offset=715298, length=59],offset=0,name=tmp_nifi_update_flow_ver.py,size=59] to nonzero status.
Executable command /usr/bin/python3.6 ended in an error: Traceback (most recent call last):
File "tmp_nifi_update_flow_ver.py", line 61, in <module>
update_flow_ver(pg_id, bucket_name, flow_name, target_version)
File "tmp_nifi_update_flow_ver.py", line 45, in update_flow_ver
sys.exit(-1)
NameError: name 'sys' is not defined
... View more
06-19-2019
06:48 AM
@Archon Gum Thanks for this detailed and informative answer! So my admin solved the SSL-problem among others by installing Python3.6. Now I come until this error: ExecuteStreamCommand[id=e0bc9898-7dfb-3992-9e35-46fd7e983658] Transferring flow file StandardFlowFileRecord[uuid=f680a1c1-5ba9-426e-8b4f-a1d89e1a3f30,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1560791493717-1, container=default, section=1], offset=646526, length=-1],offset=0,name=tmp_nifi_update_flow_ver.py,size=0] to nonzero status.
Executable command /usr/bin/python3.6 ended in an error: Traceback (most recent call last):
File "tmp_nifi_update_flow_ver.py", line 56, in <module> update_flow_ver(pg_id, bucket_name, flow_name, target_version)
File "tmp_nifi_update_flow_ver.py", line 42, in update_flow_ver flow = nipyapi.versioning.get_flow_in_bucket(bucket_id=bucket.identifier, identifier=flow_name, identifier_type='name')
AttributeError: 'NoneType' object has no attribute 'identifier' Can you give me some advice on this?
... View more
06-13-2019
05:11 PM
Hi @Patrick Hochstenbach Did you find the cause for this behavior? Facing the same issue since update 1.9.2. First time a restart solved the problem, second time it worked again after 1 day without restart or doing anything in this context. Strange...
... View more