Member since
04-07-2022
38
Posts
11
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1004 | 03-17-2025 01:46 AM | |
| 6450 | 06-12-2024 08:32 PM |
11-21-2023
08:45 AM
@SAMSAL thank you, that solved it
... View more
11-21-2023
05:55 AM
I am trying to transform a json using jolt where i am moving the array inside serviceCharacteristic to outside of service {}, eventually i plan to move to respective positions {
"serviceOrderItem": [
{
"action": "modify",
"id": "0965f63b-db26-4397-b222-190a891838f0",
"service": {
"href": "serviceInventory/v4/service/adaptiveNetworkSite/1b9a3fae-f50b-4310-887d-60988f8d5645",
"id": "adaptiveNetworkSite",
"serviceCharacteristic": [
{
"name": "modifyPath",
"value": [
{
"op": "add",
"path": "$.supportingService",
"value": {}
},
{
"op": "sub",
"path": "$.supportingService",
"value": {}
}
]
}
]
}
},
{
"action": "create",
"id": "0965f63b-db26-4397-b222-190a891838f0",
"service": {
"href": "serviceInventory/v4/service/adaptiveNetworkSite/1b9a3fae-f50b-4310-887d-60988f8d5645",
"id": "adaptiveNetworkSite",
"serviceCharacteristic": [
{
"name": "modifyPath",
"value": [
{
"op": "abcd",
"path": "$.supportingService",
"value": {
"state": "active"
}
},
{
"op": "efgh",
"path": "$.supportingService",
"value": {
"state": "active"
}
}
]
}
]
}
}
]
} desired output is {
"serviceOrderItem": [
{
"action": "modify",
"id": "0965f63b-db26-4397-b222-190a891838f0",
"service": {
"href": "serviceInventory/v4/service/adaptiveNetworkSite/1b9a3fae-f50b-4310-887d-60988f8d5645",
"id": "adaptiveNetworkSite"
}
"modifyPath": [
{
"op": "add",
"path": "$.supportingService",
"value": {}
},
{
"op": "sub",
"path": "$.supportingService",
"value": {}
}
]
},
{
"action": "create",
"id": "0965f63b-db26-4397-b222-190a891838f0",
"service": {
"href": "serviceInventory/v4/service/adaptiveNetworkSite/1b9a3fae-f50b-4310-887d-60988f8d5645",
"id": "adaptiveNetworkSite"
}
"modifyPath": [
{
"op": "abcd",
"path": "$.supportingService",
"value": {
"state": "active"
}
},
{
"op": "efgh",
"path": "$.supportingService",
"value": {
"state": "active"
}
}
]
}
}
]
} my Jolt is working only when there is single array in modifyPath [{
"operation": "shift",
"spec": {
"*": "&",
"serviceOrderItem": {
"*": {
"*": "serviceOrderItem[&1].&",
"service": {
"*": "serviceOrderItem[&2].service.&",
"serviceCharacteristic": {
"*": {
"value": "@(1,name)"
}
}
}
}
}
}
}, {
"operation": "shift",
"spec": {
"*": "&",
"modifyPath": {
"@(1,modifyPath)": "serviceOrderItem[0].modifyPath"
},
"supportingService": {
"@(1,supportingService)": "serviceOrderItem[0].service.supportingService"
}
}
}] when multiple array is given it shuffles the array, "serviceOrderItem" : [ {
"action" : "modify",
"id" : "0965f63b-db26-4397-b222-190a891838f0",
"modifyPath" : [ {
"op" : "add",
"path" : "$.supportingService",
"value" : { }
}, {
"op" : "sub",
"path" : "$.supportingService",
"value" : { }
}, [ {
"op" : "abcd",
"path" : "$.supportingService",
"value" : {
"state" : "active"
}
}, {
"op" : "efgh",
"path" : "$.supportingService",
"value" : {
"state" : "active"
}
} ] ],
"service" : {
"href" : "serviceInventory/v4/service/adaptiveNetworkSite/1b9a3fae-f50b-4310-887d-60988f8d5645",
"id" : "adaptiveNetworkSite"
}
}, {
"action" : "create",
"id" : "0965f63b-db26-4397-b222-190a891838f0",
"service" : {
"href" : "serviceInventory/v4/service/adaptiveNetworkSite/1b9a3fae-f50b-4310-887d-60988f8d5645",
"id" : "adaptiveNetworkSite"
}
} ]
} please advise , thank you for you time
... View more
Labels:
- Labels:
-
Apache NiFi
11-20-2023
09:52 PM
Changed the jolt to [
{
"operation": "shift",
"spec": {
"relatedParty": "relatedParty",
"serviceOrderItem": {
"*": {
"id": "serviceOrderItem[&1].id",
"action": "serviceOrderItem[&1].action",
"service": {
"*": "serviceOrderItem[&2].service.&",
"supportingService": {
"#supportingService": "serviceOrderItem[&3].service.serviceCharacteristic[1].name",
"*": "serviceOrderItem[&3].service.serviceCharacteristic[1].value"
}
},
"modifyPath": {
"#modifyPath": "serviceOrderItem[&2].service.serviceCharacteristic[1].name",
"@(1,modifyPath)": "serviceOrderItem[&2].service.serviceCharacteristic[1].value"
}
}
}
}
},
{
"operation": "modify-overwrite-beta",
"spec": {
"*": "=recursivelySquashNulls"
}
}] Is there a way to remove the indexing from serviceCharacteristic[1].name / serviceCharacteristic[0].name ?
... View more
11-20-2023
07:43 PM
I am new to Jolt, I am trying to transform JSON to a particular format using nifi, where i am moving some objects into another object, I think i am close This is my JSON {
"relatedParty": [
{
"id": "B2BSmallandMedium"
}
],
"serviceOrderItem": [
{
"id": "e26205f4-e144-4ded-9e64-82ede0b26f4",
"action": "add",
"service": {
"name": "cust",
"state": "active",
"supportingService": [
{
"id": "e26205f4-e144-4ded-9e64-82ede0b26e99",
"href": "serviceInventory/v4/service"
},
{
"id": "7313c885-a075-4188-a6bc-c3fbfb4c7422",
"href": "activationAndConfiguration/v4/service"
}
]
}
}
]
} I am new to Jolt, I am trying to transform JSON to a particular format, where i am moving some objects into another object, I think i am close. This is my JSON {
"relatedParty": [
{
"id": "B2BSmallandMedium"
}
],
"serviceOrderItem": [
{
"id": "e26205f4-e144-4ded-9e64-82ede0b26f4",
"action": "add",
"service": {
"name": "cust",
"state": "active",
"supportingService": [
{
"id": "e26205f4-e144-4ded-9e64-82ede0b26e99",
"href": "serviceInventory/v4/service"
},
{
"id": "7313c885-a075-4188-a6bc-c3fbfb4c7422",
"href": "activationAndConfiguration/v4/service"
}
]
}
}
]
}
I am using the JOLT [
{
"operation": "shift",
"spec": {
"relatedParty": "relatedParty",
"serviceOrderItem": {
"*": {
"id": "serviceOrderItem[&1].id",
"action": "serviceOrderItem[&1].action",
"service": {
"*": "serviceOrderItem[&2].service.&",
"#modifyPath": "serviceOrderItem[&2].service.serviceCharacteristic[0].name",
"@(1,modifyPath)": "serviceOrderItem[&2].service.serviceCharacteristic[0].value",
"#supportingService": "serviceOrderItem[&2].service.serviceCharacteristic[1].name",
"@(1,service.supportingService)": "serviceOrderItem[&2].service.serviceCharacteristic[1].value"
}
}
}
}
},
{
"operation": "modify-default-beta",
"spec": {
"serviceOrderItem": {
"*": {
"service": {
"serviceCharacteristic": {
"*": {
"value": []
}
}
}
}
}
}
},
{
"operation": "remove",
"spec": {
"serviceOrderItem": {
"*": {
"service": {
"supportingService": ""
}
}
}
}
}] this is now giving me "service" : {
"name" : "cust",
"serviceCharacteristic" : [ {
"name" : "modifyPath",
"value" : "[]"
}, {
"name" : "supportingService",
"value" : [ {
"href" : "serviceInventory/v4/service",
"id" : "e26205f4-e144-4ded-9e64-82ede0b26e99"
}, {
"href" : "activationAndConfiguration/v4/service",
"id" : "7313c885-a075-4188-a6bc-c3fbfb4c7422"
} ]
} ],
"state" : "active"
}
} ] This is working for me but, I would like to know if it is possible replace: "#modifyPath": "serviceOrderItem[&2].service.serviceCharacteristic[0].name" "#supportingService": "serviceOrderItem[&2].service.serviceCharacteristic[1].name" to update the Json only when the key is actually available. I do not want an empty array. Thank you for your time
... View more
Labels:
- Labels:
-
Apache NiFi
09-12-2023
07:03 AM
Turns out there is a workaround, there is an option to change the first processor to execute in single node, this will execute flowfiles from single node.
... View more
09-12-2023
04:24 AM
I have NiFi 1.23.2 running in cluster mode in K8s. As List SFTP does not take an incoming request have written a script that is called from ExecuteStreamCommand to count the number of files in a directory. I am using expect to make this happen. #!/bin/sh -e USER=$1 export PASSWORD="${SFTP_PASSWORD}" REMOTE_DIR=$3 HOST=$4 BATCH_FILE=$(mktemp) echo "spawn sftp $USER@$HOST" > $BATCH_FILE echo "expect \"*fingerprint*\"" >> $BATCH_FILE echo "send \"yes\\r\"" >> $BATCH_FILE echo "expect \"password:\"" >> $BATCH_FILE echo "send \"\$env(PASSWORD)\\r\"" >> $BATCH_FILE echo "expect \"sftp>\"" >> $BATCH_FILE echo "send \"ls -l $REMOTE_DIR \\r\"" >> $BATCH_FILE echo "expect \"sftp>\"" >> $BATCH_FILE echo "send \"exit\\r\"" >> $BATCH_FILE #expect $BATCH_FILE dirs=$(expect $BATCH_FILE| sed -n '/sftp>/, /sftp>/p' | grep -v '^sftp>'| awk '$0 ~ /^d/ {print $NF}') rm $BATCH_FILE total=0 SAVEIFS=$IFS IFS=$'\n' directories=($dirs) IFS=$SAVEIFS for value in "${directories[@]}"; do val=$(echo "$value" | tr -d '[:blank:]' | tr -d '[:space:]') if [[ "$val" == "Completed" || "$val" == "Failed" || "$val" == "Rejected" || "$val" == "Error" ]]; then dir=$REMOTE_DIR/$val BATCH_FILE=$(mktemp) echo "spawn sftp $USER@$HOST" > $BATCH_FILE echo "expect \"*fingerprint*\"" >> $BATCH_FILE echo "send \"yes\\r\"" >> $BATCH_FILE echo "expect \"password:\"" >> $BATCH_FILE echo "send \"\$env(PASSWORD)\\r\"" >> $BATCH_FILE echo "expect \"sftp>\"" >> $BATCH_FILE echo "send \"ls -l $dir\\r\"" >> $BATCH_FILE echo "expect \"sftp>\"" >> $BATCH_FILE echo "send \"exit\\r\"" >> $BATCH_FILE count=$(expect $BATCH_FILE| sed -n '/sftp>/, /sftp>/p' | grep -v '^sftp>'| wc -l) total=$((total + count)) rm $BATCH_FILE fi done echo "$total" This script is working fine when I run inside this manually with in the container, also is working fine when NiFi runs on standalone mode in 1.23.0 bash-4.4$ ./test_files.sh 4 test_files.sh has the same script but HOST, USER and REMOTE_DIR hardcoded. When I call the same script from NiFi processor I am getting execution.error send: spawn id exp3 not open while executing "send "yes\r"" (file "/tmp/tmp.eatmyc06B6" line 3) The fingerprint expects had to be added inside the container as it is being asked. when running outside NiFi it is not required. There are times when it works, in the list. one has an error , the other doesn't I have tried the header with both /bash and /sh. what could be the reason? Please advise.
... View more
Labels:
- Labels:
-
Apache NiFi
09-03-2023
10:42 PM
1 Kudo
Update : This is working as I hoped for With this configuration the FileProcessor group will take the next flowfile only after completely processing the flowfile that is inside the group. Thank you @SAMSAL, @pvillard
... View more
09-03-2023
09:36 PM
@SAMSAL : Thank you for this. This could be what I am looking for, Will try with this and update here.
... View more
08-31-2023
08:57 AM
Thanks for the reply, I cannot use list sftp as it does not allow incoming request hence I have created an execute stream command processor that fetches the list. I will then feed these list into fetch sftp and then process these files. I want to fetch these file one by one only after the previous file has processed completely. So far with my implementation the files are picked almost immediately and processed, I cannot use the control rate processor in between because each file has different set of data and have varying processing time
... View more
08-31-2023
02:35 AM
I have a flowfile that is a list of files, that gets split into individual flowfiles (10 in this case) I am using FetchSFTP to fetch the file from a sftp location, the content of this fetched file is then split and is processed and stored. In this example: I want to start processing ep.txt only after i have processed bulk.txt after that names.txt and so on. I was referring to these links: NiFi workflow monitoring – Wait/Notify pattern with split and merge – Pierre Villard and How to wait for all fragments to be processed, then do something? (ijokarumawak.github.io) I was able to implement something like this. but this is not what I was looking for, I want to implement a wait before fetchSFTP I tried release signal identifier with filename , but that was stopping any flowfile from moving forward. Please advise, thank you
... View more
Labels:
- Labels:
-
Apache NiFi
- « Previous
- Next »