Member since
11-16-2015
902
Posts
664
Kudos Received
249
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 238 | 09-30-2025 05:23 AM | |
| 658 | 06-26-2025 01:21 PM | |
| 503 | 06-19-2025 02:48 PM | |
| 753 | 05-30-2025 01:53 PM | |
| 10982 | 02-22-2024 12:38 PM |
06-05-2017
01:58 PM
1 Kudo
You can use the JoltTransformJSON processor for this. Here is a Chain spec that should get you what you want: [
{
"operation": "shift",
"spec": {
"0": {
"*": "header"
},
"*": "data[]"
}
},
{
"operation": "shift",
"spec": {
"data": {
"*": {
"*": {
"@0": "[&2].@(4,header[&])"
}
}
}
}
}
] The first operation splits the top array into "header" (containing the first element) and "data" (containing the rest). The second operation creates an array of objects, each object having the keys from the header associated with the values from each element in the "data" array. With your input above, I get the following output: [ {
"host" : "DUSTSADMIN.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "28",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
}, {
"host" : "DUSTSVMDC01.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "34",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
}, {
"host" : "DUSTSVMDC02.ads.xyz.de",
"host_icons" : "menu",
"host_state" : "UP",
"num_services_crit" : "0",
"num_services_ok" : "34",
"num_services_pending" : "0",
"num_services_unknown" : "0",
"num_services_warn" : "0"
} ]
... View more
06-01-2017
03:27 PM
1 Kudo
FetchElasticsearch is used to get a single document from an ES cluster. Each document in ES has a document identifier (or "_id") associated with it, and that identifier is what should be supplied to the Document Identifier property. If you don't know the document identifier for the document(s) you're looking for, then QueryElasticsearchHttp is your best bet. It allows you to use the Query String "mini-language" to search for fields with desired values (see here for more information). You can then parse the results using any number of processors, such as EvaluateJsonPath to get individual fields from the results, SplitJson if there are multiple results, etc.
... View more
05-31-2017
03:26 PM
You can add dynamic properties in the InvokeHttp processor's configuration dialog as described here.
... View more
05-24-2017
01:42 PM
2 Kudos
As of NiFi 1.2.0 you may be able to use ConvertRecord to do this, with a JsonTreeReader and a CSVRecordSetWriter (with a Record separator of comma and a value separator of a single space). Prior to 1.2.0 (or if the above approach doesn't work), you can use ExecuteScript. Here is a sample Groovy script that will read all the "term" values from the incoming JSON object and add an attribute called "terms" containing the comma-separated list: def flowFile = session.get()
if(!flowFile) return
def input = session.read(flowFile)
def json = new groovy.json.JsonSlurper().parse(input)
def terms = json.results.collect { it.term }.join(',')
input.close()
flowFile = session.putAttribute(flowFile, 'terms', terms)
session.transfer(flowFile, REL_SUCCESS) If instead you need to replace the content of the flow file with the comma-separated list: def flowFile = session.get()
if(!flowFile) return
flowFile = session.write(flowFile, { inputStream, outputStream ->
def json = new groovy.json.JsonSlurper().parse(inputStream)
def terms = json.results.collect { it.term }.join(',')
outputStream.write(terms.bytes)
} as StreamCallback)
flowFile = session.putAttribute(flowFile, 'mime.type', 'text/csv')
session.transfer(flowFile, REL_SUCCESS)
... View more
05-19-2017
07:16 PM
I believe you'll also need the following in your my.cnf: binlog_format=row
... View more
05-18-2017
02:10 PM
Have you performed any INSERT, UPDATE, DELETE events since you enabled binary logging? You probably don't need to include Begin/Commit events unless you are doing auditing or your target DB needs them. In general, should you ever want to "reset" the CDC processor to get all the binlog records, set Retrieve All Records to true and clear the state of the processor (i.e. right-click on the stopped processor, choose View State, then Clear State).
... View more
05-16-2017
09:22 PM
You can still use RouteOnAttribute to route the flow files with missing values somewhere separate from the files that have all the values populated. In UpdateAttribute, you can call the attribute whatever you want the message to be, and for the JOLT transform, you can have multiple entries in the spec, each matching a separate entry from the JSON.
... View more
05-16-2017
09:17 PM
After ConvertJSONToSQL you should not have JSON in the flowfile content, rather it should be a SQL statement. From the screenshot, you are sending all relationships to PutSQL, when you should only send the "sql" relationship. The "original" and "failure" relationships contain the input JSON, the "sql" relationship contains the corresponding SQL statement(s).
... View more
05-15-2017
07:09 PM
In response to your comments/requirements on my other answer, you could do the following: 1) EvaluateJsonPath (to read the JSON data), let's say you get attributes field_1, field_2, field_3, field_4. 2) UpdateAttribute, to set attributes "Missing data field_1", "Missing data field_2", "Missing data field_3", "Missing data field_4" to true or false, based on your aforementioned "field_N:isEmpty()" expression 3) AttributesToJSON to put the "Missing data" attributes into the content 4) JoltTransformJSON, to find the missing fields and add an error message, using the following ShiftR spec: {
"operation": "shift",
"spec": {
"Missing data field_*": {
"true": {
"$1": "errors[].errorMessage"
}
}
}
}
That gives the following output: {
"errors" : [ {
"errorMessage" : "Missing data field_2"
}, {
"errorMessage" : "Missing data field_4"
} ]
}
... View more
05-15-2017
01:45 PM
What are your attributes named, and what are their values? AttributesToJSON will not create an array for you, but you can use JoltTransformJSON after AttributesToJSON to create an array from the attributes. Given this example input JSON: {
"error_message_1": "missing_field_1",
"error_message_2": "missing_field_2"
} You can use the following Shift spec in the JoltTransformJSON processor: {
"operation": "shift",
"spec": {
"error_message_*": "errors[].errorMessage"
}
} This gives the following output (which matches your goal above): {
"errors" : [ {
"errorMessage" : "missing_field_1"
}, {
"errorMessage" : "missing_field_2"
} ]
}
... View more