Member since
11-16-2015
905
Posts
666
Kudos Received
249
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 450 | 09-30-2025 05:23 AM | |
| 785 | 06-26-2025 01:21 PM | |
| 693 | 06-19-2025 02:48 PM | |
| 883 | 05-30-2025 01:53 PM | |
| 11475 | 02-22-2024 12:38 PM |
05-16-2017
09:17 PM
After ConvertJSONToSQL you should not have JSON in the flowfile content, rather it should be a SQL statement. From the screenshot, you are sending all relationships to PutSQL, when you should only send the "sql" relationship. The "original" and "failure" relationships contain the input JSON, the "sql" relationship contains the corresponding SQL statement(s).
... View more
05-15-2017
07:09 PM
In response to your comments/requirements on my other answer, you could do the following: 1) EvaluateJsonPath (to read the JSON data), let's say you get attributes field_1, field_2, field_3, field_4. 2) UpdateAttribute, to set attributes "Missing data field_1", "Missing data field_2", "Missing data field_3", "Missing data field_4" to true or false, based on your aforementioned "field_N:isEmpty()" expression 3) AttributesToJSON to put the "Missing data" attributes into the content 4) JoltTransformJSON, to find the missing fields and add an error message, using the following ShiftR spec: {
"operation": "shift",
"spec": {
"Missing data field_*": {
"true": {
"$1": "errors[].errorMessage"
}
}
}
}
That gives the following output: {
"errors" : [ {
"errorMessage" : "Missing data field_2"
}, {
"errorMessage" : "Missing data field_4"
} ]
}
... View more
05-15-2017
01:45 PM
What are your attributes named, and what are their values? AttributesToJSON will not create an array for you, but you can use JoltTransformJSON after AttributesToJSON to create an array from the attributes. Given this example input JSON: {
"error_message_1": "missing_field_1",
"error_message_2": "missing_field_2"
} You can use the following Shift spec in the JoltTransformJSON processor: {
"operation": "shift",
"spec": {
"error_message_*": "errors[].errorMessage"
}
} This gives the following output (which matches your goal above): {
"errors" : [ {
"errorMessage" : "missing_field_1"
}, {
"errorMessage" : "missing_field_2"
} ]
}
... View more
05-15-2017
01:40 PM
1 Kudo
I am having trouble importing the "etree" module, I have tried with brew-installed Python 2.7 and Anaconda 2.7 (where I believe the etree submodule is part of "xml" not "lxml"). Do I need any additional configuration? Looking in the lxml package, I see some native libraries (.so files, e.g.). If lxml is a native library, Jython (the "python" script engine in ExecuteScript) will not be able to load/execute it. All imported modules (and their dependencies) must be pure Python (no native code like CPython for example) for Jython to execute the script successfully. Perhaps there is a different library you can use? If you don't have a requirement on Jython/Python, consider using Javascript, Groovy, or Clojure instead. Their Module Directory allows you to use third-party Java libraries to accomplish this conversion, such as NekoHTML, JTidy, or JSoup.
... View more
05-13-2017
02:38 AM
In a JOLT spec, if you don't explicitly provide a transformation for a particular field, it will be excluded. So you can include matching rules for the fields you care about (i.e. those that have a certain value), the rest will be discarded. Check the "Filter data from an Array, based on a leaf level value" example at the JOLT Demo online app.
... View more
05-13-2017
02:33 AM
As of NiFi 1.2.0, after GetFile you can use the PutDatabaseRecord processor with a CSVReader, giving it a schema if you know the layout, or if the CSV file has a header row you can get the column names from that using "Use String Fields From Header" for the Schema Access Strategy property. Prior to NiFi 1.2.0, after GetFile you probably want a SplitText to get each row in its own flow file. Do you know the number of columns for the CSV file? If so then you can use a regex in ExtractText; assuming there were four columns you might have a dynamic property called "column" set to something like: ([^,]+),([^,]+),([^,]+),([^,]+) That should give you attributes "column.1", "column.2", "column.3", and "column.4". Then you can use ReplaceText to generate a SQL statement, perhaps something like: INSERT INTO myTable VALUES(${column.1},${column.2},${column.3},${column.4}) Then send that to PutSQL. Another option (prior to NiFi 1.2.0) is to convert the CSV to Avro (see this HCC article), then ConvertAvroToJSON, then ConvertJSONToSQL, then PutSQL.
... View more
05-11-2017
03:17 PM
1 Kudo
Since a template is XML, you could use an XSLT to replace the values for those properties. Alternatively, scripting languages such as Python and Groovy can handle XML fairly easily, you could write a script to replace the values. Ideally these properties would support Expression Language, I have written NIFI-3867 to cover this improvement.
... View more
05-10-2017
12:52 PM
There are too many IOUtils.toString() calls there, the "text" line should read: text = IOUtils.toString(inputStream, StandardCharsets.ISO_8859_1))
... View more
05-09-2017
03:27 PM
2 Kudos
Try the following for the XPath: string(/queryResponse/@last) Also ensure the Destination property is "flowfile-content" and Return is "string", this will ensure the value of the attribute is written as the contents of the outgoing flow file.
... View more
05-03-2017
01:15 PM
Can you provide some sample input? I tried with a tab-separated file that contained a \n in the column (with the line ending in \n\r), and your script worked fine. I tried replacing the delimiter value with \t instead of an actual tab character, and it seemed to work fine too.
... View more