Member since
08-01-2021
59
Posts
14
Kudos Received
7
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 3803 | 11-18-2022 09:06 AM | |
| 5848 | 11-15-2022 05:46 PM | |
| 3977 | 10-12-2022 03:18 AM | |
| 2981 | 10-11-2022 08:52 AM | |
| 7316 | 10-08-2022 08:23 AM |
09-20-2026
10:23 PM
Greetings dear Apache NiFi community. I've recently had the opportunity to use NiFi to read and write JSON data to PostgresSQL using its native JSONB data type. While researching how to do this, I found very limited information, mainly just one helpful StackOverflow question and reply by @mpayne. My use case was fairly simple: I had JSON files that I wanted to store as-is and be able to retrieve them later in the exact same form. This presented three main challenges:
JSONB is a column, meaning that when writing to it, you need to wrap up your JSON under a new field with a name matching the column.
PutDatabaseRecord will not work if the data under the new wrapper field is a valid JSON, since it will try to parse it as record values. Therefore, you need to escape your JSON data before writing to the DB.
When reading the data back, you need to restore the escaped JSONs back to their original form and remove the wrapper field from step 1.
Writing to JSONB column
In my case, the PostgreSQL table had two columns:
data of type JSONB
id of type UUID (generated from the contents of data)
The table DDL is as such:
CREATE TABLE IF NOT EXISTS <schema>.<table> (
data JSONB NOT NULL,
id UUID GENERATED ALWAYS AS ((data->>'id')::uuid) STORED,
PRIMARY KEY (id)
)
To wrap my JSONs in the 'data' field and escape them, I used a single ReplaceText processor configured as such:
Replacement Strategy
Regex Replace
Search Value
(?s)(^.*$)
Replacement Value
{"data": "${'$1':escapeJson():replace('\\', '\\\\')}"}
Character Set
UTF-8
Maximum Buffer Size
1 MB
Evaluation Mode
Entire text
Line-by-Line Evaluation Mode
All
Credit to Mark Payne (@mpayne) for suggesting that replacement value in the StackOverflow thread mentioned above. The expression matches your entire file content (careful if your JSONs are bigger than the Maximum Buffer Size), making it accessible via the '$1' attribute in the Replacement Value property. This allows you to run Expression Language on your entire file's contents. We use this to run :escapeJson():replace('\\', '\\\\') on the JSON data to prepare it for insertion. There is also a simple {"data": "...."} literal text wrapper so that the following PutDatabaseRecord processor will know to write this escaped json to the 'data' column.
No special configuration is needed in the PutDatabaseRecord processor. This is the overall flow:
Note that although it appears you insert escaped JSONs to your DB, they are actually stored correctly and can be queried using standard JSONB operators (->, ->>, etc.)
Reading the Data back
To read the data back, I use ExecuteSQLRecord to fetch rows (making sure to specify Array Output in the JSON RecordWriter), followed by another ReplaceText processor to edit the results back to their correct shape.
I define a basic SQL query to fetch just the data column:
SELECT data FROM <schema>.<table>
The resulting FlowFile content might look something like:
[
{
"data": "{\"id\": \"488b5b63-6ccf-4816-9a38-38c3befd248f\", \"myKey\": \"myValue\", \"keyTwo\": \"bababa\"}"
},
{
"data": "{\"id\": \"2fec9cad-5246-4d92-aa41-9a57cfc4e8ab\", \"myKey\": \"myValue23\", \"keyTwo\": \"21312\"}"
},
{
"data": "{\"id\": \"6efbac98-3a10-422d-a2a2-ac6d79ffef6b\", \"myKey\": \"myValue23\", \"keyTwo\": {\"nested\": [\"abc\", \"def\"]}}"
}
]
As you can see, each row gets read as a flat JSON where the key is 'data' like the column name, and the value is a stringified/escaped version of the original content.
To fix it now, I use this config in the following replace text:
Replacement Strategy
Regex Replace
Search Value
"data"\s*:\s*"\{((?:\\.|[^"\\])*)\}"
Replacement Value
${'$1':unescapeJson()}
Character Set
UTF-8
Maximum Buffer Size
1 MB
Evaluation Mode
Line-By-Line
Line-by-Line Evaluation Mode
All
Note that I use Line-By-Line mode this time, since ExecuteSQLRecord seems to fetch rows prettified with newlines separating them (Even if Pretty Print is set to false in the RecordWriter!). Because it scans line-by-line, it ignores the braces/brackets at the top and bottom of the response and in-between the different rows, and you can instead focus on just the content of the rows, which looks like:
"data" : "{\"id\": \"488b5b63-6ccf-4816-9a38-38c3befd248f\", \"myKey\": \"myValue\", \"keyTwo\": \"bababa\"}"
Thus, we match everything within the value of data's braces "data" : "{ <this part> }" and once again use ReplaceText's special expression syntax '$1' to refer to our escaped version of the original JSON content and then run :unescapeJson() on it to restore it back to its valid form.
The output will look like
[ {
"id" : "488b5b63-6ccf-4816-9a38-38c3befd248f",
"myKey" : "myValue",
"keyTwo" : "bababa"
}, {
"id" : "2fec9cad-5246-4d92-aa41-9a57cfc4e8ab",
"myKey" : "myValue23",
"keyTwo" : "21312"
}, {
"id" : "6efbac98-3a10-422d-a2a2-ac6d79ffef6b",
"myKey" : "myValue23",
"keyTwo" : {
"nested" : [ "abc", "def" ]
}
} ]
And now you have valid JSONs to work with again for the rest of your flows. DISCLAIMER: An external user contributed to this article. Cloudera may not verify that the steps may be applicable for all use cases and may be very specific to a particular distribution. Please follow with caution and at your own risk. If needed, raise a support case to get confirmation.
... View more
Labels:
01-27-2026
08:32 AM
Hello community I was wondering if anyone has any experience with writing entries into redis with dynamic TTLs. From what I can tell, it is only possible to set the TTL via the DistributedMapCacheClientService and as such it is set 'globally' per flow. For me the ideal case would've been if it was possible to set the TTL based on flowfile attributes. Any advice would be greatly appreciated. Thanks 🙂
... View more
Labels:
- Labels:
-
Apache NiFi
01-14-2026
06:30 AM
1 Kudo
@MattWho Wow! I think this pattern would work best for my usecase. I hadn't even considered the first challenge you brought up of production flows having their Parameter Context unassigned if I were to update their version. That would've been painful to find out after deploying many instances. Back in NiFi 1 I used to handle situations such as this with variables, since they could just be directly attached to Process Groups and so I never had to worry about creating separate objects (parameters) and ensuring they get attached, or that every new instance of a versioned flow had to have its own unique context created. It's been a couple years but I believe I even questioned Pierre about this in one of his appearances in the Israeli NiFi meet-ups. In regards to product work, I've ran into this case of trying to use NiFi as the underlying tool for different SaaS platforms multiple times already. There could definitely be some QoL changes made to make such a use-case easier to implement with NiFi's flow registry, I guess the responsibility lies in people like me opening issues to bring them though 🙂 Thank you very much for the suggestions Matt! Green
... View more
01-13-2026
06:08 AM
@MattWho So just to verify I understand everything: 1) I start with a dev process group somewhere in my canvas (let us say it is named my-flow), with a parameter context attached to it, say PC-DEV, and then I commit this PG to the registry so I will be able to deploy thousands of unique production copies of it 2) When I deploy my-flow from the registry in the same nifi environment, the newly created process group will automatically have PC-DEV attached to it (<-- this is what I'm worried about) 3) In order to uniquely set up this flow, I now need to create a new PC which mimics PC-DEV so that I will be able to configure a unique set of parameters for this deployed instance Currently testing on nifi 2.7.2, I actually see that there's a toggle when importing from registry to "Keep existing Parameter Contexts", which partially answers my concern. If it is toggled ON, then the new deployment will automatically use an existing PC. If toggled OFF, it will instead automatically create a new copy of the commit-time PC (without sensitive values) which will be named "<original pc name> (1)" with the number increasing per copy. The OFF behaviour is more in-line with what I wish to do in my environment, though the naming is still problematic as I'd prefer the PC name to more directly correlate to the deployed instance (when I'll deploy instances via my code, I'll probably add the instance ID to the PC name) Your 2nd note has raised some concerns for me. Did I understand correctly that if I commit a new version for a flow and in it I added a new paramter somewhere in the procsesors, then automatically when I update the other flows that new parameter will be added to all of their respective parameter contexts? (E.G. I update my dev flow with a new parameter, all production copies of the dev parameter context will now have a new parameter added, though with the value being the same as the dev value I commited) Frankly I still feel as though my best course of action is to simply commit the flows without any PC attached but with parameter strings (#{..}) pre-configured. This way when I deploy a flow with my code via the REST api, I can more easily tailor a new parameter context to the new flow. This way there are also no concerns of accidentally attaching default/dev values to production flows. Though if I add new parameters, I will need to manually add them to all the existing PCs. I'm not sure there's any one clear answer to my question so I'll just mark yours as the solution. Thanks Matt 🙂
... View more
01-11-2026
12:15 AM
Hi @MattWho , thank you for the detailed reply 🙂 I might not have been very clear in my original question so I'll rephrase with more context- My use-case is not deploying a single instance of a flow from NiFi A to NiFi B, rather I've got just one NiFi instance and I'd like to deploy thousands of instances of the same versioned flow (and perhaps in the future, I'll have more clusters in different regions and would like to deploy the flows there too). In this context, it is problematic for me that Parameter Contexts get created/picked based on the name of the original PC at commit time, since I want each flow instance to have its own unique values for the parameters. Therefore, my current plan is commiting the flows without a PC attached (though with parameters configured in the processors), and at creation time (via my codebase) creating a new PC (uniquely named per flow) with the specific values for that instance. My question was whether there is a more preferable way to do this - creating many instances of a parameterized versioned flow in the same nifi environment and ensuring each instance can get its own unique set of parameters. Thanks for helping out, Green
... View more
01-05-2026
11:55 PM
I am not clear on what kind of example I could share with you. 403 seems to imply the user you got the token for does not have permissions to run the operation. You should verify if your user has the correct permissions, specifically "Operate the component" permissions, though I believe these are included if the user has modify permissions.
... View more
01-01-2026
01:15 AM
1 Kudo
So just to verify, in the UI you are able to start/stop process groups? If your user is able to do so by clicking on the PG, it should be able to do so with the REST API
... View more
12-29-2025
01:46 AM
1 Kudo
So GET requests work fine for you but POSTs fail? Perhaps your user does not have write permissions? I recommend opening devtools and looking at the Network tab to see what requests get sent when you do certain operations (starting/stopping a process group) in the UI. You should be able to replicate them with the REST API if you can do them manually in the UI.
... View more
12-29-2025
12:46 AM
Hello team & community 🙂 I am looking into deploying many instances of a flow that is versioned in a git registry via the REST API. My current issue is that I would like to configure each instance of this flow dynamically, though if I version the flow with a Parameter Context attached to it, whenever I deploy instances of it they all end up attaching to a single Parameter Context. I understand this is the expected behaviour as described in the documentation. Therefore, I'd like to know if there is any recommended practice for situations such as mine. The best solution I could think of was perhaps configuring the processors in the flow to use parameters but not to actually attach a PC to the process group. This way I could deploy flows in two steps by first importing them and then creating a new parameter context & configuring/attaching it. I would greatly appreciate any advice for this use-case, Thank you.
... View more
Labels:
- Labels:
-
Apache NiFi
12-29-2025
12:33 AM
1 Kudo
Hi @MuruganFinastra , Are you sure you are attaching your token correctly to your requests? You should be using an Authorization header with a value of "Bearer <token>" for all requests once you fetch the token.
... View more