Member since
06-26-2015
515
Posts
138
Kudos Received
114
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 2269 | 09-20-2022 03:33 PM | |
| 6036 | 09-19-2022 04:47 PM | |
| 3258 | 09-11-2022 05:01 PM | |
| 3727 | 09-06-2022 02:23 PM | |
| 5799 | 09-06-2022 04:30 AM |
06-23-2022
08:46 PM
1 Kudo
The NiFi community maintains a NiFi Docker image that comes very handy when you want to play with NiFi on your laptop. When you develop flows to run in a NiFi cluster with 2 or more nodes, though, there are things that you need to check and test differently from a single-node NiFi cluster. Recently, @Arqui shared a docker-compose configuration in this forum question that creates a 2-node NiFi cluster on Docker. The compose configuration still had a few issues to be ironed, though, and there were some manual steps you still had to perform before launching the cluster, like creating the TLS certificates for it. I found the idea interesting and spent some time working out the missing details. I also automated the TLS certificate creation and added a load-balancer to the compose configuration. Now, to use your own NiFi sanbox cluster, all you need to do is this: Download the docker-compose.yml file (see repo link below) Run "docker compose up -d" Connect to your own local 2-node cluster on http://localhost:8443/nifi. I've been pretty happy with the result and that's what I now use on my laptop when I want to test something out quickly. I hope this can be useful for others too: https://github.com/asdaraujo/nifi-stuff/tree/main/nifi-docker-compose Cheers, André
... View more
Labels:
06-22-2022
07:31 PM
1 Kudo
@baymax277 , Try this: This is the regex: (?s)(^\[|\]$) You may have to increase the buffer size to accommodate your file. Cheers, André
... View more
06-22-2022
06:49 PM
@baymax277 , You should look into using PutDatabaseRecord instead of ExecuteStreamCommand. This would be more efficient and it would eliminate the need to manually remove those brackets. Cheers, André
... View more
06-22-2022
06:20 PM
@Arqui , I've added nginx as a load balancer/proxy to the compose file. Cheers, André
... View more
06-22-2022
06:18 PM
@baymax277 Ok. Can you share a screenshot of the part of the flow that insert the data into your database? Cheers, André
... View more
06-22-2022
06:01 PM
@baymax277 , Could you also share the screenshot of the part of the flow where the error is happening? I think that's after the Decompress File step, right? André
... View more
06-22-2022
05:51 PM
@baymax277 , Could you share a screenshot of the part of you flow that reads and processes the JSON content? André
... View more
06-22-2022
04:46 PM
@baymax277 , The problem with your file is not the square brackets ([ ]). The issue is that single quotes are not a valid JSON quote. The correct JSON would be: [
{"a":1,"b":2},
{"a":3,"b":4}
] The content above is processed correctly by NiFi if you use record-based processors (like, for example QueryRecord). Ideally your source data should be fixed to provide a valid JSON. For example, if those files are being generated by a Python script like this: my_object = [{'a': 1, 'b': 2}, {'a': 3, 'b': 4}]
print(my_object) The Python script should be updated to do this instead: import json
my_object = [{'a': 1, 'b': 2}, {'a': 3, 'b': 4}]
print(json.dumps(my_object)) If you cannot have the source data fixed, you can try to replace the single quotes with double quotes using a ReplaceText processor with the following configuration. This is a simplistic approach, though, and can cause problems in some complex JSON objects if you have strings with unexpected sequences of characters in them. Cheers, André
... View more
06-22-2022
04:17 PM
@Sayed016 , At the time of the alert, if you run a cat /proc/sys/kernel/random/entropy_avail on that exact host, what values are you getting? Cheers, André
... View more