Member since
07-29-2020
574
Posts
320
Kudos Received
175
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
241 | 12-20-2024 05:49 AM | |
276 | 12-19-2024 08:33 PM | |
287 | 12-19-2024 06:48 AM | |
239 | 12-17-2024 12:56 PM | |
230 | 12-16-2024 04:38 AM |
06-30-2022
03:14 PM
Hi, Similar question was asked before. Can you please check this post: https://community.cloudera.com/t5/Support-Questions/How-to-load-json-record-to-postgres-as-json-datatype/m-p/345779#M234644
... View more
06-29-2022
11:28 AM
Thank you so much Mr @MattWho for comprehensive explanation on what Mr @SAMSAL earlier proposed. I really learn alot & appreciate.
... View more
06-29-2022
06:20 AM
@Tryfan You mention this file comes in daily. You also mention that this file arrives through a load-balancer so you don't know which node will receive it. This means you can't configure your source processor for "primary node" only execution as you have done in your shared sample flow with the ListFile. As Primary Node only, the elected primary node will be the only node that executes that processor. So if the source file lands on any other node, it would not get listed. You could handle this flow in the following manor: GetFile ---> (7 success relationships) PostHTTP or InvokeHTTP (7 of these with one configured for each node in your cluster cluster) ListenHTTP --> UpdateAttribute --> PutFile So in this flow, no matter which node receives your source file, the GetFile will consume it. It will then get cloned 6 times (7 copies then exist) with one copy of the FlowFile getting routed to 7 unique PostHttp processors. Each of these targets the ListenHTTP processor listening on each node in your cluster. That ListenHTTP processor will receive all 7 copies (one copy per node) of the original source file. Then use the UpdateAttribute to set your username and location info before the putFile which place each copy in the desired location on the source node. If you add or remove nodes from your cluster, you would need to modify this flow accordingly which is a major downside to such a design. Thus the best solution is still one where the source file is placed somewhere all nodes can retrieve it from so it scales automatically. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-28-2022
06:47 AM
Thanks. I think your approach is better to push all overheads to the Oracle Db rather than NIFI. 👍
... View more
06-24-2022
10:44 PM
@HTalha , Another way to do this is to use the ExecuteScript processor with the following python script: from org.apache.commons.io import IOUtils
from java.nio.charset import StandardCharsets
from org.apache.nifi.processor.io import StreamCallback
import json
import re
class SplitAndConvertToJson(StreamCallback):
def __init__(self):
pass
def process(self, inputStream, outputStream):
input = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
input = re.sub(r'(^\s*\{\s*|\s*\}\s*)', '', input)
fields = input.split(',')
obj = dict([('val_%s' % (i,), v.rstrip()) for i, v in enumerate(fields)])
outputStream.write(bytearray(json.dumps(obj).encode('utf-8')))
flowfile = session.get()
if(flowfile != None):
flowfile = session.write(flowfile, SplitAndConvertToJson())
session.transfer(flowfile, REL_SUCCESS) Cheers, André
... View more
06-22-2022
09:18 AM
Dear @MattWho Thanks a lot for the response. I installed Java SDK 11 and that helps to run NiFi on my laptop. I also change the environmental variables. I add JAVA_HOME under user variables and defined it like %JAVA_HOME%\bin in the path and now it is working fine. For my desktop I did the same thing and I notice NI applications (LabView) installed on my desktop was also using port 8080, so I fixed that one as well. NiFi is working fine on my desktop as well. I also found the following link relevant and useful for any other community members getting the same error: https://stackoverflow.com/questions/69830860/nifi-will-run-but-wont-open-in-browser-possible-error-with-java All the best Alireza
... View more
06-21-2022
10:55 AM
Hi @Techie123
Hopefully after reading the link that @SAMSAL shared earlier, you've reached the understanding that no, you don't have to install the database on the same server as the server that NiFi is installed on, but of course the server hosting the database must be accessible to the NiFi server via the JDBC protocol, regardless of where it is located. There are any number of things that could be causing the error message that you are getting.
Given what you have shared in this thread, I would recommend that you begin by talking to your DBA or perhaps the person that installed the database on the server. The string that you have entered on the properties tab, as shown in your page shot:
jdbc:oracle:thin@hostname:portnumber:sname
…needs to be replaced with the values appropriate for the server where your actual database is installed. To be a bit more specific, you have to get the actual literal values for hostname, portnumber and sname necessary to access your database and enter all of them correctly. In my personal experience, you'd also be much better off if you can verify that the database server is accessible via JDBC using some other tool so you have a way to validate the connection string before you get involved with NiFi-specific details.
... View more
06-20-2022
06:40 AM
I apologize for that, I hope someone else step in and help you with better solution. May I just know why it did not help?
... View more
06-16-2022
10:29 PM
Thanks to everyone especially @SAMSAL. I got to know it is the firewall that was blocking the communications among the ports. I tried to add those ports to firewall but no much success. When i tried to reach NIFI GUI, i ran into this error: java.net.NoRouteToHostException: No route to host (Host unreachable). So i disable the firewall & everything worked perfectly. FIRST METHOD: NO SUCCESS firewall-cmd --zone=public --permanent --add-port 9991/tcp firewall-cmd --zone=public --permanent --add-port 2888/tcp firewall-cmd --zone=public --permanent --add-port 3888/tcp firewall-cmd --zone=public --permanent --add-port 2181/tcp firewall-cmd --reload SECOND METHOD: SUCCESS systemctl stop firewalld
... View more
06-16-2022
07:24 AM
Sorry I forgot to mention that in the PutSql processor set the Batch Size property to 1
... View more