Member since
03-10-2017
82
Posts
28
Kudos Received
19
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
160 | 11-07-2022 02:37 AM | |
170 | 11-02-2022 03:04 AM | |
248 | 09-14-2022 07:54 AM | |
937 | 07-26-2022 05:44 AM | |
391 | 07-25-2022 05:49 AM |
01-18-2023
01:26 AM
@jacektrocinski Yes, Sir, you are right, I put more focus on read attribute section of the document which has none but missed seeing Additional details. Kudos!!
... View more
01-17-2023
05:58 AM
ConnectWebSocket does not read any details from Flow file attribute from an incoming flow file. If you are a Cloudera customer, feel free to file a support ticket with us, we should be able to help you on this with a longer engagement. Thank you
... View more
01-17-2023
04:04 AM
Ah my bad, for the WebSocket end point did you try ConnectWebSocket ?
... View more
01-17-2023
03:01 AM
You can use InvokeHTTP processor wich provide options to handle Header Thank you
... View more
11-07-2022
04:21 AM
2 Kudos
You can refer to this article and tune the logback.xml file to delete and rollover based log files based on you space constraints https://community.cloudera.com/t5/Community-Articles/Understanding-how-the-logback-xml-configuration-in-NiFi/ta-p/248662 If you found this response assisted with your issue, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Chandan
... View more
11-07-2022
04:16 AM
Before starting processor ListenHHTP , please check and make sure port 8888 is not used by any other process at host where nifi service is running. You can use netstat command to check and confirm. If you found this response assisted with your issue, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Chandan
... View more
11-07-2022
02:37 AM
When using SplitXML to split the flowfiles, SpilitXML writes the details of the following attributes in each split flow files, fragment. identifier, fragment.index, fragment.count, segment.original.filename if needed to be merged later then MrgeContent should be configured with Merge Strategy : Defragment "The 'Defragment' algorithm combines fragments that are associated by attributes back into a single cohesive FlowFile" Are you doing the same ? with MergeContent? If you found this response assisted with your issue, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Chandan
... View more
11-02-2022
03:04 AM
Any config files or supported jar needed by processors or controllers in NiFi can not be automated or managed by Cloudera Manager.
... View more
09-14-2022
07:54 AM
1 Kudo
I would first check if there is a difference in reporting by CM about service by checking the process status by running #ps -eaf | grep nifi on host where nifi is service installed and started by CM. nifi logs logged under default directory /var/log/nifi/ There you should find bootstrap.log , nifi-app.log and nifi-user.log , if service is started by CM then we should see logging at bootstrap.log and later in nifi-app.log. if there is any issue while starting the service then you should able to see stdout and stderr logs in CM for nfi start up events as well, let me know if you are to more details, and you can open a support case with Cloudera technical support if you have an active subscription. Thank You.
... View more
08-29-2022
07:58 AM
Trying to understand the relationship between upload template action and the error you have seen in logs, so the error is related to a sync issue between NiFi Flow and NiFi Regsitry Version control and is not related to uploading template in NiFi UI Are you this error only appear when you try uploading templates in NiFi UI ? Just to discard the error here can you please check in NiFi UI search for component with id b2b33551-8e13-1429-bd99-94bc96edf954 and see if get any component listed in search results and verify the status of that component with version control. Now the 2nd point, is when you see unable to upload message box Is this nifi service is secure? then you can check in nifi-user.log for any events with your username after template uplod action Check if any timeout related message reported in nifi-app.log after template upload action? how big template is ? Can you please try with very small template ? The above action items can help isolate the issue further. Thank You
... View more
08-29-2022
05:04 AM
This line " The problem occurs when InvokeHttp tries to resend data and it consumes so many threads that ListenSyslog starts dropping incoming data" My understanding is that InvokeHttp will try run with max number of threads based upon what is configured concurrent task settings, beyond that it will not use any more threads at the same ListenSyslog is also running with it own concurrent task setting and max threads it can use and which can be further tune to hold more request based on settings Max Size of Message Queue and Max Size of Socket Buffer and Max Number of TCP Connections. To hold your initial anaylsys between InvokeHTTP and ListenSyslog you can provide more details around your flow design when the issue occurs like screen shots of flow, configs setting for ListenSyslg and InvokeHTTP would help more to understand. Thank You
... View more
08-18-2022
02:11 AM
Running Stored procedures from NiFi sql based processors is not possible however you can use following suggestion suggested at https://community.cloudera.com/t5/Support-Questions/Execute-Stored-Procedure-of-MSSQL-in-Nifi/m-p/213365 Thank You.
... View more
08-18-2022
02:08 AM
State can be deleted from view state option on TailFile processor , the when processor is stopped but it will clear the entire state for tailfile processor, in subsequent run , tailfile file may end up reading the same file again in case file is present. I do not think any harm in keeping the state for files even if files are deleted from the source, Do you have any specific requirements here? or any issues?
... View more
07-26-2022
08:36 AM
At moment you can not run stored procedure using ExecuteSQL, however, you can use ExecureScript to do the same with stored procedure wrapped in script with MySQL engine or any other client you can use please refer : https://community.cloudera.com/t5/Support-Questions/Does-ExecuteSQL-processor-allow-to-execute-stored-procedure/m-p/158922 If you found this response assisted with your issue, please take a moment to login and click on "Accept as Solution" below this post. Thank you
... View more
07-26-2022
05:44 AM
This is related to Jira https://issues.apache.org/jira/browse/NIFI-9993 , fixed in the latest release 1.16.3 If you found this response assisted with your issue, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Chandan
... View more
07-25-2022
05:49 AM
I mean you can have as many CaptureChangeMySQL for a number of different MySQL DB's , Yes your design is alinged with what I have updated before. If you found this response assisted with your issue, please take a moment and click on "Accept as Solution" below this post. Thank you
... View more
07-25-2022
03:45 AM
Answer to your queries : 1. You have to use the two CaptureChangeMySQL Processors if you wish to connect two different MySQL database services. 2. Since you can get the change records from two CaptureChangeMySQL processors for two different source MySQL DB's then you can insert those records into another MySQL DB using PutSQL processor. Thank you
... View more
07-25-2022
03:22 AM
2 Kudos
In nifi there is no provision to restrict resources based upon process groups or processors, so the same goes for threads. but what you can do with your current request on limiting threads, you can run the processors which are part of one process group with default concurrent task 1 settings if not already done. Thank You.
... View more
07-14-2022
05:09 AM
1 Kudo
ExtractText processor can help you grab needed information from json content and store in the flow file attribute for reuse after InvokeHTTP. Thank You
... View more
06-21-2022
08:22 AM
There is no inbuilt one but you can please refer suggestion posted by one of NiFi Engineers about running Stored procedure at https://stackoverflow.com/questions/36306764/how-to-use-procedure-call-in-apache-nifi-executesql-processor-with-both-in-and If you found this response assisted with your issue, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Chandan
... View more
06-21-2022
08:01 AM
1 Kudo
DBCPConnectionPool provides Database Connection Pooling Service. Connections can be asked from pool and returned after usage by SQL processor which are responsible to run the SQL queries . You can useListDatabaseTables processor along withDBCPConnectionPool controller to get the table metadata If you found this response assisted with your issue, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Chandan
... View more
06-21-2022
07:51 AM
I would check Hive metastore logs to get more details on aborted t ransactions with respect to transaction ids since transactions are aborted at the Hive end and not the NiFi end.
... View more
06-16-2022
06:51 AM
I suggest you have a technical support case opened with the Cloudera Technical Support team so we can dive further. Thank You.
... View more
06-14-2022
05:21 AM
So java.net.BindException: Cannot assign requested address is an error occurred while attempting to bind a socket to provided hostname and port. Typically, the port is in use, or the requested hostname could not be assigned, Are you sure port is free and no issues with the hostname you are using ?
... View more
06-14-2022
05:15 AM
I per MS SQL docs I found this example String connectionUrl = "jdbc:sqlserver://localhost:1433;" + "databaseName=AdventureWorks;integratedSecurity=true;" + "encrypt=true; trustServerCertificate=false;" + "trustStore=storeName;trustStorePassword=storePassword"; Can you please check ? also the URL which you have used works outside of nifi ?
... View more
06-14-2022
02:56 AM
1 Kudo
Details you can only get through UI or API , considering web endpoint is reachable, Seems in this case NiFi service is up but UI is not available or NiFi service is completely down , can be confirm by checking its process status. So you need find way to bring back the NIFI service along with UI in minimalistic load by stopping all flows by setting nifi.flowcontroller.autoResumeState to false and start the service, by doing this nifi will start with all the queued data but it will not do any processing, this can give you the UI but since you have said that are too much data so startup will take time. If you found this response assisted with your issue, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Chandan
... View more
06-14-2022
02:18 AM
How you are passing SSL details in DBCPConnectionPool used by ConvertJSONToSQL? in configured JDBC URL?
... View more
06-13-2022
02:49 AM
Ok, so you got the copy file in NiFi using List/FetchFile then you changed the new directory location where it has to be written using UpdateAttribute then you have used DuplicateFlowFile to create duplicates, till now what is happening since ListFile is running on Primary node (And it should be running on Primary ) rest all downstream flow will be processed/computed on Primary that why you see DuplicateFlowFile creating duplicated on node which is Primary in NiFi Cluster. I do not see any issue here. If you wish to have flow files on All NiFi nodes redistributed after DuplicateFlowFile then you need to set Load Balancing at queue connection on the connection "Round Robin " between DuplicateFlowFile & PutFile, by doing this duplicates flow files will be distributed among nifi nods in round-robin manner one by one and will be written by PutFile back on each nifi nodes where files got distributed after DuplicateFlowFile. Now one question you mention 7 HDFS data nodes, does that means you have NiFi service running as well on the same 7 data nodes ? and having 7 node nifi cluster? in that case, only all 7 nodes will receive each file copy. I see
... View more
06-09-2022
06:21 AM
The error says the Connection pool is not able to find configured driver classpath name into provided database client-jar driver. Please check and make sure configured driver classpath name is present on the provided jar file.
... View more
06-09-2022
06:09 AM
1 Kudo
Please refer answer posted by MattB on UTC time,https://stackoverflow.com/questions/65620632/why-do-executesqlrecord-and-csvrecordsetwriter-updated-the-time-zone-of-datetime. Thank You.
... View more