Member since
07-25-2022
20
Posts
7
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
523 | 11-06-2024 02:11 PM | |
552 | 08-14-2024 01:31 PM | |
1759 | 08-14-2024 12:52 PM | |
729 | 04-04-2024 08:45 AM |
11-06-2024
02:23 PM
I was able to setup GitLabFlowRegistry in controller settings successfully (which is new feature in Nifi 2.0) From Nifi UI, I am able to import Nifi flow directly from Gitlab repository. After modifications, I was able to successfully commit into Gitlab repo, from Nifi UI. Do we have any toolkit commands to perform these operations? Currently we are using toolkit commands to deploy Nifi flow changes via CICD pipeline. Can I achieve the same with new GitLabFlowRegistry? Appreciate any help/pointers. Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
11-06-2024
02:11 PM
@CommanderLaus Gitlab integration is available in latest version Nifi 2.0 (released few days ago). I was able to import and export flows directly from Gitlab repo. Note: Until Nifi 2.0M4, the file extension is "snapshot", but in 2.0 if you use Gitlab, flow config file extension should be ".json" We need to provide access token of the gitlab repository in Registry controller settings.
... View more
11-04-2024
06:09 AM
2 Kudos
@shiva239 You can create an apache "NiFi" jira in the Apache community to highlight this new feature and request modification to existing processor to support it. https://issues.apache.org/jira/ The more detail you provide in your jira the better chance someone might take this on in the community. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-09-2024
03:47 PM
Thanks @SAMSAL for the clarification and the alternatives for looping in Nifi. We will consider using DuplicateFlowFile processor. Do you think nifi documentation should be updated to explicitly mention that it is not only for load tests but also can be used in production flows when there is need to clone flow files ?
... View more
09-04-2024
02:51 PM
@AlokKumar I am not able to understand the context here. It would help to get to right solution if you can explain why the inserts into same table should be executed in certain order. However, Nifi Connection configuration settings have prioritizers. Please check screenshot below.. If you are able to set attribute "priority" with desired number, "PriorityAttributePrioritizer" can be used. Please note that these prioritizers operate on node level. You may have to test these if you have multi node nifi environment. There is a processor "EnforceOrder" which has more options to enforce execution order. You may check this processor and find out if it works. https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/nifi2.0-m4/org.apache.nifi.processors.standard.EnforceOrder/index.html
... View more
08-16-2024
08:43 AM
@Adyant001 Oracle has auto increment IDENTITY feature as below. Table definition takes care of setting primary key incremental value. No need to send this value in json payload I think all other major RDBMS have similar feature to assign auto increment value. CREATE TABLE roletab (id NUMBER GENERATED BY DEFAULT ON NULL AS IDENTITY (START WITH 1 INCREMENT BY 1 NOCYCLE), role_id NUMBER, role_name VARCHAR(100), PRIMARY KEY (id) ); INSERT INTO roletab (role_id, role_name) VALUES (10, 'Admin'); INSERT INTO roletab (role_id, role_name) VALUES (20, 'Developer'); commit; SELECT * FROM roletab; ID |ROLE_ID|ROLE_NAME| --+-------+---------+ 1 | 10 | Admin | 2 | 20 |Developer|
... View more
08-14-2024
01:31 PM
1 Kudo
@Former Member LookupRecord processor can be used in this case. Please check if this method works for you. Sample CSV input id,brand,phone 10,Samsung,1 20,Apple,2 Expected output (codes 1 and 2 are replaced with names) id,brand,phone 10,Samsung,Note 20,Apple,iPhone I tried nifi flow as below Lookup record configuration CSV Reader CSV Recordset Writer Didnt make any change. Used default settings Simple KeyValue Lookup service. There are several lookup services available in Nifi. You may use right service as per your requirement.
... View more
07-25-2024
02:11 PM
1 Kudo
Hi Nifi Experts, We are looking for best practice / suggestion for our use case below. a) There are around 100 tables in source database b) This source data needs to be synced into various target databases (after optional transformation) c) Each source tables may need to be synced to multiple target databases tables. So one extract on source table, might be used to load into multiple target tables which have similar structures d) As of today, we have implemented flows using the core processors ConsumeAMQP - ExecuteSQLRecord - PutDatabaseRecord . e) The challenge we face today is with the growing number of processors in proportion with the number of source/target tables. We are looking for a solution to minimize the number of processors by sharing ExecuteSQLRecord - PutDatabaseRecord processors for multiple tables sync. As a proof of concept, we tried to assign the database name, query, target database, target table name, keyfield etc., dynamically by using LookupRecord processor. Internally, tested with SimpleDatabase lookup service and Properfile lookup service to help assign required attributes dynamically. Please advise if this is good or any other best practice to handle data sync among tables dynamically. Core requirement is to have generic flows for multiple tables rather than dedicated flow for each table. Please let me know if more details are required. Thanking you in advance!
... View more
Labels:
- Labels:
-
Apache NiFi
04-17-2024
02:41 PM
1 Kudo
Thank you @MattWho Your recommendation worked for me. I have updated bootstrap.conf file in nifi. Was able to successfully enable DB controller service and persist data into Ignite database from Nifi. Details: Copied jvm parameters available in the file \apache-ignite-2.16.0-bin\bin\include\jvmdefaults.sh to \nifi-2.0.0-M2\conf\bootstrap.conf file Here is the format and exact list of java arguments added in nifi bootstrap java.arg.21=--add-opens=java.base/jdk.internal.access=ALL-UNNAMED
java.arg.22=--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED
java.arg.23=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
java.arg.24=--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
java.arg.25=--add-opens=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED
java.arg.26=--add-opens=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED
java.arg.27=--add-opens=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED
java.arg.28=--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
java.arg.29=--add-opens=java.base/java.io=ALL-UNNAMED
java.arg.30=--add-opens=java.base/java.nio=ALL-UNNAMED
java.arg.31=--add-opens=java.base/java.net=ALL-UNNAMED
java.arg.32=--add-opens=java.base/java.util=ALL-UNNAMED
java.arg.33=--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
java.arg.34=--add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED
java.arg.35=--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
java.arg.36=--add-opens=java.base/java.lang=ALL-UNNAMED
java.arg.37=--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
java.arg.38=--add-opens=java.base/java.math=ALL-UNNAMED
java.arg.39=--add-opens=java.sql/java.sql=ALL-UNNAMED
java.arg.40=--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
java.arg.41=--add-opens=java.base/java.time=ALL-UNNAMED
java.arg.42=--add-opens=java.base/java.text=ALL-UNNAMED
java.arg.43=--add-opens=java.management/sun.management=ALL-UNNAMED
java.arg.44=--add-opens=java.desktop/java.awt.font=ALL-UNNAMED We just need to make sure java.arg.<> numbers are unused in the bootstrap file that we are working on. Thanks again!
... View more
04-04-2024
10:01 AM
Congratulations on resolving the issue @shiva239. Can you please mark the appropriate reply as the solution to make it easier for others to find the answer in the future?
... View more