Member since
07-25-2022
23
Posts
7
Kudos Received
4
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
807 | 11-06-2024 02:11 PM | |
657 | 08-14-2024 01:31 PM | |
1895 | 08-14-2024 12:52 PM | |
800 | 04-04-2024 08:45 AM |
08-14-2024
01:31 PM
1 Kudo
@Former Member LookupRecord processor can be used in this case. Please check if this method works for you. Sample CSV input id,brand,phone 10,Samsung,1 20,Apple,2 Expected output (codes 1 and 2 are replaced with names) id,brand,phone 10,Samsung,Note 20,Apple,iPhone I tried nifi flow as below Lookup record configuration CSV Reader CSV Recordset Writer Didnt make any change. Used default settings Simple KeyValue Lookup service. There are several lookup services available in Nifi. You may use right service as per your requirement.
... View more
08-14-2024
12:52 PM
1 Kudo
I would do this way. Please see if this works for you. 1. Use JolttransformJSON processor to alter the input JSON Spec: [ { "operation": "shift", "spec": { "id": ["role.role_id", "user.id"], "name": "user.name", "age": "user.age", "role": { "role": "role.role_name" } } } ] 2. Pass the transformed JSON to first PutDatabaseRecord processor to insert into user table Set "Data Record Path" = user 3. Add another PutDatabaseRecord processor to insert into role table Set "Data Record Path" = role
... View more
08-13-2024
12:17 PM
@Adyant001 Can you share sample json structure ? If source json contains both parent and child information, you can have two PutDatabaseRecord processors connected in series with right "Data Record Path" mentioned. If child record needs information from parent level, JoltTransformJson processor can be used to create transformed json before passing to PutDatabaseRecord.
... View more
08-13-2024
11:58 AM
Hi, Currently we are using ConsumeAMQP processor to fetch messages from Rabbitmq queues. There are some use cases in our system where Rabbitmq streams seem to be a good fit. I have tested ConsumeAMQP to consume messages from Rabbitmq streams queue, but it doesn't work as expected. I believe the new functionality related to OFFSET management is not available in ConsumeAMQP processor. Any insights on this would be helpful. Am I missing something? If functionality is not part of current processor and would like it to be enhanced, how should I request? Thanks in advance.
... View more
Labels:
- Labels:
-
Apache NiFi
07-25-2024
02:11 PM
1 Kudo
Hi Nifi Experts, We are looking for best practice / suggestion for our use case below. a) There are around 100 tables in source database b) This source data needs to be synced into various target databases (after optional transformation) c) Each source tables may need to be synced to multiple target databases tables. So one extract on source table, might be used to load into multiple target tables which have similar structures d) As of today, we have implemented flows using the core processors ConsumeAMQP - ExecuteSQLRecord - PutDatabaseRecord . e) The challenge we face today is with the growing number of processors in proportion with the number of source/target tables. We are looking for a solution to minimize the number of processors by sharing ExecuteSQLRecord - PutDatabaseRecord processors for multiple tables sync. As a proof of concept, we tried to assign the database name, query, target database, target table name, keyfield etc., dynamically by using LookupRecord processor. Internally, tested with SimpleDatabase lookup service and Properfile lookup service to help assign required attributes dynamically. Please advise if this is good or any other best practice to handle data sync among tables dynamically. Core requirement is to have generic flows for multiple tables rather than dedicated flow for each table. Please let me know if more details are required. Thanking you in advance!
... View more
Labels:
- Labels:
-
Apache NiFi
04-17-2024
02:41 PM
1 Kudo
Thank you @MattWho Your recommendation worked for me. I have updated bootstrap.conf file in nifi. Was able to successfully enable DB controller service and persist data into Ignite database from Nifi. Details: Copied jvm parameters available in the file \apache-ignite-2.16.0-bin\bin\include\jvmdefaults.sh to \nifi-2.0.0-M2\conf\bootstrap.conf file Here is the format and exact list of java arguments added in nifi bootstrap java.arg.21=--add-opens=java.base/jdk.internal.access=ALL-UNNAMED
java.arg.22=--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED
java.arg.23=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
java.arg.24=--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
java.arg.25=--add-opens=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED
java.arg.26=--add-opens=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED
java.arg.27=--add-opens=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED
java.arg.28=--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
java.arg.29=--add-opens=java.base/java.io=ALL-UNNAMED
java.arg.30=--add-opens=java.base/java.nio=ALL-UNNAMED
java.arg.31=--add-opens=java.base/java.net=ALL-UNNAMED
java.arg.32=--add-opens=java.base/java.util=ALL-UNNAMED
java.arg.33=--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
java.arg.34=--add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED
java.arg.35=--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
java.arg.36=--add-opens=java.base/java.lang=ALL-UNNAMED
java.arg.37=--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
java.arg.38=--add-opens=java.base/java.math=ALL-UNNAMED
java.arg.39=--add-opens=java.sql/java.sql=ALL-UNNAMED
java.arg.40=--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
java.arg.41=--add-opens=java.base/java.time=ALL-UNNAMED
java.arg.42=--add-opens=java.base/java.text=ALL-UNNAMED
java.arg.43=--add-opens=java.management/sun.management=ALL-UNNAMED
java.arg.44=--add-opens=java.desktop/java.awt.font=ALL-UNNAMED We just need to make sure java.arg.<> numbers are unused in the bootstrap file that we are working on. Thanks again!
... View more
04-17-2024
10:55 AM
@MattWho Here is the log from nifi-app.log 2024-04-17 12:50:13,061 ERROR [Timer-Driven Process Thread-2] o.a.n.c.s.StandardControllerServiceNode StandardControllerServiceNode[service=DBCPConnectionPool[id=025a6b87-e362-3761-5695-a2871b444c4b], name=Ignite-db-localhost, active=true] Failed to invoke @OnEnabled method
java.lang.NoClassDefFoundError: Could not initialize class org.apache.commons.dbcp2.BasicDataSource
at org.apache.nifi.dbcp.AbstractDBCPConnectionPool.onConfigured(AbstractDBCPConnectionPool.java:139)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:145)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:133)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:78)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:55)
at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:655)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: Could not initialize class org.apache.ignite.IgniteJdbcThinDriver [in thread "Timer-Driven Process Thread-2"]
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:534)
at java.base/java.lang.Class.forName(Class.java:513)
at java.sql/java.sql.DriverManager.isDriverAllowed(DriverManager.java:559)
at java.sql/java.sql.DriverManager.isDriverAllowed(DriverManager.java:551)
at java.sql/java.sql.DriverManager.getDrivers(DriverManager.java:452)
at java.sql/java.sql.DriverManager.getDrivers(DriverManager.java:429)
at org.apache.commons.dbcp2.BasicDataSource.<clinit>(BasicDataSource.java:75)
... 15 common frames omitted
... View more
04-16-2024
01:34 PM
@Jim_Steinebrey @MattWho Thanks for your response. As mentioned, I have created another folder "thirdparty_lib" and updated the new location of jar file in "Database Driver Location(s)" field. I am still getting the same error. Please advise.
... View more
04-15-2024
04:11 PM
1 Kudo
@ckumar Yes I've placed ignite-core-2.16.0.jar file in nifi lib directory.
... View more
04-04-2024
03:02 PM
1 Kudo
Hi, I have Nifi 2.0-M2 and Apache Ignite 2.16 database running on my localhost. http://localhost:8443/nifi jdbc:ignite:thin://localhost:10800 (I am able to connect to Ignite localhost db from Dbeaver) I have copied ignite-core-2.16.0.jar file to nifi lib directory (before starting nifi) and tried to create DBCPConnectionPool controller service in Nifi to connect to Ignite database. DB Controller service returns the below error. Failed to invoke @OnEnabled method: java.lang.NoClassDefFoundError: Could not initialize class org.apache.commons.dbcp2.BasicDataSource - Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: Could not initialize class org.apache.ignite.IgniteJdbcThinDriver [in thread "Timer-Driven Process Thread-10"] Can you let me know what am I missing here? Please let me know if more details are required. Screenshot of DB controller service
... View more
Labels:
- Labels:
-
Apache NiFi
- « Previous
-
- 1
- 2
- Next »