Member since
03-10-2017
170
Posts
80
Kudos Received
32
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1160 | 08-12-2024 08:42 AM | |
| 1984 | 05-30-2024 04:11 AM | |
| 2568 | 05-29-2024 06:58 AM | |
| 1728 | 05-16-2024 05:05 AM | |
| 1325 | 04-23-2024 01:46 AM |
05-30-2024
06:17 AM
@Vikas-Nifi @ckumar is 100% correct. Only fields explicitly marked as supporting NiFi Expression Language (NEL) can support a NEL expression "${schedule}". I am however curious about your use case as to why you would even being trying to do this. From what you shared you are extracting a cron schedule from the json content of some FlowFiles traversing an EvaluateJsonPath processor. That "schedule" is added on to the NiFi FlowFile as a FlowFile attribute (key=value pair). This would not make that key=value pair accessible to any other NiFi component unless that FlowFile containing the FlowFile attribute was processed by that other component. However, in your shared dataflow you do not mention that EvaluateJsonPath connects to your invokeHTTP processor via an inbound dataflow connection (Keep in mind that even if you did do this, it does not change the fact that the run schedule property does not support NEL). I just wanted to clarify how FlowFile attributes are and can be used. Also keep in mind that the "run schedule" is a scheduler only. The run schedule set on a processor controls when the NiFi controller will schedule the execution of the processors code. It does not mean that they the processor will immediately execute at time of scheduling (It may be delayed on execution waiting for an available execution thread from the thread pool). All scheduled components share a thread pool and NiFi framework will also handle assigning threads to next scheduled component as thread become available. So the NiFi framework needs to know the scheduling for a component when it is started; otherwise, NiFi would never know when to schedule it to execute. Unless a component property has an explicit tool tip that tells you it support NEL, then it does not. For NiFi processor components, you will find that only some processor specific properties within the "PROPERTIES" tab support NEL. This is not only available through property tooltips, but also in the processors documentation. Examples: Even when NEL is supported there is a scope. It may support FlowFile attributes, Variable Registry (going away in NiFi 2.x releases), or both. Thank you, Matt
... View more
05-16-2024
05:43 AM
1 Kudo
yes I think that was the issue. I am still new to Nifi and slowly learning to work with it. Thank you @ckumar
... View more
04-23-2024
01:46 AM
1 Kudo
Not possible, there is a reason for that if we allow to add a property to set an initial offset, every time that processor was stopped and started we would again start at that configured offset, with console consumer, you only pass the initial offset on the first execution then subsequent executions use the last stored offset in Kafka, which is not the case here with the processor. Thus only two options are "earliest" or "latest". Thank you
... View more
04-17-2024
02:41 PM
1 Kudo
Thank you @MattWho Your recommendation worked for me. I have updated bootstrap.conf file in nifi. Was able to successfully enable DB controller service and persist data into Ignite database from Nifi. Details: Copied jvm parameters available in the file \apache-ignite-2.16.0-bin\bin\include\jvmdefaults.sh to \nifi-2.0.0-M2\conf\bootstrap.conf file Here is the format and exact list of java arguments added in nifi bootstrap java.arg.21=--add-opens=java.base/jdk.internal.access=ALL-UNNAMED
java.arg.22=--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED
java.arg.23=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
java.arg.24=--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
java.arg.25=--add-opens=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED
java.arg.26=--add-opens=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED
java.arg.27=--add-opens=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED
java.arg.28=--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
java.arg.29=--add-opens=java.base/java.io=ALL-UNNAMED
java.arg.30=--add-opens=java.base/java.nio=ALL-UNNAMED
java.arg.31=--add-opens=java.base/java.net=ALL-UNNAMED
java.arg.32=--add-opens=java.base/java.util=ALL-UNNAMED
java.arg.33=--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
java.arg.34=--add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED
java.arg.35=--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
java.arg.36=--add-opens=java.base/java.lang=ALL-UNNAMED
java.arg.37=--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
java.arg.38=--add-opens=java.base/java.math=ALL-UNNAMED
java.arg.39=--add-opens=java.sql/java.sql=ALL-UNNAMED
java.arg.40=--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
java.arg.41=--add-opens=java.base/java.time=ALL-UNNAMED
java.arg.42=--add-opens=java.base/java.text=ALL-UNNAMED
java.arg.43=--add-opens=java.management/sun.management=ALL-UNNAMED
java.arg.44=--add-opens=java.desktop/java.awt.font=ALL-UNNAMED We just need to make sure java.arg.<> numbers are unused in the bootstrap file that we are working on. Thanks again!
... View more
04-15-2024
07:33 PM
1 Kudo
I agree with ckumar's point and there is one more thing you could do if it makes you feel safer. You could put the HadoopConfigurationResources file path string into a sensitive parameter and refer to the that parameter (using #{paramname}) in the HiveCatalogService property.
... View more
04-15-2024
06:22 AM
1 Kudo
Do you see any exceptions in the logs before you see the Initiating shutdown of Jetty web server?
... View more
04-15-2024
06:20 AM
2 Kudos
Load the data into NiFi in an iterative manner, for this you can evaluate GenerateTableFetch -->ExecuteSQL. Loading a large table in one go with ExecuteSQL could result in high heap usage. on the other side, CPU usage is high due to running too many parallelism by increasing the processor concurrent task from default 1 to 8 which is not required. Thank you
... View more
04-07-2024
02:45 AM
1 Kudo
Hi, @schrippe can you please run the ldapsearch command on this particular OU "OU=Zentral,OU=Gruppen,DC=bk,DC=datev,DC=de" and check if you are getting your missing group here or not, it could be the group is present on different OU level. This is your Group search base config "OU=Zentral,OU=Gruppen,DC=bk,DC=datev,DC=de" so run the ldapsearch and verify the o/p.
... View more
04-04-2024
10:31 AM
Hi @Ytch , The way I was able to get to work is through the following instruction: 1- After you have download the jdbc drive from here: https://learn.microsoft.com/en-us/sql/connect/jdbc/download-microsoft-jdbc-driver-for-sql-server?view=sql-server-ver15 you should be able to find the following dll "mssql-jdbc_auth-8.2.2.x64.dll" under the path: ..\sqljdbc_8.2\enu\auth\x64 2- Copy the dll and place under the JAVA Home path bin folder: .. \Java\jdk-21\bin 3- Adjust the DB Connection URL in the DBCPconnectionpool controller service by adding and setting "integrated security" flag to true: jdbc:sqlserver://{SQL SEVER NAME};databaseName={DB NAme};integratedSecurity=true; 4- Restart Nifi If that helps please accept solution. Thanks
... View more
04-04-2024
06:15 AM
Can you please try these two NIFI_JVM_HEAP_INIT NIFI_JVM_HEAP_MAX Thank you
... View more