About DigitalPlumber

DigitalPlumber · ‎01-21-2022

Without knowing how the FTP directory is populated, you might have an issue with the ListFTP state and might consider changing its its Listing Strategy, However assuming that is no issue and you want to only perform an action based on any file with a naming structure like this: xxxx-xxxx-xxxx-xxxx-20220121110000-xxxx-xxxx.csv and only if it is older than two hours from "now" I would run the output of your ListFTP to an UpdateAttribute and add these 2 properties: property name fileTime value ${filename:getDelimitedField(5,'-'):trim():toDate('yyyyMMddHHmmss'):toNumber()} property name timeNow value ${now():toNumber():minus(7200000)} Then route that to a RouteOnAttribute and add a new property: property name value 2 hours old ${fileTime:le(${timeNow})} Then you can drag that connection to follow on processing and the unmatched connection to other processing or terminate it. Explanation: filelName ${filename:getDelimitedField(5,'-'):trim():toDate('yyyyMMddHHmmss'):toNumber()} This grabs the time out of your filename and converts it to a Date object so it can be converted to its epoch representation timeNow ${now():toNumber():minus(7200000)} Sets a value 2 hours ago from current time ${fileTime:le(${timeNow})} If attribute fileName is less than or equal to timeNow it means that it is 2 hours old.

raniaa · ‎01-21-2022

Indeed @DigitalPlumber , EvaluateXpath can't handle variables, only an Xpath. Queryrecord might be an alternative if i was using another type of file but since im using an XML file and this processor performs an SQL Query what can be my parameters in the query ?

DigitalPlumber · ‎01-20-2022

Deleting that state directory should not be a normal maintenance function. What you initially described is a very odd case. The fact that your node went down 003 and if it was the primary node or the coordinator node, internally there would have been a election to nominate a new cluster member node to perform those functions. In your case node 003 is a member of the cluster but it is not connected. Why it is not connected could be the cause of n reasons typically node is down or it was manually disconnected. When you see that message how many member nodes do you have? I expect the UI to show 2/3 because node 3 is not connected. The solution is to connect it by fixing the issue of why node is down or connect it through the UI

nilyin · ‎08-16-2021

Hi, can you clarify your answer a bit more. "PutDistributedMapCache" processor can store only Content of FlowFile, not key and an arbitrary attribute (from documentation: "Gets the content of a FlowFile and puts it to a distributed map cache"). So how is it possible to cache simple values like UUID in the flow without necessity of storing a “heavy” payload (content)? Can you give an example of such configuration? Can you give an example of such configuration?

Enri · ‎09-02-2020

Same error, different port (8080): the problem was fixed by deleting the /var/lib/nifi/state directory (you need to stop all Nifi instances first). You can have this problem everytime you switch from TLS to non-secure and viceversa.

yogesh_shisode · ‎08-13-2020

This issue resolved. External ZK did the trick. I am not able to figure the solution with embedded ZK. But using External ZK I am able to achieve the use case which I was trying. Below is link where you will get few more details on this thread, https://stackoverflow.com/questions/63339452/apache-nifi-not-working-with-multiple-notes-in-zookeeper-string Thanks.

SAMSAL · ‎08-07-2020

Thank you very much! After installing the proper java version the Groovy script started to work.

DigitalPlumber · ‎07-20-2020

To put Kerberos DEBUG logging for HDF managed Schema Registry ( SR ) clusters, do the following depending on your environment: Navigate to Ambari schema registry > configs > advanced > registry-env template Add the below line: export REGISTRY_OPTS="$REGISTRY_OPTS -Dsun.security.krb5.debug=true" Example: Restart SR. You will then be able to view Kerberos DEBUG messages under /var/log/registry (default).

DigitalPlumber · ‎07-15-2020

When NiFi is secured for TLS server authentication, at UI login time first it tries to use TLS certificates if loaded on the browser and then, it tries to use SPNEGO authentication, and finally, it falls back to your configured login provider. If you KERBERISE the cluster via AMBARI and want to use login-providers like LDAP or KERBEROS, it automatically sets the following properties which enable SPNEGO authentication. nifi.kerberos.spnego.keytab.location nifi.kerberos.spnego.principal Furthermore, SPNEGO properties through AMBARI are greyed out for: COMMAND: From your Amabari manager host, change the setting for NiFi, where the text in red is tailored to your unique environment: nifi.kerberos.spnego.keytab.location to be blank: ./configs.py -a set -s http -l c2288-node1.squadron.support.hortonworks.com -t 8080 -n c2288 -u admin -p AdminPassword -c nifi-properties -k 'nifi.kerberos.spnego.keytab.location' -v '' nifi.kerberos.spnego.principal to be blank ./configs.py -a set -s http -l c2288-node1.squadron.support.hortonworks.com -t 8080 -n c2288 -u admin -p AdminPassword -c nifi-properties -k 'nifi.kerberos.spnego.principal' -v '' -a set -s http or https -l fqdn of ambari host -t port number ambari is listening on -n ambari cluster name ( you can get that from top right UI ) -u user that has edit privileges on ambari -p the password for that user -c the config type in this case nifi.properties -k the key to change -v the value to change You can also do this for NiFi Registry with the following sample commands: ./configs.py -a set -s http -l c2288-node1.squadron.support.hortonworks.com -t 8080 -n c2288 -u admin -p AdminPassword -c nifi-registry-properties -k 'nifi.registry.kerberos.spnego.principal' -v '' ./configs.py -a set -s http -l c2288-node1.squadron.support.hortonworks.com -t 8080 -n c2288 -u admin -p AdminPassword -c nifi-registry-properties -k 'nifi.registry.kerberos.spnego.principal' -v '' Restart NiFi and/or NiFi Registry and ensure that you clear your browser cache. You should see the following on Ambari config sections of NiFi and/or NiFi registry:

DamD · ‎05-07-2020

Hi, I did restart nifi , but the problem still persists. The next release of nifi will add a new DBCP connection Pool: Hadoop DBCP connection pool: https://issues.apache.org/jira/browse/NIFI-7257 It will, hopefully, solve the issue. For my part, I implemented a specific connector which modifies the classpath: https://github.com/dams666/nifi-dbcp-connectionpool In short: public static final PropertyDescriptor DB_DRIVER_LOCATION = new PropertyDescriptor.Builder() .name("database-driver-locations") .displayName("Database Driver Location(s)") .description("Comma-separated list of files/folders and/or URLs containing the driver JAR and its dependencies (if any). For example '/var/tmp/mariadb-java-client-1.1.7.jar'") .defaultValue(null) .required(false) .addValidator(StandardValidators.createListValidator(true, true, StandardValidators.createURLorFileValidator())) .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY) .dynamicallyModifiesClasspath(true) .build(); ... dataSource = new BasicDataSource(); dataSource.setDriverClassName(drv); dataSource.setDriverClassLoader(this.getClass().getClassLoader()); But I still got same error message: PutSQL[id=94d192a9-fd1d-3c59-99be-d848f8902968] Failed to process session due to java.sql.SQLException: Cannot create PoolableConnectionFactory (ERROR 103 (08004): Unable to establish connection.): org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Cannot create PoolableConnectionFactory (ERROR 103 (08004): Unable to establish connection.) My setup: Database Connection URL : jdbc:phoenix:zk4-xxx.ax.internal.cloudapp.net,zk5-xxx.ax.internal.cloudapp.net,zk6-xxx.ax.internal.cloudapp.net:2181:/hbase-unsecure Database Driver Class Name: org.apache.phoenix.jdbc.PhoenixDriver Damien

Online	Offline
Last Visited	‎12-11-2024 01:24 AM

Member Since	‎09-04-2019 01:31 PM
Last Visited	‎12-11-2024 01:24 AM
Posts	62
Kudos received	17

Cloudera Community

Re: Stateful processors in test flows vs. deployme...

Re: How to split large json file into multiple jso...

Re: Template import error

Re: how to replace value in file

Re: Can Launch Apache NiFi on Windows Laptop

Re: How to get files two hours ago in FTP file sys...

Re: Use a variable in a processor

Re: NiFi Node showing 2 nodes and not respecting n...

Re: Storing dynamic data in Apache NiFi

Re: (NiFi) Cannot replicate request to Node nifi.d...

Re: Apache Nifi not working with multiple notes in...

Re: ExecuteScrip Groovy ClassNotFound Exception

How to put Schema Registry on DEBUG for Kerberos J...

How to disable SPNEGO login on HDF (Ambari) NiFi a...

Re: Connecting to Phoenix on kerberized cluster fr...