Cloudera Community
DigitalPlumber
user rank
user rank
Expert Contributor

Re: How to get files two hours ago in FTP file sys...

by Expert Contributor Expert Contributor in Support Questions
‎01-21-2022 09:46 AM
1 Kudo
‎01-21-2022 09:46 AM
1 Kudo
Without knowing how the FTP directory is populated, you might have an issue with the ListFTP state and might consider changing its its Listing Strategy, However assuming that is no issue and you want to only perform an action based on any file with a naming structure like this: xxxx-xxxx-xxxx-xxxx-20220121110000-xxxx-xxxx.csv and only if it is older than two hours from "now" I would run the output of your ListFTP to an UpdateAttribute and add these 2 properties:   property name        fileTime          value ${filename:getDelimitedField(5,'-'):trim():toDate('yyyyMMddHHmmss'):toNumber()} property name timeNow                      value ${now():toNumber():minus(7200000)}   Then route that to a RouteOnAttribute and add a new property:   property name              value 2 hours old                ${fileTime:le(${timeNow})}   Then you can drag that connection to follow on processing and the unmatched connection to other processing or terminate it.   Explanation: filelName ${filename:getDelimitedField(5,'-'):trim():toDate('yyyyMMddHHmmss'):toNumber()} This grabs the time out of your filename and converts it to a Date object so it can be converted to its epoch representation   timeNow ${now():toNumber():minus(7200000)} Sets a value 2 hours ago from current time     ${fileTime:le(${timeNow})} If attribute fileName is less than or equal to timeNow it means that it is 2 hours old.       ... View more

Re: Use a variable in a processor

by Explorer in Support Questions
‎01-21-2022 05:44 AM
‎01-21-2022 05:44 AM
Indeed @DigitalPlumber , EvaluateXpath can't handle variables, only an Xpath. Queryrecord might be an alternative if i was using another type of file but since im using an XML file and this processor performs an SQL Query what can be my parameters in the query ?  ... View more

Re: NiFi Node showing 2 nodes and not respecting n...

by Expert Contributor Expert Contributor in Support Questions
‎01-20-2022 09:43 AM
‎01-20-2022 09:43 AM
Deleting that state directory should not be a normal maintenance function. What you initially described is a very odd case. The fact that your node went down 003 and if it was the primary node or the coordinator node, internally there would have been a election to nominate a new cluster member node to perform those functions.   In your case node 003 is a member of the cluster but it is not connected. Why it is not connected could be the cause of n reasons typically node is down or it was manually disconnected. When you see that message how many member nodes do you have? I expect the UI to show 2/3 because node 3 is not connected. The solution is to connect it by fixing the issue of why node is down or connect it through the UI     ... View more

Re: Storing dynamic data in Apache NiFi

by New Contributor in Support Questions
‎08-16-2021 09:26 AM
‎08-16-2021 09:26 AM
Hi, can you clarify your answer a bit more. "PutDistributedMapCache" processor can store only Content of FlowFile, not key and an arbitrary attribute (from documentation: "Gets the content of a FlowFile and puts it to a distributed map cache").    So how is it possible to cache simple values like UUID in the flow without necessity of storing a “heavy” payload (content)?   Can you give an example of such configuration?   Can you give an example of such configuration? ... View more

Re: (NiFi) Cannot replicate request to Node nifi.d...

by Explorer in Support Questions
‎09-02-2020 06:46 AM
‎09-02-2020 06:46 AM
Same error, different port (8080):  the problem was fixed by deleting the /var/lib/nifi/state directory (you need to stop all Nifi instances first).     You can have this problem everytime you switch from TLS to non-secure and viceversa.   ... View more

Re: Apache Nifi not working with multiple notes in...

by Explorer in Support Questions
‎08-13-2020 07:15 AM
‎08-13-2020 07:15 AM
This issue resolved. External ZK did the trick. I am not able to figure the solution with embedded ZK. But using External ZK I am able to achieve the use case which I was trying.   Below is link where you will get few more details on this thread, https://stackoverflow.com/questions/63339452/apache-nifi-not-working-with-multiple-notes-in-zookeeper-string    Thanks. ... View more

Re: ExecuteScrip Groovy ClassNotFound Exception

by Super Guru in Support Questions
‎08-07-2020 10:14 AM
‎08-07-2020 10:14 AM
Thank you very much! After installing the proper java version the Groovy script started to work. ... View more

How to put Schema Registry on DEBUG for Kerberos J...

by Expert Contributor Expert Contributor in Community Articles
‎07-20-2020 07:41 AM
‎07-20-2020 07:41 AM
To put Kerberos DEBUG logging for HDF managed Schema Registry ( SR ) clusters, do the following depending on your environment: Navigate to Ambari schema registry > configs > advanced > registry-env template Add the below line: export REGISTRY_OPTS="$REGISTRY_OPTS -Dsun.security.krb5.debug=true" Example: Restart SR. You will then be able to view Kerberos DEBUG messages under /var/log/registry (default).      ... View more

How to disable SPNEGO login on HDF (Ambari) NiFi a...

by Expert Contributor Expert Contributor in Community Articles
‎07-15-2020 12:18 PM
‎07-15-2020 12:18 PM
When NiFi is secured for TLS server authentication, at UI login time first it tries to use TLS certificates if loaded on the browser and then, it tries to use SPNEGO authentication, and finally, it falls back to your configured login provider. If you KERBERISE the cluster via AMBARI and want to use login-providers like LDAP or KERBEROS, it automatically sets the following properties which enable SPNEGO authentication. nifi.kerberos.spnego.keytab.location nifi.kerberos.spnego.principal    Furthermore, SPNEGO properties through AMBARI are greyed out for: COMMAND: From your Amabari manager host, change the setting for NiFi, where the text in red is tailored to your unique environment: nifi.kerberos.spnego.keytab.location to be blank: ./configs.py -a set -s http -l c2288-node1.squadron.support.hortonworks.com -t 8080 -n c2288 -u admin -p AdminPassword -c nifi-properties -k 'nifi.kerberos.spnego.keytab.location' -v '' nifi.kerberos.spnego.principal to be blank ./configs.py -a set -s http -l c2288-node1.squadron.support.hortonworks.com -t 8080 -n c2288 -u admin -p AdminPassword -c nifi-properties -k 'nifi.kerberos.spnego.principal' -v '' -a set -s http or https -l fqdn of ambari host -t port number ambari is listening on -n ambari cluster name ( you can get that from top right UI ) -u user that has edit privileges on ambari -p the password for that user -c the config type in this case nifi.properties -k the key to change -v the value to change You can also do this for NiFi Registry with the following sample commands: ./configs.py -a set -s http -l c2288-node1.squadron.support.hortonworks.com -t 8080 -n c2288 -u admin -p AdminPassword -c nifi-registry-properties -k 'nifi.registry.kerberos.spnego.principal' -v '' ./configs.py -a set -s http -l c2288-node1.squadron.support.hortonworks.com -t 8080 -n c2288 -u admin -p AdminPassword -c nifi-registry-properties -k 'nifi.registry.kerberos.spnego.principal' -v '' Restart NiFi and/or NiFi Registry and ensure that you clear your browser cache. You should see the following on Ambari config sections of NiFi and/or NiFi registry:   ... View more

Re: Connecting to Phoenix on kerberized cluster fr...

by Explorer in Support Questions
‎05-07-2020 06:35 AM
‎05-07-2020 06:35 AM
Hi,   I did restart nifi , but the problem still persists.   The next release of nifi will add a new DBCP connection Pool: Hadoop DBCP connection pool: https://issues.apache.org/jira/browse/NIFI-7257 It will, hopefully, solve the issue.   For my part, I implemented a specific connector which modifies the classpath: https://github.com/dams666/nifi-dbcp-connectionpool   In short:   public static final PropertyDescriptor DB_DRIVER_LOCATION = new PropertyDescriptor.Builder() .name("database-driver-locations") .displayName("Database Driver Location(s)") .description("Comma-separated list of files/folders and/or URLs containing the driver JAR and its dependencies (if any). For example '/var/tmp/mariadb-java-client-1.1.7.jar'") .defaultValue(null) .required(false) .addValidator(StandardValidators.createListValidator(true, true, StandardValidators.createURLorFileValidator())) .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY) .dynamicallyModifiesClasspath(true) .build(); ... dataSource = new BasicDataSource(); dataSource.setDriverClassName(drv); dataSource.setDriverClassLoader(this.getClass().getClassLoader());   But I still got same error message:   PutSQL[id=94d192a9-fd1d-3c59-99be-d848f8902968] Failed to process session due to java.sql.SQLException: Cannot create PoolableConnectionFactory (ERROR 103 (08004): Unable to establish connection.): org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Cannot create PoolableConnectionFactory (ERROR 103 (08004): Unable to establish connection.)   My setup:   Database Connection URL : jdbc:phoenix:zk4-xxx.ax.internal.cloudapp.net,zk5-xxx.ax.internal.cloudapp.net,zk6-xxx.ax.internal.cloudapp.net:2181:/hbase-unsecure   Database Driver Class Name: org.apache.phoenix.jdbc.PhoenixDriver   Damien ... View more
Contact Me
Online
Offline
Last Visited
‎11-25-2024 09:53 AM
Community Statistics
Member Since ‎09-04-2019 01:31 PM
Last Visited ‎11-25-2024 09:53 AM
Posts 62
Kudos received 17