Q&A

I have a pipeline running on a daily basis, I wish to receive an email alert whenever any of the processor fails. I tried testing put email processor, I am receiving success flow files after PutEmail but email is not being generated despite entering correct SMTP details. Kindly suggest if I am missing on anything. My nifi pipeline flow is as follows :getMongo > flatteJson > evaluateJsonPath > attributeToJson > queryRecord > updateAttribute > PutS3For eg : If I get an error in QueryRecord : column 'x' not found in list of columns. This failure alert should be sent on my email.PFB the properties I have used for configuration of PutEmail processor 

Dears,I install NIFI 1.14 with a secure cluster. When open NIFI GUI, the user access policy and Users list is missing. I try single mode but the result is the same. First screen my new NIFI 1.14 version.Second screen old NIFI 1.11. How to add policy and users on NIFI1.14? NIFI 1.14 new versionNIFI 1.1 old version

I am consuming a messages from IBM MQ using processor (ConsumeJMS), then once you explore as a "List queue" you will see the following type and format:  Hence, I added a following processor (ExtractText 1.16.1) in order to search for some texts using regular expression as following: (ACT.A.EMS..ST...AIR|ACT.X.EMS..ST...AIR) Finally, Sometimes the results of regular expression are valid and sometimes it satisfies a half of my condition because:1) if it does not match anything hence it match only the part of my condition. 2) or if the half of my condition is valid; hence it will match! Is there something wrong I am doing in my regular expression or something else? or I need to change type, format, or content type (application/octet-stream) to text so that I can search correctly instead of viewed as a hex? I hope my explanation was fine and clear for everyone.Thanks!

I am having a HDP hadoop cluster. I took snapshot of phoenix table from this cluster & I cloned that table to a CDP cluster. Hbase table was created from the snapshot & the hfiles are imported to the table's data directory . While selecting the table from phoenix shell, empty data it is showing. If I do a scan operation from hbase shell data is displaying (encoded). Is there any workaround for this issue. Thanks

I had a 3 node NiFi cluster and created a NiFi registry client with this a URL that goes through Knox and looks something like this https://xyz.com/gateway/registry/nifi-registry-app/nifi-registry-api. However, when I restart NiFi the registry client in NiFi strips the URL path so it comes back as https://xyz.com which isn't correct and can't connect to the registry. Has anyone experienced this before and/or solved this issue?

Hi there, any support would be much appreciated. First things first, here is our setup. We are running nifi on aws ec2 in a docker container. It's a two node cluster setup. We are planning to do an in place upgrade by upgrading the docker container and changing the nifi.properties parameters needed to get 1.19 running. changing the port number to 8443using https instead of httpsetting the sensitive properties, was empty in 1.13we are using the web.proxy.host with 3 entries, the hostname of the 2 nodes and the clusternamewe are running anonymously, without ldap. We'll set this up later.We made a backup of the flow.xml.gz file and deleted the original. We are planning to use the nifi-registry to import our flow on the clean canvas. So this works great! We have succesfully upgraded to 1.19. Now what we're trying to figure out is our fallback plan, just in case upgrading doesn't work in the production environment.When reverting all the changes above and deleting the flow.xml.gz,

 Hi, I work with spatial data which is in netCDF format. To manipulate and analyze the data I use the NCO toolkit, which consists of a bunch of commands. I run these in the Windows CMD. Though this works fine, it is somewhat slow and cumbersome. Thus I would like to automate this with NiFi. I figured the ExecuteStreamCommand would do the job. The netCDF data comes in via InvokeHTTP (generates a flowfile with ‘Content Type:application/x-netcdf’), and is subsequently fed to the ExecuteStreamCommand. I have it configured as follows    ‘ncks’ is the executable command from the NCO Toolkit. This is on my path, such that I am sure NiFi can reach it. When I run the flow as follows  I receive errors on the ExecuteStreamCommand processor. These are as follows 2023-04-09 20:46:43,879 ERROR [Thread-187] o.a.n.p.standard.ExecuteStreamCommand ExecuteStreamCommand[id=65610950-0187-1000-7ef5-c5d0d5057e63] Failed to write FlowFile to Standard Input Streamjava.io

I have Installed CDP Private Cloud 7.1.7getting 500 error code while accessing 'http://hostname:50111/templeton/v1/status'  

Hello, I have a scenario where I need to copy the file from HDFS to local file system on edge server. When i run the code in cluster mode with spark submit , I am getting below kind of error:   User class threw exception: java.io.IOException: Mkdirs failed to create/some_specific/path/in_edge_server (exists=false, cwd=file:/some/path/yarn/nm/usercache/1234/appcache/application_123456744/container_t1234)   It seems like it is trying to find the destination path where the execution takes place and since it is not finding the specific path it is trying to create a directory and failing?   Below is the function being used:fs.copyToLocalFile(new Path(src_HDFSPath), new Path(dest_edgePath))   Do we have a solution to copy the file from HDFS to specific path of edge server when we run the program in cluster mode?

When I try to put files to HDFS with Talend, it creates the file but the file is empty I get this error :File /user/hdfs/data/file.xlsx could only be written to 0 of the 1 minReplication nodes. There are 1 datanode(s) running and 1 node(s) are excluded in this operation. I'm really struggling hope someone can help 

Hello, Is there any provision to write the Kafka data in to cloud. This feature is available in Apache and Confluent Kafka. Thanks,Uday

Hello, i'm using the cetic helm chart to deploy nifi to GCP gke cluster. I have configured SSL and OIDC with azure AD, i was able to see the AD page and log in, after that i'm seeing this error  Values:   replicaCount: 1image: repository: apache/nifi tag: "1.16.3" pullPolicy: "IfNotPresent"securityContext: runAsUser: 1000 fsGroup: 1000sts: podManagementPolicy: Parallel AntiAffinity: soft useHostNetwork: null hostPort: null pod: annotations: security.alpha.kubernetes.io/sysctls: net.ipv4.ip_local_port_range=10000 65000 #prometheus.io/scrape: "true" serviceAccount: create: false #name: nifi annotations: {} hostAliases: [] startupProbe: enabled: false failureThreshold: 60 periodSeconds: 10properties: sensitiveKey: changeMechangeMe externalSecure: true isNode: false httpsPort: 8443 httpPort: 8080 webProxyHost: nifi-test.qa.cloud.paycomplete.com clusterPort: 6007 provenanceStorage: "8 GB" siteToSite: port: 100

Hello TeamI have a small development POC HDP Cluster installed on 4 machinesI was using the Public Repositories of Hortonworks/Cloudera that are no longer validhttp://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0/repodata/repomd.xmlhttp://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/repodata/repomd.xmlhttp://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/repodata/repomd.xmlhttp://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/repodata/repomd.xmlQuote"Starting with the HDP 3.1.5 release and Ambari 2.7.5 access to HDP, Ambari repositories requires authentication. Authentication credentials for new customers are sent sent from Cloudera to registered support contacts"To be a registered support contact I suppose one needs a subscription. How do I obtain the subscription for access to the repositories and how much does it cost annually and monthlyapproximately. Are they still available for legacy HDP and Ambari Platforms

Hello friends, I wrote a custom processor and now wanted to add a custom controller. The problem is, that the properties I wrote for the controller do not appear when I create the service. I followed the structure of this controller: https://github.com/Automna/NiFi/blob/master/nifi-rel-nifi-1.1.2/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java  Here is what I did so far:public class MyNewControllerService extends AbstractControllerService implements MyControllerService { public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder() .name("Username") .expressionLanguageSupported(ExpressionLanguageScope.NONE) .displayName("Set username") .required(true) .defaultValue("User") .description("Username to login to graph service") .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)

Hello All, I have configured nifi flow with three node with external zookeepernode1.domain.comnode2.domain.comnode3.domain.com with three zookeeper nodeszk1.domain.comzk2.domain.comzk3.domain.com Everything is working fine but the requirement here is my flow has trigger points from external client systems to whom I have give a common URL likenode.domain.com to avoid any changes on the client teams if any of nodes goes down. Any idea how to configure this.

Hi all, I have HDFS service running on my CDP 7.1.8 private cloud base cluster with Kerberos enabled. Recently, I got two issues with my HDFS NameNode, here is the screen capture:The first one The second one:When looking into the role log, it shows Could anyone point out the root cause and the solution for this issue for me please? Thanks in advance. Please let me know if I need to provide more information.

Tried to add ambari metrics to a cluster and the metrics collector keeps failing on a node due to some issuehere are the logs:stderr:Traceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 90, in <module>AmsCollector().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 51, in startams_service('collector', action = 'start')File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunkreturn fn(*args, **kwargs)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/ams_service.py", line 84, in ams_serviceuser=params.ams_userFile "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__self.env.run()File "/usr/lib/ambari-agent

I have a 3 node NiFi cluster set up and running in our environment.  I used a user called "ec2-user" to perform the NiFi install. For this initial install (in a development environment) I used the NiFi Toolkit to create the certificates for the 3 nodes as well as for the ec2-user   I would now like to use the NiFi registry tool to create additional NiFi users. I have been following this post by alim: Setting Up a Secure Apache NiFi Registry - Cloudera Community - 247753.   I started with "Registry Configuration" in this post since I had already used the Toolkit to generate the keystore, truststore and a client certificate for the ec2-user. I copied the keystore and truststore to the conf directory of the Registry install. I then copied the values rom the keystore and truststore properties from the nifi.properties file into the corresponding values in the nifi-registry.properties file. I also modified the HTTP and HTTPS web properties as indicated in the post. I then

Hello,Does nipyapi package works with windows 10 (64bit)?I have tried to install nipyapi on two machines, one with winpython (manual installation of downloaded packages) another with anaconda (installation with pip).Both failed, obviously due a number of missing obscure dependencies like libxml2. I have only a little knowledge, but the installation looked like a tough stuff under windows 10. Is nipyapi recommended at all or are there better alternatives? Best regardsJaro

Has anyone already configured the processor for twitter?GetTwitter return 404ConsumeTwitter return 403  

Hello,I built a flow to pull data from the source (Oracle) and push it into the data warehouse. (Oracle). The ETL procedure log must be created in a database, for example. FLOW_ID, FLOW_NAME, START_TIME, END_TIME, STATUS, and ERROR_MESSAGE How do I go about doing this? Flow Example:  Log message into database like below     

First attempt:Command Arguments = ${absolute.path:append('*.log')};${absolute.path:append('logfiles')}Command Path = /bin/mvNo working directory setExecution error = /bin/mv: cannot stat â/path/input/*.logâ: No such file or directory So I ran "/bin/mv /path/input/*.log /path/input/logfiles" from putty — no errors, files moved. Second attempt:Based on some online troubleshooting, e.g., https://unix.stackexchange.com/questions/77007/mv-cannot-stat-no-such-file-or-directory-in-shell-scriptSame as first, but first argument changed to "${absolute.path}"*.logExecution error = same as above, except double quotes around first path Third attempt:Command Arguments = *.log;logfilesCommand Path = /bin/mvWorking Directory = ${absolute.path}Execution error = /bin/mv: cannot stat â*.logâ: No such file or directory So in putty, I cd into /path/input and type "/bin/mv *.log logfiles" — no errors, files moved. Many folks online having the same/similar problem say "my bad, I

Hi Team,  Prod master node is not coming up. Getting below error, could you pls tell me how to resolve the issue as the data is very important.  2023-01-23 09:40:50,748 ERROR [master/ctrlsu-hbaseRS1:16000:becomeActiveMaster] master.HMaster: Failed to become active masterjava.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILEDat org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:379)at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:319)at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1324)at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1055)at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2184)at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:519)at

I am getting following error message while trying to set up ODBC. I am using SSL connection to connect. [Cloudera] [ImpalaODBC] (100) Error from the Impala Thrift API: authorize: cannot authorize peer