Q&A

Hi Team,Below are the basic steps that I am following to upgrade Nifi from version 1.23.2 to version 2.3.0 on my RHEL 9 server:unzip nifi newer version: unzip nifi-2.3.0-bin.zipcopying flow files from older to newer version: cp nifi-1.23.2/conf/flow.* nifi-2.3.0/conf/copying state directory from older to newer version: cp -r nifi-1.23.2/state nifi-2.3.0/state/update nifi properties for sensitive prop key: nifi.sensitive.props.key=<<same as older version>>start nifi: sh nifi-2.3.0/bin/nifi.sh startProblem Statement:I am getting the Nifi UI console for newer verion (Nifi-2.3.0). All the processGroups Or Processors those were in older verion copied to new version as well. But few processors (like joltTransformation, publishKafka and etc), it still showing that those are using 1.23.2 nar file.Troubleshooting steps, but did not work:Though I have deleted the work directory and revalidated that lib is not using any nar file of older version.Delete work directory:&n

Hello All,I am new to Apache Nifi where we have a requirement to convert a user input date into Nifi date format and use it in an sql query running inside an executeSQL processor.I am facing challenge while converting the user input date, Any help would be really appreciated.Below are my flowfilesGenerateFlowfile with a Custom Date properties with value as "2023-01-15 10:30:00"UpdateAttribute using the Custom Date as End_date with value as ${'Custom Date':plus(864000000):format('yyyyMMddHHmmss')}ExecuteSQL using End_date in Oracle query as " TO_DATE ('${End_Date}', 'YYYYMMDDHH24MISS')"Also is there any better way to handle this. 

Hi Team,Could you please provide detailed steps on how to connect to my Cloudera trial account from the command line on a Mac?The host address is: coordinator-jwemca0ltng5.dw-jwemca0ltng5.t7gcn3.cloudera.siteThank you, Venkat

I have deployed a single node cluster in my lab.  It consists of two virtual machines implemented in VMware: one manager and one host.Trying to Add Cloudera Management Service to address warning “Unable to issue query: could not connect to host monitor”.This is a test of my database:    [root@cloudera-manager ~]# hostname -A    cloudera-manager.interop.local cloudera-manager    [root@cloudera-manager ~]# psql -d scmdb -U scm -W    Password for user scm:    psql (10.23)    Type "help" for help.I entered the following in the Setup Database  GUI:    Type: PostgreSQL    Database Hostname: cloudera-manager.interop.local    Database Name: scmdb    Usename: scm    Password: xxxxxxxSelecting Test Connection fails.Any ideas?Dalier.

Hi! I am testing the integration between Cloudera Manager and HSM .For that purpose I have deployed a singe node cluster in my lab.  It consists of two virtual machines implemented in VMware: one manager and one host.Getting and error trying to complete Installing Cloudera Navigator Key HSM.Here are the steps I took: 1. mkdir -p /var/www/html/cloudera-repos/cm72. wget https://xxxxxx:xxxxxx@archive.cloudera.com/p/cm7/7.13.1.0/repo-as-tarball/cm7.13.1.0-redhat8.tar.gz -P /var/www/html/cloudera-repos/cm7/3. tar xvfz /var/www/html/cloudera-repos/cm7/cm7.13.1.0-redhat8.tar.gz -C /var/www/html/cloudera-repos/cm7 --strip-components=14. sudo chmod -R ugo+rX /var/www/html/cloudera-repos/cm75. In another ssh session, start a Python http.server in the /var/www/html directory. 6. Through vSphere launch a WEB console.7. Launch the browser and go to https://localhost:8900/cloudera-repos/ and verify the extracted files are present. Checked OK.8. cat /

Hi everyone,I'm a very new user and am currently using Nifi to design a system that will automatically download files from an SFTP server. I will be utilizing "Tracking Entities" as the listing strategy.However, sometimes I might need to re-download files from the server, so I need a method to clear/reset the state of the ListSFTP processor. All of the replies I looked at said to right-click, select "View State" and click on the "Clear State" link.But there's no such link, at least for 2.5.0 on my machine: I have the "filter" text box, a list where I can sort things by "key" or "value" and the "close" button. I'm at a loss as to how to do this.Another post suggested that I delete the directories storing the data used by ListSFTP, but a) I feel that that is not a very elegant solution, and b) can someone point out to me which directories are these?All help would be greatly appreciated.

Hello,I installed NiFI 2.4.0. I added LDAP accounts and made the necessary settings for users to log in. However, users receive an “Insufficient Permissions” error after the login step. I granted permissions to users from the policy tab on the UI screen. What other settings do I need to make?Here are the configs I have setup for my authorizers.xml:Spoiler (Highlight to read)<userGroupProvider><identifier>file-user-group-provider</identifier><class>org.apache.nifi.authorization.FileUserGroupProvider</class><property name="Users File">./conf/users.xml</property></userGroupProvider><userGroupProvider><identifier>ldap-user-group-provider</identifier><class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class><property name="Authentication Strategy">SIMPLE</property><property name="Manager DN">LDAP_USER</property><property name="Manager Password">Password1</property><pro

Hi Expert,I've the incoming flowfile have a attribute somethings like cdc = 12. And I'd like to list file in the folder looks like /home/user/cdc=${cdc}.Because the listfile not required an incomming flowfile. So How can I achieve this?Thank you so much.

I cloned a cdp 7.3.1-1 one node that is kerb enabled, to connect to a kdc server in a different domain.During the kerberos wizard, I choose "Administration" | "security" | "Kerberos Credentials" | "Import KDC Account Manager Credentials", and put in valid username and password, then I get "kinit: Preauthentication failed while getting initial credentials"the same user name / pwd pair, I can do kinit xx@TEST.LAN, and get valid results from klist. So I know the credential is good. I have validated on the AD server, use delegation wizard to give add, modify delete rights to the user xx.If I run Generate Missing Credentials, I see "SIMPLE_PWD_STR='-x -D cloudera@old kdc server -w REDACTED'", and this refers to the old kdc server.So I need to know how to change the setting on ldaps and how I can solve the issue with kinit: Preauthentication failed while getting initial credentials

Dears, Trying to sign up for the free trial to try and test the dataflow, unfortunately, I got this error: 403There was a problem authenticating your request.Account '001Hr00001myQliIAE' not foundRequest ID: 8b78e3a6-ff83-4bdb-a193-9af3a2c35c22Contact supportCheck service status How to solve this?BR,Mohammad 

Hi there,I am using a NiFi flow to create an SQL statement dynamically and execute it with a PutHive3QL processor to delete records from a Hive table that supports ACID transactions. PutHive3Streaming works correctly to write new records to the table and apparently the table is working fine and there are no credentials issues.The error I am getting with PutHive3QL is puzzling me and I really do not understand what is going on. Any help is appreciated! 11:32:04 CESTERROR5b0a3ca0-21d2-16cf-a426-a2b4518fb710All NodesPutHive3QL[id=5b0a3ca0-21d2-16cf-a426-a2b4518fb710] Failed to update Hive for FlowFile[filename=f77e9d70-415b-4037-9b35-5376192d6d8c] due to Error while compiling statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1747050143707_0616_1_00, diagnostics=[Vertex vertex_1747050143707_0616_1_00 [Map 1] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Inp

Unable to start dfsrouter daemon. Cluster is kerberized and using hadoop 3.3.4. Trying to implement hdfs router based federation. Configured hdfs-rbf-site.xml with required properties.It is a working cluster. Zookeeper quorum is already configured in core-site.xml.Below is the error from log. "Zookeeper connection string cannot be null".ERROR router.FederationUtil - Could not instantiate: ZKDelegationTokenSecretManagerImpljava.lang.reflect.InvocationTargetExceptionat sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:423)at org.apache.hadoop.hdfs.server.federation.router.FederationUtil.newInstance(FederationUtil.java:164)at org.apache.hadoop.hdfs.server.federation.router.FederationUtil.newSecretManager(Federation

Just add a new node into the current cluster, and now I'm running the Balancer command to balance disk space into the new node.But the Balancer runs for so long, we're still waiting for it to complete before doing anything. We fear that reading/writing to HDFS while the Balancer is running will cause corrupted data files.Should we wait, or just work like normal

Hi everyone,I am having an issue where the aggregated logs are not being generated/saved for Spark Jobs.There is the below error but I don't know what can be causing this.The user has permissions on Ranger for ADLS.Failed to setup application log directory for application_1751535521169_0953Failed to acquire a SAS token for get-status on /oplogs/yarn-app-logs/csso_luis.simoes/bucket-logs-ifile/0953/application_1751535521169_0953 due to org.apache.hadoop.security.AccessControlException: org.apache.ranger.raz.intg.RangerRazException: <!doctype html><html lang="en"><head><title>HTTP Status 401 – Unauthorized</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 401 – U

When we are trying to upgrade apache nifi from 2.0.0 m4 to 2.4.0, We are getting this error "/opt/nifi/nifi-current/bin/nifi.sh: 276: exec: 04:10:55,565: not found"The nifi.sh file is taking some time stamp/junk value in its on of the exec command.this is snippet from nifi.sh which is causing issue.if [ "$1" = "run" ]; thenRUN_COMMAND=$(eval "${run_bootstrap_cmd} get-run-command")RUN_COMMAND_STATUS=$?if [ $RUN_COMMAND_STATUS = 0 ]; thenexec $RUN_COMMANDelseecho "Failed to get run command"echo "${RUN_COMMAND}"exit 1fiWe are upgrading nifi in EKS, we can not edit or touch files from official docker image.Can some one help to fix this issue?

Hi..I don't see the Users and Policies items in the Global MenuGoogle Chrome ver99Apache NiFi 1.15.3  

Dear Cloudera team,@Bharati @jagadeesan We are currently using Cloudera 7.11.3, and Spark actions added via the UI are still being treated as Spark2. We would like to use Spark3 by default instead. How can we migrate or configure the system to ensure Spark3 is used for these actions? Best Regards,Modhi

We need help to get token to execute Apache NiFi API which is running on Linux and OAuth authentication.                 The below two steps we have taken.Get token from Microsoft OAuth API call which gives us the token. <Success>Using the token above and we are trying to get token from NiFi Api call which is not success  Example:                Step 1:                                               Token from Azure: curl -X POST https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token \  -H "Content-Type: application/x-www-form-urlencoded" \  -d "grant_type=client_credentia

Hi,I'm attempting to run a simple join query between two tables in an Apache Phoenix database on a Spark cluster. However, when I try to read from the resulting DataFrame after executing the query, I encounter a NullPointerException.Query : (SELECT CONNTEST.EMPLOYEEFILTERJOIN.EMP_ID,CONNTEST.EMPLOYEEFILTERJOIN.NAME,CONNTEST.EMPLOYEEFILTERJOIN.SALARY,CONNTEST.DEPARTMENTFILTERJOIN.DEPT_ID,CONNTEST.DEPARTMENTFILTERJOIN.DETP_NAME FROM CONNTEST.EMPLOYEEFILTERJOIN,CONNTEST.DEPARTMENTFILTERJOIN WHERE CONNTEST.EMPLOYEEFILTERJOIN.EMP_ID=CONNTEST.DEPARTMENTFILTERJOIN.EMP_ID) query_aliasBelow is the code I'm executing,val spark = SparkSession.builder().appName("PhoenixWriteIssueSimulator").master("local[*]").config("spark.sql.catalogImplementation", "in-memory") .getOrCreate()val jdbcProperties = new Properties()jdbcProperties.setProperty("driver","org.apache.phoenix.jdbc.PhoenixDriver")var df = spark.read.jdbc(phoenixJdbcUrl, phoenixSourceQuery, jdbcProperties)var df = spark.read.jdbc(

hiveserver2 is working on ubuntu but not on RHEL7(Redhat)Checked port with lsof -f:10000 and beeline -u jdbc:hive2://localhost:10000in redhat, It is just giving me WARNJNDI lookup classis not available because this JRE does not supportJNDI.JNDI string lookups will not be available, continuing configuration.Ignoring java.lang.ClassNotFoundException: org.apache.logging.log4j.core.lookup.JndiLookupchecked log4j2.properties and it is printing to console. I dont see any other log. 

Hi, I was trying to enable Azure AD based OIDC Authentication for Apache Nifi rest API when accessing from a daemon service. I registered the App in Azure AD and  generated a bearer token   and tried passing it to the nifi rest API, but got the following error "Unauthorized error="invalid_token", error_description="An error occurred while attempting to decode the Jwt: Signed JWT rejected: Another algorithm expected, or no matching key(s) found", error_uri="https://tools.ietf.org/html/rfc6750#section-3.1" The token was generated with the grant type as "client_credentials". Does NIFI OIDC implementation support this flow ? If yes why am I getting the error and how can I solve it? Thanks

I configured the Atlas version 2.2.0 on AWS EMR, and did a basic search without prior importing the data in atlas, but it gives me the invalid expression error (in popup). What could be the possible reasons for that?

In have Atlas setup with Cassandra and ES. In ES entity data gets indexed in janusgraph_vertex_index. I want a way to add relationship data as well, so that I can support queries having filters on both Vertex and Edge attributes. How can I enable this use case using Atlas?

Hello.  I would like to get more information about how to create Lineage in Apache Atlas.  Specifically, what is the right way to create lineage that includes multiple Processes and Data Sets.  Should I just be creating multiple Entities for the Process (using entity/bulk api endpoint)?  This seems to work, but it does create multiple Entities instead of just one complex entity: {"entities": [{"typeName": "mysystem_dataMovement","attributes": {"id": "1305f6c4-f0da-4929-be21-dd0798dc2086","qualifiedName": "1305f6c4-f0da-4929-be21-dd0798dc2086@mysystem_dataegress","displayName": "Data Egress Use Case 1","inputs": [{"uniqueAttributes": {"qualifiedName": "c72375fb-34a5-4a22-895c-0d55435fdf26@mysystem_datasource"},"typeName": "mysystem_datasource"}],"outputs": [{"uniqueAttributes": {"qualifiedName": "b8e4ced9-f3f4-451a-8b24-3fa4d7970824@mysystem_computer"},"typeName": "mysystem_computer"}],"description": "Data Egress from Data Sour

Hi,I choose the wrong company name when I register. How can I change it. It cannot be edited in Profile.Thank you