Solved

All, I have two separate NiFi instance and I want to push provenance events from one instance to another. Source NiFi's hostname is nifi8443 and Target NiFi's hostname is nifi8444.Both of these instances are running over HTTPS. On the "Target NiFi" I set up an input port called Test. On source NiFi, here's what my SiteToSiteProvenanceReportingTask looks like. Here's a couple things I did. I realized that the reporting task on NiFi8443 is essentially acting like an client and NiFi8444 is acting as the server. I added the NiFi8444 server certificate into a truststore.p12 file on NiFi8443 and created the corresponding SSLContextService to point to that Truststore file. I resolved the SSL issue and then I was reading on the target node (ie. NiFi8444) I had to make an identity for the full DN of the source (ie NiFi8443) certificate. So on NiFI8444, I created a new user called this - CN=nifi8443, O=NiFi Server, L=San Francisco, ST=California, C=USThen I added th

I have requirement where I need to append and prepend the attribute with "$$$". how ever, Nifi expression language is appending or prepending with "\$" For example: if attribute named "myattr" value is "This is Test" when I do ${myattr:prepend('$$$'):append('$$$')} I am expecting "$$$This is Test$$$" but the output is "\$This is Test\$" Is there any way to append or prepend the attribute with "$$$"

I have the AES decryption working in a Nodejs script using crypto locally on my computer, with the following parts:Text to decrypt: c6 c7 4b 49 0d cf 5c 20 87 0a e0 cd c4 a7 bf 94 d8Key: 3E 9B 26 FE 46 4F 6D 2D 2F 69 5D 87 8A 07 93 74Initialization Vector (IV): 2d 2c 83 42 00 74 1b 16 20 c0 7d 13 20 00 00 00Correct result: 14 25 79 ed a8 ff a7 00 00 e5 03 00 00 be 03 00 00So I know both key and IV are correct. I am using AES128, CTR and NoPaddingNow, I need to do it in Apache Nifi, and I am trying to use the DecryptContent processor. I am confused about how to make it happen and have tried quite a few things, the latest being:Cipher Algorithm Mode is set to CTRCipher Algorithm Padding is set to NoPaddingKey Specification Format: RAWI have set the content of the incoming flowfile content toc6c74b490dcf5c20870ae0cdc4a7bf94d84E69466949562d2c834200741b1620c07d1320000000(With 4E6946694956 as the NiFiIV delimiter)This results in an error from the DecryptContent processor: "Wrong IV length: m

I would like to know how to obtain the values of total_io_mb, mapreduce_inputBytes, and mapreduce_outputBytes for each application within a certain period of time.I believe that these values are written to the yarn log file. So, I would like to know SME's advice on how to do this, that is, how to configure these values to be written to the log.By Cloudera Manager, we are not able to get but the rate(ex. yarn_application_hdfs_bytes_read_rate). However, alternative ideas are welcome.The purpose is to evaluate the disk IO of the entire CDH cluster, which is made up of several hundred bare metal machines, and to estimate the disk IO required for the next generation of bare metal and/or storage nodes.

I am running into an issue where I am unable to successfully use a PutSQL or ExecuteSQL after returning from a Batch Processor group.The general scenario is this: I need to watch a folder for files that will be created with information to be uploaded to a PostgreSQL database. The files will have multiple lines, each line is to be inserted as a separate record into a temporary/staging table. When all the records from that file have been uploaded, I need to call a stored procedure that will process all those records and update other permanent tables all at once.I came across the concept of using a processor group set with 'Process Group Outbound Policy' set to 'Batch Output'.So, I am able to use GetFile to get each file as it is produced, then pass it into my batch processor group, where it splits the lines into individual flow files, and performs the inserts into my staging table. That all works perfectly.I then use a MergeContent processor to reduce those split files back into a single

Both Nifi and Nifi Registry services are configured to use LDAP, the user authentication in both of them are working just fine. The communication between Nifi and Registry is made by calling IP and Port and I use the same addresses to access their UI and login with my LDAP user.But when I try to start version control in one of the Nifi access groups it can connects to the Nifi Registry but returns "No available buckets" even with the buckets already been created in Nifi Registry.This is the Registry logs when I use the "Start version control" option inside Nifi and it can't find the buckets:  2024-10-17 17:39:54,289 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using X509IdentityProvider2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.x.X509CertificateExtractor No client certificate found in request.2024-10-17 17:39:54,290 DEBUG [NiFi Registry Web Server-15] o.a.n.r.w.s.a.IdentityFilter Attempt

Hi, experts,I need to enrich flowfile with data from database table, but before this table should be filtered based on attribute value. Is it possible to filter data within lookup processor? If not, please advise any solution for my task.Best regards,Alexey  

Hello,We have done this before but now we are trying to make a change on a table column and we are getting the follow error:Error while compiling statement: FAILED: Execution Error, return code 40013 from org.apache.hadoop.hive.ql.ddl.DDLTask. Unable to alter table. Cannot change stats state for a transactional table pro_con.dmcrp001_th_importe_consolidado_contrato without providing the transactional write state for verification (new write ID 327, valid write IDs null; current state null; new state {}The statement that we are executing is:alter table `pro_con`.`dmcrp001_th_importe_consolidado_contrato` change column `fecha_movimiento` `fecha_proceso_dia` timestamp comment 'Fecha de referencia de los datos.'; 

Hello Cloudera Community,I'm encountering an issue with my NiFi deployment where users are getting frequent "Session Expired" messages approximately every 30 minutes. When this happens, they have to refresh their browser, which causes them to lose any unsaved modifications in the process.Setup details:NiFi is integrated with Azure Active Directory .Below are the relevant configurations from my nifi.properties file (please let me know if any more details are needed). nifi.security.user.oidc.additional.scopes=profilenifi.security.user.oidc.claim.identifying.user=upnnifi.security.user.oidc.client.id=********************nifi.security.user.oidc.client.secret=***************nifi.security.user.oidc.connect.timeout=180 secsnifi.security.user.oidc.discovery.url=https://login.microsoftonline.com/************nifi.security.user.oidc.preferred.jwsalgorithm=nifi.security.user.oidc.read.timeout=180 secsnifi.web.war.directory= ./libnifi.web.proxy.host= nifi.**************.comnifi.web.https.p

I am facing below issue in my flow.. flowfile is currently penalized and data cannot be processed at this time .Any idea how to solve this issue .Thanks in advance . 

Hi Folks,This is likely a multipart problem, but I've been trying to get the following sample JSON to distribute the "id" value from the root of the JSON object into a flattened array of objects with the "id" value and the "code" value from each category (or subcategory in the nested structure).  For example, given the following JSON payload: { "id": "1234", "categories": [ { "code": "A", "name": "Alpha", "subcategories": [ { "code": "AA", "name": "Alpha Alpha" }, { "code": "AB", "name": "Alpha Bravo" } ] }, { "code": "B", "name": "Bravo" }, { "code": "C", "name": "Charlie" }, { "code": "D", "name": "Delta" } ]} The Desired Output would be:  [ { "individualId": "1234", "categoryCode": "A", "categoryName": "Alpha" }, { "individualId": "1234", "categoryCode": "AA", "categoryName": "Alpha Alpha" }, {

Hello!The general task is to transform xml to csvFor this aim I created following pipeline:       Source xml content is:     <Email E-mail="AR-BIK@GMAIL.COM"> <Data GRN="3214600023849" Date="2024-07-28"></Data></Email>Parameters of EvaluateXPath are:        ReplaceText parameters:      Attribute grn successfully produced with expected value:    But value "3214600023849" is not translated to csv. Only ; delimiter. Encoding is UTF-8 What am I doing wrong?

Hello,Trying to configure Datanode balancer network bandwidth to 4 GB since we have a good 25 GBPS network dedicated to the cluster. However, when set via Cloudera Manager, I get an alert stating max allowed in 1 GB.Is it something hard coded and cannot be increased beyond 1 GB via CM? OR there is a different way to do that? Thankssnm1523

When using the InvokeHttp process for all 4xx HTTP status codes, we receive the HTTP response in the request flow file attribute 'invokehttp.response.body'. However, the attribute holds only 256 characters, and any response bigger than that causes truncation.I would like to hold the entire response body , even it is beyond 256 chars for 4xx status code.

Hi ,I have NIFI cluster with three nodes .Each node has 16 CPU and 100G ram .Currently, the JVM heap memory on the primary node gradually grows until it reaches full RAM capacity and crashes on OOM. Can you please help me to address this issue .Why does the memory on the primary node keep growing ? Is the GC not supposed to clean the heap ?  bootstrap.conf # Java command to use when running NiFijava=java# Username to use when running NiFi. This value will be ignored on Windows.run.as=nifi# Preserve shell environment while runnning as "run.as" userpreserve.environment=false# Configure where NiFi's lib and conf directories livelib.dir=/opt/nifi/libconf.dir=/opt/nifi/conf# How long to wait after telling NiFi to shutdown before explicitly killing the Processgraceful.shutdown.seconds=20# Disable JSR 199 so that we can use JSP's without running a JDKjava.arg.1=-Dorg.apache.jasper.compiler.disablejsr199=true# JVM memory settingsjava.arg.2=-Xms40gjava.arg.3=-Xmx40g# En

Hi Team, After enabling ranger in Kafka, I am not able to bring up the Kafka broker instance. The error is:[kafka-raft-io-thread]: Encountered fatal fault: Unexpected error in raft IO threadorg.apache.kafka.common.errors.ClusterAuthorizationException: Received cluster authorization error in response InboundResponse(correlationId=0, data=FetchResponseData(throttleTimeMs=0, errorCode=31, sessionId=0, responses=[]), sourceId=155) Please help resolve this. Thanks in advance.

HiIam new to Nifi and trying to get this resolved.we have a S3 bucket with below structure. and using below pattern to get the file recursively.Though this works if the files are there in home directory of the bucket.But it doesn't work for folder or subfolder.Please let me know how to get the files recursively.Note: Also the same structure has to be created in local with PutFile, which I think is possible once i get the files from FetchS3Object.  

Hello Team,can we import/export hive table to s3 location or nfs share ? as per what i searched it is only supported for hdfs location. can you please help me to know if we can do it and if yes how can we achieve it ?

Hello,Could someone please help me with this caseI want to copy data from an Oracle table to another table that has the same schema. I want to count number of record read in and number of records succesfully inserted to the target, and put those counts into a tracking table. My idea is as belowIn which, the left branch I count the records in, the right branch I insert data to target table and then count records.-QueryRecord: count number of record. Output from left and right branch are the same and as below-ExtractText: to extract just the count to record_in and record_out. The output attribute is as below-MergeContent: Merge these 2 attributes before put SQL. Result and output attributes are as below -UpdateAttribute: to escape user-based inputMy query is something like:INSERT INTO TABLE_1 (TABLE_NAME, RECORD_IN, RECORD_OUT) VALUES ('manually_input', ?, ?)I got the error: ORA-00911: invalid character SQL = INSERT INTO TABLE_1 (TABLE_NAME, RECORD_IN, RECORD_OUT) VALUES ('manually_

Hello All,I'm running a Nifi cluster with 3 nodes.  Below are some questions in regards to backup. 1. What's the best way to backup Nifi without stop \ starting?2. Is there a way to backup all the processes running using a backup method in the event I need to restore 1 process I don't have to restore all the processes available?Thanks!

Hi Community,I have recently upgraded our Cloudera Manager from CM 7.6.7 to CM 7.11.3.  After the upgrade, we have not been able to successfully restart many of our services, which include, hue, hdfs, and hbase.We are currently on CDP 7.1.7 SP2. We intend to upgrade the CDP 7.1.7 SP2 to CDP 7.1.7 SP3 after the issues are resolved.As a pre-requisite, we have installed Python3.8 (Installation was via anaconda) and created an alternatives pointing python3.8 to the custom installation location.So, we can type python3.8 --version and get the right result (Python 3.8.18).After the upgrade, we tried to restart a hue service but get the following exception.   [07/Oct/2024 16:58:09 +0100] 10345 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/hue/hue.sh runcpserver[07/Oct/2024 16:58:09 +0100] 10345 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 10620[07/Oct/2024 16:58:10 +0100] 10621 MainThread re

Hi, here is a summary of our blocking situation on our CDH Express 6.3.1 installation:- A crash happened on the storage where all the virtual nodes composing our Cluster insist- After solving the issue on the storage, our Cloudera cluster started again working, but with a critical error on HDFS Service Nodename- In fact, we have High Availability configured on 2 nodes, and exploring the situation we saw that Namenode 2 was active and Namenode 1 was DOWN- But the cluster worked- In an attempt to fix the issue, we tried several times to manually start the Namenode 1 from "HDFS --> Instances --> federation --> Namenode1 --> Start" but it was impossible to start manually- So, we decided to disable High Availability but the new configuration has not completed and actually is in pending status- It's in fact impossible to start HDFS service at all because an error pops up saying that "Nameservice nameservice1 has no SecondaryNameNode or High-Availability partner"It is interesting

hi community cloudera, i need to read a Hive table and write the result to an Oracle tablethis flow needs to be executed every hour to update the data in the Oracle tabledoes anyone have a test flow for this example?

I renamed a Hive partitioned table DB1.Tab1 to DB1.Tab1_old and then moved another table with the same name but from another database DB2.Tab1 to the first database DB1.Tab1, using the same rename command. It looks like the partitioned directories were merged because when I deleted the first renamed one (DB1.Tab1_old), the moved table partitions (DB1.Tab1) were deleted. I would like to understand why this happened. Regards

I have a CML project using a JupyterLab Runtime with Python 3.10 and I want to start a Spark cluster with my CDP Datalake. I'm using the predefined Spark Data Lake Connection in CML which looks like this:```import cml.data_v1 as cmldata# Sample in-code customization of spark configurations#from pyspark import SparkContext#SparkContext.setSystemProperty('spark.executor.cores', '1')#SparkContext.setSystemProperty('spark.executor.memory', '2g')CONNECTION_NAME = "hiaa-dl"conn = cmldata.get_connection(CONNECTION_NAME)spark = conn.get_spark_session()# Sample usage to run query through sparkEXAMPLE_SQL_QUERY = "show databases"spark.sql(EXAMPLE_SQL_QUERY).show()``` When I execute this I get the error:IllegalArgumentException: The value of property spark.app.name must not be nullI'm using the predefined spark-defaults.conf which looks like this:```spark.executor.memory=1gspark.executor.cores=1spark.yarn.access.hadoopFileSystems=abfs://[container]@[storage-account].dfs.core.windows.net```Is