Solved

I am trying to update parameter context within Nifi which is allocated to several process groups. We would like to update the value of one parameter within the parameter context. Please provide some sample payload to do to update parameter contexts either PUT or POST method

Hi All,I would like to know what is heap value that I need to set for SNN ?I have a big cluster with ~600 nodes and ~150M blocks so I had set NN heap to be 150GB but what value should I keep for SNN ?Currently its running with 8GB, I feel its low as it needs more MEM to perform checkpointing. Each hosts are running with 750GB of MEM.Please suggest.

I'm fascinated by Apache Airflow, but I can't operate it locally. Is there anywhere I can test it on the cloud? Almost all of the other posts in this sub are about Airflow/Dagster.For example, we have replit to experiment with Python and other programming languages; is there a comparable infrastructure for Airflow/Dagster? Thank You.

When I make some changes to some service configurations, Cloudera asks me to "restart stale service". And there is a window pop-up asking me should I "Redeploy Client Configuration" too like below:The default option is No (not tick), so does this mean this is optional? When should I tick this? Since if I tick this, it will take a long time to restart everything, even I only make a small change. I wonder if is it necessary.

             i'm trying to split the array values to seperate list of object values but not able to get it.input:{"FatturaElettronicaBody": {"DatiGenerali": {"DatiGeneraliDocumento": {"TipoDocumento": "TD01","Numero": "126561","Data": "05.05.2023","Divisa": "EUR","ImportoTotaleDocumento": "9382.29"},"DatiDDT": {"NumeroDDT": "086708,086704,086705,086706,086707"}}}} using spec:[{"operation": "shift","spec": {"FatturaElettronicaBody": {"DatiGenerali": {"DatiDDT": {"*": {"NumeroDDT": "FatturaElettronicaBody.DatiGenerali.DatiDDT[&1].NumeroDDT"}}}}}}] but getting result as null. expected output:{"FatturaElettronicaBody": {"DatiGenerali": {"DatiGeneraliDocumento": {"TipoDocumento": "TD01","Numero": "126561","Data": "05.05.2023","Divisa": "EUR","ImportoTotaleDocumento": "9382.29"},"DatiDDT": {"NumeroDDT":{"086708},"NumeroDDT":{086704},"NumeroDDT":{086705},"NumeroDDT":{086706},"NumeroDDT":{086707"}}}}}    Regards,Pradeep

I can't connect to HDFS kerberized and SSL enabled though I generated a keytab and certificate, Any help?

I've been testing the phoenixdb library in Python with CDP Operational DataStore cluster. Following the example here:https://community.cloudera.com/t5/Community-Articles/Using-phoenixdb-to-connect-Cloudera-Machine-Learning-to/ta-p/310130I can make a connection and return a single row but the connection is very slow to establish and execute. I'm running phoenixdb from the python interpreter on the COD cluster gateway host. The timestamped python execution is below. Python 3.6.8 (default, Nov 16 2020, 16:55:22)[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux>>> import phoenixdb>>> from datetime import datetime>>> opts = {}>>> opts["authentication"] = "BASIC">>> opts["serialization"] = "PROTOBUF">>> opts["avatica_user"] = "workload-username">>> opts["avatica_password"] = "workload-password">>> jbdc_connection_url = "https://cod--<clusterID>-gateway0.<envID>.<custID>.cloudera.site/cod--<cluste

Hello, i'm installing a CDP Private Cloud Base 7.1.7 cluster and i have to enable Auto-TLS feature (Use case 2) as described in the cloudera documentation: https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/security-encrypting-data-in-transit/topics/cm-security-use-case-2.html At the step 5 of the procedure is possibile to use the optional argument --trusted-ca-certs that as described is an optional argument, and if it is given, then ca-certs.pem should point to a PEM-formatted file containing one or more root CA certificates.About this i have two questions: 1. Do have i to add also the RootCA used to sign the Intermediate CA certificate? 2. In general why should I add more trusted root CAs? Is it necessary for external client connections that use a specific RootCA? Thanks.

I am new to Spark and wanted to read/write data to/from HBase tables. I encountered an error while reading from HBase tables.Versions: Spark: 2.4.7; HBase: 1.4.13; Scala: 2.11.12Command: spark-shell --jars /usr/lib/hbase/shc/core/target/shc-core-1.1.3-2.4-s_2.11.jar,/usr/lib/hbase/lib/htrace-core4-4.1.0-incubating.jar,/usr/lib/hbase/hbase-client-2.4.0.jar,/usr/lib/hbase/hbase-common-2.4.0.jar,/usr/lib/hbase/hbase-server-2.4.0.jar,/usr/lib/hbase/hbase-protocol-2.4.0.jar,/usr/lib/hbase/lib/htrace-core4-4.1.0-incubating.jar,/usr/lib/hbase/hbase-shaded-miscellaneous-2.2.1.jar,/usr/lib/hbase/hbase-protocol-shaded-2.4.0.jar Error:java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:232) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:128) at org.apache.spark.sql.execution.datasources.hbase.HBaseConnectionCache$$anonfun$g

I am trying to use Nifi to automatically create external Impala tables if there are new subfolders under a certain directory. For this I have the directory structuredata/<timestamp>/<tables>/*.parquetI am therefore currently trying to first get a list of folders under data. Then I want to query all subfolders in these folders again, so that I can use them as table names in a SQL query that I send to Impala.However, the ListFiles processor only returns files, I am not interested in the file names, I only need the folders.Is there a way to do this with NiFi or is my plan complete nonsense?

Hi All, I am trying to send slack alerts from Nifi whenever there is a failure in flow.I have some 10 different processor flows, I want to use a single putslack for all those flows but the message in slack is dynamic..Example:record failed for processor1record failed for processor2meaning just a variable changes in slack message. and this variable i should send from processor flows.How can putslack processor know from which previous processor it got the request ? But I don't know how to make it possible. I kept a output port that collects all flowfiles of failures. Now i created a putslack processor that connects to this output port.. but i dont see a option on how can i send variables dynamically. Can someone please guide me with suggestions, or correct me if this ideation is wrong.. 

Hi community, Do you known where is the path for the files configuration of zeppelin? I see the path /opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/etc/zeppelin/conf.dist/ is this the correct path? thanks

Existe alguma forma de deixar dois tipos de autenticação no Nifi funcionando ao mesmo tempo? Exemplo: SAML + Autenticação Local (próprio nifi).Ou ele só permite o uso de uma única fonte de autenticação configurada por vez?

I have a cluster of 15 nodes and I'm a bit headache when try to assign nodes. I'm following this Role distribution instruction on Cloudera.But I wonder can I just add a Gateway role to every node for some services? Even if it's name node, master node or whatever?I just want to make sure that everyone can access services on every node in the cluster. I always wonder something like "this is a hive name node, do I need to add Gateway role?, oh that is a hbase master node, do I need to add Gateway role?,....". Now I don't want to think too much.Is there any performance issue, or incompatibility issue if I add Gateway role to every node for every service just for the sake on simplicity?

Hi, Where would I be able to request credentials to download HDP? Thanks. https://archive.cloudera.com/p/ambari/2.x/2.7.3.0/ubuntu18/ambari.list 

I have a cluster of 3 nodes (all brand new with centos7, no java, no MySQL, nothing at all). I'm following this official install guide to install CDH 6.2.0 on the first node (called node1). Everything was fine, but do I need to install everything in the guide the same way for node2 and node3? I mean do I need to run sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-serverand sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm and all other commands in all 3 nodes? The instruction is unclear. I read some article on the internet that said I only need to install "cloudera-manager-agent", "cloudera-manager-daemons" on all nodes, "cloudera-manager-server" only need to install on 1 node or something like that. Which step should I execute on all nodes, and which step I only need to execute on 1 node? Or do I only need to install cdh on 1 node, then I use the Cloudera Manager UI website and add a new node, i

Hello all,I am trying to use QueryRecord processor to filter a json file but i am getting 0kb flowfile on my filter relationship.My input flowfile is:{"data": [{"room": "A","path": "/old/can"},{"room": "B","path": "/old/can"},{"room": "C","path": "/old/can"}]} My filter query:SELECT * FROM FLOWFILE WHERE RPATH_STRING(data, '/room[*]')='${'ip'}'NOTE: ip is an attribute using to compare which class to filter, so if ip = A my output after filtering should be{"data": [{"room": "A","path": "/old/can"}            ]}Thank you all.

i'm trying to perform Importo= importo/Quantita  for each line item value. but unable get the division even though i've used =divide function. Could you please help meinput:{"FatturaElettronicaBody": {"DatiBeniServizi": {"DettaglioLinee": [{"NumeroLinea": 1,"Quantita": "3.0","PrezzoUnitario": "311.55000000","AliquotaIVA": "22.00","ScontoMaggiorazione": [{"Tipo": "SC","Percentuale": "28.00","Importo": "261.69"}],"Descrizione": "B66450458 SACCA DA GOLF","PrezzoTotale": "672.96000000"},{"NumeroLinea": 2,"Quantita": "1.00000000","PrezzoUnitario": "311.55000000","AliquotaIVA": "22.00","ScontoMaggiorazione": [{"Tipo": "SC","Percentuale": "28.00","Importo": "87.23000000"}],"Descrizione": "B66450458 SACCA DA GOLF","PrezzoTotale": "224.32000000"},{"NumeroLinea": 3,"Quantita": "1.00000000","PrezzoUnitario": "311.55000000","AliquotaIVA": "22.00","ScontoMaggiorazione": [{"Tipo": "SC","Percentuale": "28.00","Importo": "87.23000000"}],"Descrizione": "B66450458 SACCA DA GOLF","PrezzoTotale"

Hello!So I am having a problem with the script I wrote. I am trying to save  content of a flowfiles I am getting, which are excel files.But I am getting an error(had to delete image due to corporate safety)Here's my script:   import java.nio.charset.StandardCharsetsimport org.apache.nifi.controller.ControllerServiceimport groovy.sql.Sqldef ff = session.get()if(!ff)returndef lookup = context.controllerServiceLookup//def dbServiceName = databaseConnectionPoolName.value def dbcpServiceId = lookup.getControllerServiceIdentifiers(ControllerService).find{ cs -> lookup.getControllerServiceName(cs) == 'DBCPConnectionPool_GP_prj_gistek_preprod' }def conn = lookup.getControllerService(dbcpServiceId)?.getConnection()def sql = new Sql(conn)session.read(ff, {inputStream -> def statement = "INSERT INTO publish.templates (excel_template) VALUES (?)" def params = [inputStream] sql.executeInsert(statement, params)} as InputStreamCallback)conn?.close()session.transfer(

My Nifi Flow starts with capturechangeMysql processor. I am trying to enhance my flows by adding email alerts whenever a processor fails. I understand I can use PutEmail for this purpose. my flow fails most of the time at the capturechangeMysql because of Binlogs or connections or some other random issues. My thinking is to create a link out of failure relationship of this processor to PutEmail, But there is no failure relationship for this processor. (It only has success relation.)In this scenario how can i alert when there is a failure in capturechangeMysql processor ?Please share your thoughts or if i am missing something or other alternatives.

I need to create a flow that rejects csv files if there are invalid records determined using ValidateRecord processor. I would like to just report those invalid records and stop it there and  do not want to process the valid records. Could someone please help with the flow

has anyone developed data flows that ingest document(csv+blobs) into ECM system like Nuxeo

We are using ListSFTP and FetchSFTP to get the files from a vendor SFTP location. There are no explicit delete commands from our end to delete the file . However the file gets deleted and the vendor is pushing it on us saying we are having some process to delete the files . Will FetchSFTP delete the files after its downloaded from the Vendor SFTP server ?

Hi Jolt Experts,Iam looking to use the Jolt transformation in NIFI and looking for a help.{"header": {"timeStamp": "1681441547144","activityId": "5142917973836294867"},"resource": {"drniId": "261968731599587823","@type": "Order","subtype": "Retail","name": "CCCCC","resourceCharacteristic": [{"propname": "aaa","value": "P_1655442"},{"name": "latest","value": [11,200,300,16,2,3]}]}} To have an output like{"records" : [ {"timeStamp" : "1681441547144","activityId" : "5142917973836294867","drniId" : "261968731599587823","type" : "Order","subtype" : "Retail","name" : "CCCCC","propname" : "aaa","value" : "P_1655442"}, {"timeStamp" : "1681441547144","activityId" : "5142917973836294867","drniId" : "261968731599587823","type" : "Order","subtype" : "Retail","name" : "CCCCC","propname" : "latest","value" : "11,200,300,16,2,3"} ]} Basically copying all the top header and resource into to new by flattening it. Any help appreciated.  

I am running a cluster with tlswhat commands I used to create tls, perhaps something was not indicated in them:.I have crt and key my companyopenssl pkcs12 -export -in /etc/nifi/certs/wibe.t.crt -inkey /etc/nifi/certs/wibe.t.key -out /etc/nifi/certs/pkcs12_file.p12 -name nifi_alias -CAfile /etc/pki/CA.pem -caname root -password pass:qwerty231keytool -importkeystore -deststorepass "keystore_password" -destkeypass "keystore_password" -destkeystore /etc/nifi/certs/nifi_keystore.jks -srckeystore pkcs12_file.p12 -srcstoretype PKCS12 -srcstorepass "pkcs12_password" -alias nifi_alias keytool -import -trustcacerts -alias root -file /etc/pki/CA.pem -noprompt -keystore /etc/nifi/certs/nifi_truststore.jks -storepass "truststore_password"at first I could not collect a quorum, then this problem went away and the cluster chose a leader, work is done with external zookeepers on three serversno errors, only warning2023-05-04 16:06:34,116 WARN [main] o.a.nifi.controller.StandardFlowService Failed