Q&A

I've been testing the phoenixdb library in Python with CDP Operational DataStore cluster. Following the example here:https://community.cloudera.com/t5/Community-Articles/Using-phoenixdb-to-connect-Cloudera-Machine-Learning-to/ta-p/310130I can make a connection and return a single row but the connection is very slow to establish and execute. I'm running phoenixdb from the python interpreter on the COD cluster gateway host. The timestamped python execution is below. Python 3.6.8 (default, Nov 16 2020, 16:55:22)[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux>>> import phoenixdb>>> from datetime import datetime>>> opts = {}>>> opts["authentication"] = "BASIC">>> opts["serialization"] = "PROTOBUF">>> opts["avatica_user"] = "workload-username">>> opts["avatica_password"] = "workload-password">>> jbdc_connection_url = "https://cod--<clusterID>-gateway0.<envID>.<custID>.cloudera.site/cod--<cluste

i am unable to start the resourcemanager, kindly help me to fix 

With NiFi toolkit we can use below command to fetch parameters from parameter provider, apply the parameters to parameter group with the mention of sensitive properties:cli.sh nifi fetch-params --paramProviderId $PCP --applyParameters --sensitiveParamPattern "sens_.*" How can I achieve the same thing using NiFi api. I am planning to use NiFi version 1.21.0.

Hello -   I have installed a 4 node Spark-3.3.1 running on YARN.  The nodes are:  node-master, node1, node2, node3 and node4.  I am using JAVA 8.   Spark is working.   I have configured Spark to use 3rd party PKI certificates and successfully created the keystore and truststore.  I copied the keystore and truststore to the Spark/conf directory.   I added the following to the spark-defaults.conf file:   spark.ssl.enabled                                           true spark.ssl.enabledAlgorithms                          TLS_RSA_WITH_AES_256_GCM_SHA384 spark.ssl.protocol                                            TLSv1.2 spark.ssl.keyPassword      &n

Hello, i'm installing a CDP Private Cloud Base 7.1.7 cluster and i have to enable Auto-TLS feature (Use case 2) as described in the cloudera documentation: https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/security-encrypting-data-in-transit/topics/cm-security-use-case-2.html At the step 5 of the procedure is possibile to use the optional argument --trusted-ca-certs that as described is an optional argument, and if it is given, then ca-certs.pem should point to a PEM-formatted file containing one or more root CA certificates.About this i have two questions: 1. Do have i to add also the RootCA used to sign the Intermediate CA certificate? 2. In general why should I add more trusted root CAs? Is it necessary for external client connections that use a specific RootCA? Thanks.

I am new to Spark and wanted to read/write data to/from HBase tables. I encountered an error while reading from HBase tables.Versions: Spark: 2.4.7; HBase: 1.4.13; Scala: 2.11.12Command: spark-shell --jars /usr/lib/hbase/shc/core/target/shc-core-1.1.3-2.4-s_2.11.jar,/usr/lib/hbase/lib/htrace-core4-4.1.0-incubating.jar,/usr/lib/hbase/hbase-client-2.4.0.jar,/usr/lib/hbase/hbase-common-2.4.0.jar,/usr/lib/hbase/hbase-server-2.4.0.jar,/usr/lib/hbase/hbase-protocol-2.4.0.jar,/usr/lib/hbase/lib/htrace-core4-4.1.0-incubating.jar,/usr/lib/hbase/hbase-shaded-miscellaneous-2.2.1.jar,/usr/lib/hbase/hbase-protocol-shaded-2.4.0.jar Error:java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:232) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:128) at org.apache.spark.sql.execution.datasources.hbase.HBaseConnectionCache$$anonfun$g

I am trying to use Nifi to automatically create external Impala tables if there are new subfolders under a certain directory. For this I have the directory structuredata/<timestamp>/<tables>/*.parquetI am therefore currently trying to first get a list of folders under data. Then I want to query all subfolders in these folders again, so that I can use them as table names in a SQL query that I send to Impala.However, the ListFiles processor only returns files, I am not interested in the file names, I only need the folders.Is there a way to do this with NiFi or is my plan complete nonsense?

Hi All, I am trying to send slack alerts from Nifi whenever there is a failure in flow.I have some 10 different processor flows, I want to use a single putslack for all those flows but the message in slack is dynamic..Example:record failed for processor1record failed for processor2meaning just a variable changes in slack message. and this variable i should send from processor flows.How can putslack processor know from which previous processor it got the request ? But I don't know how to make it possible. I kept a output port that collects all flowfiles of failures. Now i created a putslack processor that connects to this output port.. but i dont see a option on how can i send variables dynamically. Can someone please guide me with suggestions, or correct me if this ideation is wrong.. 

Hi community, Do you known where is the path for the files configuration of zeppelin? I see the path /opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/etc/zeppelin/conf.dist/ is this the correct path? thanks

Existe alguma forma de deixar dois tipos de autenticação no Nifi funcionando ao mesmo tempo? Exemplo: SAML + Autenticação Local (próprio nifi).Ou ele só permite o uso de uma única fonte de autenticação configurada por vez?

I have a cluster of 15 nodes and I'm a bit headache when try to assign nodes. I'm following this Role distribution instruction on Cloudera.But I wonder can I just add a Gateway role to every node for some services? Even if it's name node, master node or whatever?I just want to make sure that everyone can access services on every node in the cluster. I always wonder something like "this is a hive name node, do I need to add Gateway role?, oh that is a hbase master node, do I need to add Gateway role?,....". Now I don't want to think too much.Is there any performance issue, or incompatibility issue if I add Gateway role to every node for every service just for the sake on simplicity?

Hi, Where would I be able to request credentials to download HDP? Thanks. https://archive.cloudera.com/p/ambari/2.x/2.7.3.0/ubuntu18/ambari.list 

I have a cluster of 3 nodes (all brand new with centos7, no java, no MySQL, nothing at all). I'm following this official install guide to install CDH 6.2.0 on the first node (called node1). Everything was fine, but do I need to install everything in the guide the same way for node2 and node3? I mean do I need to run sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-serverand sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm and all other commands in all 3 nodes? The instruction is unclear. I read some article on the internet that said I only need to install "cloudera-manager-agent", "cloudera-manager-daemons" on all nodes, "cloudera-manager-server" only need to install on 1 node or something like that. Which step should I execute on all nodes, and which step I only need to execute on 1 node? Or do I only need to install cdh on 1 node, then I use the Cloudera Manager UI website and add a new node, i

Hello all,I am trying to use QueryRecord processor to filter a json file but i am getting 0kb flowfile on my filter relationship.My input flowfile is:{"data": [{"room": "A","path": "/old/can"},{"room": "B","path": "/old/can"},{"room": "C","path": "/old/can"}]} My filter query:SELECT * FROM FLOWFILE WHERE RPATH_STRING(data, '/room[*]')='${'ip'}'NOTE: ip is an attribute using to compare which class to filter, so if ip = A my output after filtering should be{"data": [{"room": "A","path": "/old/can"}            ]}Thank you all.

I have installed nifi on 3 hosts and zookeeper on one, for now.  In the nifi-app.log, I keep seeing these errors.org.apache.nifi.priority.Rul;esManager Exception occurred when trying to update rules from teh stateManageProvider.  java.io.  Failed to obtain value from zookeeper for component with ID RulesManagerID with the exception code CONNECTIONLOSS.  I'm running zookeeper-3.4.13 until I can get a later release.   In addition, in my zookeeper.out log, These messages keep rolling in the log:Accepted socket connection from /IP/portException causing close of sessions 0x0 Len error 369296128Closed socket connection for client /IP/port (no session established for client)Any help would be greatly appreciated.Cathy  

I need to convert timestamp value coming in flowfile of  cst/cdt  time or anytime to UTC in nifi . But I am unable to convert it is not working. Can anybody help on this

Dears, We have a small issue with excel sheet  data loaded into DB table.We are unable to load the data from excel in nifi if  the field name have space or symbols.ex: In excel field name is (Cost Center) , In DB (Cost Center) got error like column not matched.if we send fileld name like Cost_Center then it is worked fine.So, we request you that is there any possible in Nifi we can change (Cost Center (field name )  in excel  to add(_) in that space to make it work?ex:2 P&L item  in excel field name also  not work out because of & symbol. Could you please suugest any approach for this if possible?

Hi all, I have a CDP 7.1.8 cluster running in my environment. Recently, I would like to try enabling CM server High Availability. I studied the instruction in this link https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/managing-clusters/topics/cm-ha-configure-steps.html. My question is, currently I have tried migrating the CM server to another host and I cannot see my cluster on the new CM server console until I manually restore the cluster. I really doubt that after I enabled CM server HA, when the system failover to the passive CM server, I will not see my cluster as well. Could someone clarify for me please? Thank you very much. 

Hi all,I have several jobs processing daily files.In short, I load zipped files from a S3 directory, extract the files, and reload them into the same directory into a specific folder created by NiFi.The workflow works great, but it looks like some files are not being processed. I checked NiFi for any errors or warnings but it seems like everything is working fine. Has anyone experience a similar problem?If so, how do you fixed it? Thank you

i'm trying to perform Importo= importo/Quantita  for each line item value. but unable get the division even though i've used =divide function. Could you please help meinput:{"FatturaElettronicaBody": {"DatiBeniServizi": {"DettaglioLinee": [{"NumeroLinea": 1,"Quantita": "3.0","PrezzoUnitario": "311.55000000","AliquotaIVA": "22.00","ScontoMaggiorazione": [{"Tipo": "SC","Percentuale": "28.00","Importo": "261.69"}],"Descrizione": "B66450458 SACCA DA GOLF","PrezzoTotale": "672.96000000"},{"NumeroLinea": 2,"Quantita": "1.00000000","PrezzoUnitario": "311.55000000","AliquotaIVA": "22.00","ScontoMaggiorazione": [{"Tipo": "SC","Percentuale": "28.00","Importo": "87.23000000"}],"Descrizione": "B66450458 SACCA DA GOLF","PrezzoTotale": "224.32000000"},{"NumeroLinea": 3,"Quantita": "1.00000000","PrezzoUnitario": "311.55000000","AliquotaIVA": "22.00","ScontoMaggiorazione": [{"Tipo": "SC","Percentuale": "28.00","Importo": "87.23000000"}],"Descrizione": "B66450458 SACCA DA GOLF","PrezzoTotale"

Hello!So I am having a problem with the script I wrote. I am trying to save  content of a flowfiles I am getting, which are excel files.But I am getting an error(had to delete image due to corporate safety)Here's my script:   import java.nio.charset.StandardCharsetsimport org.apache.nifi.controller.ControllerServiceimport groovy.sql.Sqldef ff = session.get()if(!ff)returndef lookup = context.controllerServiceLookup//def dbServiceName = databaseConnectionPoolName.value def dbcpServiceId = lookup.getControllerServiceIdentifiers(ControllerService).find{ cs -> lookup.getControllerServiceName(cs) == 'DBCPConnectionPool_GP_prj_gistek_preprod' }def conn = lookup.getControllerService(dbcpServiceId)?.getConnection()def sql = new Sql(conn)session.read(ff, {inputStream -> def statement = "INSERT INTO publish.templates (excel_template) VALUES (?)" def params = [inputStream] sql.executeInsert(statement, params)} as InputStreamCallback)conn?.close()session.transfer(

I have an input JSON file and I want to convert it to a fixed-width length type file. If the length is less than the required length, trailing spaces should be added to the field.Here is the sample Input JSON: [ { "orderId": "1234567890", "orderName": "Test1" }, { "orderId": "12235", "orderName": "Test2" }, { "orderId": "12236", "orderName": "Test3" }] Here is the desired Output: 1234567890 Test1 12235 Test2 12236 Test3    The condition here is if the length of orderId(in input JSON) is less than 10 then add the required spaces.Can this be done in Nifi?I want to have fixed-width length files.Thanks in advance!

My Nifi Flow starts with capturechangeMysql processor. I am trying to enhance my flows by adding email alerts whenever a processor fails. I understand I can use PutEmail for this purpose. my flow fails most of the time at the capturechangeMysql because of Binlogs or connections or some other random issues. My thinking is to create a link out of failure relationship of this processor to PutEmail, But there is no failure relationship for this processor. (It only has success relation.)In this scenario how can i alert when there is a failure in capturechangeMysql processor ?Please share your thoughts or if i am missing something or other alternatives.

I need to create a flow that rejects csv files if there are invalid records determined using ValidateRecord processor. I would like to just report those invalid records and stop it there and  do not want to process the valid records. Could someone please help with the flow

Hi everyone,We installed ranger user-sync and able to sync all external users from FreeIpa.But this user-sync can only sync users with groups thoses dont exist in ranger.If a user exists in ranger, this user-sync cannot sync its groups.We tried to call Ranger Api /service/xusers/secure/users/%sto add user's group.But  when we restart user-sync manually, the user's group added before is gone.How can we solve this problem ?