Find answers, ask questions, and share your expertise
Recently active
I want to connect opensearch with nifi but it is showing error while running processor of sending data. it shows failure of requests . i have also tried different version of nifi with opensearch but still same error . How can i solve it? screenshot of error
I followed this guide on the Cloudera website, I've finished "Installing and Upgrading the Sentry Service" step. Now what do I do next?I tried to start Beeline and execute some queries as normal but get a privilege errorkinit -k -t /home/vgdata/vega.keytab vega@BI.VEGA.COM (get tgt for user vega)beeline -u "jdbc:hive2://data-node01:10000/test;principal=hive/data-node01.vega.com@BI.VEGA.COM" (access beeline with kerberos principal)select * from test; (execute query) But I got an error like belowError: Error while compiling statement: FAILED: SemanticException No valid privilegesUser vega does not have privileges for SWITCHDATABASEThe required privileges: Server=server1->Db=*->Table=+->Column=*...User "vega" is hdfs superuser, and also the main user that we use to connect to all Hadoop services (there is a Kerberos principal "vega" too)I tried to execute some commands likecreate role admin;grant role ....But all get an error like "No groups found for user ve
Hi All,I would like to know what is heap value that I need to set for SNN ?I have a big cluster with ~600 nodes and ~150M blocks so I had set NN heap to be 150GB but what value should I keep for SNN ?Currently its running with 8GB, I feel its low as it needs more MEM to perform checkpointing. Each hosts are running with 750GB of MEM.Please suggest.
I'm fascinated by Apache Airflow, but I can't operate it locally. Is there anywhere I can test it on the cloud? Almost all of the other posts in this sub are about Airflow/Dagster.For example, we have replit to experiment with Python and other programming languages; is there a comparable infrastructure for Airflow/Dagster? Thank You.
When I make some changes to some service configurations, Cloudera asks me to "restart stale service". And there is a window pop-up asking me should I "Redeploy Client Configuration" too like below:The default option is No (not tick), so does this mean this is optional? When should I tick this? Since if I tick this, it will take a long time to restart everything, even I only make a small change. I wonder if is it necessary.
i'm trying to split the array values to seperate list of object values but not able to get it.input:{"FatturaElettronicaBody": {"DatiGenerali": {"DatiGeneraliDocumento": {"TipoDocumento": "TD01","Numero": "126561","Data": "05.05.2023","Divisa": "EUR","ImportoTotaleDocumento": "9382.29"},"DatiDDT": {"NumeroDDT": "086708,086704,086705,086706,086707"}}}} using spec:[{"operation": "shift","spec": {"FatturaElettronicaBody": {"DatiGenerali": {"DatiDDT": {"*": {"NumeroDDT": "FatturaElettronicaBody.DatiGenerali.DatiDDT[&1].NumeroDDT"}}}}}}] but getting result as null. expected output:{"FatturaElettronicaBody": {"DatiGenerali": {"DatiGeneraliDocumento": {"TipoDocumento": "TD01","Numero": "126561","Data": "05.05.2023","Divisa": "EUR","ImportoTotaleDocumento": "9382.29"},"DatiDDT": {"NumeroDDT":{"086708},"NumeroDDT":{086704},"NumeroDDT":{086705},"NumeroDDT":{086706},"NumeroDDT":{086707"}}}}} Regards,Pradeep
I can't connect to HDFS kerberized and SSL enabled though I generated a keytab and certificate, Any help?
I've been testing the phoenixdb library in Python with CDP Operational DataStore cluster. Following the example here:https://community.cloudera.com/t5/Community-Articles/Using-phoenixdb-to-connect-Cloudera-Machine-Learning-to/ta-p/310130I can make a connection and return a single row but the connection is very slow to establish and execute. I'm running phoenixdb from the python interpreter on the COD cluster gateway host. The timestamped python execution is below. Python 3.6.8 (default, Nov 16 2020, 16:55:22)[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux>>> import phoenixdb>>> from datetime import datetime>>> opts = {}>>> opts["authentication"] = "BASIC">>> opts["serialization"] = "PROTOBUF">>> opts["avatica_user"] = "workload-username">>> opts["avatica_password"] = "workload-password">>> jbdc_connection_url = "https://cod--<clusterID>-gateway0.<envID>.<custID>.cloudera.site/cod--<cluste
i am unable to start the resourcemanager, kindly help me to fix
With NiFi toolkit we can use below command to fetch parameters from parameter provider, apply the parameters to parameter group with the mention of sensitive properties:cli.sh nifi fetch-params --paramProviderId $PCP --applyParameters --sensitiveParamPattern "sens_.*" How can I achieve the same thing using NiFi api. I am planning to use NiFi version 1.21.0.
Hello - I have installed a 4 node Spark-3.3.1 running on YARN. The nodes are: node-master, node1, node2, node3 and node4. I am using JAVA 8. Spark is working. I have configured Spark to use 3rd party PKI certificates and successfully created the keystore and truststore. I copied the keystore and truststore to the Spark/conf directory. I added the following to the spark-defaults.conf file: spark.ssl.enabled true spark.ssl.enabledAlgorithms TLS_RSA_WITH_AES_256_GCM_SHA384 spark.ssl.protocol TLSv1.2 spark.ssl.keyPassword &n
Hello, i'm installing a CDP Private Cloud Base 7.1.7 cluster and i have to enable Auto-TLS feature (Use case 2) as described in the cloudera documentation: https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/security-encrypting-data-in-transit/topics/cm-security-use-case-2.html At the step 5 of the procedure is possibile to use the optional argument --trusted-ca-certs that as described is an optional argument, and if it is given, then ca-certs.pem should point to a PEM-formatted file containing one or more root CA certificates.About this i have two questions: 1. Do have i to add also the RootCA used to sign the Intermediate CA certificate? 2. In general why should I add more trusted root CAs? Is it necessary for external client connections that use a specific RootCA? Thanks.
I am new to Spark and wanted to read/write data to/from HBase tables. I encountered an error while reading from HBase tables.Versions: Spark: 2.4.7; HBase: 1.4.13; Scala: 2.11.12Command: spark-shell --jars /usr/lib/hbase/shc/core/target/shc-core-1.1.3-2.4-s_2.11.jar,/usr/lib/hbase/lib/htrace-core4-4.1.0-incubating.jar,/usr/lib/hbase/hbase-client-2.4.0.jar,/usr/lib/hbase/hbase-common-2.4.0.jar,/usr/lib/hbase/hbase-server-2.4.0.jar,/usr/lib/hbase/hbase-protocol-2.4.0.jar,/usr/lib/hbase/lib/htrace-core4-4.1.0-incubating.jar,/usr/lib/hbase/hbase-shaded-miscellaneous-2.2.1.jar,/usr/lib/hbase/hbase-protocol-shaded-2.4.0.jar Error:java.io.IOException: java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:232) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:128) at org.apache.spark.sql.execution.datasources.hbase.HBaseConnectionCache$$anonfun$g
I am trying to use Nifi to automatically create external Impala tables if there are new subfolders under a certain directory. For this I have the directory structuredata/<timestamp>/<tables>/*.parquetI am therefore currently trying to first get a list of folders under data. Then I want to query all subfolders in these folders again, so that I can use them as table names in a SQL query that I send to Impala.However, the ListFiles processor only returns files, I am not interested in the file names, I only need the folders.Is there a way to do this with NiFi or is my plan complete nonsense?
Hi All, I am trying to send slack alerts from Nifi whenever there is a failure in flow.I have some 10 different processor flows, I want to use a single putslack for all those flows but the message in slack is dynamic..Example:record failed for processor1record failed for processor2meaning just a variable changes in slack message. and this variable i should send from processor flows.How can putslack processor know from which previous processor it got the request ? But I don't know how to make it possible. I kept a output port that collects all flowfiles of failures. Now i created a putslack processor that connects to this output port.. but i dont see a option on how can i send variables dynamically. Can someone please guide me with suggestions, or correct me if this ideation is wrong..
Hi community, Do you known where is the path for the files configuration of zeppelin? I see the path /opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/etc/zeppelin/conf.dist/ is this the correct path? thanks
Existe alguma forma de deixar dois tipos de autenticação no Nifi funcionando ao mesmo tempo? Exemplo: SAML + Autenticação Local (próprio nifi).Ou ele só permite o uso de uma única fonte de autenticação configurada por vez?
I have a cluster of 15 nodes and I'm a bit headache when try to assign nodes. I'm following this Role distribution instruction on Cloudera.But I wonder can I just add a Gateway role to every node for some services? Even if it's name node, master node or whatever?I just want to make sure that everyone can access services on every node in the cluster. I always wonder something like "this is a hive name node, do I need to add Gateway role?, oh that is a hbase master node, do I need to add Gateway role?,....". Now I don't want to think too much.Is there any performance issue, or incompatibility issue if I add Gateway role to every node for every service just for the sake on simplicity?
Hi, Where would I be able to request credentials to download HDP? Thanks. https://archive.cloudera.com/p/ambari/2.x/2.7.3.0/ubuntu18/ambari.list
I have a cluster of 3 nodes (all brand new with centos7, no java, no MySQL, nothing at all). I'm following this official install guide to install CDH 6.2.0 on the first node (called node1). Everything was fine, but do I need to install everything in the guide the same way for node2 and node3? I mean do I need to run sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-serverand sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm and all other commands in all 3 nodes? The instruction is unclear. I read some article on the internet that said I only need to install "cloudera-manager-agent", "cloudera-manager-daemons" on all nodes, "cloudera-manager-server" only need to install on 1 node or something like that. Which step should I execute on all nodes, and which step I only need to execute on 1 node? Or do I only need to install cdh on 1 node, then I use the Cloudera Manager UI website and add a new node, i
Hello all,I am trying to use QueryRecord processor to filter a json file but i am getting 0kb flowfile on my filter relationship.My input flowfile is:{"data": [{"room": "A","path": "/old/can"},{"room": "B","path": "/old/can"},{"room": "C","path": "/old/can"}]} My filter query:SELECT * FROM FLOWFILE WHERE RPATH_STRING(data, '/room[*]')='${'ip'}'NOTE: ip is an attribute using to compare which class to filter, so if ip = A my output after filtering should be{"data": [{"room": "A","path": "/old/can"} ]}Thank you all.
I have installed nifi on 3 hosts and zookeeper on one, for now. In the nifi-app.log, I keep seeing these errors.org.apache.nifi.priority.Rul;esManager Exception occurred when trying to update rules from teh stateManageProvider. java.io. Failed to obtain value from zookeeper for component with ID RulesManagerID with the exception code CONNECTIONLOSS. I'm running zookeeper-3.4.13 until I can get a later release. In addition, in my zookeeper.out log, These messages keep rolling in the log:Accepted socket connection from /IP/portException causing close of sessions 0x0 Len error 369296128Closed socket connection for client /IP/port (no session established for client)Any help would be greatly appreciated.Cathy
I need to convert timestamp value coming in flowfile of cst/cdt time or anytime to UTC in nifi . But I am unable to convert it is not working. Can anybody help on this
Dears, We have a small issue with excel sheet data loaded into DB table.We are unable to load the data from excel in nifi if the field name have space or symbols.ex: In excel field name is (Cost Center) , In DB (Cost Center) got error like column not matched.if we send fileld name like Cost_Center then it is worked fine.So, we request you that is there any possible in Nifi we can change (Cost Center (field name ) in excel to add(_) in that space to make it work?ex:2 P&L item in excel field name also not work out because of & symbol. Could you please suugest any approach for this if possible?
Hi all, I have a CDP 7.1.8 cluster running in my environment. Recently, I would like to try enabling CM server High Availability. I studied the instruction in this link https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/managing-clusters/topics/cm-ha-configure-steps.html. My question is, currently I have tried migrating the CM server to another host and I cannot see my cluster on the new CM server console until I manually restore the cluster. I really doubt that after I enabled CM server HA, when the system failover to the passive CM server, I will not see my cluster as well. Could someone clarify for me please? Thank you very much.
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.