Q&A

Hello, I could use some help... I am using nifi in windows and I downloaded Nifi Registry in windows too. Although the official Nifi Registry web sais that Nifi Registry is not supported for windows I have created a Repository in github and it looks like the data flows were updated correctly into my repository. Nevertheless, If I do a git clone of the repository and try to use the new versios nifi is not able to import the flows. I wanted to ask why this error might be happening, and how to solve it. I have thought in two options: 1) Setting up a secure nifi and a secure nifi registry (I have not done that and probably is the cause of the error)2) Setting up nifi registry in docker (or a VM) and try to run nifi registry in linux. (If I have to use this option would you also recommend setting up nifi in docker (or the VM)?) Many thanks in advance for the help. PS: When I shut down Nifi and run it again the connection between nifi and nifi registry fails: "Failed to s

client :C++ client (librdkafka)Used below config securely connect  to  cloudera kafka broker . SASL_SSL/GSSAPI - truststore/keystore  JKS files ssl.truststore.location=< JKS File >ssl.truststore.password ='xxxxxx'C++ client not accepting config as it is specific to  java .Is there any alternate  mechanism available securely connect to cloudera broker and config supported by c++ clients?  

Hi, I want to connect connect two different MySQL database services(mysql 5.7 and Mysql 8.0 ). I am using two  two CaptureChangeMySQL Processors for this. Mysql 5.7 is install in my local machine and CaptureChangeMySQL Processors connecting successfully.  Mysql 8.0 is install on docker container and on host machine port 3310 like (sudo docker run --name mysql-8.0-container -p 127.0.0.1:3310:3306 -e MYSQL_ROOT_PASSWORD=password -d mysql:8.0). CaptureChangeMySQL connecting with mysql 8.0 is giving  error. Can any suggest why i am getting this error and how to resolve it.Note: I am able to connect mysql 8.0 using mysql workbench.  Error:2022-08-03 15:59:58,863 ERROR [Timer-Driven Process Thread-4] o.a.n.c.m.processors.CaptureChangeMySQL CaptureChangeMySQL[id=352daba9-0182-1000-a02a-faf160ab1329] Processing failedorg.apache.nifi.processor.exception.ProcessException: Could not connect binlog client to any of the spec

As per the accepted solution posted in this question, export_snapshot is not supported ( or may not work ) between major versions of Hbase. However, this CDP documentation mentions the supportability of export_snapshot from CDH 5 (Hbase 1.2.0) to CDP 7 (HBase 2.2.3).  We are working on migrating Hbase tables from Hbase 1.2 to Hbase 2 and would like to know if "export_snapshot" work in this scenario. 

Hi guys,I have a Data lake (Hive Managed tables base) and I would like to do an Incremental approach to the Warehouse (Hive Managed tables base) using spark v3.2, and I faced an issue connection to Hive Managed tables with spark3. And I would like to know,How to connect to Hive ACID Tables? Using JDBC if yes how? Or there are other ways? Thank you!

Hey guys, i've been a quite reader in this forums last months and you all provide me a lot of hints and solutions for multiple questions. Now i've search the documentation and this community, but i'm the first one with this problem i think. We store all our remote-server ips in a database, the login is the same for all. We want to fetch files (names changes possible) from remote servers and download them.The only (built in) sftp processor with input connection is "GetSFTP", but i have to know the filename to perform a download. i cannot guarantee that the filenames will never change, so this is no way to solve my problem ... I know, i can create an processor on my own with some development, but my first step is to get it work with built-in ones.Does anybody got a good idea how to solve it or have i to create a processor on my own? Greetings from Dortmund  

Hi all, We have a NiFi 1.15 cluster with five nodes, after running for a while there is a node with below error, I couldn't find the root reason, please help me, thanks. 2021-12-08 19:58:35,803 ERROR [Load-Balance Server Thread-4385] o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer p-edi-nifi-003.gz.cvte.cn/10.21.46.32:55354org.apache.nifi.controller.queue.clustered.server.TransactionAbortedException: Attempted to receive FlowFiles from Peer p-edi-nifi-003 for Connection with ID bf3245df-e6d7-15a9-0000-00005fd59914 but no Connection exists with that ID at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:205) at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:134) at org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadB

Hi All, I have two cases :Case A : i have a kerberised production cluster  i want to copy files from this linux box  to gcp storage ( non kerberised).Currently performing manully : downloading linux files to local system using winscp and uploading to google cloud storageif can be done using Distcp provinding steps will be helpful Case B :  from the same  kerberised cluster want to copy data of HDFS to Google cloud storage( non kerberised) Thanks,Syed.   

Hi, I have a background in traditional data warehousing (SQL, 3NF Inmon, Star schema Kimball). I used to design data models then create it in a SQL RDMS and then populate it with ETL data flows. Paying a particular attention to slowly changing dimensions. With Cloudera, how to build the equivalent of a data warehouse? and which feeding treatments? What are the components to use? Thanks.

How to check if one node is connected to another node?

When I tried to run simple commands os Zeppelin like this:%livy.pyspark sc.version I'm getting this error:Caused by: org.apache.zeppelin.livy.LivyException: org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://SERVER:8998/version": PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target We changed the Livy Server instance to other hosts. How can I solve this?

Hi,Query 1: How to connect to two different source DB instants using captureChangeMySql processor?.Query 2: is it possible to configure two different source DB instants using captureChangeMySql processor to get data from both db instance and put it in destination DB?, If yes, Please suggest the configuration approach.Below is my current configuration screenshot to read the data from single instant connecting two source DB. 

I have a table with an ID, key and 2 columns with each contains pipe separated values. How can i parse column into multiple rows with the respective values? Table that I have: IDkeyitemnovalue1A1|2|310|20|301B1|2|340|50|601C1|2|370|80|90 Expected output: IDkeyitemnovalue1A1101A2201A3301B1401B2501B3601C1701C2801C390 Really appreciate the help! Thanks

Hello Experts, We have HandleHttpRequests processor running with 16 concurrency.Container queue size configured is 1000.While running a load test suddenly HandleHttpRequests stopped emitting flowfiles recieved and started throwing error show in below image - So we stopped the load test and now even after 30 mins when we send requests it will just emit 2 or 3 flow files and again shows same error and does not release/emit any request flowfiles.Ideally when no request is getting hit to that port (which HttpRequestListener is configured) container should get cleared and flow files should get created from queue and emit right?What could be the reason this processor is just trhowing container full error and not emitting any flow files ?It started working only when we manually cleanup all repositories (content, provenance, ff) and restart the nifi !Any suggestion would be appreciated. Thanks,Mahendra

Hello,We would like to create a Hive table in the ussign pyspark dataframe cluster.We have the script below, which has run well several times in the past on the same cluster. After some configuration changes in the cluster, the same script is showing the error below.We were unable to identify what changes are made to the cluster to trigger this error in this script (we rearange some services at cluster, etc) The simple script is:# pyspark --master=yarndata = [("Java", "20000"), ("Python", "100000"), ("Scala", "3000")]rdd = spark.sparkContext.parallelize(data)dfFromRDD1 = rdd.toDF(columns)dfFromRDD1.printSchema()dfFromRDD1.show()from pyspark.sql import SQLContextfrom pyspark.sql import HiveContextsqlContext = HiveContext(sc)dfFromRDD1.registerTempTable("evento_temp")sqlContext.sql("use default").show() ERROR:Hive Session ID = bd9c459e-1ec8-483e-9543-c1527b33feec22/07/30 13:55:45 WARN metastore.PersistenceManagerProvider: datanucleus.autoStartMechanismMode is set to unsupported

Does anyone know the process for setting up URL aliases in NiFi registry when using Cloudera Manager to manage the instance?  The property: nifi.registry.registry.alias.configuration.file is not available in the UI.  And while there are options to ignore properties, there is nothing I can find to add arbitrary properties to the configuration. In addition to the above, you also need to add a registry-aliases.xml file to the server and I see no way to accomplish that.  Ideally this would all be auto-configured as the registry connection URL information is required in this latter file.  And the path to these configuration files is managed by Cloudera Manager. For reference, this is the feature of NiFi Registry I am attempting to make use of:https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#url-aliasing  Thanks for any advice anyone can give.

Hello!I've read this and this and this but explanations are pretty scarce. I understand what "&" means. But I can't wrap my head around what [&] means =\Can someone please explain me what this [&] does applicable to this example? What exactly happening in lines? : "@1": "outer[&4].inner[&2].t", "@(3,eventTypeCode)": "outer[&4].inner[&2].etc" JSON:{ "uID": 1000358546, "events": [{ "eventTypeCode": "FEEDBACK", "transports": ["PUSH", "SMS"] }, { "eventTypeCode": "MARKETING", "transports": ["PUSH", "EMAIL"] }, { "eventTypeCode": "ORDER_STATUS", "transports": ["SOC_VK"] } ]} SPEC:[{ "operation": "shift", "spec": { "events": { "*": { "transports": { "*": { "*": { "@1": "outer[&4].inner[&2].t", "@(3,eventTypeCode)": "outer[&4].inne

Hello all, I am trying to run spark job using spark-submit with a docker image over yarn.  I followed the instructions in the Blog provided by cloudera in the following link:https://blog.cloudera.com/introducing-apache-spark-on-docker-on-top-of-apache-yarn-with-cdp-datacenter-release/ and I ran into an error that I couldn't fine an answer to.Note: I already did all the configurations required from the post. I ran this command :spark-submit \--master yarn \--deploy-mode cluster \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=faresdev8/python3:v5 \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/data1/opt/cloudera/parcels/:/data1/opt/cloudera/parcels/:ro" \--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \--conf spark.executorEnv.YARN_CONTAINER_

Hi,I have install a NiFi setup and deign the flow file to move data from source mysql db to destination mysql db.Now I want to know how can I setup it in dev or QA environments. I was going to NiFi admin guide and came to know about NiFi Registry and NiFi toolkit feature.I have below doubts:1. Is it mandatory  to install NiFi toolkit along with NiFi to configure clustering and managing the cluster node? or without installing a NiFi toolkit, we can configure cluster and manage it? 2.  Is it mandatory  to install NiFi Registry along with NiFi to manage the flowfile change restriction? 

Hi All, I enabled sentry and all services are healthy but unable to create roles and not getting 0: jdbc:hive2://wa1cog3mgmt2200.cdp.centuryli> CREATE ROLE huesuperrole;going to print operations logsprinted operations logsgoing to print operations logsprinted operations logsgoing to print operations logsprinted operations logsgoing to print operations logsINFO : Compiling command(queryId=hive_20220731105224_c56c786a-bfaf-4270-a2a0-e62936b67777): CREATE ROLE huesuperroleINFO : Semantic Analysis CompletedINFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)INFO : Completed compiling command(queryId=hive_20220731105224_c56c786a-bfaf-4270-a2a0-e62936b67777); Time taken: 2.529 secondsINFO : Executing command(queryId=hive_20220731105224_c56c786a-bfaf-4270-a2a0-e62936b67777): CREATE ROLE huesuperroleINFO : Starting task [Stage-0:DDL] in serial modeprinted operations logsgoing to print operations logsprinted operations logsgoing to print operations logsprinted o

Hi I have a flow file with huge string content and I want to create a JSON and add this string to the JSON as a value with a static key. Example Flow file string content: "Example string"Needed output :{"Static key":"Example string"} I tried to add the string content to the flow file attribute using ExtractText and use attribute to JSON processor to generate the needed output but I have 2 issues with this approach:-1- The string content size is not under my control and I dont know how much I need to configure the Maximum Buffer Size and Maximum Capture Group Length.2- Adding the content to the flow file attribute will duplicate the flow file size and this will impact the performance.3- Finally I feel it is a workaround solution :)) Is there any other way to solve this issue? 

ExampleAvailability : four/five 9’s uptimeDurability : 10 9's Reliability :  15 9's Can anyone  pls clarify on this

Is there a simple way to restrict number of threads to a NIFI process group?. 

When I try to start a spark-shell with YARN I'm getting this error: 22/07/29 15:36:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable22/07/29 15:36:08 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.22/07/29 15:36:09 WARN security.HBaseDelegationTokenProvider: Fail to invoke HBaseConfigurationjava.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfigurationat scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)at org.apache.spark.deploy.security.HBaseDelegationTokenProvider.hbaseConf(HBaseDelegationTokenProvider.scala:117)at org.apache.spark.deploy.security.HBaseDelegationTokenProvider.delegationTokensRequired(HBaseDelegationTokenProvider.s

Hi, I am trying to add flow attribute from my json file which looks like:[ {"name" : "id","value" : "29879"}, {"name" : "file","value" : ""}, {"name" : "barcode","value" : ""}, {"name" : "roll no","value" : "3032"}, {"name" : "name","value" : "Daniel"}] Now I want to add roll no. as a flow atrribute using   Can someone please help me with it? Thanks