Solved

Hi guys,I have a Data lake (Hive Managed tables base) and I would like to do an Incremental approach to the Warehouse (Hive Managed tables base) using spark v3.2, and I faced an issue connection to Hive Managed tables with spark3. And I would like to know,How to connect to Hive ACID Tables? Using JDBC if yes how? Or there are other ways? Thank you!

Hey guys, i've been a quite reader in this forums last months and you all provide me a lot of hints and solutions for multiple questions. Now i've search the documentation and this community, but i'm the first one with this problem i think. We store all our remote-server ips in a database, the login is the same for all. We want to fetch files (names changes possible) from remote servers and download them.The only (built in) sftp processor with input connection is "GetSFTP", but i have to know the filename to perform a download. i cannot guarantee that the filenames will never change, so this is no way to solve my problem ... I know, i can create an processor on my own with some development, but my first step is to get it work with built-in ones.Does anybody got a good idea how to solve it or have i to create a processor on my own? Greetings from Dortmund  

How to check if one node is connected to another node?

When I tried to run simple commands os Zeppelin like this:%livy.pyspark sc.version I'm getting this error:Caused by: org.apache.zeppelin.livy.LivyException: org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://SERVER:8998/version": PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target We changed the Livy Server instance to other hosts. How can I solve this?

Hi,Query 1: How to connect to two different source DB instants using captureChangeMySql processor?.Query 2: is it possible to configure two different source DB instants using captureChangeMySql processor to get data from both db instance and put it in destination DB?, If yes, Please suggest the configuration approach.Below is my current configuration screenshot to read the data from single instant connecting two source DB. 

I have a table with an ID, key and 2 columns with each contains pipe separated values. How can i parse column into multiple rows with the respective values? Table that I have: IDkeyitemnovalue1A1|2|310|20|301B1|2|340|50|601C1|2|370|80|90 Expected output: IDkeyitemnovalue1A1101A2201A3301B1401B2501B3601C1701C2801C390 Really appreciate the help! Thanks

Hello Experts, We have HandleHttpRequests processor running with 16 concurrency.Container queue size configured is 1000.While running a load test suddenly HandleHttpRequests stopped emitting flowfiles recieved and started throwing error show in below image - So we stopped the load test and now even after 30 mins when we send requests it will just emit 2 or 3 flow files and again shows same error and does not release/emit any request flowfiles.Ideally when no request is getting hit to that port (which HttpRequestListener is configured) container should get cleared and flow files should get created from queue and emit right?What could be the reason this processor is just trhowing container full error and not emitting any flow files ?It started working only when we manually cleanup all repositories (content, provenance, ff) and restart the nifi !Any suggestion would be appreciated. Thanks,Mahendra

Hello,We would like to create a Hive table in the ussign pyspark dataframe cluster.We have the script below, which has run well several times in the past on the same cluster. After some configuration changes in the cluster, the same script is showing the error below.We were unable to identify what changes are made to the cluster to trigger this error in this script (we rearange some services at cluster, etc) The simple script is:# pyspark --master=yarndata = [("Java", "20000"), ("Python", "100000"), ("Scala", "3000")]rdd = spark.sparkContext.parallelize(data)dfFromRDD1 = rdd.toDF(columns)dfFromRDD1.printSchema()dfFromRDD1.show()from pyspark.sql import SQLContextfrom pyspark.sql import HiveContextsqlContext = HiveContext(sc)dfFromRDD1.registerTempTable("evento_temp")sqlContext.sql("use default").show() ERROR:Hive Session ID = bd9c459e-1ec8-483e-9543-c1527b33feec22/07/30 13:55:45 WARN metastore.PersistenceManagerProvider: datanucleus.autoStartMechanismMode is set to unsupported

Does anyone know the process for setting up URL aliases in NiFi registry when using Cloudera Manager to manage the instance?  The property: nifi.registry.registry.alias.configuration.file is not available in the UI.  And while there are options to ignore properties, there is nothing I can find to add arbitrary properties to the configuration. In addition to the above, you also need to add a registry-aliases.xml file to the server and I see no way to accomplish that.  Ideally this would all be auto-configured as the registry connection URL information is required in this latter file.  And the path to these configuration files is managed by Cloudera Manager. For reference, this is the feature of NiFi Registry I am attempting to make use of:https://nifi.apache.org/docs/nifi-registry-docs/html/administration-guide.html#url-aliasing  Thanks for any advice anyone can give.

Hello!I've read this and this and this but explanations are pretty scarce. I understand what "&" means. But I can't wrap my head around what [&] means =\Can someone please explain me what this [&] does applicable to this example? What exactly happening in lines? : "@1": "outer[&4].inner[&2].t", "@(3,eventTypeCode)": "outer[&4].inner[&2].etc" JSON:{ "uID": 1000358546, "events": [{ "eventTypeCode": "FEEDBACK", "transports": ["PUSH", "SMS"] }, { "eventTypeCode": "MARKETING", "transports": ["PUSH", "EMAIL"] }, { "eventTypeCode": "ORDER_STATUS", "transports": ["SOC_VK"] } ]} SPEC:[{ "operation": "shift", "spec": { "events": { "*": { "transports": { "*": { "*": { "@1": "outer[&4].inner[&2].t", "@(3,eventTypeCode)": "outer[&4].inne

Hello all, I am trying to run spark job using spark-submit with a docker image over yarn.  I followed the instructions in the Blog provided by cloudera in the following link:https://blog.cloudera.com/introducing-apache-spark-on-docker-on-top-of-apache-yarn-with-cdp-datacenter-release/ and I ran into an error that I couldn't fine an answer to.Note: I already did all the configurations required from the post. I ran this command :spark-submit \--master yarn \--deploy-mode cluster \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=faresdev8/python3:v5 \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/data1/opt/cloudera/parcels/:/data1/opt/cloudera/parcels/:ro" \--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \--conf spark.executorEnv.YARN_CONTAINER_

Hi,I have install a NiFi setup and deign the flow file to move data from source mysql db to destination mysql db.Now I want to know how can I setup it in dev or QA environments. I was going to NiFi admin guide and came to know about NiFi Registry and NiFi toolkit feature.I have below doubts:1. Is it mandatory  to install NiFi toolkit along with NiFi to configure clustering and managing the cluster node? or without installing a NiFi toolkit, we can configure cluster and manage it? 2.  Is it mandatory  to install NiFi Registry along with NiFi to manage the flowfile change restriction? 

ExampleAvailability : four/five 9’s uptimeDurability : 10 9's Reliability :  15 9's Can anyone  pls clarify on this

Is there a simple way to restrict number of threads to a NIFI process group?. 

When I try to start a spark-shell with YARN I'm getting this error: 22/07/29 15:36:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable22/07/29 15:36:08 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.22/07/29 15:36:09 WARN security.HBaseDelegationTokenProvider: Fail to invoke HBaseConfigurationjava.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfigurationat scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)at org.apache.spark.deploy.security.HBaseDelegationTokenProvider.hbaseConf(HBaseDelegationTokenProvider.scala:117)at org.apache.spark.deploy.security.HBaseDelegationTokenProvider.delegationTokensRequired(HBaseDelegationTokenProvider.s

Hi, I am trying to add flow attribute from my json file which looks like:[ {"name" : "id","value" : "29879"}, {"name" : "file","value" : ""}, {"name" : "barcode","value" : ""}, {"name" : "roll no","value" : "3032"}, {"name" : "name","value" : "Daniel"}] Now I want to add roll no. as a flow atrribute using   Can someone please help me with it? Thanks

How do I get a CDP Private Cloud Base license?

Hello, I would like to know if this CVE which impacts Apache Hadoop is already resolve into HDP or CDP products ? In Apache Hadoop 2.2.0 to 2.10.1, 3.0.0-alpha1 to 3.1.4, 3.2.0 to 3.2.2, and 3.3.0 to 3.3.1, a user who can escalate to yarn user can possibly run arbitrary commands as root user. Users should upgrade to Apache Hadoop 2.10.2, 3.2.3, 3.3.2 or higher. https://nvd.nist.gov/vuln/detail/CVE-2021-33036Thanks in advance for your help.

We are getting this below errors all the time in Hbase  Could some give me the suggestions ?

Hi,I am getting error "Record does not have a value for the PrimaryKey column 'HOST' " when I update a data in source db table. This issue is happening when destination DB table's name: 'user' or 'users'. It is working fine when destination table's name is change to 'report_user'(I mean when we change the table name 'user'or 'users' to something else which is not used as mysql internal  table) I am not sure why it is trying to insert or update on internal mysql table as I am giving the destination DBName and tableName.Can anyone suggest how can we fix this issue without changing the destination table name?  Error Log:PutDatabaseRecord[id=ff8ca68c-8252-3652-8f55-1044ad3f1bab] Failed to put Records to database for StandardFlowFileRecord[uuid=43e64992-a3d0-4a75-bb85-8d70c1f556d7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1658747398462-4, container=default, section=4], offset=20549, length=1214],offset=0,name=43e64992-a3d0-4a75-bb85-8d70c1f556d7,

Hi folks, Can I use the below script in Ozzie workflow? WITH order_data (order_date, product_name, amount)AS (SELECT  order_date,  product_name,  SUM(i.quantity * i.list_price * (1 - discount))FROM sales.orders oINNER JOIN sales.order_items i  ON i.order_id = o.order_idINNER JOIN production.products p  ON p.product_id = i.product_idGROUP BY order_date,         product_name) SELECT * FROM order_data;

Hello,Please kindly help to resolve this error.I am trying to learn how to create a Nifi custom processor following this tutorial:https://help.syncfusion.com/data-integration/how-to/create-a-custom-processor#_Project_Creation%22%22 But i don't know the class to import to solve the below error: Below are all the imports i have on the fileimport org.apache.nifi.components.PropertyDescriptor;import org.apache.nifi.flowfile.FlowFile;import org.apache.nifi.annotation.behavior.ReadsAttribute;import org.apache.nifi.annotation.behavior.ReadsAttributes;import org.apache.nifi.annotation.behavior.WritesAttribute;import org.apache.nifi.annotation.behavior.WritesAttributes;import org.apache.nifi.annotation.lifecycle.OnScheduled;import org.apache.nifi.annotation.documentation.CapabilityDescription;import org.apache.nifi.annotation.documentation.SeeAlso;import org.apache.nifi.annotation.documentation.Tags;import org.apache.nifi.processor.AbstractProcessor;import org.apache.nifi.processor.Process

Hi All, Wish you a healthy life!!I am trying to load a DWH server (mysql) with incoming data from 100 mysql databases using NiFi.I am using CaptureChangeMySQL (clients) of NiFi to capture changes and applying them incrementally. I am successful in my PoC which loads a single DWH table from 2 different databases.(DB1.table1 -> DWH.table and DB2.table1->DWH.table) Now, in order to do this in realtime, I have 100 different databases, all these databases have their own table which needs to be loaded into a single table at DWH.Do I need to create 100 different processors and configure each to connect to different database ?or is there an alternative way to deal with 100 databases ?If there is an alternative can someone help me understanding the solution and a sample xml file.. much appreciated..Also I am looking for your suggestions on how to do the same CDC for 15 other tables of 100 databases..

Experts,I am continuously seeing this log in Nifi app log - 2022-07-25 06:12:40,788 INFO [Timer-Driven Process Thread-219] o.a.n.c.repository.FileSystemRepository Unable to write to container default due to archive file size constraints; waiting for archive cleanup Any idea what is the reason and will there be any impact of this ?ThanksMahendra

Hi, We are using spark.sql.sources.partitionOverwriteMode=dynamic" in our pyspark scripts in our CDH 6.3.2 cluster with spark version 2.4.0, but when we are trying it to CDP 7.1.4 with Spark 2.4.0 version and it is not working, is there anyway to have the config spark.sql.sources.partitionOverwriteMode=dynamic" work in CDP? is there any alternatives on it? Just to highlight that both our CDP 7.1.4 and CDH 6.3.2 clusters are having the same Spark version of 2.4.0