Q&A

Hey Team ,i am using DAS most of the time its very slow , i would like to know how i will find App_Id application_1654575913119_3886895 From hive_20220725051805_76a0f15d-c15f-4c90-b73f-eb8e6bc1c4c1 Hive Queryi have Hive_Query_id want to find App_IdHive_Query_id :- hive_20220725051805_76a0f15d-c15f-4c90-b73f-eb8e6bc1c4c1 App_Id :-application_1654575913119_3886895 ThanksImran Chaush

How do I get a CDP Private Cloud Base license?

Hello, I would like to know if this CVE which impacts Apache Hadoop is already resolve into HDP or CDP products ? In Apache Hadoop 2.2.0 to 2.10.1, 3.0.0-alpha1 to 3.1.4, 3.2.0 to 3.2.2, and 3.3.0 to 3.3.1, a user who can escalate to yarn user can possibly run arbitrary commands as root user. Users should upgrade to Apache Hadoop 2.10.2, 3.2.3, 3.3.2 or higher. https://nvd.nist.gov/vuln/detail/CVE-2021-33036Thanks in advance for your help.

We are getting this below errors all the time in Hbase  Could some give me the suggestions ?

 HDP-3.1.0.0​2022-07-28 00:19:45,289 - Package Manager failed to install packages: No package found for hive-${stack_version}(expected name: hive-3-1-0-0-78)Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 486, in <module> InstallPackages().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 149, in actionexecute raise Fail("Failed to distribute repositories/install packages")resource_management.core.exceptions.Fail: Failed to distribute repositories/install packagesstdout: /var/lib/ambari-agent/data/output-6991.txt2022-07-28 00:18:33,442 - Will install packages for repository version 3.1.0.0-782022-07-28 00:18:33,443 - Repository['HDP-UTILS-1.1.0.22-repo-1'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/ubunt

Hi,I am getting error "Record does not have a value for the PrimaryKey column 'HOST' " when I update a data in source db table. This issue is happening when destination DB table's name: 'user' or 'users'. It is working fine when destination table's name is change to 'report_user'(I mean when we change the table name 'user'or 'users' to something else which is not used as mysql internal  table) I am not sure why it is trying to insert or update on internal mysql table as I am giving the destination DBName and tableName.Can anyone suggest how can we fix this issue without changing the destination table name?  Error Log:PutDatabaseRecord[id=ff8ca68c-8252-3652-8f55-1044ad3f1bab] Failed to put Records to database for StandardFlowFileRecord[uuid=43e64992-a3d0-4a75-bb85-8d70c1f556d7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1658747398462-4, container=default, section=4], offset=20549, length=1214],offset=0,name=43e64992-a3d0-4a75-bb85-8d70c1f556d7,

Hi folks, Can I use the below script in Ozzie workflow? WITH order_data (order_date, product_name, amount)AS (SELECT  order_date,  product_name,  SUM(i.quantity * i.list_price * (1 - discount))FROM sales.orders oINNER JOIN sales.order_items i  ON i.order_id = o.order_idINNER JOIN production.products p  ON p.product_id = i.product_idGROUP BY order_date,         product_name) SELECT * FROM order_data;

Hello,Please kindly help to resolve this error.I am trying to learn how to create a Nifi custom processor following this tutorial:https://help.syncfusion.com/data-integration/how-to/create-a-custom-processor#_Project_Creation%22%22 But i don't know the class to import to solve the below error: Below are all the imports i have on the fileimport org.apache.nifi.components.PropertyDescriptor;import org.apache.nifi.flowfile.FlowFile;import org.apache.nifi.annotation.behavior.ReadsAttribute;import org.apache.nifi.annotation.behavior.ReadsAttributes;import org.apache.nifi.annotation.behavior.WritesAttribute;import org.apache.nifi.annotation.behavior.WritesAttributes;import org.apache.nifi.annotation.lifecycle.OnScheduled;import org.apache.nifi.annotation.documentation.CapabilityDescription;import org.apache.nifi.annotation.documentation.SeeAlso;import org.apache.nifi.annotation.documentation.Tags;import org.apache.nifi.processor.AbstractProcessor;import org.apache.nifi.processor.Process

Hi All, Wish you a healthy life!!I am trying to load a DWH server (mysql) with incoming data from 100 mysql databases using NiFi.I am using CaptureChangeMySQL (clients) of NiFi to capture changes and applying them incrementally. I am successful in my PoC which loads a single DWH table from 2 different databases.(DB1.table1 -> DWH.table and DB2.table1->DWH.table) Now, in order to do this in realtime, I have 100 different databases, all these databases have their own table which needs to be loaded into a single table at DWH.Do I need to create 100 different processors and configure each to connect to different database ?or is there an alternative way to deal with 100 databases ?If there is an alternative can someone help me understanding the solution and a sample xml file.. much appreciated..Also I am looking for your suggestions on how to do the same CDC for 15 other tables of 100 databases..

We have been using CDH version 6.2.1 and OS is RHEL 7.7 and like to upgrade to 7.9. Can you help on this to steps to be followed in the Cloudera distribution 6.3.3?

Hi everyone,I have signed up for the CDP 60-days trial verison because I will work with Cloudera on the job soon and therefore I would like to get familiar with it in advance and do some of these tutorials: https://www.cloudera.com/tutorials.html#?fq=When I go to Cloudera Data Platform I dont see any environments and I cant create/register news ones, because I am not the admin. During the registration process, there was no way to choose any specific user rights. It seems I have confused a few things, as I thought the Public Cloud trial would be some kind of sandbox where I could try out all kind of stuff like connecting to my AWS databases etc., but I don't see how this works if I lack the admin rights. Can anyone tell me if I can create an admin account by myself or if there is any other way to get free temporary access to some sandbox-like environment?Thanks a lot in advance!

Experts,I am continuously seeing this log in Nifi app log - 2022-07-25 06:12:40,788 INFO [Timer-Driven Process Thread-219] o.a.n.c.repository.FileSystemRepository Unable to write to container default due to archive file size constraints; waiting for archive cleanup Any idea what is the reason and will there be any impact of this ?ThanksMahendra

Hi, We are using spark.sql.sources.partitionOverwriteMode=dynamic" in our pyspark scripts in our CDH 6.3.2 cluster with spark version 2.4.0, but when we are trying it to CDP 7.1.4 with Spark 2.4.0 version and it is not working, is there anyway to have the config spark.sql.sources.partitionOverwriteMode=dynamic" work in CDP? is there any alternatives on it? Just to highlight that both our CDP 7.1.4 and CDH 6.3.2 clusters are having the same Spark version of 2.4.0

I'm trying to run a Yarn cluster by this command:spark-submit --deploy-mode cluster --master yarn main.py and i return this message:INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) Server address is not correct but in yarn-site.xml there is the correct FQDN. How can i solve this?

After following this article, I tried to create a flow where email attachments, subject, body, to, from, cc, bcc etc. can be extracted.   However the flow is not proceeding beyond the first processor:ConsumeIMAP is showing 0(0 bytes) in its Out, as shown in screenshot above. 

Hi All, Do we have any tool or utility to mask all the sensitive information (like hostname, ip, etc)  inside the logfiles and Diagnostic bundle before uploading that to Cloudera cases? Also, I can see some of the service logs suffixes with hostname, I need to mask them too. like those below example:- hadoop-cmf-hdfs-NAMENODE-cdp-master.log.out to -> hadoop-cmf-hdfs-NAMENODE-xxx-xxxxx.log.out 

Hi,Recently, in our production environment, there was an abnormal exit of the Hbase Master. In the log, I found that the ZK connection had some problems.According to the sessionid, I found that the sessionid of one of our applications was the same as the sessionid established by the master at the time in question, and then the application side closed the session, and then the master hung. Is this possible? Why does a session with the same ZooKeeper sessionid appear?

Hi, I am very frequently getting the following exception whenever I call the URL seen in the message.  2022-07-06 15:24:56,138 ERROR [Timer-Driven Process Thread-10] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=33351c02-15ee-1ccc-6c00-dda0e16110ca] Routing to Failure due to exception: javax.net.ssl.SSLPeerUnverifiedException: Hostname creatorapp.zoho.com not verified (no certificates): javax.net.ssl.SSLPeerUnverifiedException: Hostname creatorapp.zoho.com not verified (no certificates)javax.net.ssl.SSLPeerUnverifiedException: Hostname creatorapp.zoho.com not verified (no certificates) at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.kt:396) at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.kt:337) at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:209) at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226) at okhttp3.internal.connection.ExchangeFinder.findHealth

after completion of schematool successfullyWhen I tried to run or open hive I got this errorHive Session ID = 2398ab4d-e3ad-4750-b299-0543ea26332eException in thread "main" java.lang.ClassCastException: class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and java.net.URLClassLoader are in module java.base of loader 'bootstrap')at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:413)at org.apache.hadoop.hive.ql.session.SessionState.<init>(SessionState.java:389)at org.apache.hadoop.hive.cli.CliSessionState.<init>(CliSessionState.java:60)at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705)at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)at java.base/java.lang.reflect.Method.invoke(Method.java:577)at org.apache.hadoop.u

Hi Team ,                 Could you please help me in working with XML tag's in Nifi. I've a scenario where i need to look for the parent xml tag value in entire document, the corresponding child xml tag value should match. if it is not matching we should copy the parent xml tag value to the corresponding specified child tag.here the parent tag is <OriginalQTY name=" 5.000">    and the child tag should match is <OriginalORDEREDPURCHASEQUANTITY>   So every time <OriginalORDEREDPURCHASEQUANTITY> should match <OriginalQTY> value in name. <Document><PURCHPURCHASEORDERHEADERV2ENTITY><VENDORACCOUNT>XXXXX</VENDORACCOUNT><CURRENCYCODE>ZAR</CURRENCYCODE><OriginalQTY name=" 1.000"><PURCHPURCHASEORDERLINEV2ENTITY><PURCHASEORDERNUMBER>PO-XXXXXXX</PURCHASEORDERNUMBER><LINENUMBER>0000000100</LINENUMBER><PURCHASEPRICE&g

Hi Everyone, i am trying to create Hive table with data from mongodb. I have installed mongodb in cloudera quickstart.  loaded json data. sample: { "_id" : ObjectId("59a47286cfa9a3a73e51e749"), "theaterId" : 1034, "location" : { "address" : { "street1" : "3404 W 13th St", "city" : "Grand Island", "state" : "NE", "zipcode" : "68803" }, "geo" : { "type" : "Point", "coordinates" : [ -98.382164, 40.933624 ] } }}  i have create Hive table as below  CREATE EXTERNAL TABLE IF NOT EXISTS THEATER (id STRING,theaterId INT,loc STRUCT<address:STRUCT<street1:STRING,city:STRING,city:STRING,city:STRING,street2:STRING>, geo:STRUCT<type:STRING,coordinates:ARRAY<DOUBLE>> >)STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler'WITH SERDEPROPERTIES ('mongo.columns.mapping'='{"id":"_id","theaterId":"theaterId","loc":"location"}')TBLPROPERTIES('mongo.uri'='mongodb://localhost:27017/movie_flix.theater'

Hi,how can I alter the table using NIFI Processor where source dbname : org, tableName:status, destination dbName:report and tableName: orgStatus.I am using captureChangeMySql Processor to read the binary-bin log file.Below json i am getting in updateAtribute processor:{"type":"ddl","timestamp":1658078637000,"binlog_filename":"mysql-bin.000030","binlog_position":2727,"database":"nrpsubscriptiondb2306","table_name":null,"table_id":null,"query":"ALTER TABLE `nrpuserorgdb2306`.`status` \nADD COLUMN `is_deleted` TINYINT NULL AFTER `code`"}Please find my processor screenshot for this flow.    

It seems that when we use DbTxnManager in CDH6.X, if SQL query does not complete successfully or is terminated abruptly (ex. ctrl+c), then the locks it implicitly acquired will not be automatically released, and other new SQL queries which references the same table/partition will be blocked trying to acquire necessary locks, and hence can't be executed successfully.   we can't manually clear the locks by issuing unlock statements under DbTxnManager,because it will pop up error: Current transaction manager does not support explicit lock requests.Transaction manager:org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;   it seems currently the only way to clear above obsolete locks is to log into the metastore db and use sql statements like below to delete records from table hive_locks:  select hl_lock_ext_id from HIVE_LOCKS where HL_TABLE=’prcs_task’;delete from hive_locks where hl_lock_ext_id = 125542;   I am wondering, if this is the correct way to ma

HI,I want NIFI to be run with the minimum resource needed. my concern is whether I can run NIFI with JRE or JDK is also required. will I face any issues in future while scaling up? I am using NIFI 1.15.3.

Hello,I know that there are processors such as CosumIMAP and ConsumPOP3 which can receive email by NiFi. These processors need to provide password.Is there another Processor which utilise Office 365 Oath2 authentification?Considering these two links which show using IMAP will be expired .... https://docs.microsoft.com/en-us/lifecycle/announcements/exchange-online-basic-auth-deprecatedhttps://www.bleepingcomputer.com/news/microsoft/microsoft-warns-exchange-online-basic-auth-will-be-disabled/ Thanks for your supports.