Find answers, ask questions, and share your expertise
Recently active
Hello all, I am trying to run spark job using spark-submit with a docker image over yarn. I followed the instructions in the Blog provided by cloudera in the following link:https://blog.cloudera.com/introducing-apache-spark-on-docker-on-top-of-apache-yarn-with-cdp-datacenter-release/ and I ran into an error that I couldn't fine an answer to.Note: I already did all the configurations required from the post. I ran this command :spark-submit \--master yarn \--deploy-mode cluster \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=faresdev8/python3:v5 \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/data1/opt/cloudera/parcels/:/data1/opt/cloudera/parcels/:ro" \--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \--conf spark.executorEnv.YARN_CONTAINER_
Hi,I have install a NiFi setup and deign the flow file to move data from source mysql db to destination mysql db.Now I want to know how can I setup it in dev or QA environments. I was going to NiFi admin guide and came to know about NiFi Registry and NiFi toolkit feature.I have below doubts:1. Is it mandatory to install NiFi toolkit along with NiFi to configure clustering and managing the cluster node? or without installing a NiFi toolkit, we can configure cluster and manage it? 2. Is it mandatory to install NiFi Registry along with NiFi to manage the flowfile change restriction?
Hi All, I enabled sentry and all services are healthy but unable to create roles and not getting 0: jdbc:hive2://wa1cog3mgmt2200.cdp.centuryli> CREATE ROLE huesuperrole;going to print operations logsprinted operations logsgoing to print operations logsprinted operations logsgoing to print operations logsprinted operations logsgoing to print operations logsINFO : Compiling command(queryId=hive_20220731105224_c56c786a-bfaf-4270-a2a0-e62936b67777): CREATE ROLE huesuperroleINFO : Semantic Analysis CompletedINFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)INFO : Completed compiling command(queryId=hive_20220731105224_c56c786a-bfaf-4270-a2a0-e62936b67777); Time taken: 2.529 secondsINFO : Executing command(queryId=hive_20220731105224_c56c786a-bfaf-4270-a2a0-e62936b67777): CREATE ROLE huesuperroleINFO : Starting task [Stage-0:DDL] in serial modeprinted operations logsgoing to print operations logsprinted operations logsgoing to print operations logsprinted o
Hi I have a flow file with huge string content and I want to create a JSON and add this string to the JSON as a value with a static key. Example Flow file string content: "Example string"Needed output :{"Static key":"Example string"} I tried to add the string content to the flow file attribute using ExtractText and use attribute to JSON processor to generate the needed output but I have 2 issues with this approach:-1- The string content size is not under my control and I dont know how much I need to configure the Maximum Buffer Size and Maximum Capture Group Length.2- Adding the content to the flow file attribute will duplicate the flow file size and this will impact the performance.3- Finally I feel it is a workaround solution :)) Is there any other way to solve this issue?
ExampleAvailability : four/five 9’s uptimeDurability : 10 9's Reliability : 15 9's Can anyone pls clarify on this
Is there a simple way to restrict number of threads to a NIFI process group?.
When I try to start a spark-shell with YARN I'm getting this error: 22/07/29 15:36:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable22/07/29 15:36:08 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.22/07/29 15:36:09 WARN security.HBaseDelegationTokenProvider: Fail to invoke HBaseConfigurationjava.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfigurationat scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)at org.apache.spark.deploy.security.HBaseDelegationTokenProvider.hbaseConf(HBaseDelegationTokenProvider.scala:117)at org.apache.spark.deploy.security.HBaseDelegationTokenProvider.delegationTokensRequired(HBaseDelegationTokenProvider.s
Hi, I am trying to add flow attribute from my json file which looks like:[ {"name" : "id","value" : "29879"}, {"name" : "file","value" : ""}, {"name" : "barcode","value" : ""}, {"name" : "roll no","value" : "3032"}, {"name" : "name","value" : "Daniel"}] Now I want to add roll no. as a flow atrribute using Can someone please help me with it? Thanks
Hey Team ,i am using DAS most of the time its very slow , i would like to know how i will find App_Id application_1654575913119_3886895 From hive_20220725051805_76a0f15d-c15f-4c90-b73f-eb8e6bc1c4c1 Hive Queryi have Hive_Query_id want to find App_IdHive_Query_id :- hive_20220725051805_76a0f15d-c15f-4c90-b73f-eb8e6bc1c4c1 App_Id :-application_1654575913119_3886895 ThanksImran Chaush
How do I get a CDP Private Cloud Base license?
Hello, I would like to know if this CVE which impacts Apache Hadoop is already resolve into HDP or CDP products ? In Apache Hadoop 2.2.0 to 2.10.1, 3.0.0-alpha1 to 3.1.4, 3.2.0 to 3.2.2, and 3.3.0 to 3.3.1, a user who can escalate to yarn user can possibly run arbitrary commands as root user. Users should upgrade to Apache Hadoop 2.10.2, 3.2.3, 3.3.2 or higher. https://nvd.nist.gov/vuln/detail/CVE-2021-33036Thanks in advance for your help.
We are getting this below errors all the time in Hbase Could some give me the suggestions ?
HDP-3.1.0.02022-07-28 00:19:45,289 - Package Manager failed to install packages: No package found for hive-${stack_version}(expected name: hive-3-1-0-0-78)Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 486, in <module> InstallPackages().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 149, in actionexecute raise Fail("Failed to distribute repositories/install packages")resource_management.core.exceptions.Fail: Failed to distribute repositories/install packagesstdout: /var/lib/ambari-agent/data/output-6991.txt2022-07-28 00:18:33,442 - Will install packages for repository version 3.1.0.0-782022-07-28 00:18:33,443 - Repository['HDP-UTILS-1.1.0.22-repo-1'] {'base_url': 'http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/ubunt
Hi,I am getting error "Record does not have a value for the PrimaryKey column 'HOST' " when I update a data in source db table. This issue is happening when destination DB table's name: 'user' or 'users'. It is working fine when destination table's name is change to 'report_user'(I mean when we change the table name 'user'or 'users' to something else which is not used as mysql internal table) I am not sure why it is trying to insert or update on internal mysql table as I am giving the destination DBName and tableName.Can anyone suggest how can we fix this issue without changing the destination table name? Error Log:PutDatabaseRecord[id=ff8ca68c-8252-3652-8f55-1044ad3f1bab] Failed to put Records to database for StandardFlowFileRecord[uuid=43e64992-a3d0-4a75-bb85-8d70c1f556d7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1658747398462-4, container=default, section=4], offset=20549, length=1214],offset=0,name=43e64992-a3d0-4a75-bb85-8d70c1f556d7,
Hi folks, Can I use the below script in Ozzie workflow? WITH order_data (order_date, product_name, amount)AS (SELECT order_date, product_name, SUM(i.quantity * i.list_price * (1 - discount))FROM sales.orders oINNER JOIN sales.order_items i ON i.order_id = o.order_idINNER JOIN production.products p ON p.product_id = i.product_idGROUP BY order_date, product_name) SELECT * FROM order_data;
Hello,Please kindly help to resolve this error.I am trying to learn how to create a Nifi custom processor following this tutorial:https://help.syncfusion.com/data-integration/how-to/create-a-custom-processor#_Project_Creation%22%22 But i don't know the class to import to solve the below error: Below are all the imports i have on the fileimport org.apache.nifi.components.PropertyDescriptor;import org.apache.nifi.flowfile.FlowFile;import org.apache.nifi.annotation.behavior.ReadsAttribute;import org.apache.nifi.annotation.behavior.ReadsAttributes;import org.apache.nifi.annotation.behavior.WritesAttribute;import org.apache.nifi.annotation.behavior.WritesAttributes;import org.apache.nifi.annotation.lifecycle.OnScheduled;import org.apache.nifi.annotation.documentation.CapabilityDescription;import org.apache.nifi.annotation.documentation.SeeAlso;import org.apache.nifi.annotation.documentation.Tags;import org.apache.nifi.processor.AbstractProcessor;import org.apache.nifi.processor.Process
Hi All, Wish you a healthy life!!I am trying to load a DWH server (mysql) with incoming data from 100 mysql databases using NiFi.I am using CaptureChangeMySQL (clients) of NiFi to capture changes and applying them incrementally. I am successful in my PoC which loads a single DWH table from 2 different databases.(DB1.table1 -> DWH.table and DB2.table1->DWH.table) Now, in order to do this in realtime, I have 100 different databases, all these databases have their own table which needs to be loaded into a single table at DWH.Do I need to create 100 different processors and configure each to connect to different database ?or is there an alternative way to deal with 100 databases ?If there is an alternative can someone help me understanding the solution and a sample xml file.. much appreciated..Also I am looking for your suggestions on how to do the same CDC for 15 other tables of 100 databases..
We have been using CDH version 6.2.1 and OS is RHEL 7.7 and like to upgrade to 7.9. Can you help on this to steps to be followed in the Cloudera distribution 6.3.3?
Hi everyone,I have signed up for the CDP 60-days trial verison because I will work with Cloudera on the job soon and therefore I would like to get familiar with it in advance and do some of these tutorials: https://www.cloudera.com/tutorials.html#?fq=When I go to Cloudera Data Platform I dont see any environments and I cant create/register news ones, because I am not the admin. During the registration process, there was no way to choose any specific user rights. It seems I have confused a few things, as I thought the Public Cloud trial would be some kind of sandbox where I could try out all kind of stuff like connecting to my AWS databases etc., but I don't see how this works if I lack the admin rights. Can anyone tell me if I can create an admin account by myself or if there is any other way to get free temporary access to some sandbox-like environment?Thanks a lot in advance!
Experts,I am continuously seeing this log in Nifi app log - 2022-07-25 06:12:40,788 INFO [Timer-Driven Process Thread-219] o.a.n.c.repository.FileSystemRepository Unable to write to container default due to archive file size constraints; waiting for archive cleanup Any idea what is the reason and will there be any impact of this ?ThanksMahendra
Hi, We are using spark.sql.sources.partitionOverwriteMode=dynamic" in our pyspark scripts in our CDH 6.3.2 cluster with spark version 2.4.0, but when we are trying it to CDP 7.1.4 with Spark 2.4.0 version and it is not working, is there anyway to have the config spark.sql.sources.partitionOverwriteMode=dynamic" work in CDP? is there any alternatives on it? Just to highlight that both our CDP 7.1.4 and CDH 6.3.2 clusters are having the same Spark version of 2.4.0
I'm trying to run a Yarn cluster by this command:spark-submit --deploy-mode cluster --master yarn main.py and i return this message:INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) Server address is not correct but in yarn-site.xml there is the correct FQDN. How can i solve this?
After following this article, I tried to create a flow where email attachments, subject, body, to, from, cc, bcc etc. can be extracted. However the flow is not proceeding beyond the first processor:ConsumeIMAP is showing 0(0 bytes) in its Out, as shown in screenshot above.
Hi All, Do we have any tool or utility to mask all the sensitive information (like hostname, ip, etc) inside the logfiles and Diagnostic bundle before uploading that to Cloudera cases? Also, I can see some of the service logs suffixes with hostname, I need to mask them too. like those below example:- hadoop-cmf-hdfs-NAMENODE-cdp-master.log.out to -> hadoop-cmf-hdfs-NAMENODE-xxx-xxxxx.log.out
Hi,Recently, in our production environment, there was an abnormal exit of the Hbase Master. In the log, I found that the ZK connection had some problems.According to the sessionid, I found that the sessionid of one of our applications was the same as the sessionid established by the master at the time in question, and then the application side closed the session, and then the master hung. Is this possible? Why does a session with the same ZooKeeper sessionid appear?
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.