Solved

Hi,I am provisionning a CDP 7.1.5 cluster with CM 7.2.4  and encountered a connexion problem between the agents and cloudera manager server after tls setup through API.There is nothing in the server log and there is the following error in the agent log.Could you help me understand where the problem stands ? Activating further logging or doing some tests upon the provided certificates ?Traceback (most recent call last):File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1430, in _send_heartbeatself.cfg.max_cert_depth)File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 185, in __init__self.conn.connect()File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connectsock.connect((self.host, self.port))File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connectret = self.connect_ssl()File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/C

I have many zip files each containing a csv file. The header of the csv file contain fields and some field names have space between words e.g "field one" instead of "field_one" which is the one in the database.Please, what is the optima method to change the csv field names to match the ones in the database.NOTE: I am using MSSQL as the DB. And my NIFI implementation is1) GetFile : from a repository that contain zip files of very large size2)UnpackContent: to extract the csv from the zipped file3) PutDabaseRecord: to persist the csv data into db because of its large size.  Thank you for your kind help.

Hello Experts, I want to run a simple Hive Insert SQL statement in Nifi periodically for example:insert overwrite Table1 select * from Table2; All SQL values are fixed hardcoded and don't need to be changed dynamically in the flow. As a newbie I was thinking I could write the flow as: ReplaceText->success->PutHiveQL Search Value:  (?s)(^.*$)Replacement Value:  insert overwrite Table1 select * from Table2;  But I have an error in ReplaceText which probably needs an incoming flowfile which I don't have since the SQL is fixed hardcoded:"Upstream Connections is invalid because Processor requires an upstream connection but currently has none."  The other option I could try but not sure it will work is:GenerateFlowFile -> success -> PutHiveQLGenerateFlowFile Custom Text: insert overwrite Table1 select * from Table2;     

From Cloudera Manager it appears like Atlas Service is in good health, but it doesn't start at all. When I try to access the Web UI, it shows an Http Error 503 Service Unavailable but the application.log doesn't show any error; it just halts. Here is the application.log whe I restart Atlas service from Cloudera Manager:2022-01-31 12:04:28,657 INFO - [shutdown-hook-0:] ~ ==> Shutdown of Atlas (Atlas$1:63)2022-01-31 12:04:28,668 INFO - [shutdown-hook-0:] ~ <== Shutdown of Atlas (Atlas$1:69)2022-01-31 12:09:49,319 INFO - [main:] ~ Loading atlas-application.properties from file:/var/run/cloudera-scm-agent/process/6754-atlas-ATLAS_SERVER/conf/atlas-application.properties (ApplicationProperties:137)2022-01-31 12:09:49,327 INFO - [main:] ~ Using graphdb backend 'janus' (ApplicationProperties:317)2022-01-31 12:09:49,327 INFO - [main:] ~ Using storage backend 'hbase2' (ApplicationProperties:328)2022-01-31 12:09:49,327 INFO - [main:] ~ Using index backend 'solr' (ApplicationProperties

Error: Package: cloudera-manager-agent-7.4.4-15850731.el8.x86_64 (cloudera-manager)          Requires: libc.so.6(GLIBC_2.28)(64bit)this is centos 7 which comes with gcc 217 [root@cdp-master yum.repos.d]# cat /etc/centos-releaseCentOS Linux release 7.9.2009 (Core)I'm trying the install direct from cloudera. Thoughts?  It shouldnt be this type of issue

Hello,I have one big table in one database that I have to sink on destination.Firs I have flow with incremental insert and it works fine. Than in other flow I have incremental delete by ID column.I am trying to delete 3179958 records from table.  I have one table on source and I have to do increment delete on destination.  This is my flow. I am sending every processor separate:generatetablefetch:execute sql: putdatabaserecord - delete When I have much less records it all works fine. But now flow can't finished... Is there any other solution. Thanks in advance

我刚加入CDH社区。如何申请archive.cloudera.com/cdh6 谢谢您的帮助！

I'm trying to use kafka data using pyspark but I having difficult because it's in Hashmap type The Question is, how can I convert this to a useful df to be treated in pyspark? This is the output and my actual code: This is the outputThis is my actual code Any suggestion and steps? 

We have file names that start with a dot (e.g. ".~tempfile.bin") that we are attempting to list with the ListFile processor.  We've tried setting the File Filter to an empty string, as well as various RegEx's (e.g. ".*") and it will not list file names starting with a dot.  As soon as we remove the starting dot from the file name, it lists the files.  Any ideas on how to list these files starting with a dot?

I have a Spark v2.4.5 Spark-Streaming application running on CDP 7.1.7 that listens to relatively small JSON messages from a Kafka topic, de-serializes them, and then writes to an appropriate unpartitioned Hive table with parquet records. This issue also occurred with CDH 5.16 and CDH 6.3.4.The code looks similar to this: def processStream(stream: DStream[ConsumerRecord[String, KafkaMessageWrapper]]): Unit = { stream.foreachRDD(rdd => { // Peek at the message to extract the event type and raw payload val rowRdd = rdd.flatMap(MessageProcessor.processRecord) val rootDF: DataFrame = rowRdd .toDF("topic", "partition", "offset", "eventName", "version", "payload") rootDF.createOrReplaceTempView("data") // Process only the USER records spark.sql(s"select eventName, version, payload from data where eventName = 'USER_CREATE'") .na.drop() // Map the records to the expected schema // This flatten the incoming JSON and does some validation /

Hi, I am seeing some situations where I have two Hive SQL commands running concurrently and I'm getting a lost update.  I am running Hive 2.3.6 on EMR with hive.support.concurrency = true and I believe this shouldn't be happening based on what I understand about Hive table locking.  (I am not using ACID transactions but the table locking should still prevent lost update as far as I know;) Specifically I have a "load data" statement loading data into table T from an S3 location.  I have an "insert overwrite T select * from T" table running concurrently from another Hive connection that deletes some rows from T but should not be affecting rows from the load data statement.  I am seeing that the data from the load data statement disappears after the insert overwrite finishes.  My understanding is that the load data and insert overwrite should create an exclusive table lock on T so they should allow each other to finish before reading or writing data from T.&n

Hi guys,    Is there a way to isolate impala daemon resources in one cluster?    eg. 6 daemons in 1 Impala Cluster. Is there a method to isolate 2 daemons' resources for testing, 4 daemons' resources for production? Testing queries only use the memory resource in their daemons.    HAProxy can forward requests to different daemons, but memory resources are not isolated. If query from testing takes too much memories, production query may failed while executing.    The only way I can figure out is setup another impala cluster. I know impala provides admission control, what if the query execution plan estimate memory cost wrong, the actual memory cost is far more than the estimation(eg. 3G VS 150G)? What parameters limit the actual memory usage?

Hi,  I am trying to run this import command: sqoop import --connect jdbc:postgresql://[host]:[port]/[db] --username [user] --password-file [path_to_pass] -table [table] --hive-import -hive-drop-import-delims --as-parquetfile --parquet-configurator-implementation hadoop --hive-table district_startdates_test --target-dir [target_with_db] -m 1 --hs2-url="jdbc:hive2://[cdp]/;ssl=true;transportMode=http;httpPath=[path_to_proxy];user=[user];password=[cluster_passw]" --verbose but it gets stuck at   DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection. ...any idea?thanks

Hello.I would like to use the kafka connect to connect the kafka brokers provided by cdp and on-premise kafka brokers. To do that, I need to set the configuration value "connector.client.config.override.policy".But I could not find the menu to set custom configuration of kafka connect. Could you let me know how to set the custom configuration of kafka connect especially "connector.client.config.override.policy".?

Sometimes the repo has all the versions of logredactor, sometimes it only has 2. This varies every time I hit refresh. I'd guess there's servers out of sync with each other behind a load balancer maybe? https://repository.cloudera.com/artifactory/cloudera-repos/org/cloudera/logredactor/logredactor/Our project relies on 1.0.3 for a Spark dependency, is there any chance of this being fixed or should we start just hosting it locally?

Hi. I was wondering if anyone can confirm or deny that kerberized hive can (how?) work with not kerberized Zookeeper (+kerberized HDFS and Yarn). Unless I overlooked something (which is very possible), it seems like in line 93 of attached log hive is trying to auth against ZK, and since there is any ZK security enabled this attempt failed. If I'm right, is there any property that I can set up to disable auth attempts on hive side? hdfs/yarn seems to have zero problems working with unsecured ZK.Or perhaps we need to enable kerberos on ZK nodes because hive won't work otherwise, period?  

Hi All, I am trying to achieve incremental sqoop import into hive when a new record is created as well when updates happen to the existing records. We have tables having both created date column and modified date column. But the modified date column will be updated only when there is any update in the existing records and will remain null when a new record is inserted. Only the created date will be updated when new records are created.  So, when we try the incremental lastmodified option it always looks for the modified date column but our source tables doesnt have values in the modified date column always as it keeps NULL for new records and only updates the value when an existing table gets modified.  So, for this kind of source DB tables with such columns is there any solution in sqoop import.  Also, sqoop incremental with lastmodifed doesnt work with hive tables and we have to create a table already and then run this incremental lastmodified&

Example : Input : 10/12/2019 I need to get subsequent Sunday of given input date

I saw this kind of notation/style using double curly braces everywhere in HDP(Ambari) or CDP (CMS) UI. Below is a configuration value under zeppelin.shiro.knox.main.block for Zeppelin configuration. (This is a random sample I picked and this question is not about Zeppelin.)++krbRealm.signatureSecretFile={{CONF_DIR}}/http_secret++I understand that I can simply overwrite {{CONF_DIR}} with the actual path. However, I wonder whether {{CONF_DIR}} an ansible variable? If yes, how do I define the variable CONF_DIR in CDP Cloudera Manager?https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#defining-simple-variables Regards,

I am a beginner in NIFI. And i would like to read live logs from a remote server to a Nifi flow on another system. I have no access to install Nifi on that remote server, please what is the best method to implement the above scenario.Thanking you so much for your reply.

We're trying to upgrade from TLSv1.0 to TLSv1.2 for the below configuration but getting error as attached.These are the installed versionsHDF: 3.1.0Nifi: 1.5.0.3.1.0.0-564Ambari: 2.6.2.2HDP: 2.6.5.1100. we have tried following the steps from this page and it didn't worked as well.https://community.cloudera.com/t5/Support-Questions/Nifi-SSL-TLS-qestion/td-p/285528Kindly help me to resolve this issue.

In the guide "Cloudera Enterprise Reference Architecture for Azure Deployments" ( https://docs.cloudera.com/documentation/other/reference-architecture/topics/ra_azure_deployment.html), it mentions " Using deployment scripts hosted on the Azure GitHub Repository".  However, the link of "Azure GitHub Repository" (https://github.com/Azure/azure-quickstart-templates/tree/master/cloudera-on-centos) is not valid anymore.   Do you have the valid link or could you fix the documentation? Thanks.

Hi,   I am trying to create 5 node cluster using AWS AMI - Cloudera Director CSC (ami-c318a8a8). However, while launching it via launcher node the error pops up in configuration as below:- Found errors in cluster configuration:* No parcel found for product CDH   as I checked in the configuration file, the parcel repository is: parcelRepositories: ["http://archive.cloudera.com/cdh5/parcels/5.4.7/"]   Can you please suggest the troubleshoot?   Thanks

When trying to access the kudu metrics from a master or worker node, in the browser, Kerberos authentication is needed.In the browser thats no problem, after navigating tohttps://<kudu_master>:8050/metrics Im asked for my credentials via pop-up, after entering them and then it shows me the metrics. I'd like to connect the KUDU API via NIFI and use a InvokeHTTP processor to access the metrics. Of course i get the error that authentication is needed. entering username and pw in the GET request header did not help. How do I make acquire the metrics via a GET request?

Dear experts, I notice when I try to load a hbase data in pyspark, it tells mejava.io.IOException: Expecting at least one region for table : myhbasetable at org.apache.hadoop.hbase.mapreduce.MultiTableInputFormatBase.getSplits(MultiTableInputFormatBase.java:195) at org.locationtech.geomesa.hbase.jobs.GeoMesaHBaseInputFormat.getSplits(GeoMesaHBaseInputFormat.scala:43) at org.apache.spark.rdd.NewHadoopRDD.getPartitions(NewHadoopRDD.scala:130)  It looks like it is telling me the table has to at least have some data in at least 1 region. This is the relevant piece of code: -->https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.java   try (Connection conn = ConnectionFactory.createConnection(context.getConfiguration())) { while (iter.hasNext()) { Map.Entry<TableName, List<Scan>> entry = (Map.Entry<TableName, List<Scan>>) iter.next()