Q&A

I'm using Machine Learning Workspace in Cloudera. I created a session with 4vCPU/16 GiB Memory and enabled Spark 3.2.0.I'm using spark to load data of one month (the whole month data size is around 12 GB) and do some transformation, then write the data as parquet files on AWS S3.My Spark session configuration looks like this:SparkSession .builder .appName(appName) .config("spark.driver.memory", "8G") .config("spark.dynamicAllocation.enabled", "true") .config("spark.dynamicAllocation.minExecutors", "4") .config("spark.dynamicAllocation.maxExecutors", "20") .config("spark.executor.cores", "4") .config("spark.executor.memory", "8G") .config("spark.sql.shuffle.partitions", 500)......Before the data are written to parquet files, they are repartitioned:df.withColumn("salt", math.floor(rand() * 100)).repartition("date_year", "date_month", "date_day", "salt").drop("salt").write.partitionBy("date_year", "date_month").mode("

can I get official support if I install the CDH on hygon server

Hi All, Need advice, how we can export all the DDL of one scheme from Hive ( from QA region, a schema with around 2000 tables) to Dev or ACPT region, only DDL's  needed to be in sync , not worried about the data, so do we have any option or script to import / export all the DDL's from one region to another region in one go, please suggest, thank you. PK

Hi All, Do we have any script to export all the DDL's of the table for a schema in Hive ? we have a schema name - staging_DB and its contains around 2000 tables, now we need to export all the DDL's of this tables to a txt file so that we can run this DDL's in another setup. Please suggest if we have any script or any easy way to do it, thank you. - PK

I have a situation, where I have to get the combined data from two tables  from Mysql database, ExecuteSql processor throwing out bellow error,  where as the same query is working fine from Mysql Workbench.   2020-04-22 14:55:06,597 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.ExecuteSQL ExecuteSQL[id=9d81c7ae-0171-1000-f638-2f1e60bfe48e] Unable to execute SQL select query selectt1.CODE_CLOCK_ID,t1.COMPANY_CD,t1.GROUP_SEGMENT_L1,t1.GROUP_SEGMENT_CD,t2.CODE_CLOCK_ID,t2.COMPANY_CD,t2.GROUP_SEGMENT_L1,t2.GROUP_SEGMENT_CDfromsheet26 t1INNER JOIN sheet27 t2on t1.CODE_CLOCK_ID = t2.CODE_CLOCK_ID and t1.COMPANY_CD = t2.COMPANY_CD LIMIT 0, 1000; due to org.apache.nifi.processor.exception.ProcessException: org.apache.avro.AvroRuntimeException: Duplicate field CODE_CLOCK_ID in record any.data.sheet26: CODE_CLOCK_ID type:UNION pos:4 and CODE_CLOCK_ID type:UNION pos:0.. No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.a

Hi , I have a spark cluster deployed on windows. I'm trying to submit a simple spark job using the rest api. The job is just python code that does simple hello world sentence as follows :from pyspark.sql import SparkSessiondef main(args): print('hello world') return 0if __name__ == '__main__': main(None)The url Im using to submit the job is:http://<Master-IP>:6066/v1/submissions/createWith the following Post Body: { "appResource": "file:../../helloworld.py", "sparkProperties": { "spark.executor.memory": "2g", "spark.master": "spark://<Master IP>:7077", "spark.app.name": "Spark REST API - Hello world", "spark.driver.memory": "2g", "spark.eventLog.enabled": "false", "spark.driver.cores": "2", "spark.submit.deployMode": "cluster", "spark.driver.supervise": "true" }, "clientSparkVersion": "3.3.1", "mainClass": "org.apache.spark.deploy.SparkSubmit", "environmentVariables": { "SPARK_

Hi. I haven't found any documentation for these internals about the InvokeHTTP processor in NiFi. My requirement is to throttle requests to and endpoint (which we also control) that will accept, let's say, 5 simultaneous connections in an address and port, but I need to wait for each of their responses before letting another flowfile to be sent, so: Does this translate to simply using 1 InvokeHTTP processor configured to 5 "Concurrent Tasks" and that's it?Will the processor wait for the remote endpoint's request before sending the next one?How does the "Run Schedule" works together with the previous settings? (if I had, e.g.: 1 sec)I've been proposed with splitting the incoming queue and put 5 InvokeHTTP processors in parallel, each one attending 1/5 of the incoming flowfiles (I'd do the pre-partitioning before with some RouteOnAttribute trick), but I think it's exactly the same outcome as the 1. above. Is it?This all boils down to not knowing exactly how this processor work

Hi,I'm looking at using Apache Nifi as a data integration hub for our case/archive management solution used in the Nordics and our rest-based API. One use case is to first read an XML from one source with a few values and then use these values in a larger JSON payload that creates a new case in our solution with InvokeHTTP.All values in the JSON arrays are inside a parameter{} node and can have objects with child properties. I've done a lot of research and testing with the different processors, like ReplaceText, and AttributesToJSON, but can't find how I easily can do this. Is it needed to use some complex JSON generator like Groovy or Jolt to achieve this with ExecuteScript? With a online tool like make.com I was able to do this really easy, but find it complex to just replace multiple single values in the JSON payload before InvokeHHTP when the JSON payload is nested. Example of a JSON payload for the API can be something like below, but can also contain many

In CDP Public Cloud Machine Learning, we can create a new session with reserved resource, for example 4vCPU and 16 GiB Memory.We can also create spark session inside the machine learning workbench with some memory configuration. For example:spark = (SparkSession.builder.appName(appName).config("spark.driver.memory", "16G").config("spark.executor.instances", "10").config("spark.executor.cores", "4").config("spark.executor.memory", "20G").getOrCreate())My question is, how will the memory be allocated to Spark session now? Is the reserved resource (4vCPU and 16 GiB Memory) in machine learning session the maximal limitation for total spark memory usage? How many work nodes and executors can I configure for the spark session? 

Folks, I have setup the secured nifi cluster on development env and there are few things hitting my mind wrt to production use. - There are few configurations related to database, flow repository, content repository, provenance, components etc and I'm wondering what should be the best practices to manage these files. Should I use the persistence volume/storage on K8 to have these centralized for whole cluster.                    If yes, would it interfere with internal replication? Wouldn't it be SPOF?                    If not, what if my cluster goes down, I will loose all the state and data or shall I use a replica set explicitly? Can someone helps understanding the best practices on production wrt above scenarios and also anything in general Thanks,

we are getting a lot of "Network Interface Speed Unknown" and "Agent Status Bad" alerts on two specific hosts in the cluster and when accessing these hosts through cloudera manager we collect the details. error details: "Network Interface Speed UnknownThe health test result for HOST_NETWORK_INTERFACES_SLOW_MODE has become unknown: Not enough data to test: Test of whether the host has network interfaces that appear to be operating at less than full speed. Agent Status BadThe health test result for HOST_SCM_HEALTH has become bad: This host is in contact with the Cloudera Manager Server. This host is not in contact with the Host Monitor."  we checked the cloudera agent logs on both hosts and found no alerts that could be impacting these alerts. we would like to know what may be impacting to generate this alert?

Hello, My NiFi flow has two process groups that process data from kafka:main - ~20 processors that processes 100 json messages per secondsupport - ~900 processors that processes 1-2 json messages in a day When the main group is running (started) without message processing the CPU load is ~5%. When the support group is additionally started the CPU load is ~20% (without messages). In addition the main group handles messages with 1-2 sec latency if the support group is not running, but latency increases dramatically (x10-20) if i just start the support group. Is it possible to set additional priorities for the main group or somehow configure the nifi scheduler? Thanks.

Hello, I cannot find an answer on the www to my following question. I have tried splitjson and jolttransform to achieve my desired outcome, but nothing does the trick. My input file looks like this: {"BVjsoBHDSI" : {"date" : "2021-04-08","description" : "blabla1","number" : 40},"FudNSk" : {"date" : "2021-09-26","description" : "blabla2","number" : 80},"Cfjdsl54fd" : {"date" : "2022-05-10","description" : "blablubb","number" : 31}} Actually I just want to split the stupid thing, but since the root level has a different individual value nifi doesn't understand what I want to do. The idea was to use a jolt transform to add a key "id" with the original key as value. I can't figure out a proper specification though. Glad to hear your ideas. Desired output: {"BVjsoBHDSI" : {"date" : "2021-04-08","description" : "blabla1","number" : 40,"id" : "BVjsoBHDSI"},"FudNSk" : {"date" : "2021-09-26","description" : "blabla2","number" : 80,"id" : "FudNSk"},"Cfjdsl54fd" : {"date" : "2

we have huge production Hadoop cluster, with HDP version 2.6.5 and ambari version 2.6.2.2 , and all machines are with OS RHEL 7.6 version the cluster size is as the following : Total workers machines - 425 ( each worker include data node and node manager service ) from time to time we get indication of lost one or two **node-manager** and this identified from Ambari as ( 424/425 when total node-manager  are 425 ) in order to fix it we just restart the **node-manager** and this action fix the problem and as results we get 425/425 after some googling , we found the following parameters that maybe should be tune better yarn.client.nodemanager-connect.max-wait-ms ( its configured to 60000 ms and we think to increase it )yarn.client.nodemanager-connect.retry-interval-ms ( its configured to 10 sec ms and we think to increase it )yarn.nm.liveness-monitor.expiry-interval-ms ( this parameter not configured yet and we think to add it with value of 1500000

hey,when I run a sqoop program. sqoop import "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" --driver org.apache.phoenix.jdbc.PhoenixDriver --connect jdbc:phoenix:172.18.1.16:2181 --query "select sid from test.PV WHERE 1=1 and \$CONDITIONS limit 10" --hive-import --hive-database tmp --hive-table system_stats --target-dir /hivetable/phoenixdb --delete-target-dir --split-by sid --hive-overwrite --null-string '\\N' --null-non-string '\\N' it's out this error  023-01-06 16:53:15,737 INFO [ReadOnlyZKClient-172.18.1.16:2181@0x062577d6] org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/yarn/nm/usercache/root/appcache/application_1672990653606_0007/container_e14_1672990653606_0007_01_0000022023-01-06 16:53:15,738 INFO [ReadOnlyZKClient-172.18.1.16:2181@0x062577d6] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=172.18.1.16:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$16/1413879487

Hi guys, how could use a command curl add the --cert, i have the below error: curl: (58) NSS: client certificate not found (nickname not specified)

I found the problem while i run the hive BDR and it took 2 hours.Cloudera support inform us that's the bug and workaround is to disable   The possible way to work around is to disable the replication of table column statistics but we don't know how? Anyone has idea? This slowness is most probably because of a fix (OPSAPS-62843) that was introduced in 7.1.7 SP1 to have a hotfix of a functional error (reported in ENGESC-8654), but having the disadvantage of decline in performance.The performance decline has already been reported and discussed in several other escalations, e.g. ENGESC-15274. So the possible ways to go forward with the solution is ---> to wait until the fix for the performance decline (OPSAPS-65104) gets released, ---> or to apply a workaround on the nodes of the target cluster that would disable the replication of table column statistics, but would enhance the performance. For that the file /opt/cloudera/cm-agen

Hi All,   Can someone past procedure to decommission disk from Ambari cluster. It will helpful. As we have 3 data node's and we have 3 disks configured each node and we want to remove one disk from it.    Thanks, Amit Mayekar

Hi team, How to troubleshoot always particular nodemanagers nodes memory reaches >95% when job is running Regards,Hanu

Encountered error with /opt/cloudera/cm/bin/gen_credentials.sh: Cannot access generated keytab file /var/run/cloudera-scm-server/cmf32289673674897789.keytab

Hi Team, We are trying to enable logging at each processor group level. But we are unable to do it. From nifi-app.log we are getting as a general but we need log for each processor in a processor group level. We cannot use bulletin as well as bulletin will stay only for 5 min interval. Can anyone can help achieving it.

Hi Team, I am using Jolt for the input:{"headers": {"query": "NA","param": "false"},"data": [{"SEQ_NUM": [162,162,162,162,162,162,162,162]},{"SEQ_NUM": [162,162,162,162,162,162,162,162]}]}  Jolt Spec:[{"operation": "shift","spec": {"data": {"*": {"SEQ_NUM": {"*": {"@": "[&1].SEQ_NUM"}}}}}}] Output:[ {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]}, {"SEQ_NUM" : [ 162, 162 ]} ] Expected Output:[ {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162}, {"SEQ_NUM" : 162} ] Can you please help me with the same.Thank you

Is there a setting or option that can be used to prevent an appltication  from creating huge temp files?We recently had an application that due to bad syntax, began to cycle and create files in TB temp almost lowering the cluster for space, there is a way to prevent /tmp from overflowing at that level

Hi, After configuration changes in alert publisher for mail alerts the stale configuration restart is required is it impact the cloudera server ?    Regards,Datta Nagargoje

Hi All,I'm not able to access BDF (HIVE/IMPALA) tables from my local machine. I'm using below query. Could you please suggest. from impala.dbapi import connectconn = connect(host='xxxximpala.rxcorp.com', port=21000,user='xxxxxx', password='xxxxx',auth_mechanism='PLAIN')cursor = conn.cursor()cursor.execute('SELECT COUNT(1) FROM my_table LIMIT 100')results = cursor.fetchall() TTransportException Traceback (most recent call last)<ipython-input-199-b13e7182755e> in <module> 1 from impala.dbapi import connect----> 2 conn = connect(host='xxxmpala.rxcorp.com', port=21000,user='xxxx', password='xxxx',auth_mechanism='PLAIN') 3 cursor = conn.cursor() 4 cursor.execute('SELECT COUNT(1) FROM my_table LIMIT 100') 5 #print cursor.description # prints the result set's schema~\AppData\Local\Continuum\anaconda3\lib\site-packages\impala\dbapi.py in connect(host, port, database, timeout, use_ssl, ca_cert, auth_mechanism, user, password, k