Q&A

Hi Team, I have below HADOOP Server details in our environment.#1 Node Cluster working - >Nodemanagers:166>Datanodes:159#2 64 Cores per Node#3 503 GB RAM per node. According to above node and core details"I WANT TO SET SPARK SUBMIT below parameter--driver-memory --driver-cores --num-executors--executor-memory --executor-cores " for that Please suggest me how to calculate it and also please share the calculation logic for the same.  Also #2 question is, In shell script we are calling the .py Python code using given spark parameter as - spark-submit--conf spark.maxRemoteBlockSizeFetchToMem=2G--conf hive.exec.dynamic.partition=true--conf hive.enforce.bucketing=true--conf hive.exec.dynamic.partition.mode=nonstrict--master yarn--deploy-mode client--driver-memory 30G --driver-cores 4 --num-executors 99--executor-memory 40G --executor-cores 4 --conf spark.sql.shuffle.partitions=800--conf spark.shuffle.compress=true--conf s

I believe below mentioned CVEs are either addressed or fixed through patching in CDH 6.3.4 -CVE-2021-4104 (Log4j1) - as per this article, CDH user doesn't need to do anything to fix this vulnerability.CVE-2021-44228 (Log4j2) -  as per this article, patches are available for this vulnerability for CDH 6.3.4.But apart from above vulnerabilities, there are few more vulnerabilities of critical, high and moderate severity in Log4j1 and Log4j2 which are - Log4j1 - https://logging.apache.org/log4j/1.2/index.htmlCVE-2019-17571 is a high severity issue targeting the SocketServer.CVE-2022-23302 is a high severity deserialization vulnerability in JMSSink.CVE-2022-23305 is a high serverity SQL injection flaw in JDBCAppender that allows the data being logged to modify the behavior of the component.CVE-2022-23307 is a critical severity against the chainsaw component in Log4j 1.x.Log4j2 - https://logging.apache.org/log4j/2.x/security.htmlCVE-2021-4504

I was trying to control apache nifi processors to start/stop by calling apache nifi-api. I am using Postman to make the call, for example to start a GenerateFlowFile processor:https://localhost:8443/processors/{processor id} (PUT Request)But I got the response status 403(forbidden) as shown in the image below.                I was able to make a get request. I have also a client self-signed certificate generated using nifi-toolkit by following this https://kylo.readthedocs.io/en/v0.10.0/security/ConfigureNiFiWithSSL.html from nifi.properties file:ifi.security.autoreload.enabled=falsenifi.security.autoreload.interval=10 secsnifi.security.keystore=./conf/kylo/localhost/keystore.jksnifi.security.keystoreType=jksnifi.security.keystorePasswd=.............nifi.security.keyPasswd=.............nifi.security.truststore=./conf/kylo/localhost/truststore.jksnifi.security.truststoreType=jksnifi.secur

After restarting all the services trough ambari, only two services don't start and are facing the same error : ranger and infrasolr /usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")resource_management.core.exceptions.Fail: Configuration parameter 'kerberos-env' was not found in configurations dictionary!  Any idea ?Thanks

While converting the input flow file JSON to any other format using query record (CSVwriter or AVRO Writer) using inferschema strategy the NIFI processor is trying to convert to Date based on first few characters of the incoming string.NIFI Error OutputSuccessfully parsed a JSON object from input but failed to convert into a Record object with the given schema\n- Caused by: org.apache.nifi.serialization.MalformedRecordException: Successfully parsed a JSON object from input but failed to convert into a Record object with the given schema\n- Caused by: org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Failed Conversion of Field [sharepoint_documents_name__c] from String [10-7-20 2x10GE LAN PHY XXXXXXXXXXXX] to LocalDate\n- Caused by: java.time.format.DateTimeParseException: Text '10-7-20 2x10GE LAN PHY XXXXXXXXXXX' could not be parsed at index 0:   Any workaround/fixes available in the newer version. Iam using NIFI 16.1

Hi Team, Can you please explain me how to interpret  'Cluster CPU' graph from CM.  

We have an MYSQL HA cluster,In case of failover, for the secondary master to take the place of the failed master, the hive metastore connection string configuration would need to be changed to point to the new master, and the services restarted. is it possible to have true HA on the Hive Metastore without changing the connection string manually ? 

Log files below--------------------/var/log/cloudera-scm-server/cloudera-scm-server.log 2022-11-10 07:12:56,046 WARN C3P0PooledConnectionPoolManager[identityToken->1hgex1yasqqqybleeujro|abff8b7]-HelperThread-#0:com.mchange.v2.resourcepool.BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@74294c1a is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.2022-11-10 07:12:56,046 WARN C3P0PooledConnectionPoolManager[identityToken->1hgex1yasqqqybleeujro|abff8b7]-HelperThread-#2:com.mchange.v2.resourcepool.BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5c69b2c -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:com.mysql.jdbc.exceptions.jdbc4.CommunicationsExcep

I am trying to create several tables with hundreds of millions of columns.Tables are created and data is added while changing the orc.stripe.size value (64MB or 256MB) for each table because the size of each table is different. (If there is a lot of table data, an error occurred when setting 64MB, so the value was increased to 256MB.)However, if the orc.stripe.size value is set large (ex. 256MB) for a table with a relatively small number of data, the following error occurs.I don't know how to set this value and create it. Is there a way to set the value according to the table size? SessionState: Vertex failed, vertexName=Map 1, vertexId=vertex_1666932355626_0180_236_01, diagnostics=[Task failed, taskId=task_1666932355626_0180_236_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1666932355626_0180_236_01_000000_0:java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOE

How to resove given ERROR - ERROR - Unknown Exception:07<class 'py4j.protocol.Py4JJavaError'>An error occurred while calling o1140.insertInto.: org.apache.spark.SparkException: Job aborted.

Hello team, my workflow has been failing for several days in SSH actions randomly.the error found in the SSH logs is:   Causé par : GSSException : aucune information d'identification valide fournie (niveau de mécanisme : TGT a été révoqué (20)) sur sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:772) sur sun.security.jgss.GSSContextImpl.initSecContext (GSSContextImpl.java:248) sur sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) sur com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ... 45 plus Causé par : KrbException : TGT a été révoqué (20) sur sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:70) sur sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:226) sur sun .security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:237)à sun.security.krb5.internal.CredentialsUtil.serviceCredsSingle(CredentialsUtil.java:482) à sun.security.krb5.internal.CredentialsUtil.serviceCreds(C

I am facing  641 MainThread agent ERROR Heartbeating to m:7182 failed. Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1430, in _send_heartbeat self.cfg.max_cert_depth) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 185, in _init_ self.conn.connect() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect sock.connect((self.host, self.port)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect ret = self.connect_ssl() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl return m2.ssl_connect(self.ssl, self._timeout)SSLError: certificate verify failed. 

i am using hive and tez.whenever i am performing insert query its returns following error: execution error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.teztask 

Hello,I could not split a JSON due to a special ETX character existing in a JSON value, SplitJson processor returns an error that it is not a Valid JSON.The JSON looks like that:[ {"key_data":"val_data"}, {"key_data":"val_ETXdata"}, {"key_data":"val_data"}, {"key_data":"val_data"},... ] I used The next regex expression to replace it but it doesn't work:\x03(?=[^"]*")also[\x03](?=[^"]*") can someone show me a trick to remove it please, I would appreciate it.   

I understand NiFi stores its FlowFile attributes in memory and large attributes are not recommended because it takes up space in the JVM. What I am having trouble grasping is the consequences. Can someone explain different use cases what could happen and also, how long those attributes are stored in memory for?

Hello everyone,  I'm tinkering with a test cluster. Test cluster has the trial license and so far I have installed HDFS, Oozie, Yarn, Spark and Zookeeper (all services are on one node). I tried to install Yarn Queue Manager, but when starting the YARN Queue Manager Webapp service, it errors out with   CM Internal user credentials are not available as system environment variables. Please make sure that both CM_USER and CM_PASSWORD are present as environment variables for the application to work correctly.  Currently I have Cloudera Manager:Version: Cloudera Enterprise Trial 7.1.4Java VM Name: OpenJDK 64-Bit Server VMJava Version: 1.8.0_232 and when installing the cluster, I could only choose from Cloudera Runtime 7.1.8, no other choices. Are there any possible fixes for the issue?

Hello and thanks for reading  I have an issue with hadoop hbase while trying to find the issue I can see below memstore flush logs for all tables and its non stop MemstoreFlusherChore requesting flush of table 1. because K has an old edit so flush to free WALs after random delay 65889ms 022-10-29 12:22:13,176 INFO [regionserver/uhn-hw-fores201b:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of table 1. because K has an old edit so flush to free WALs after random delay 65889ms2022-10-29 12:22:13,177 INFO [regionserver/uhn-hw-fores201b:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of tabl 2 . because K has an old edit so flush to free WALs after random delay 181952ms2022-10-29 12:22:13,177 INFO [regionserver/uhn-hw-fores201b:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of table 3 . because K has an old edit so flush to free WALs after random delay 150462ms2

Hi All, We are enabling hbase hook in apache atlas 2.2.0 , we followed below steps ,https://atlas.apache.org/2.0.0/Hook-HBase.html After that , when we start to start hbase , failing with below error.ROD.2022-09-05 14:02:23,481 INFO [master/vmazewdranger01:16000:becomeActiveMaster] atlas.ApplicationProperties: Setting solr.wait-searcherproperty 'true'2022-09-05 14:02:23,481 INFO [master/vmazewdranger01:16000:becomeActiveMaster] atlas.ApplicationProperties: Setting index.search.map-name property 'false'2022-09-05 14:02:23,482 INFO [master/vmazewdranger01:16000:becomeActiveMaster] hook.AtlasHook: Failed to load application propertiesorg.apache.atlas.AtlasException: Failed to load application propertiesat org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:150)at org.apache.atlas.ApplicationProperties.get(ApplicationProperties.java:103)at org.apache.atlas.hook.AtlasHook.<clinit>(AtlasHook.java:80)at org.apache.atlas.hbase.hook.HBaseAtlasCoprocessor.<in

Hi,  I have the requirement where I am converting the base64 Image content to attribute and after that I am preparing the payload including the base64 Image content. But the problem is while converting the base64 Image content (approx. 6-7 MB size ) it is getting truncated in the attribute Value.  Can you please tell me what is the length limit of attribute value in NIFI.Appreciate your help. Thank you.

I've added ranger as service in Cloudera manager, but when I click 'audit' from the menu i get the following error : 

Hello,I am trying to create an Operational DB and getting the error in Subject. Appreciate any inputs.Thanks in advance.  

Hi Cloudera community,   I have been using MapR for sometime and have some regular data pipelines in place. I want to move out from MapR to Couldera.    My clarification with the experts here is, I want to retain the system & data architecture created in MapR(servers, nodes, clusters, data storage) and implement couldera on top of hadoop HDFS. Is this possible?   basically i want to replace MapR with Couldera without disturbing my existing hardware and data storage setup.    Looking forward for some sharing.

Problem: I have set up a listSFTP processor to run between 2am to 2pm everyday.  But the files are not being picked up by the processor.  Existing setup and configuration: The files are generated at the SFTP server at 12 am to 1am every day. ListSFTP ConfigurationSchedule: listing strategy: tracking timestampsother tracking configuration  When I start this processor, it runs as expected for 1 day or 2 and after that, the files are not picked.  Is it the tracking time window of 3 hours that is affecting the file being listed? 

Hello,I have a question about Cloudera Data Platform and SDX :Is data sharing available accros my company only?If I want to share a specific file or a specific table in my datawarehouse with my client outside my company (who has his own account in Cloudera), is it possible? Regards,Mohamed

Need your expertise on this. I would like to know concerning about disabling the ansible tower job isolation security bubble .I am aware that there is a security bubble integration with Ansible Tower and CyberArk using Central credential provider (CCP) approach .However, with the credential provider (CP) approach, this security bubble integration will be bypassed