Find answers, ask questions, and share your expertise
Recently active
Hi Team, We have 1 table which has more than 1 lakh partitions. Support team moved data from HDFS to other path and try to do MSCK but it is not working even drop table also. Getting below errors.. FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask While drooping partitions also getting same error after 10min can one help me on this..
Hive Fails to install kerberos authentication hive. But after uninstall kerberos , caused by an error occurs during the hive startup Unable to read HiveServer2 configs from Zookeeper
Hi, I have a flow that receives JSON arrays as input. I would like to validate each of these jsons' schema, however using the ValidateRecord processor doesn't quite seem to do the job. I need to validate things such as certain fields being Enum values, having a max/min length, and ensuring required fields are present (sometimes inside of optional nested jsons). It seems an avro schema does not allow some of these functionalities and as such the Record processors can't quite validate my data as I need it. I would love to hear if anyone has had a similar use case and what they did to solve it. I am considering using the ScriptedValidateRecord processor, however I would prefer to avoid that and might instead opt for using EvaluateJsonPath to extract all the fields I want to validate and then using RouteOnAttribute with the expression language to filter out bad records. If there is a more appropriate way to validate records like this then I'm all ears. Thanks I'm advance!
I have a NIFI workflow running, how do I start and stop the workflow from the CLI. Is it possible to only start/stop one workflow from multiple workflows?
when I try to read parquet file from Azure datalake container from databricks, I am getting spark exemption. Below is my query import pyarrow.parquet as pqfrom pyspark.sql.functions import *from datetime import datetimedata = spark.read.parquet(f"/mnt/data/country/abb/countrydata.parquet") org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 14.0 failed 4 times, most recent failure: Lost task 0.3 in stage 14.0 (TID 35) (10.135.39.71 executor 0): org.apache.spark.SparkException: Exception thrown in awaitResult: what does this mean? What I need to do for this?
Hi Team, We have 3 DN's and Data Directories mount point size is around 1TB on each Node , total Data Directories size was 3TB. I deleted 500GB data from HDFS but space got release from only DN3 Data Directory not from other DN's. Please advise..
hello cloudera community, we are trying to use pyspark to interact with kudu We were able to successfully execute the command below: import kudu but when running the command below: from kudu.client import Partitioning returns the error ImportError: libkudu_client.so.0: cannot open shared object file: No such file or directory ps: we are using cdp, cdh 7.1.7 and cm 7.6.1
Hello everybody,I'm facing with a very weird behaviour with an UPSERT instruction: the operation doesn't have effect on columns that was previously stored with null value; I have 'tab1' defined as create table IF NOT EXISTS tab1(id VARCHAR not null, eventdate TIMESTAMP not null, category VARCHAR, history VARBINARY); UPSERT into tab1(id,eventdate,category,history) values ('id1',to_timestamp('2022-10-20 00:00:00.0','test',null); now I just try to execute the following statement:UPSERT into tab1(id,eventdate,category,history) values ('id1',to_timestamp('2022-10-20 00:00:00.0','edited test','ABC'); this is the resulting table:SELECT * FROM tab1+-----+-----------------------+-------------+---------+| id | eventdate | category | history |+-----+-----------------------+-------------+---------+| id1 | 2022-10-20 00:00:00.0 | edited test | |+-----+--------------------
Hi Team, I have below HADOOP Server details in our environment.#1 Node Cluster working - >Nodemanagers:166>Datanodes:159#2 64 Cores per Node#3 503 GB RAM per node. According to above node and core details"I WANT TO SET SPARK SUBMIT below parameter--driver-memory --driver-cores --num-executors--executor-memory --executor-cores " for that Please suggest me how to calculate it and also please share the calculation logic for the same. Also #2 question is, In shell script we are calling the .py Python code using given spark parameter as - spark-submit--conf spark.maxRemoteBlockSizeFetchToMem=2G--conf hive.exec.dynamic.partition=true--conf hive.enforce.bucketing=true--conf hive.exec.dynamic.partition.mode=nonstrict--master yarn--deploy-mode client--driver-memory 30G --driver-cores 4 --num-executors 99--executor-memory 40G --executor-cores 4 --conf spark.sql.shuffle.partitions=800--conf spark.shuffle.compress=true--conf s
I believe below mentioned CVEs are either addressed or fixed through patching in CDH 6.3.4 -CVE-2021-4104 (Log4j1) - as per this article, CDH user doesn't need to do anything to fix this vulnerability.CVE-2021-44228 (Log4j2) - as per this article, patches are available for this vulnerability for CDH 6.3.4.But apart from above vulnerabilities, there are few more vulnerabilities of critical, high and moderate severity in Log4j1 and Log4j2 which are - Log4j1 - https://logging.apache.org/log4j/1.2/index.htmlCVE-2019-17571 is a high severity issue targeting the SocketServer.CVE-2022-23302 is a high severity deserialization vulnerability in JMSSink.CVE-2022-23305 is a high serverity SQL injection flaw in JDBCAppender that allows the data being logged to modify the behavior of the component.CVE-2022-23307 is a critical severity against the chainsaw component in Log4j 1.x.Log4j2 - https://logging.apache.org/log4j/2.x/security.htmlCVE-2021-4504
I was trying to control apache nifi processors to start/stop by calling apache nifi-api. I am using Postman to make the call, for example to start a GenerateFlowFile processor:https://localhost:8443/processors/{processor id} (PUT Request)But I got the response status 403(forbidden) as shown in the image below. I was able to make a get request. I have also a client self-signed certificate generated using nifi-toolkit by following this https://kylo.readthedocs.io/en/v0.10.0/security/ConfigureNiFiWithSSL.html from nifi.properties file:ifi.security.autoreload.enabled=falsenifi.security.autoreload.interval=10 secsnifi.security.keystore=./conf/kylo/localhost/keystore.jksnifi.security.keystoreType=jksnifi.security.keystorePasswd=.............nifi.security.keyPasswd=.............nifi.security.truststore=./conf/kylo/localhost/truststore.jksnifi.security.truststoreType=jksnifi.secur
After restarting all the services trough ambari, only two services don't start and are facing the same error : ranger and infrasolr /usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")resource_management.core.exceptions.Fail: Configuration parameter 'kerberos-env' was not found in configurations dictionary! Any idea ?Thanks
While converting the input flow file JSON to any other format using query record (CSVwriter or AVRO Writer) using inferschema strategy the NIFI processor is trying to convert to Date based on first few characters of the incoming string.NIFI Error OutputSuccessfully parsed a JSON object from input but failed to convert into a Record object with the given schema\n- Caused by: org.apache.nifi.serialization.MalformedRecordException: Successfully parsed a JSON object from input but failed to convert into a Record object with the given schema\n- Caused by: org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Failed Conversion of Field [sharepoint_documents_name__c] from String [10-7-20 2x10GE LAN PHY XXXXXXXXXXXX] to LocalDate\n- Caused by: java.time.format.DateTimeParseException: Text '10-7-20 2x10GE LAN PHY XXXXXXXXXXX' could not be parsed at index 0: Any workaround/fixes available in the newer version. Iam using NIFI 16.1
Hi Team, Can you please explain me how to interpret 'Cluster CPU' graph from CM.
We have an MYSQL HA cluster,In case of failover, for the secondary master to take the place of the failed master, the hive metastore connection string configuration would need to be changed to point to the new master, and the services restarted. is it possible to have true HA on the Hive Metastore without changing the connection string manually ?
Log files below--------------------/var/log/cloudera-scm-server/cloudera-scm-server.log 2022-11-10 07:12:56,046 WARN C3P0PooledConnectionPoolManager[identityToken->1hgex1yasqqqybleeujro|abff8b7]-HelperThread-#0:com.mchange.v2.resourcepool.BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@74294c1a is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests.2022-11-10 07:12:56,046 WARN C3P0PooledConnectionPoolManager[identityToken->1hgex1yasqqqybleeujro|abff8b7]-HelperThread-#2:com.mchange.v2.resourcepool.BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@5c69b2c -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:com.mysql.jdbc.exceptions.jdbc4.CommunicationsExcep
I am trying to create several tables with hundreds of millions of columns.Tables are created and data is added while changing the orc.stripe.size value (64MB or 256MB) for each table because the size of each table is different. (If there is a lot of table data, an error occurred when setting 64MB, so the value was increased to 256MB.)However, if the orc.stripe.size value is set large (ex. 256MB) for a table with a relatively small number of data, the following error occurs.I don't know how to set this value and create it. Is there a way to set the value according to the table size? SessionState: Vertex failed, vertexName=Map 1, vertexId=vertex_1666932355626_0180_236_01, diagnostics=[Task failed, taskId=task_1666932355626_0180_236_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1666932355626_0180_236_01_000000_0:java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOE
How to resove given ERROR - ERROR - Unknown Exception:07<class 'py4j.protocol.Py4JJavaError'>An error occurred while calling o1140.insertInto.: org.apache.spark.SparkException: Job aborted.
Hello team, my workflow has been failing for several days in SSH actions randomly.the error found in the SSH logs is: Causé par : GSSException : aucune information d'identification valide fournie (niveau de mécanisme : TGT a été révoqué (20)) sur sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:772) sur sun.security.jgss.GSSContextImpl.initSecContext (GSSContextImpl.java:248) sur sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) sur com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ... 45 plus Causé par : KrbException : TGT a été révoqué (20) sur sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:70) sur sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:226) sur sun .security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:237)à sun.security.krb5.internal.CredentialsUtil.serviceCredsSingle(CredentialsUtil.java:482) à sun.security.krb5.internal.CredentialsUtil.serviceCreds(C
I am facing 641 MainThread agent ERROR Heartbeating to m:7182 failed. Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1430, in _send_heartbeat self.cfg.max_cert_depth) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 185, in _init_ self.conn.connect() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect sock.connect((self.host, self.port)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect ret = self.connect_ssl() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl return m2.ssl_connect(self.ssl, self._timeout)SSLError: certificate verify failed.
i am using hive and tez.whenever i am performing insert query its returns following error: execution error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.teztask
Hello,I could not split a JSON due to a special ETX character existing in a JSON value, SplitJson processor returns an error that it is not a Valid JSON.The JSON looks like that:[ {"key_data":"val_data"}, {"key_data":"val_ETXdata"}, {"key_data":"val_data"}, {"key_data":"val_data"},... ] I used The next regex expression to replace it but it doesn't work:\x03(?=[^"]*")also[\x03](?=[^"]*") can someone show me a trick to remove it please, I would appreciate it.
I understand NiFi stores its FlowFile attributes in memory and large attributes are not recommended because it takes up space in the JVM. What I am having trouble grasping is the consequences. Can someone explain different use cases what could happen and also, how long those attributes are stored in memory for?
Hello everyone, I'm tinkering with a test cluster. Test cluster has the trial license and so far I have installed HDFS, Oozie, Yarn, Spark and Zookeeper (all services are on one node). I tried to install Yarn Queue Manager, but when starting the YARN Queue Manager Webapp service, it errors out with CM Internal user credentials are not available as system environment variables. Please make sure that both CM_USER and CM_PASSWORD are present as environment variables for the application to work correctly. Currently I have Cloudera Manager:Version: Cloudera Enterprise Trial 7.1.4Java VM Name: OpenJDK 64-Bit Server VMJava Version: 1.8.0_232 and when installing the cluster, I could only choose from Cloudera Runtime 7.1.8, no other choices. Are there any possible fixes for the issue?
Hello and thanks for reading I have an issue with hadoop hbase while trying to find the issue I can see below memstore flush logs for all tables and its non stop MemstoreFlusherChore requesting flush of table 1. because K has an old edit so flush to free WALs after random delay 65889ms 022-10-29 12:22:13,176 INFO [regionserver/uhn-hw-fores201b:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of table 1. because K has an old edit so flush to free WALs after random delay 65889ms2022-10-29 12:22:13,177 INFO [regionserver/uhn-hw-fores201b:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of tabl 2 . because K has an old edit so flush to free WALs after random delay 181952ms2022-10-29 12:22:13,177 INFO [regionserver/uhn-hw-fores201b:16020.Chore.1] regionserver.HRegionServer: MemstoreFlusherChore requesting flush of table 3 . because K has an old edit so flush to free WALs after random delay 150462ms2
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.