Solved

Failed to regenerate the missing credentials in cm, the error log is as follows:Key log information:com.cloudera.cmf.command.CmdExecException: java.lang.IllegalArgumentExceptionCaused by: java.lang.IllegalArgumentExceptionERROR CommandPusher-1:com.cloudera.cmf.model.DbCommand: Command 61246(GenerateCredentials) has completed. finalstate: CANCELLED,success:false,msg:java.lang.IllegalArgumentException It is worth noting that the import of kerberos account manager credentials in CM is successful.And the keytab in the /var/run/cloudera-scm-server directory can be kinit successfully /var/log/cloudera-scm-server/cloudera-scm-server.log The log error is as follows:Key log information:ERROR Staleness-Detector-1: com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Failed to check staleness for service client configs : DbService{id=75,name=atlas} com.cloudera.cmf.service.config.ConfigGenException: Failed to generate client configs Caused by: java.la

We are migrating from Hortonworks (HDP) to Cloudera (CDP) and having difficulties connecting to the hbase.The following code works when using a HDP host:   from thrift.transport import TSocket, TTransport, THttpClientfrom thrift.transport.TSSLSocket import TSSLSocketfrom thrift.protocol import TBinaryProtocol, TCompactProtocolimport app.hbased.Hbase as Hbasehost = "hdp.mycompany.com"port = 9090sock = TSocket.TSocket(host, port)saslServiceName = "hbase"transport = TTransport.TSaslClientTransport(sock, host, saslServiceName, mechanism="GSSAPI")protocol = TBinaryProtocol.TBinaryProtocol(transport)client = Hbase.Client(protocol)transport.open()table_names = client.getTableNames()print table_names   However, when changing the host to a CDP server:host = "cdp.mycompany.com"we get the following error: Traceback (most recent call last):File "C:\Users\myuser\apps\app\test\test.py", line 20, in <module>table_names = client.getTableNames()File "C:\Users\my

I am trying to learn the basics in cloudera and Impala and by reading this documentation :https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_cluster_util_custom.html#concept_jp4_4bh_hxI saw that with tsquery I can monitor Impala. I want to ask :1) where can I run these kind of queries?2) where can I find the attributes like category and clusterName in cloudera? Thanks in advance.

Hi,We have recently made our external ZooKeeper's implement SSL, Now Nifi is failing to startup.  2021-09-23 15:46:01,711 ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave uporg.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:972) at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:943) at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66) at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut

Hello, We are running single node nifi in 8 core machine and configured 'Maxmum Timer driven thread count' as 300.Once in a while Nifi get totally stuck (wont accept flowfiles) and also no app log will generate till we restart nifi manually. So wondering if its due to nifi thread starvation... In below article 'NiFi Thread starvation' section its mention that system will stuck in resource competition cycle.https://community.cloudera.com/t5/Community-Articles/NiFi-HDF-Dataflow-Optimization-Part-2-of-2/ta-p/245113 So any idea if nifi needs more threads than system available core then will there be any chance of nifi to get stuck ?  @Wynner   Thanks,Mahendra

We have a job that was running fine in 7.1.4, but is now failing after we went to 7.1.6.  It fails with permission issues writing to a directory in /user/yarn.  The job runs as a different user, and we are wondering why it started writing what looks like temporary output to a directory under /user/yarn, that the user running job doesn't have access to?

Hi,I have 6 disks installed on each server with 3 data nodes. I am having 3 disks of 1TB and 3 disk of 4TB. Now what is happening that the smaller disk 1 TB are getting full frequently and Impala stops working. Is there a way that I can replicate those disks with some percent of free space?Like none of the 6 disk  should exceed 80% of usage until the balancer completes.

Hello   I have a Clouder Manager 4.8 cluster with Kerberos enabled   When I configured HDFS/Hbase/Zookeeper to use Kerberos, Cloudera Manager generated all the principals and keytabs for all the services on each node   However, I have two unmanaged namenodes that I want to use as part of my cluster - Cloudera Manager does not know about these nodes, so it did not generate the principals and keytabs for the services on these nodes   I followed the steps for generating the keytabs/principals as part of the manual CDH setup, but the guide told me to move my keys tabs to /etc/hadoop/conf, but there are no other keytabs in that folder   So I need to know where does Cloudera Manager store its generated keytabs   Or am I supposed to merge the manually created keytabs with the generated keytabs somehow?   Thanks for any help in advance

Hi,   I have enabled kerberos in my cloudera cluster using Cloudera Manager. Kerberized cluster is running fine. But I'm unable to get the location of keytab files (eg. hdfs.keytab, yarn.keytab etc). The keytab location property is also missing from the *-site.xml files.   Where can I get the location of those keytab files.   Regards, Amit

Hi. I am trying to write a text file from execute script with python. Whenever the processor executes, it creates a locked text file. How can I overcome this problem?Following is the codefrom time import gmtime, strftimeimport datetimenow = datetime.datetime.now()current_time = now.strftime("%H:%M:%S")with open('**path to directory**/time.txt', "a") as f:current_time = str(current_time) + ('\n') f.write(current_time)f.close()

How can I see the memory % that is used right now in Impala? Also I want to check the whole memory size

Two night ago a server that contains iSCSI/lvm volumes had a crash. that server had three volumes from one Cloudera node that has HDFS NameNode, DataNode and JournalNode roles, beside other Cloudera/Hadoop roles. Those volumes contain all the data from all the roles that node has. I figure it out how to solve the volume problem and Kudu went Up (the data is in Kudu). But HDFS didn't, so Impala and other services are not running either.I've try a lot of possible solutions with no luck at all. Now CM says all three journal nodes are up, but none of the NameNodes are up because they can't connect to the Journal. This is the error I see:  2021-09-21 11:28:04,945 FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [10.241.109.57:8485, 10.241.109.81:8485, 10.241.109.76:8485], stream=null))java.io.IOException: Timed out waiting 120000ms for a quorum of nodes to respond. at org.apache.hadoo

Dear,I am getting the below error on atlas role log, kindly support.how to know if the vertex_index exists? GraphBackedSearchIndexer.initialize() failed (GraphBackedSearchIndexer:376)org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://xxxxxxxxxxxxxx:8993/solr: Can not find the specified config set: vertex_index at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:668) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368) at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296) at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1143) at  Regards,

Ex: como gerar através deste Json abaixo e gerar 4 flow files?[ {"ID_WA_DESTINATARIO" : "5599999999999","DES_MENS_NMO" : "Você está recebendo anexa 1a via de sua fatura.","COD_IDEN_MENS_ENV_NMO" : "xptoqwerzxcv"}, {"ID_WA_DESTINATARIO" : "5599999999999","DES_MENS_NMO" : "Você está recebendo anexa 1a via de sua fatura.","COD_IDEN_MENS_ENV_NMO" : "xptoqwerzxcvb"}, {"ID_WA_DESTINATARIO" : "5599999999999","DES_MENS_NMO" : "Você está recebendo anexa 1a via de sua fatura.","COD_IDEN_MENS_ENV_NMO" : "xptoqwerzxcvc"}, {"ID_WA_DESTINATARIO" : "5599999999999","DES_MENS_NMO" : "Você está recebendo anexa 1a via de sua fatura.","COD_IDEN_MENS_ENV_NMO" : "xptoqwerzxcvd"} ] 

Hi, I'm pretty new to NiFi and don't have much experience with it, so it's the reason I'm here. Yesterday, NiFi was running smoothly and for some reason it stopped working. Every time I tried to start : ./nifi.sh start or any other command it showed me "killed". Later on, I found that java was killed by OOM. Fixed this by restarting server, but it won't start. In the "nifi-bootstrap.log" I found that it's getting shut down due to this error:I haven't installed any packages, so don't know why it's giving me this error.Setup:Ubuntu 20openjdk 11.0.11nifi 1.13.2Connector: mysql-connector-java-8.0.25.jarWhat should I check? How to fix this error and start nifi?Thanks in advance.

Hello, Wanted to know what features/components in CM/CDH 6.3.3 would stop working once the enterprise license is expired. Thanks Wert

I've been trying to run an SQL that comes from a GenerateTableFetch processor, so when the SQL will be executed the following error is thrown:ExecuteSQL[id=ba6f39d7-4ea4-19bd-984a-1b3176886957] Unable to execute SQL select query SELECT * FROM MY_SPATIAL_TABLE WHERE 1=1 for StandardFlowFileRecord[uuid=fafa1a55-5345-4588-b0be-9e216fc86099,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1631702321511-1, container=default, section=1], offset=688879, length=46],offset=0,name=0e20eaaa-f1a7-4ce8-8cf2-e4a7858c6adc,size=46] due to java.lang.IllegalArgumentException: createSchema: Unknown SQL type 2002 / MDSYS.SDO_GEOMETRY (table: NiFi_ExecuteSQL_Record, column: GEOM) cannot be converted to Avro type; routing to failure: java.lang.IllegalArgumentException: createSchema: Unknown SQL type 2002 / MDSYS.SDO_GEOMETRY (table: NiFi_ExecuteSQL_Record, column: GEOM) cannot be converted to Avro type↳ causes: org.apache.nifi.processor.exception.ProcessException: java.lang.IllegalArgument

Hello All, Can some one please help me fix this OOM error||heap issue , while running query from hive Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "client DomainSocketWatcher"-----------Map1 - stays in initializing state and not moving to running state with OOM errorERROR: java.sql.SQLExceptionPyRaisable: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Application application_1616507270167_1067 failed 2 times due to AM Container for appattempt_1616507270167_1067_000002 exited with  exitCode: 255Failing this attempt.Diagnostics: [2021-09-07 09:49:05.213]Exception from container-launch.Container exited with a non-zero exit code 255. Error file: prelaunch.err.Exit code: 255 --------------------------------------------------------------------------java.sql.SQLExceptionPyRaisable        &

Hello,   For an application, I need to extract the maximum depth from an hdfs directory. I know how to do this action in shell: we can execute         find /tmp -type d -printf '%d\n' | sort -rn | head -1         So I wanted to do the same with the find function of hdfs:         hdfs dfs -find /tmp -type d           but the -type argument does not exist on hdfs, here is the error:         find: Unexpected argument: -type           Does anyone have any solution or advice for this problem ? ps: my hadoop version Hadoop 2.6.0-cdh5.13.   regards, thanks in advance

when i install cm agent using cloudera manager,got "Installation failed. Failed to copy installation files." click the ditails:Failed to copy installation files. View/tmp/scm_prepare_node.SkukORUR Could not verify key bundle check the log/var/log/cloudera-scm-server# tail -f cloudera-scm-server.log2019-02-27 21:26:32,543 INFO scm-web-227:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Install on ip-172-22-1-148.2019-02-27 21:26:32,543 INFO scm-web-227:com.cloudera.server.cmf.node.NodeConfiguratorService: Adding password-based configurator for ip-172-22-1-1482019-02-27 21:26:32,544 INFO scm-web-227:com.cloudera.server.cmf.node.NodeConfiguratorService: Submitted configurator for ip-172-22-1-148 with id 692019-02-27 21:26:32,547 INFO NodeConfiguratorThread-68-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: ip-172-22-1-148: Transitioning from INIT (PT0.004S) to CONNECT2019-02-27 21:26:32,548 INFO NodeConfiguratorThread-68-0:net.schmizz.sshj.transport.TransportI

Dear team, We are getting the below error in the CDP - 7.1.6 HiveServer logs. Can you please share what's the cause of this issue and any possible solution?  2021-09-03 13:12:25,571 ERROR org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-16886]: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get checksum, since file /warehouse/tablespace/managed/hive/xxxxx/xxxxx/xxxxx/xxxxx/delta_0000003_0000003_0000/xxxxx.xxxxx is under construction.2021-09-03 13:12:25,571 INFO org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-16886]: Completed executing command(queryId=hive_20210903131225_70117bf2-c60f-4564-83e9-8a60be421f63); Time taken: 0.12 seconds2021-09-03 13:12:25,572 INFO org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-16886]: OK2021-09-03 13:12:25,572 INFO org.apache.hadoop.hive.ql.lockmgr.DbTxnManager: [HiveServer2-Background-Pool: Thread-1

HI I have exported a table from a hadoop envrionment using the following command:export table department to 'hdfs_exports_location/department'; I tried to import the same table into another hadoop environment using the command: import from 'hdfs_exports_location/department';i get the following error:Error: Error while compiling statement: FAILED: SemanticException [Error 10027]: Invalid path (state=42000,code=10027)i tried usingimport table imported_dept from 'hdfs_exports_location/department';i get the following error:Error: Error while compiling statement: FAILED: SemanticException [Error 10324]: Import Semantic Analyzer Error (state=42000,code=10324)Any idea what could be the issue?i am using hive 3.1.0.3.0.1.0-187.Thank you

I am trying to parse a nested json document using RDD rather than DataFrame. The reason I cannot use DataFrame (the typical code is like spark.read.json) is that the document structure is very complicated. The schema detected by the reader is useless because child nodes at the same level have different schemas. So I try the script below. import jsons='{"key1":{"myid": "123","myname":"test"}}'rdd=sc.parallelize(s).map(json.loads) My next step will be using map transformation to parse json string but I do not know where to start. I tried the script below but it failed. rdd2=rdd.map(lambda j: (j[x]) for x in j) I would appreciate any resource on using RDD transformation to parse json.

We are trying to implement best practices in regards export/import via NiFi Registry. Currently we have one DEV site and multiple operational sites, ideally we would export from DEV and import into our various OPS sites. With that in mind I have a few questions:   What is the best way to provide site-specific configuration for the controllers/Processors? Should we have a unique properties file for each site and we select which one to deploy per site? According to some documentation, later versions of NiFi are planning to remove support for this kind of unique property file loading. Is there a way in the controllers/Processors to provide conditional values based on deployed site? Of particular concern is our Solr "Zookeeper Host List" This is a property value on our Query and Ingest processors, and they currently aren't enabled to support the Variable Registry, so the value is explicitly provided (no variable lookup/replacement). Do we need to set up and use a "Variable Re

On CDP we are using both the Hive (for the Hive Metastores) and Hive on Tez (for the HiveServers). We are getting the below error while trying to run a query based on a condition. I can't share the table information and the exact query but it looks something as below. CREATE EXTERNAL TABLE IF NOT EXISTS XXX ( `1` string, `6` varchar(30), `7` varchar(5), `8` varchar(10)) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES('hbase.columns.mapping'=':key, xx:1, xx:5, xx:6')TBLPROPERTIES ( 'hbase.table.name'='YYYYY');  the query looks as follows: select * from XXX where 8 = '1990-10-10';  And we see the below error from the HiveServer [a3ed3b7b-d225-43af-9ac0-76917911a742 HiveServer2-Handler-Pool: Thread-128-EventThread]: Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1d7573cd rejected from java.util.concurrent.ThreadPoolExecutor@194