Q&A

We use the Cloudera ODBC Driver for Impala from c# .Net using an OdbcConnection.  We are using MIT Kerberos on Windows and keytab authentication.  Occasionally, the process begins throwing the bellow Exception.  Over time, the exception seems to correct itself, however the only known fix to us is to restart the process (which is a problem..) Exception:[Cloudera][ThriftExtension] (3) Error occurred while contacting server: ETIMEDOUT. The connection has been configured to use a SASL mechanism for authentication. This error might be due to the server is not using SASL for authentication. Connection String:Driver={Cloudera ODBC Driver for Impala};Host=[ImpalaHostName];Port=21050;AuthMech=1;KrbRealm=[KerberosRealm];KrbServiceName=impala;UseKeytab=1;KrbFQDN=_HOST;UID=[UserName];UPNKeytabMappingFile=[jsonKeyTabMappingFile];UseNativeQuery=1;CurrentSchemaRestrictedMetadata=1;Schema=[Database];SocketTimeout=120 We have also occasionally seen the bellow exceptio

Hi,I have 6 disks installed on each server with 3 data nodes. I am having 3 disks of 1TB and 3 disk of 4TB. Now what is happening that the smaller disk 1 TB are getting full frequently and Impala stops working. Is there a way that I can replicate those disks with some percent of free space?Like none of the 6 disk  should exceed 80% of usage until the balancer completes.

Hello   I have a Clouder Manager 4.8 cluster with Kerberos enabled   When I configured HDFS/Hbase/Zookeeper to use Kerberos, Cloudera Manager generated all the principals and keytabs for all the services on each node   However, I have two unmanaged namenodes that I want to use as part of my cluster - Cloudera Manager does not know about these nodes, so it did not generate the principals and keytabs for the services on these nodes   I followed the steps for generating the keytabs/principals as part of the manual CDH setup, but the guide told me to move my keys tabs to /etc/hadoop/conf, but there are no other keytabs in that folder   So I need to know where does Cloudera Manager store its generated keytabs   Or am I supposed to merge the manually created keytabs with the generated keytabs somehow?   Thanks for any help in advance

Hi,   I have enabled kerberos in my cloudera cluster using Cloudera Manager. Kerberized cluster is running fine. But I'm unable to get the location of keytab files (eg. hdfs.keytab, yarn.keytab etc). The keytab location property is also missing from the *-site.xml files.   Where can I get the location of those keytab files.   Regards, Amit

Hi. I am trying to write a text file from execute script with python. Whenever the processor executes, it creates a locked text file. How can I overcome this problem?Following is the codefrom time import gmtime, strftimeimport datetimenow = datetime.datetime.now()current_time = now.strftime("%H:%M:%S")with open('**path to directory**/time.txt', "a") as f:current_time = str(current_time) + ('\n') f.write(current_time)f.close()

How can I see the memory % that is used right now in Impala? Also I want to check the whole memory size

Hi All, I get the following issue when logging in to nifi UI , user success login but UI not showing NIFI Menu :2021-09-01 13:24:02,942 INFO [NiFi Web Server-1911] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[cdpadmindev@DOMAIN], groups[] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response. I'm using cloudera private base with kerberos and ADThe NIFI configuration is still the default, is there anything that needs to be changed? Thanks WB

Two night ago a server that contains iSCSI/lvm volumes had a crash. that server had three volumes from one Cloudera node that has HDFS NameNode, DataNode and JournalNode roles, beside other Cloudera/Hadoop roles. Those volumes contain all the data from all the roles that node has. I figure it out how to solve the volume problem and Kudu went Up (the data is in Kudu). But HDFS didn't, so Impala and other services are not running either.I've try a lot of possible solutions with no luck at all. Now CM says all three journal nodes are up, but none of the NameNodes are up because they can't connect to the Journal. This is the error I see:  2021-09-21 11:28:04,945 FATAL org.apache.hadoop.hdfs.server.namenode.FSEditLog: Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [10.241.109.57:8485, 10.241.109.81:8485, 10.241.109.76:8485], stream=null))java.io.IOException: Timed out waiting 120000ms for a quorum of nodes to respond. at org.apache.hadoo

Hi Everyone, I have hive partitions folder at HDFS location, but all the partitions folders are in upper case.i.e. YEAR=2021/MONTH=07/DAY=31/HOUR=00 like this. in hive when i am creating table it taking partition columns in lowercase/year=2021/month=07/day=31/hour=00 like this. since hdfs is case sensitive and hive is case insensitive so hive is expecting partitions column in lowercase at hdfs location and I am not able to see any partition in my hive table. so is there any way to handle this case.either hive column in uppercase or Recursively change all hdfs partitions column in lowercase. I have  8000+ partitions (1day=24 hours=30 days=12 months(24*30*12=8640)  for 1 year so not able to rename every folder manually. someone kindly suggest.

Dear,I am getting the below error on atlas role log, kindly support.how to know if the vertex_index exists? GraphBackedSearchIndexer.initialize() failed (GraphBackedSearchIndexer:376)org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://xxxxxxxxxxxxxx:8993/solr: Can not find the specified config set: vertex_index at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:668) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368) at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296) at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1143) at  Regards,

Ex: como gerar através deste Json abaixo e gerar 4 flow files?[ {"ID_WA_DESTINATARIO" : "5599999999999","DES_MENS_NMO" : "Você está recebendo anexa 1a via de sua fatura.","COD_IDEN_MENS_ENV_NMO" : "xptoqwerzxcv"}, {"ID_WA_DESTINATARIO" : "5599999999999","DES_MENS_NMO" : "Você está recebendo anexa 1a via de sua fatura.","COD_IDEN_MENS_ENV_NMO" : "xptoqwerzxcvb"}, {"ID_WA_DESTINATARIO" : "5599999999999","DES_MENS_NMO" : "Você está recebendo anexa 1a via de sua fatura.","COD_IDEN_MENS_ENV_NMO" : "xptoqwerzxcvc"}, {"ID_WA_DESTINATARIO" : "5599999999999","DES_MENS_NMO" : "Você está recebendo anexa 1a via de sua fatura.","COD_IDEN_MENS_ENV_NMO" : "xptoqwerzxcvd"} ] 

Dear all I have installed CDP private cloud on single node, Also i added hive, hive-on-tez and Hue services but unfortunately Hue not working with hive database and hiveserver2 of hive-on-tez stopping frequently.On hue UI this error is shown while choosing hive databaseCould not connect to any of [('xx.xx.xxx.xxx', 10000)] (code THRIFTTRANSPORT): TTransportException("Could not connect to any of [('xx.xx.xxx.xxx', 10000)]",) So what shall i do to work on hive using hue editor? thanks in advance 

Hi, I'm pretty new to NiFi and don't have much experience with it, so it's the reason I'm here. Yesterday, NiFi was running smoothly and for some reason it stopped working. Every time I tried to start : ./nifi.sh start or any other command it showed me "killed". Later on, I found that java was killed by OOM. Fixed this by restarting server, but it won't start. In the "nifi-bootstrap.log" I found that it's getting shut down due to this error:I haven't installed any packages, so don't know why it's giving me this error.Setup:Ubuntu 20openjdk 11.0.11nifi 1.13.2Connector: mysql-connector-java-8.0.25.jarWhat should I check? How to fix this error and start nifi?Thanks in advance.

Hi,Firstly, I make Hue LDAP integration and then Cloudera manager user interface administration->security enter  LDAP information  and cloudera-scm-server restart via console  I am reaching the Cloudera manager interface but it does not accept any users (admin-admin) that I have used before and LDAP users.  I am getting a username password error for all users and passwords.I try to reset username &password via below link but it did not work. http://mynotesonhadoop.blogspot.com/2017/07/reset-admin-password-on-cloudera-manager.html  I can see below logs in  /var/log/cloudera-scm-server/cloudera-scm-server.log Caused by: org.springframework.ldap.BadLdapGrammarException: Failed to parse DN; nested exception is org.springframework.ldap.core.TokenMgrError: Lexical error at line 1, column 1. Encountered: "(" (40), after : ""at org.springframework.ldap.core.DistinguishedName.parse(DistinguishedName.java:232)at org.springframework.ld

Hello, Wanted to know what features/components in CM/CDH 6.3.3 would stop working once the enterprise license is expired. Thanks Wert

I've been trying to run an SQL that comes from a GenerateTableFetch processor, so when the SQL will be executed the following error is thrown:ExecuteSQL[id=ba6f39d7-4ea4-19bd-984a-1b3176886957] Unable to execute SQL select query SELECT * FROM MY_SPATIAL_TABLE WHERE 1=1 for StandardFlowFileRecord[uuid=fafa1a55-5345-4588-b0be-9e216fc86099,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1631702321511-1, container=default, section=1], offset=688879, length=46],offset=0,name=0e20eaaa-f1a7-4ce8-8cf2-e4a7858c6adc,size=46] due to java.lang.IllegalArgumentException: createSchema: Unknown SQL type 2002 / MDSYS.SDO_GEOMETRY (table: NiFi_ExecuteSQL_Record, column: GEOM) cannot be converted to Avro type; routing to failure: java.lang.IllegalArgumentException: createSchema: Unknown SQL type 2002 / MDSYS.SDO_GEOMETRY (table: NiFi_ExecuteSQL_Record, column: GEOM) cannot be converted to Avro type↳ causes: org.apache.nifi.processor.exception.ProcessException: java.lang.IllegalArgument

We noticed for few of the hive tables, during the  maintenance window, just after the cluster restart, impala does not show the data accurately  but it shows properly in hive. Once we perform invalidate metadata and refresh on the underlying table, it starts working fine. It was okay to use the work around but this is happening every month consistently after the cluster restart and so we needed to find the root cause and solution. Surprisingly issue is reported for only two tables, one of them is view. These are external hive table with parquet format.  How to investigate the issue any further? 

How can we get 1 MD5 value for a hive table? Do not need a MD5 value for each record in the table.  Is there a way to query the MD5() at a table level. Thanks

Hello All, Can some one please help me fix this OOM error||heap issue , while running query from hive Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "client DomainSocketWatcher"-----------Map1 - stays in initializing state and not moving to running state with OOM errorERROR: java.sql.SQLExceptionPyRaisable: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Application application_1616507270167_1067 failed 2 times due to AM Container for appattempt_1616507270167_1067_000002 exited with  exitCode: 255Failing this attempt.Diagnostics: [2021-09-07 09:49:05.213]Exception from container-launch.Container exited with a non-zero exit code 255. Error file: prelaunch.err.Exit code: 255 --------------------------------------------------------------------------java.sql.SQLExceptionPyRaisable        &

Hello, We have one oozie server in our lower environment and I was assigned to add one more oozie server. I added one oozie server in ambari but  oozie server installation stuck due to keytab/kerberos issues. Error--> Execution of '/usr/bin/kinit -l 5m20s -c /var/lib/ambari-agent/tmp/oozie_alert_cc_7579 -kt /etc/security/keytabs/oozie.service.keytab oozie/FQDN; ' returned 1. kinit: Client 'oozie/FQDN' not found in Kerberos database while getting initial credentials  Any help? How to I fix this issue to add one more oozie server? Thank in advance.      

Hello,   For an application, I need to extract the maximum depth from an hdfs directory. I know how to do this action in shell: we can execute         find /tmp -type d -printf '%d\n' | sort -rn | head -1         So I wanted to do the same with the find function of hdfs:         hdfs dfs -find /tmp -type d           but the -type argument does not exist on hdfs, here is the error:         find: Unexpected argument: -type           Does anyone have any solution or advice for this problem ? ps: my hadoop version Hadoop 2.6.0-cdh5.13.   regards, thanks in advance

when i install cm agent using cloudera manager,got "Installation failed. Failed to copy installation files." click the ditails:Failed to copy installation files. View/tmp/scm_prepare_node.SkukORUR Could not verify key bundle check the log/var/log/cloudera-scm-server# tail -f cloudera-scm-server.log2019-02-27 21:26:32,543 INFO scm-web-227:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Install on ip-172-22-1-148.2019-02-27 21:26:32,543 INFO scm-web-227:com.cloudera.server.cmf.node.NodeConfiguratorService: Adding password-based configurator for ip-172-22-1-1482019-02-27 21:26:32,544 INFO scm-web-227:com.cloudera.server.cmf.node.NodeConfiguratorService: Submitted configurator for ip-172-22-1-148 with id 692019-02-27 21:26:32,547 INFO NodeConfiguratorThread-68-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: ip-172-22-1-148: Transitioning from INIT (PT0.004S) to CONNECT2019-02-27 21:26:32,548 INFO NodeConfiguratorThread-68-0:net.schmizz.sshj.transport.TransportI

Dear team, We are getting the below error in the CDP - 7.1.6 HiveServer logs. Can you please share what's the cause of this issue and any possible solution?  2021-09-03 13:12:25,571 ERROR org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-16886]: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.IOException: Fail to get checksum, since file /warehouse/tablespace/managed/hive/xxxxx/xxxxx/xxxxx/xxxxx/delta_0000003_0000003_0000/xxxxx.xxxxx is under construction.2021-09-03 13:12:25,571 INFO org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-16886]: Completed executing command(queryId=hive_20210903131225_70117bf2-c60f-4564-83e9-8a60be421f63); Time taken: 0.12 seconds2021-09-03 13:12:25,572 INFO org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-16886]: OK2021-09-03 13:12:25,572 INFO org.apache.hadoop.hive.ql.lockmgr.DbTxnManager: [HiveServer2-Background-Pool: Thread-1

Hi, I am looking to install nifi on 3 clusters, and below is the loop which kept on running: 2021-09-16 13:34:35,324 INFO [Curator-Framework-0] o.a.c.f.state.ConnectionStateManager State change: SUSPENDED2021-09-16 13:34:36,836 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected Leader for role 'Cluster Coordinator' due to org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /nifi/leaders/Cluster Coordinator; assuming no leader has been elected2021-09-16 13:34:36,837 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting2021-09-16 13:34:37,438 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role.2021-09-16 13:34:37,439 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager CuratorLeaderElectionManager[stopped=true] Registered new Leader Selector for role Cluster

HI I have exported a table from a hadoop envrionment using the following command:export table department to 'hdfs_exports_location/department'; I tried to import the same table into another hadoop environment using the command: import from 'hdfs_exports_location/department';i get the following error:Error: Error while compiling statement: FAILED: SemanticException [Error 10027]: Invalid path (state=42000,code=10027)i tried usingimport table imported_dept from 'hdfs_exports_location/department';i get the following error:Error: Error while compiling statement: FAILED: SemanticException [Error 10324]: Import Semantic Analyzer Error (state=42000,code=10324)Any idea what could be the issue?i am using hive 3.1.0.3.0.1.0-187.Thank you