@Skyler ,   There were no changes to archive.cloudera.com that we are aware of that should have brought this type of performance problem about.   I've done a few time curls of various sections of that repo, but of course, I am internal to our network and performance looks fine.   When this problem happens, could you please run a few   time curl https://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/pool/contrib/e/enterprise/   in addition to files in the archive like:   time curl https://archive.cloudera.com/cm5/ubuntu/precise/amd64/cm/pool/contrib/e/enterprise/cloudera-manager-server_5.9.3-1.cm593.p0.6~precise-cm5_all.deb   ping archive.cloudera.com  nslookup (or dig) archive.cloudera.com   and let us know what the response rates are?   One off the top of my head suggestions that I have is please make sure DNS resolution of archive.cloudera.com is working and that you don't have a broken DNS server in /etc/resolv.conf which is producing a DNS timeout lag everytime apt-get update gets a file. ... View more

Hi @mokkan ,   Oozie High Availability is an Active - Active type of high availability, with a load balancer in front of both servers to distribute the load:   https://oozie.apache.org/docs/5.0.0/AG_Install.html#High_Availability_HA ... View more

@Rak ,   I'm suspecting you are using Impala, because of the statement of the values returning null, but please confirm so I can properly label this question so others can see it.   Please see the following in the Impala documentation regarding the proper format for casting a string into a timestamp datatype:   https://docs.cloudera.com/documentation/enterprise/latest/topics/impala_timestamp.html   Restrictions: If you cast a   STRING   with an unrecognized format to a   TIMESTAMP , the result is   NULL   rather than an error. Make sure to test your data pipeline to be sure any textual date and time values are in a format that Impala   TIMESTAMP   can recognize. Currently, Avro tables cannot contain   TIMESTAMP   columns. If you need to store date and time values in Avro tables, as a workaround you can use a   STRING   representation of the values, convert the values to   BIGINT   with the   UNIX_TIMESTAMP()   function, or create separate numeric columns for individual date and time fields using the   EXTRACT()   function. Examples: The following examples demonstrate using   TIMESTAMP   values with built-in functions: select cast('1966-07-30' as timestamp); select cast('1985-09-25 17:45:30.005' as timestamp); select cast('08:30:00' as timestamp); select hour('1970-01-01 15:30:00'); -- Succeeds, returns 15. select hour('1970-01-01 15:30'); -- Returns NULL because seconds field required. select hour('1970-01-01 27:30:00'); -- Returns NULL because hour value out of range. select dayofweek('2004-06-13'); -- Returns 1, representing Sunday. select dayname('2004-06-13'); -- Returns 'Sunday'. select date_add('2004-06-13', 365); -- Returns 2005-06-13 with zeros for hh:mm:ss fields. select day('2004-06-13'); -- Returns 13. select datediff('1989-12-31','1984-09-01'); -- How many days between these 2 dates? select now(); -- Returns current date and time in local timezone.   ... View more

Hi @Rak ,   Could you please elaborate on what component you are specifically doing this (Hive, Impala, etc..) so we can properly label this question and answer accordingly? ... View more

Hi @sandy605,   Per our recent announcements at Strata last week, the Cloudera Data Platform has been released for AWS.  There is nothing download, other than the command line client to manage these installations in AWS.   CDP for on premise is still pending release.   I would suggest discussing this timeline and roadmap with sales at the link at the bottom of this page:   https://www.cloudera.com/products/cloudera-data-platform.html   Thanks, ... View more

I wanted to interject that while both the above are definite valid possible causes of Oozie jobs stuck in PREP state, there may be several other possible causes which may need to be resolved such as:   1.  Issues with the Yarn Resource Manager / MR Job Tracker, lack of resources either for the RM or queues for the user running the job. 2.  Problems with the Oozie server getting to the oozie database server, the database server itself, or locks on tables. 3.  Lack of resources to Oozie such as callable queues, java heap, GC thrashing, etc.   The above is a brief shortlist from review of support cases relating to Oozie jobs stuck in PREP.   I want to emphasize that deleting records from the Oozie database should be ONLY done the last resort to solving this problem, and only needed if you have a very large mass of oozie workflows that cannot be killed in a timely fashion by an oozie CLI script.   This should be only done at the direction of support, people knowledgeable with SQL, and the relationship between tables, columns, and rows in the oozie database as referential integrity and constraints are lacking in the schema design.  The above post from 2017 also missed one key table COORD_ACTIONS, where if this data was not properly cleaned up, would break your Oozie purge and possibly cause other serious problems. ... View more

Hi @mmcd ,   You would need to provide either hcat or hive2 credentials within your hive or hive2 action (respectively), as now that your cluster is kerberized, the oozie hive or hive2 action needs to authenticate:   https://oozie.apache.org/docs/4.2.0/DG_ActionAuthentication.html   Since you mentioned the Hue scheduler, here is some more information about Hue, it's a little dated (2014) and specific to only hcat, but the idea is to check the appropriate credential in the advanced properties of the action:   http://gethue.com/hadoop-tutorial-oozie-workflow-credentials-with-a-hive-action-with-kerberos/ ... View more

I would say this depends on if your hostname is set FQDN at time of CM/cluster install, if your hostname is set as fully qualified in the OS by the hostname command, or if you are still using TLS/SSL your certificates have the FQDN hostname.   Kerberos and several other things are very sensitive to DNS.   Run the hosts inspector under CM to check for DNS / host resolution problems.   Since you state you are not using DNS, what I would suggest is making sure your /etc/hosts file on all hosts contains all hosts of the cluster and CM, and are set as fully qualified hostname first, then aliased to the short hostname.  You can use rsync to keep this file accurate across the cluster.   Also make sure /etc/nsswitch.conf has files first for the hosts: line, and /etc/hosts will get used first.   Finally, if you supect the internal hostname was changed away from FQDN to short, either change it back or follow the following article to get CM configuration back in sync with what it was at time of install (Check the Name column beneath CM hosts tab to see what CM has in it's database):   https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ag_change_hostnames.html   [root@cm1 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.2.3 cm1.rgjustice.com cm1 192.168.2.4 node1.rgjustice.com node1 192.168.2.5 node2.rgjustice.com node2   [root@cm1 ~]# cat /etc/nsswitch.conf |grep hosts #hosts: db files nisplus nis dns hosts: files dns myhostname ... View more

Hi @bridgor , if we were just to take the N6apache6thrift9transport13TSSLException originally posted, this typically ends in a hostname mistmatch in the .pem files used / DNS problems.   Check statestore logs / web UI for issues as well.   I agree with @AcharkiMed that this seems to be network / SSL related. ... View more

As upsert is not SQL-92 syntax, the Impala JDBC driver does not currently support it.   However, we are currently testing a newer version of JDBC driver and hope to support upsert in the near future.  For the time being, please break these statements into a combination of insert and update statements. ... View more

Hello,   Most likely, either you need to kinit and use curl -k, if kerberos is enabled, and/or additionally specify the 11443 oozie port with https, if TLS/SSL is enabled.    With TLS/SSL, please remember to import the certificate into the cacerts file for the JDK so commands such as oozie and curl trust your oozie server's certificate.   https://blog.cloudera.com/blog/2013/06/how-to-use-the-apache-oozie-rest-api/ https://www.cloudera.com/documentation/enterprise/latest/topics/cm_sg_ssl_oozie.html https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cm_sg_tls_troubleshoot.html ... View more

Interesting.   I'll make sure I test on HA with OpenJDK.   Just to be sure as you mentioned HA, you did this unlink on all oozie servers behine your haproxy instance, correct? ... View more

My apologies as there were two additional jars in the final bugfix that I missed in my original suggestion.   Could you please also do the following in the /opt/cloudera/parcels/CDH/lib/oozie/embedded-oozie-server/dependency directory:   unlink ./jetty-runner-9.3.20.v20170531.jar unlink ./javax.servlet.jsp-api-2.3.1.jar   Restart the oozie server, and please let me know if your oozie web console then functioning.   ... View more

I confirmed this problem definitely does occur, even with fresh install.   Removing the link to javax.servlet.jsp-2.3.2.jar in the /opt/cloudera/parcels/CDH/lib/oozie/embedded-oozie-server/dependency directory, and restarting, does solve it in my test environment.  The below is for the latest - CDH 6.1.1.   We will be resolving this in the CDH 6.2 release.   Could you please confirm the link is gone, on the oozie server, and confirm the oozie server was restarted?    [root@host-10-17-100-72 dependency]# ls -ltr total 0 lrwxrwxrwx 1 root root 47 Feb 7 23:58 jetty-xml-9.3.20.v20170531.jar -> ../../../../jars/jetty-xml-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-util-9.3.20.v20170531.jar -> ../../../../jars/jetty-util-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-server-9.3.20.v20170531.jar -> ../../../../jars/jetty-server-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 44 Feb 7 23:58 javax.servlet-api-3.1.0.jar -> ../../../../jars/javax.servlet-api-3.1.0.jar lrwxrwxrwx 1 root root 35 Feb 7 23:58 commons-el-1.0.jar -> ../../../../jars/commons-el-1.0.jar lrwxrwxrwx 1 root root 46 Feb 7 23:58 jetty-io-9.3.20.v20170531.jar -> ../../../../jars/jetty-io-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-http-9.3.20.v20170531.jar -> ../../../../jars/jetty-http-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 55 Feb 7 23:58 jetty-annotations-9.3.20.v20170531.jar -> ../../../../jars/jetty-annotations-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 31 Feb 7 23:58 mail-1.4.7.jar -> ../../../../jars/mail-1.4.7.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 apache-jsp-9.3.20.v20170531.jar -> ../../../../jars/apache-jsp-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 53 Feb 7 23:58 jetty-util-ajax-9.3.20.v20170531.jar -> ../../../../jars/jetty-util-ajax-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 52 Feb 7 23:58 jetty-security-9.3.20.v20170531.jar -> ../../../../jars/jetty-security-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 javax.servlet.jsp-api-2.3.1.jar -> ../../../../jars/javax.servlet.jsp-api-2.3.1.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 taglibs-standard-spec-1.2.5.jar -> ../../../../jars/taglibs-standard-spec-1.2.5.jar lrwxrwxrwx 1 root root 51 Feb 7 23:58 jetty-rewrite-9.3.20.v20170531.jar -> ../../../../jars/jetty-rewrite-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 taglibs-standard-impl-1.2.5.jar -> ../../../../jars/taglibs-standard-impl-1.2.5.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-runner-9.3.20.v20170531.jar -> ../../../../jars/jetty-runner-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-plus-9.3.20.v20170531.jar -> ../../../../jars/jetty-plus-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-jndi-9.3.20.v20170531.jar -> ../../../../jars/jetty-jndi-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 47 Feb 7 23:58 apache-log4j-extras-1.2.17.jar -> ../../../../jars/apache-log4j-extras-1.2.17.jar lrwxrwxrwx 1 root root 38 Feb 7 23:58 apache-jsp-8.0.33.jar -> ../../../../jars/apache-jsp-8.0.33.jar lrwxrwxrwx 1 root root 57 Feb 7 23:58 org.eclipse.jdt.core-3.8.2.v20130121.jar -> ../../../../jars/org.eclipse.jdt.core-3.8.2.v20130121.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-webapp-9.3.20.v20170531.jar -> ../../../../jars/jetty-webapp-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 44 Feb 7 23:58 javax.servlet.jsp-2.3.2.jar -> ../../../../jars/javax.servlet.jsp-2.3.2.jar lrwxrwxrwx 1 root root 51 Feb 7 23:58 jetty-servlet-9.3.20.v20170531.jar -> ../../../../jars/jetty-servlet-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 49 Feb 7 23:58 apache-jstl-9.3.20.v20170531.jar -> ../../../../jars/apache-jstl-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 37 Feb 7 23:58 apache-el-8.0.33.jar -> ../../../../jars/apache-el-8.0.33.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-client-9.3.20.v20170531.jar -> ../../../../jars/jetty-client-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-jaas-9.3.20.v20170531.jar -> ../../../../jars/jetty-jaas-9.3.20.v20170531.jar [root@host-10-17-100-72 dependency]# unlink ./javax.servlet.jsp-2.3.2.jar   <RESTARTED OOZIE>   [root@host-10-17-100-72 dependency]# ls -l total 0 lrwxrwxrwx 1 root root 37 Feb 7 23:58 apache-el-8.0.33.jar -> ../../../../jars/apache-el-8.0.33.jar lrwxrwxrwx 1 root root 38 Feb 7 23:58 apache-jsp-8.0.33.jar -> ../../../../jars/apache-jsp-8.0.33.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 apache-jsp-9.3.20.v20170531.jar -> ../../../../jars/apache-jsp-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 49 Feb 7 23:58 apache-jstl-9.3.20.v20170531.jar -> ../../../../jars/apache-jstl-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 47 Feb 7 23:58 apache-log4j-extras-1.2.17.jar -> ../../../../jars/apache-log4j-extras-1.2.17.jar lrwxrwxrwx 1 root root 35 Feb 7 23:58 commons-el-1.0.jar -> ../../../../jars/commons-el-1.0.jar lrwxrwxrwx 1 root root 44 Feb 7 23:58 javax.servlet-api-3.1.0.jar -> ../../../../jars/javax.servlet-api-3.1.0.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 javax.servlet.jsp-api-2.3.1.jar -> ../../../../jars/javax.servlet.jsp-api-2.3.1.jar lrwxrwxrwx 1 root root 55 Feb 7 23:58 jetty-annotations-9.3.20.v20170531.jar -> ../../../../jars/jetty-annotations-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-client-9.3.20.v20170531.jar -> ../../../../jars/jetty-client-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-http-9.3.20.v20170531.jar -> ../../../../jars/jetty-http-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 46 Feb 7 23:58 jetty-io-9.3.20.v20170531.jar -> ../../../../jars/jetty-io-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-jaas-9.3.20.v20170531.jar -> ../../../../jars/jetty-jaas-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-jndi-9.3.20.v20170531.jar -> ../../../../jars/jetty-jndi-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-plus-9.3.20.v20170531.jar -> ../../../../jars/jetty-plus-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 51 Feb 7 23:58 jetty-rewrite-9.3.20.v20170531.jar -> ../../../../jars/jetty-rewrite-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-runner-9.3.20.v20170531.jar -> ../../../../jars/jetty-runner-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 52 Feb 7 23:58 jetty-security-9.3.20.v20170531.jar -> ../../../../jars/jetty-security-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-server-9.3.20.v20170531.jar -> ../../../../jars/jetty-server-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 51 Feb 7 23:58 jetty-servlet-9.3.20.v20170531.jar -> ../../../../jars/jetty-servlet-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-util-9.3.20.v20170531.jar -> ../../../../jars/jetty-util-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 53 Feb 7 23:58 jetty-util-ajax-9.3.20.v20170531.jar -> ../../../../jars/jetty-util-ajax-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-webapp-9.3.20.v20170531.jar -> ../../../../jars/jetty-webapp-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 47 Feb 7 23:58 jetty-xml-9.3.20.v20170531.jar -> ../../../../jars/jetty-xml-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 31 Feb 7 23:58 mail-1.4.7.jar -> ../../../../jars/mail-1.4.7.jar lrwxrwxrwx 1 root root 57 Feb 7 23:58 org.eclipse.jdt.core-3.8.2.v20130121.jar -> ../../../../jars/org.eclipse.jdt.core-3.8.2.v20130121.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 taglibs-standard-impl-1.2.5.jar -> ../../../../jars/taglibs-standard-impl-1.2.5.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 taglibs-standard-spec-1.2.5.jar -> ../../../../jars/taglibs-standard-spec-1.2.5.jar [root@host-10-17-100-72 dependency]# pwd /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/lib/oozie/embedded-oozie-server/dependency ... View more

Hi,   We have seen this with a few upgrades to CDH 6.X from CDH 5.X recently.   Could you please confirm if this was an upgrade (you had mentioned "just installed", so we would like to confirm)?   The solution for upgrades has been to:   1. Execute the below command on the Oozie server host command line unlink /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/oozie/embedded-oozie-server/dependency/javax.servlet.jsp-2.3.2.jar *note that we aim to only remove the link to the actual jar file 2. Restart Oozie server 3. Please check if the web ui comes up correctly.   This should be fixed in a future upgrade.   Thanks, Robert Justice ... View more

Hello,   At this current point, the Altus Data Engineering and Altus Data Warehouse platform does not support Google Compute Engines and Google Compute Storage.   Altus Director does provide these capabilities.    As of yet, support for GCE and GCS is not on the immediate roadmap, but I will add your interest for this support. ... View more

Hi,   I noticed you are using a quickstart VM, from your nameNode name ( hdfs : //quickstart.cloudera:8020) in your job.properties.   Most likely, when the oozie launcher launches and stays in RUNNING status, but no further additional hive job launches, yarn does not have enough resources to launch an additional job.  Please take a look at the Yarn Resource Manager role log and Yarn Resource Manager scheduler page for clues.   This could be due to lack of memory (AM, nodemanager, or scheduler), vcores, or several other factors in yarn tuning.   The quickstart VM is tuned for a very small demo environment.   You may need to add additional memory to the VM and/or cores, then tune yarn to have more resources for the second hive job from oozie to launch.   I will provide a link to a blog to aid you in this tuning below.      https://blog.cloudera.com/blog/2015/10/untangling-apache-hadoop-yarn-part-2/     ... View more

Please ensure that the hue, hive, impala, hue, solr, kakfa, and hbase group have not been removed from  sentry.service.admin.group, and that the hue, hive, impala, hue, hdfs, solr, kakfa, and hbase users have not been removed in sentry.service.allow.connect.  This is often the cause of the "Connection to sentry service denied due to lack of client credentials" exception.   Please see the following documentation:   https://www.cloudera.com/documentation/enterprise/latest/topics/hue_sec_sentry_auth.html#hue_sec_sentry_auth   ... View more

Just to confirm, have you enabled Impala to use the Sentry service in Cloudera Manager with the following instructions?   https://www.cloudera.com/documentation/enterprise/latest/topics/impala_authorization.html#sentry_cm ... View more

There is no new product name.  This will be a new feature within Sentry called Attribute Based Access Control using Navigator Lineage tags for metadata.  At present, there is no estimated release date, but we will be sure to announce that. ... View more

We are sorry to hear your cluster was compromised. Please see this recent security blog by Cloudera on malware attacks on un-secure Hadoop clusters [1]. Regardless of if you have a paid subscription, a distcp job can still be performed to another Hadoop cluster [2]. Please keep in mind that if the malware that has compromised your cluster has not yet been eradicated, and your target cluster is not secured with kerberos security as the blog mentions, there is a possibility your compromised cluster could be a source of attack to your target distcp cluster. As a last resort, you could possibly use hdfs dfs -copyToLocal for data that could fit into a local disk, then copy critical data elsewhere, in batches, prior to the remediation of the master node. [1] https://blog.cloudera.com/blog/2018/11/protecting-hadoop-clusters-from-malware-attacks/ [2] https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_admin_distcp_data_cluster_migrate.html [3] https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-common/FileSystemShell.html ... View more

As this is an edge node, Spark 2 is not supported in Oozie actions until CDH 6, and it sounds like these source files may be located specifically on this edgenode, I would suggest using the Oozie SSH action to SSH to this specific edgenode.   You would run a script in the SSH action to run the spark2-submit.  Please see:   https://oozie.apache.org/docs/4.1.0/DG_SshActionExtension.html ... View more

@uzubair,   Great to hear!  Thanks for getting back and letting us know. ... View more

Hi  @uzubair ,   Just touching base to see if you were able to resolve this issue, and if not, were you able to create a support case through through the Altus Web UI (Altus Web UI -> Support -> Support Center -> Technical Support;  Component = Altus Data Engineering;  Sub-Component = Clusters)?   Thanks, Robert Justice ... View more