Member since
08-14-2013
558
Posts
13
Kudos Received
8
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 3013 | 10-29-2019 11:47 AM | |
| 2971 | 07-15-2019 09:02 AM | |
| 7180 | 06-06-2019 07:56 AM | |
| 11439 | 02-28-2019 11:34 AM | |
| 10056 | 01-07-2019 01:08 PM |
10-29-2019
11:47 AM
2 Kudos
Hello @luisfeg,
The files are only listed in the security scan out if the cgroups feature is in effect (CM -> Hosts -> Configuration -> "Enable Cgroup-based Resource Management"). The default permission of the cgroup_event_control files are indeed world writable without any other permission bit set. However, this has not too much effect in accordance with this document which explains why cgroup files are typically world writable: https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
This seems to be created by the supervisor code cgroups feature, and Red Hat appears to have a subscription article on a similar concern elsewhere on the file system: https://access.redhat.com/solutions/377043. These files being world writable is an inherent part of Linux cgroups and users being able to use the cgroup API. Cloudera can only use this cgroup feature if the user configures it explicitly thru Cloudera Manager. As Red Hat states in the document mentioned above:
"Changing the world writable bit is not recommended on these files as it would break notification features of the cgroup API. Changing the permissions would have no effect. Cgroups has it's own pseudo-file-system and manages all the files therein. Each time the cgconfig init script is run, the world writable bit will be set on these files. The cgroup.event_control files are world writeable. This is because they are used in conjunction with the cgroups notification API to allow notifications to be sent about a changing status in the control group."
... View more
09-26-2019
11:15 AM
I wanted to interject that while both the above are definite valid possible causes of Oozie jobs stuck in PREP state, there may be several other possible causes which may need to be resolved such as:
1. Issues with the Yarn Resource Manager / MR Job Tracker, lack of resources either for the RM or queues for the user running the job.
2. Problems with the Oozie server getting to the oozie database server, the database server itself, or locks on tables.
3. Lack of resources to Oozie such as callable queues, java heap, GC thrashing, etc.
The above is a brief shortlist from review of support cases relating to Oozie jobs stuck in PREP. I want to emphasize that deleting records from the Oozie database should be ONLY done the last resort to solving this problem, and only needed if you have a very large mass of oozie workflows that cannot be killed in a timely fashion by an oozie CLI script. This should be only done at the direction of support, people knowledgeable with SQL, and the relationship between tables, columns, and rows in the oozie database as referential integrity and constraints are lacking in the schema design. The above post from 2017 also missed one key table COORD_ACTIONS, where if this data was not properly cleaned up, would break your Oozie purge and possibly cause other serious problems.
... View more
07-15-2019
09:02 AM
1 Kudo
Hi @Harish19 , the best place for information regarding TCP-DS tests on Impala would be (follow the README.md): https://github.com/cloudera/impala-tpcds-kit Once the data is populated in HDFS and tables are created, you likely can run most the same queries in tree/master/queries/ on Hive and/or Hive on Spark to test. IBM and Databricks have githubs with some SparkSQL tests, which you can Google for, but I have not personally evaluated them, or know if they work. Thanks,
... View more
06-06-2019
07:56 AM
1 Kudo
@Prav , This appears to have been listed as a bug (which is actually a longstanding limitation due to the definition of files and directories with _ and . being considered as "hidden" in FileInputFormat in Hadoop) of Hive since the 0.12 version: https://issues.apache.org/jira/browse/HIVE-6431 https://stackoverflow.com/questions/19830264/which-files-are-ignored-as-input-by-mapper If these files are needed to be seen, please consider using a pre-process script to rename them after loading. Thanks,
... View more
05-03-2019
02:25 PM
1 Kudo
I would say this depends on if your hostname is set FQDN at time of CM/cluster install, if your hostname is set as fully qualified in the OS by the hostname command, or if you are still using TLS/SSL your certificates have the FQDN hostname. Kerberos and several other things are very sensitive to DNS. Run the hosts inspector under CM to check for DNS / host resolution problems. Since you state you are not using DNS, what I would suggest is making sure your /etc/hosts file on all hosts contains all hosts of the cluster and CM, and are set as fully qualified hostname first, then aliased to the short hostname. You can use rsync to keep this file accurate across the cluster. Also make sure /etc/nsswitch.conf has files first for the hosts: line, and /etc/hosts will get used first. Finally, if you supect the internal hostname was changed away from FQDN to short, either change it back or follow the following article to get CM configuration back in sync with what it was at time of install (Check the Name column beneath CM hosts tab to see what CM has in it's database): https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ag_change_hostnames.html [root@cm1 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.2.3 cm1.rgjustice.com cm1 192.168.2.4 node1.rgjustice.com node1 192.168.2.5 node2.rgjustice.com node2 [root@cm1 ~]# cat /etc/nsswitch.conf |grep hosts #hosts: db files nisplus nis dns hosts: files dns myhostname
... View more
04-30-2019
02:13 PM
1 Kudo
Hi @bridgor , if we were just to take the N6apache6thrift9transport13TSSLException originally posted, this typically ends in a hostname mistmatch in the .pem files used / DNS problems. Check statestore logs / web UI for issues as well. I agree with @AcharkiMed that this seems to be network / SSL related.
... View more
02-28-2019
11:34 AM
This worked for me with Oozie HA with haproxy running OpenJDK 1.8. I did have to make sure the three jars were unlinked on both oozie servers in the /opt/cloudera/parcels/CDH/lib/oozie/embedded-oozie-server/dependency/ directory. Could it be possible that you may have these jars elsewhere in the classpath for oozie, maybe in the /var/lib/oozie directory? unlink ./jetty-runner-9.3.20.v20170531.jar unlink ./javax.servlet.jsp-api-2.3.1.jar unlink ./javax.servlet.jsp-2.3.2.jar
... View more
02-28-2019
09:05 AM
Interesting. I'll make sure I test on HA with OpenJDK. Just to be sure as you mentioned HA, you did this unlink on all oozie servers behine your haproxy instance, correct?
... View more
02-27-2019
09:03 AM
My apologies as there were two additional jars in the final bugfix that I missed in my original suggestion. Could you please also do the following in the /opt/cloudera/parcels/CDH/lib/oozie/embedded-oozie-server/dependency directory: unlink ./jetty-runner-9.3.20.v20170531.jar unlink ./javax.servlet.jsp-api-2.3.1.jar Restart the oozie server, and please let me know if your oozie web console then functioning.
... View more
02-26-2019
01:13 PM
I confirmed this problem definitely does occur, even with fresh install. Removing the link to javax.servlet.jsp-2.3.2.jar in the/opt/cloudera/parcels/CDH/lib/oozie/embedded-oozie-server/dependency directory, and restarting, does solve it in my test environment. The below is for the latest - CDH 6.1.1. We will be resolving this in the CDH 6.2 release. Could you please confirm the link is gone, on the oozie server, and confirm the oozie server was restarted? [root@host-10-17-100-72 dependency]# ls -ltr total 0 lrwxrwxrwx 1 root root 47 Feb 7 23:58 jetty-xml-9.3.20.v20170531.jar -> ../../../../jars/jetty-xml-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-util-9.3.20.v20170531.jar -> ../../../../jars/jetty-util-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-server-9.3.20.v20170531.jar -> ../../../../jars/jetty-server-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 44 Feb 7 23:58 javax.servlet-api-3.1.0.jar -> ../../../../jars/javax.servlet-api-3.1.0.jar lrwxrwxrwx 1 root root 35 Feb 7 23:58 commons-el-1.0.jar -> ../../../../jars/commons-el-1.0.jar lrwxrwxrwx 1 root root 46 Feb 7 23:58 jetty-io-9.3.20.v20170531.jar -> ../../../../jars/jetty-io-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-http-9.3.20.v20170531.jar -> ../../../../jars/jetty-http-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 55 Feb 7 23:58 jetty-annotations-9.3.20.v20170531.jar -> ../../../../jars/jetty-annotations-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 31 Feb 7 23:58 mail-1.4.7.jar -> ../../../../jars/mail-1.4.7.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 apache-jsp-9.3.20.v20170531.jar -> ../../../../jars/apache-jsp-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 53 Feb 7 23:58 jetty-util-ajax-9.3.20.v20170531.jar -> ../../../../jars/jetty-util-ajax-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 52 Feb 7 23:58 jetty-security-9.3.20.v20170531.jar -> ../../../../jars/jetty-security-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 javax.servlet.jsp-api-2.3.1.jar -> ../../../../jars/javax.servlet.jsp-api-2.3.1.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 taglibs-standard-spec-1.2.5.jar -> ../../../../jars/taglibs-standard-spec-1.2.5.jar lrwxrwxrwx 1 root root 51 Feb 7 23:58 jetty-rewrite-9.3.20.v20170531.jar -> ../../../../jars/jetty-rewrite-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 taglibs-standard-impl-1.2.5.jar -> ../../../../jars/taglibs-standard-impl-1.2.5.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-runner-9.3.20.v20170531.jar -> ../../../../jars/jetty-runner-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-plus-9.3.20.v20170531.jar -> ../../../../jars/jetty-plus-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-jndi-9.3.20.v20170531.jar -> ../../../../jars/jetty-jndi-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 47 Feb 7 23:58 apache-log4j-extras-1.2.17.jar -> ../../../../jars/apache-log4j-extras-1.2.17.jar lrwxrwxrwx 1 root root 38 Feb 7 23:58 apache-jsp-8.0.33.jar -> ../../../../jars/apache-jsp-8.0.33.jar lrwxrwxrwx 1 root root 57 Feb 7 23:58 org.eclipse.jdt.core-3.8.2.v20130121.jar -> ../../../../jars/org.eclipse.jdt.core-3.8.2.v20130121.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-webapp-9.3.20.v20170531.jar -> ../../../../jars/jetty-webapp-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 44 Feb 7 23:58 javax.servlet.jsp-2.3.2.jar -> ../../../../jars/javax.servlet.jsp-2.3.2.jar lrwxrwxrwx 1 root root 51 Feb 7 23:58 jetty-servlet-9.3.20.v20170531.jar -> ../../../../jars/jetty-servlet-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 49 Feb 7 23:58 apache-jstl-9.3.20.v20170531.jar -> ../../../../jars/apache-jstl-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 37 Feb 7 23:58 apache-el-8.0.33.jar -> ../../../../jars/apache-el-8.0.33.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-client-9.3.20.v20170531.jar -> ../../../../jars/jetty-client-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-jaas-9.3.20.v20170531.jar -> ../../../../jars/jetty-jaas-9.3.20.v20170531.jar [root@host-10-17-100-72 dependency]# unlink ./javax.servlet.jsp-2.3.2.jar <RESTARTED OOZIE> [root@host-10-17-100-72 dependency]# ls -l total 0 lrwxrwxrwx 1 root root 37 Feb 7 23:58 apache-el-8.0.33.jar -> ../../../../jars/apache-el-8.0.33.jar lrwxrwxrwx 1 root root 38 Feb 7 23:58 apache-jsp-8.0.33.jar -> ../../../../jars/apache-jsp-8.0.33.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 apache-jsp-9.3.20.v20170531.jar -> ../../../../jars/apache-jsp-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 49 Feb 7 23:58 apache-jstl-9.3.20.v20170531.jar -> ../../../../jars/apache-jstl-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 47 Feb 7 23:58 apache-log4j-extras-1.2.17.jar -> ../../../../jars/apache-log4j-extras-1.2.17.jar lrwxrwxrwx 1 root root 35 Feb 7 23:58 commons-el-1.0.jar -> ../../../../jars/commons-el-1.0.jar lrwxrwxrwx 1 root root 44 Feb 7 23:58 javax.servlet-api-3.1.0.jar -> ../../../../jars/javax.servlet-api-3.1.0.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 javax.servlet.jsp-api-2.3.1.jar -> ../../../../jars/javax.servlet.jsp-api-2.3.1.jar lrwxrwxrwx 1 root root 55 Feb 7 23:58 jetty-annotations-9.3.20.v20170531.jar -> ../../../../jars/jetty-annotations-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-client-9.3.20.v20170531.jar -> ../../../../jars/jetty-client-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-http-9.3.20.v20170531.jar -> ../../../../jars/jetty-http-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 46 Feb 7 23:58 jetty-io-9.3.20.v20170531.jar -> ../../../../jars/jetty-io-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-jaas-9.3.20.v20170531.jar -> ../../../../jars/jetty-jaas-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-jndi-9.3.20.v20170531.jar -> ../../../../jars/jetty-jndi-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-plus-9.3.20.v20170531.jar -> ../../../../jars/jetty-plus-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 51 Feb 7 23:58 jetty-rewrite-9.3.20.v20170531.jar -> ../../../../jars/jetty-rewrite-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-runner-9.3.20.v20170531.jar -> ../../../../jars/jetty-runner-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 52 Feb 7 23:58 jetty-security-9.3.20.v20170531.jar -> ../../../../jars/jetty-security-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-server-9.3.20.v20170531.jar -> ../../../../jars/jetty-server-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 51 Feb 7 23:58 jetty-servlet-9.3.20.v20170531.jar -> ../../../../jars/jetty-servlet-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 jetty-util-9.3.20.v20170531.jar -> ../../../../jars/jetty-util-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 53 Feb 7 23:58 jetty-util-ajax-9.3.20.v20170531.jar -> ../../../../jars/jetty-util-ajax-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 50 Feb 7 23:58 jetty-webapp-9.3.20.v20170531.jar -> ../../../../jars/jetty-webapp-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 47 Feb 7 23:58 jetty-xml-9.3.20.v20170531.jar -> ../../../../jars/jetty-xml-9.3.20.v20170531.jar lrwxrwxrwx 1 root root 31 Feb 7 23:58 mail-1.4.7.jar -> ../../../../jars/mail-1.4.7.jar lrwxrwxrwx 1 root root 57 Feb 7 23:58 org.eclipse.jdt.core-3.8.2.v20130121.jar -> ../../../../jars/org.eclipse.jdt.core-3.8.2.v20130121.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 taglibs-standard-impl-1.2.5.jar -> ../../../../jars/taglibs-standard-impl-1.2.5.jar lrwxrwxrwx 1 root root 48 Feb 7 23:58 taglibs-standard-spec-1.2.5.jar -> ../../../../jars/taglibs-standard-spec-1.2.5.jar [root@host-10-17-100-72 dependency]# pwd /opt/cloudera/parcels/CDH-6.1.1-1.cdh6.1.1.p0.875250/lib/oozie/embedded-oozie-server/dependency
... View more