Member since
08-14-2013
558
Posts
13
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2120 | 10-29-2019 11:47 AM | |
1964 | 07-15-2019 09:02 AM | |
5686 | 06-06-2019 07:56 AM | |
7788 | 02-28-2019 11:34 AM | |
8287 | 01-07-2019 01:08 PM |
11-01-2019
08:35 AM
1 Kudo
@Rak ,
I'm suspecting you are using Impala, because of the statement of the values returning null, but please confirm so I can properly label this question so others can see it. Please see the following in the Impala documentation regarding the proper format for casting a string into a timestamp datatype:
https://docs.cloudera.com/documentation/enterprise/latest/topics/impala_timestamp.html
Restrictions:
If you cast a STRING with an unrecognized format to a TIMESTAMP , the result is NULL rather than an error. Make sure to test your data pipeline to be sure any textual date and time values are in a format that Impala TIMESTAMP can recognize.
Currently, Avro tables cannot contain TIMESTAMP columns. If you need to store date and time values in Avro tables, as a workaround you can use a STRING representation of the values, convert the values to BIGINT with the UNIX_TIMESTAMP() function, or create separate numeric columns for individual date and time fields using the EXTRACT() function.
Examples:
The following examples demonstrate using TIMESTAMP values with built-in functions:
select cast('1966-07-30' as timestamp);
select cast('1985-09-25 17:45:30.005' as timestamp);
select cast('08:30:00' as timestamp);
select hour('1970-01-01 15:30:00'); -- Succeeds, returns 15.
select hour('1970-01-01 15:30'); -- Returns NULL because seconds field required.
select hour('1970-01-01 27:30:00'); -- Returns NULL because hour value out of range.
select dayofweek('2004-06-13'); -- Returns 1, representing Sunday.
select dayname('2004-06-13'); -- Returns 'Sunday'.
select date_add('2004-06-13', 365); -- Returns 2005-06-13 with zeros for hh:mm:ss fields.
select day('2004-06-13'); -- Returns 13.
select datediff('1989-12-31','1984-09-01'); -- How many days between these 2 dates?
select now(); -- Returns current date and time in local timezone.
... View more
10-31-2019
11:12 AM
Hi @Rak ,
Could you please elaborate on what component you are specifically doing this (Hive, Impala, etc..) so we can properly label this question and answer accordingly?
... View more
10-29-2019
11:47 AM
2 Kudos
Hello @luisfeg,
The files are only listed in the security scan out if the cgroups feature is in effect (CM -> Hosts -> Configuration -> "Enable Cgroup-based Resource Management"). The default permission of the cgroup_event_control files are indeed world writable without any other permission bit set. However, this has not too much effect in accordance with this document which explains why cgroup files are typically world writable: https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
This seems to be created by the supervisor code cgroups feature, and Red Hat appears to have a subscription article on a similar concern elsewhere on the file system: https://access.redhat.com/solutions/377043. These files being world writable is an inherent part of Linux cgroups and users being able to use the cgroup API. Cloudera can only use this cgroup feature if the user configures it explicitly thru Cloudera Manager. As Red Hat states in the document mentioned above:
"Changing the world writable bit is not recommended on these files as it would break notification features of the cgroup API. Changing the permissions would have no effect. Cgroups has it's own pseudo-file-system and manages all the files therein. Each time the cgconfig init script is run, the world writable bit will be set on these files. The cgroup.event_control files are world writeable. This is because they are used in conjunction with the cgroups notification API to allow notifications to be sent about a changing status in the control group."
... View more
09-26-2019
11:15 AM
I wanted to interject that while both the above are definite valid possible causes of Oozie jobs stuck in PREP state, there may be several other possible causes which may need to be resolved such as:
1. Issues with the Yarn Resource Manager / MR Job Tracker, lack of resources either for the RM or queues for the user running the job.
2. Problems with the Oozie server getting to the oozie database server, the database server itself, or locks on tables.
3. Lack of resources to Oozie such as callable queues, java heap, GC thrashing, etc.
The above is a brief shortlist from review of support cases relating to Oozie jobs stuck in PREP. I want to emphasize that deleting records from the Oozie database should be ONLY done the last resort to solving this problem, and only needed if you have a very large mass of oozie workflows that cannot be killed in a timely fashion by an oozie CLI script. This should be only done at the direction of support, people knowledgeable with SQL, and the relationship between tables, columns, and rows in the oozie database as referential integrity and constraints are lacking in the schema design. The above post from 2017 also missed one key table COORD_ACTIONS, where if this data was not properly cleaned up, would break your Oozie purge and possibly cause other serious problems.
... View more
07-15-2019
09:02 AM
1 Kudo
Hi @Harish19 , the best place for information regarding TCP-DS tests on Impala would be (follow the README.md): https://github.com/cloudera/impala-tpcds-kit Once the data is populated in HDFS and tables are created, you likely can run most the same queries in tree/master/queries/ on Hive and/or Hive on Spark to test. IBM and Databricks have githubs with some SparkSQL tests, which you can Google for, but I have not personally evaluated them, or know if they work. Thanks,
... View more
06-06-2019
07:56 AM
1 Kudo
@Prav , This appears to have been listed as a bug (which is actually a longstanding limitation due to the definition of files and directories with _ and . being considered as "hidden" in FileInputFormat in Hadoop) of Hive since the 0.12 version: https://issues.apache.org/jira/browse/HIVE-6431 https://stackoverflow.com/questions/19830264/which-files-are-ignored-as-input-by-mapper If these files are needed to be seen, please consider using a pre-process script to rename them after loading. Thanks,
... View more
05-03-2019
02:25 PM
1 Kudo
I would say this depends on if your hostname is set FQDN at time of CM/cluster install, if your hostname is set as fully qualified in the OS by the hostname command, or if you are still using TLS/SSL your certificates have the FQDN hostname. Kerberos and several other things are very sensitive to DNS. Run the hosts inspector under CM to check for DNS / host resolution problems. Since you state you are not using DNS, what I would suggest is making sure your /etc/hosts file on all hosts contains all hosts of the cluster and CM, and are set as fully qualified hostname first, then aliased to the short hostname. You can use rsync to keep this file accurate across the cluster. Also make sure /etc/nsswitch.conf has files first for the hosts: line, and /etc/hosts will get used first. Finally, if you supect the internal hostname was changed away from FQDN to short, either change it back or follow the following article to get CM configuration back in sync with what it was at time of install (Check the Name column beneath CM hosts tab to see what CM has in it's database): https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ag_change_hostnames.html [root@cm1 ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.2.3 cm1.rgjustice.com cm1 192.168.2.4 node1.rgjustice.com node1 192.168.2.5 node2.rgjustice.com node2 [root@cm1 ~]# cat /etc/nsswitch.conf |grep hosts #hosts: db files nisplus nis dns hosts: files dns myhostname
... View more
04-30-2019
02:13 PM
1 Kudo
Hi @bridgor , if we were just to take the N6apache6thrift9transport13TSSLException originally posted, this typically ends in a hostname mistmatch in the .pem files used / DNS problems. Check statestore logs / web UI for issues as well. I agree with @AcharkiMed that this seems to be network / SSL related.
... View more
03-12-2019
02:55 PM
As upsert is not SQL-92 syntax, the Impala JDBC driver does not currently support it. However, we are currently testing a newer version of JDBC driver and hope to support upsert in the near future. For the time being, please break these statements into a combination of insert and update statements.
... View more
02-28-2019
11:34 AM
This worked for me with Oozie HA with haproxy running OpenJDK 1.8. I did have to make sure the three jars were unlinked on both oozie servers in the /opt/cloudera/parcels/CDH/lib/oozie/embedded-oozie-server/dependency/ directory. Could it be possible that you may have these jars elsewhere in the classpath for oozie, maybe in the /var/lib/oozie directory? unlink ./jetty-runner-9.3.20.v20170531.jar unlink ./javax.servlet.jsp-api-2.3.1.jar unlink ./javax.servlet.jsp-2.3.2.jar
... View more