Member since
08-14-2013
558
Posts
13
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2268 | 10-29-2019 11:47 AM | |
2181 | 07-15-2019 09:02 AM | |
5981 | 06-06-2019 07:56 AM | |
8509 | 02-28-2019 11:34 AM | |
8782 | 01-07-2019 01:08 PM |
11-01-2019
08:35 AM
1 Kudo
@Rak ,
I'm suspecting you are using Impala, because of the statement of the values returning null, but please confirm so I can properly label this question so others can see it. Please see the following in the Impala documentation regarding the proper format for casting a string into a timestamp datatype:
https://docs.cloudera.com/documentation/enterprise/latest/topics/impala_timestamp.html
Restrictions:
If you cast a STRING with an unrecognized format to a TIMESTAMP , the result is NULL rather than an error. Make sure to test your data pipeline to be sure any textual date and time values are in a format that Impala TIMESTAMP can recognize.
Currently, Avro tables cannot contain TIMESTAMP columns. If you need to store date and time values in Avro tables, as a workaround you can use a STRING representation of the values, convert the values to BIGINT with the UNIX_TIMESTAMP() function, or create separate numeric columns for individual date and time fields using the EXTRACT() function.
Examples:
The following examples demonstrate using TIMESTAMP values with built-in functions:
select cast('1966-07-30' as timestamp);
select cast('1985-09-25 17:45:30.005' as timestamp);
select cast('08:30:00' as timestamp);
select hour('1970-01-01 15:30:00'); -- Succeeds, returns 15.
select hour('1970-01-01 15:30'); -- Returns NULL because seconds field required.
select hour('1970-01-01 27:30:00'); -- Returns NULL because hour value out of range.
select dayofweek('2004-06-13'); -- Returns 1, representing Sunday.
select dayname('2004-06-13'); -- Returns 'Sunday'.
select date_add('2004-06-13', 365); -- Returns 2005-06-13 with zeros for hh:mm:ss fields.
select day('2004-06-13'); -- Returns 13.
select datediff('1989-12-31','1984-09-01'); -- How many days between these 2 dates?
select now(); -- Returns current date and time in local timezone.
... View more
10-29-2019
11:47 AM
2 Kudos
Hello @luisfeg,
The files are only listed in the security scan out if the cgroups feature is in effect (CM -> Hosts -> Configuration -> "Enable Cgroup-based Resource Management"). The default permission of the cgroup_event_control files are indeed world writable without any other permission bit set. However, this has not too much effect in accordance with this document which explains why cgroup files are typically world writable: https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
This seems to be created by the supervisor code cgroups feature, and Red Hat appears to have a subscription article on a similar concern elsewhere on the file system: https://access.redhat.com/solutions/377043. These files being world writable is an inherent part of Linux cgroups and users being able to use the cgroup API. Cloudera can only use this cgroup feature if the user configures it explicitly thru Cloudera Manager. As Red Hat states in the document mentioned above:
"Changing the world writable bit is not recommended on these files as it would break notification features of the cgroup API. Changing the permissions would have no effect. Cgroups has it's own pseudo-file-system and manages all the files therein. Each time the cgconfig init script is run, the world writable bit will be set on these files. The cgroup.event_control files are world writeable. This is because they are used in conjunction with the cgroups notification API to allow notifications to be sent about a changing status in the control group."
... View more
09-26-2019
11:15 AM
I wanted to interject that while both the above are definite valid possible causes of Oozie jobs stuck in PREP state, there may be several other possible causes which may need to be resolved such as:
1. Issues with the Yarn Resource Manager / MR Job Tracker, lack of resources either for the RM or queues for the user running the job.
2. Problems with the Oozie server getting to the oozie database server, the database server itself, or locks on tables.
3. Lack of resources to Oozie such as callable queues, java heap, GC thrashing, etc.
The above is a brief shortlist from review of support cases relating to Oozie jobs stuck in PREP. I want to emphasize that deleting records from the Oozie database should be ONLY done the last resort to solving this problem, and only needed if you have a very large mass of oozie workflows that cannot be killed in a timely fashion by an oozie CLI script. This should be only done at the direction of support, people knowledgeable with SQL, and the relationship between tables, columns, and rows in the oozie database as referential integrity and constraints are lacking in the schema design. The above post from 2017 also missed one key table COORD_ACTIONS, where if this data was not properly cleaned up, would break your Oozie purge and possibly cause other serious problems.
... View more
07-15-2019
09:02 AM
1 Kudo
Hi @Harish19 , the best place for information regarding TCP-DS tests on Impala would be (follow the README.md): https://github.com/cloudera/impala-tpcds-kit Once the data is populated in HDFS and tables are created, you likely can run most the same queries in tree/master/queries/ on Hive and/or Hive on Spark to test. IBM and Databricks have githubs with some SparkSQL tests, which you can Google for, but I have not personally evaluated them, or know if they work. Thanks,
... View more
06-06-2019
07:56 AM
1 Kudo
@Prav , This appears to have been listed as a bug (which is actually a longstanding limitation due to the definition of files and directories with _ and . being considered as "hidden" in FileInputFormat in Hadoop) of Hive since the 0.12 version: https://issues.apache.org/jira/browse/HIVE-6431 https://stackoverflow.com/questions/19830264/which-files-are-ignored-as-input-by-mapper If these files are needed to be seen, please consider using a pre-process script to rename them after loading. Thanks,
... View more
05-06-2019
03:27 PM
1 Kudo
Thanks to everyone who replied. It turns out that references to truststores and server keys, etc., and associated passwords may be cached, so when we changed these after moving the cluster, creating new cerrts and replacing the passwords in CDH was insufficient. So, after DELETING all fields containing passwords, cert locations, key locations, etc.,unchecking SSL, restarting the cluster, and adding the references back in, everything works. Uugghhh - who knew! 🙂 B
... View more
03-12-2019
02:55 PM
As upsert is not SQL-92 syntax, the Impala JDBC driver does not currently support it. However, we are currently testing a newer version of JDBC driver and hope to support upsert in the near future. For the time being, please break these statements into a combination of insert and update statements.
... View more
03-01-2019
12:02 AM
1 Kudo
Yes, I found the jar in other places, I remove the link, now the oozie web ui work fine! Thank you! Now I try to fix the oozie ha problem.
... View more
01-18-2019
04:36 AM
In the Sentry logs, shows this error: mismatch: Client thrift version is: 1 , server thrift version is 2 In Impala, it throws this error: org.apache.impala.catalog.AuthorizationException: User 'xxxxx' does not have privileges to access: server I tried to see if there is any mismatch in the cloudera libraries but didnt find anything... Any clue about it?
... View more
01-08-2019
12:42 PM
There is no new product name. This will be a new feature within Sentry called Attribute Based Access Control using Navigator Lineage tags for metadata. At present, there is no estimated release date, but we will be sure to announce that.
... View more