Hi @MichaelPlet , yes, sure, 7.1.7 SP1 is definitely a stable and also a long term support release with lots of bugfixes over 7.1.7. Also check the  additional cumulative hotfixes which were release on top of 7.1.7 SP1: https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/runtime-release-notes/topics/chf-pvcb-sp1-overview.html#chf-pvcb-sp1-overview If you have some security vulnerability questions, then kindly raise those questions through a support case. Thank you  Miklos Szurap Customer Operations Engineer, Cloudera   ... View more

The Canary is just testing whether the basic operatins are working in Hive Metastore. If that shows "unhealthy" it does not necessarily mean that the jobs are failing due to the Hive Metastore not functioning (it may be just slow for example), it is however indeed a warning sign for you that something is not proper. Please connect with beeline to the HiveServer2 and verify what is working and what is failing, then check the HiveServer2 logs and HiveMetastore logs. You can file a support case (where you can share much more details) if this is an urgent issue. ... View more

Hi @Jarinek , Yes, in CDH/CDP every service which depends on HDFS will inherit the HDFS configuration " auth-to-local rules", in CM in HDFS Configuration see "Additional Rules to Map Kerberos Principals to Short Names". Kafka does not need HDFS so that's why it has a separate such configuration. See the documentation how to set it: https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/security-kerberos-authentication/topics/cm-security-kerberos-authentication-auth-to-local-isolate.html Best regards  Miklos ... View more

Hi @Shaswat , Without reviewing completely what (else) may be the problem, the "port=21000" is definitely not correct. Impala has two "frontend" ports to which the clients can connect: - Port 21000 is used only for "impala-shell" - Port 21050 is used for all the other client applications using JDBC, ODBC, Hue or other Python based applications using Impyla - which is also used in the above example. Please see Impyla docs for more.   Best regards  Miklos ... View more

Hi @Neel_Sharma , The message suggests that the query tried to read the table's datafiles as if the table was parquet file based. (and parquet might be the default table format in Hive in CDP - of course only if the table format is not specified during creation) However the table creation script you've shared suggests the table should be text (CSV) based. Can you please verify it with checking what is the table format, with: DESCRIBE FORMATTED GeoIP2_ISP_Blocks_IPv4; Are you in the right database? For the second issue - how do you create the  external tables from tab delimited files? How are the files uploaded to hdfs? Thanks  Miklos ... View more

Hi @Uday_Singh2022 , yes, Flume is not a supported component in CDP. You can find documentations on Flume on it's official website: https://flume.apache.org/ Have you considered to use CDF / Nifi for this usecase? https://docs.cloudera.com/cdf-datahub/latest/nifi-hbase-ingest/topics/cdf-datahub-nifi-hbase-ingest.html  Thanks, Miklos ... View more

Hi @DataMan-HJ , the requirement you're looking for with case-insensitive joins doesn't seem to be present in Hive and likely will not be implemented as Hive relies on Java's UTF-8 strings and the behavior which implicitly comes with it - without possibility to change the collation. There's a good discussion on HIVE-4070 where a similar ask is raised for the LIKE operator behavior. You can review the pros and cons there. So you will likely need to go ahead to change the individual joins to use the lower/upper functions.   Best regards  Miklos   ... View more

Please remember that 1 block is not necessarily 256 MB, it can be less. Also not all files have replica factor of 3, some might have only 1 replica too, so it can be totally fine if all of those were all single replica files. 600.000 * 256 MB = 153.6 TB as a maximum, but since blocks can be smaller than 256 MB, the 60 TB freed up is reasonable. ... View more

Hi @Amn_468 , The lock contention happens when there are too many "invalidate metadata" (IM) and "refresh" commands running. The catalog daemon's responsibility is to load the Hive Metastore metadata (hive table and partition information, including stats) and the HDFS metadata (list of files and their block locations). If a table is refreshed (or a table is loaded for the first time after an IM) then catalogd has to load these metadata information, and has some built-in limits and has a max throughput how many tables and/or partitions/files it can handle (load). While doing so it needs to maintain a lock on the "catalog update", to avoid simultaneous requests to overwrite the previously collected information. So if there are concurrent and long running "refresh" statements [1], then those can block each other and cause a delay in the publishing of the catalog information. What can be done is to: - reduce the number of IM calls - reduce the number of refresh calls - wherever it is possible, use refresh on partition level only - There were some improvements in  IMPALA-6671, which is available in CDP 7.1.7 SP1 version, so an upgrade could also help (it still cannot completely help with high frequency, heavy refreshes)   I hope this can help the discussions with the users/teams how frequently and when are they submitting the refresh queries.    Miklos Customer Operations Engieer, Cloudera   [1] https://impala.apache.org/docs/build3x/html/topics/impala_refresh.html ... View more