Member since
01-25-2019
75
Posts
10
Kudos Received
13
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 3394 | 02-25-2021 02:10 AM | |
| 1986 | 02-23-2021 11:31 PM | |
| 4031 | 02-18-2021 10:18 PM | |
| 5562 | 02-11-2021 10:08 PM | |
| 19387 | 02-01-2021 01:47 PM |
02-10-2021
09:45 PM
Hello @ryu Well you can make that user similar to hive user. hive user is mapped to hadoop group and you can make alterations to simulate a normal user to hive user but again as I mentioned earlier, you'll have to spend time managing it and eventually land up in spending more time in tshooting if things break. Remember hadoop is a complex setup with multiple components talking to each other. 🙂
... View more
02-09-2021
07:46 AM
Hello @ryu If you run the job with the end user, you will eventually end up managing internal permissions, job submission permissions your self. Also you will find difficulty integrating things as per my experience. But if you submit the job and let the hive user take care of the file creation, managing part in the backend, admin's job life become easier. You also will be able to hook/integrate things more properly. the above was just a jist, recommendations are to authenticate using the end user but then keep the impersonation off and let the hive take care of things in the backend.
... View more
02-01-2021
01:47 PM
Hello @BhaveshP The error you are seeing is because certificate CN/SAN name mismatch than that of hostname. Let me try to explain you with an example. client [A]--> Balancer[B] -->NIFI Nodes[C,D,E] Here you have a client A who wants to access NIFI nodes C,D and E and he wants to access via B. When you create SSL certificates for C,D,E, you create 3 certificates with 3 different "Common Names" C, D. E respectively. Now when you try connecting to nifi nodes C,D,E respectively directly from client A, you will not observe the issue. But when you try to access C,D,E via balancer or any proxy B, you are likely to get the error. WHY? Client A is trying to talk to Nifi Node C or D or E but to client the NIFI node is B. During SSL handshake, the certificates are passed by Nifi server to A where in client gets confused becuase it wanted to talk to B however the certificate it got is from C. Name don't match. FIX: Make use to SAN Names in certificates. (Subject Alternative Names) Issue certificate of C,D,E such that it has a SAN name as B. What I am trying to say is, the certificate should have both the name that of C and B in nifi node C, D and B in nifi node D and E and B in nifi node E. In this way client will come to know that when it talks to B and when the certificate received to A from C, it will not get confused because the certificate will have two name present in SAN that of B and C. Let me know if the above gives some clarity as to what exactly is happening.
... View more
01-26-2021
07:34 PM
Hey @saurabh707 Could you please try the below: Update hive log4j configs through CM Hive on Tez -> Configuration -> HiveServer2 Logging Advanced Configuration Snippet (Safety Valve) Hive -> Configuration -> Hive Metastore Server Logging Advanced Configuration Snippet (Safety Valve) Add the following to config: appender.DRFA.strategy.action.type=DELETE appender.DRFA.strategy.action.basepath=${log.dir} appender.DRFA.strategy.action.maxdepth=1 appender.DRFA.strategy.action.PathConditions.glob=${log.file}.* appender.DRFA.strategy.action.PathConditions.type=IfFileName appender.DRFA.strategy.action.PathConditions.nestedConditions.type=IfAccumulatedFileCount appender.DRFA.strategy.action.PathConditions.nestedConditions.exceeds=10 Let me know if the above addresses it.
... View more
11-08-2020
07:15 PM
@ateka_18 As mentioned before as well, view is just a query statement in HMS. It's not actually a table. If you want to know the size of the table, below is the best approach. hdfs dfs -du -s -h /path/to/table Let's say you have a table named test(/user/hive/warehouse/default.db/test) and have several partitions under it (part1, part2, part3). To get the size of the table. hdfs dfs -du -s -h /user/hive/warehouse/default.db/test Let me know if the above helps.
... View more
11-07-2020
07:45 AM
@ateka_18 VIEW is just a query statement saved in HMS. So basically its the size of the table you gather and to do so, the best option is below. hdfs dfs -ls <path of the table> With regards to the error you are observing, it clearly says that analyze table is not supported for views. You can get the same for tables.
... View more
11-01-2020
06:10 AM
@HanzalaShaikh Could you try the below and see if this helps or not. grant SELECT on DATABASE `_impala_builtins` to role <role-name mapped to user hadmin>
... View more
10-29-2020
09:30 AM
@sace17 No this is not right. As I mentioned, the Catalog server will be using the keytab file (impala.keytab) present inside it's process directory ssh to catalog server and run the below command to list out the principals from the keytab klist -ket $(ls -td /var/run/cloudera-scm-agent/process/*CATALOGSERVER* | head -1)/impala.keytab Also From the logs, I see the catalog server is not using authentication. Hence I would request you to focus on setting up the kerberos properly. I1029 17:29:11.249019 30770 authentication.cc:1093] Internal communication is not authenticated
I1029 17:29:11.249027 30770 authentication.cc:1114] External communication is not authenticated
... View more
10-29-2020
07:15 AM
@sace17 Your catalog server is unable to authenticate or get a kerberos ticket. Generally the ticket will be read form the impala.keytab file present under catalog process directory. /var/run/cloudera-scm-agent/process/<latest-process-number>-impala-CATALOGSERVER/implala.keytab for example, below is the keytab output from my catalog server. It contains principal of my catalog server. [root@host-10-17-102-166 6772-impala-CATALOGSERVER]# klist -ket /var/run/cloudera-scm-agent/process/6772-impala-CATALOGSERVER/impala.keytab Keytab name: FILE:/var/run/cloudera-scm-agent/process/6772-impala-CATALOGSERVER/impala.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 5 08/11/20 22:46:07 impala/host-10-17-102-166.coe.cloudera.com@COE.CLOUDERA.COM (des3-cbc-sha1) Can you see if you have the same on your end ? You can also verify by looking at the catalog server logs, when it boots. I1029 07:09:27.356434 18103 authentication.cc:730] Using internal kerberos principal "impala/email@redacted.host" I1029 07:09:27.356523 18103 authentication.cc:1083] Internal communication is authenticated with Kerberos I1029 07:09:27.360359 18103 init.cc:362] Logged in from keytab as impala/email@redacted.host (short username impala) I1029 07:09:27.360714 18103 authentication.cc:866] Kerberos ticket granted to impala/email@redacted.host I1029 07:09:27.360739 18103 authentication.cc:730] Using external kerberos principal "impala/email@redacted.host" I1029 07:09:27.360744 18103 authentication.cc:1099] External communication is authenticated with Kerberos Use the below link to setup kerberos manually. to ensure if all the process was followed correctly or not. https://plenium.wordpress.com/2018/07/17/kerberos-setup-in-cloudera-hadoop/
... View more
10-22-2020
06:07 PM
@PauloRC -->With regards to your query point 1, When a table is queried first time, impala pulls the metadata from catalog which further gets it from HMS and then Impalad stores the metadata in the cache. So, next time when the query is run on the same table, impala does not have to again reach out to catalog server to get the metadata. In your case, you were able to submit queries on other tables because those tables metadata was already available locally with impalad(check the query profile and it can be found whether metadata was pulled from catalog or fetched locally). But for table "impala_table", metadata was not present locally and it was trying to fetch this from catalog server. Now due to failover, HMS was somehow still pointing to standby NN which is why you see the below error. CAUSED BY: TableLoadingException: Failed to load metadata for table: impala_db.impala_table
CAUSED BY: RemoteException: Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error -->For your point 2, I would like to check if you followed the below or not. https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/cdh_hag_hdfs_ha_cdh_components_config.html#topic_2_6_3 -->For your point 3, here is the solution below in the link. https://docs.cloudera.com/documentation/enterprise/latest/topics/impala_metadata.html#auto_poll_hms_notification
... View more
- « Previous
-
- 1
- 2
- Next »