Member since
03-06-2020
171
Posts
15
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
127 | 02-04-2022 05:00 AM | |
99 | 01-03-2022 04:37 AM | |
333 | 11-01-2021 04:13 AM | |
219 | 10-15-2021 08:48 PM | |
285 | 07-29-2021 07:09 AM |
04-28-2022
11:45 PM
Hi @yagoaparecidoti , I do not think we have such option of blocking but as i know i think you can make it using sentry authorisation. If sentry is enabled in the cluster you should grant only "SELECT" privilege to all the users for all the databases/tables so if the users try any other DDL commands it will throw 'Authorisation exceptions". Do refer the below documentation: https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_sg_sentry_service.html If you are using Ranger same can be achieved from ranger also. Regards, Chethan YM
... View more
04-19-2022
01:28 AM
1 Kudo
Hi @yagoaparecidoti , Each host that runs the Impala Daemon acts as both a coordinator and as an executor and if you want to deploy dedicated coordinators and executors please refer the below documentation for complete details: https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/impala_dedicated_coordinator.html Regards, Chethan YM
... View more
04-12-2022
05:59 AM
Hi, Can you regenerate the kerberos credentials for impala from CM and see if it helps? Regards, Chethan YM
... View more
04-07-2022
01:48 AM
Hi @swapko , > Please provide the error stacktrace that you are seeing when you refresh the impala tables. > was it working earlier? Regards, Chethan YM
... View more
04-04-2022
03:41 AM
Hi @jagad , I think you are facing the above error for an impala query am i right? If Yes please increase the memory limit for that query using set MEM_LIMIT=xg; parameter and try to rerun the query. This usually occurs if there is not enough memory for a query. Regards, Chethan YM
... View more
04-04-2022
03:32 AM
Hi @dmharshit , This error can occures if you have deployed the "Hive Server 2" and "WebHCat" roles under the Hive Service. Can you stop and delete the "Hive Server 2" and "WebHCat" instances from Cloudera Manager -> Hive -> Instances? Restart if any stale configurations are present in the cluster and try the hive query. Regards, Chethan YM
... View more
04-04-2022
03:19 AM
Hi, As per the above Jira- HUE-2782 the resolution is "won't fix" so issue cannot be fixed. We do not know you are facing the same issue or not so please attach the error screenshot and error stacktrcae to look into it. Regards, Chethan YM
... View more
03-23-2022
05:26 AM
1 Kudo
Hi @PratCloudDev , "Spill to disk" happens when there is no enough memory available for a running query, Below is the example. Suppose you are running a query which is using 10gb ( per node peak memory) of memory and in case this query needs 12 gb of memory in this situation spill to disk happen on the configured scratch directories. you can see the directory by searching the "Impala Daemon Scratch Directories" property in the impala configurations. If you do not want to fail the query then you need to make sure the configured scratch directories/disk has enough space to store spilling information, this can potentially be large amounts of data. Check the query profile for "per node peak memory" it is the actual memory used for that query on each daemon, suppose if it is 15GB then set the MEM_LIMIT to 10gb or 12gb to see the spill to disk functionality. To understand why you are seeing the error[1] i need few details from your side. 1. Screenshot of impala admission control pool settings. 2. How much memory you are setting and seeing the below error[1]? 3. Which pool you are using to run the query? 4. If possible you can provide the query profile. Regards, Chethan YM [1]. Rejected query from pool root.centos: minimum memory reservation is greater than memory available to the query for buffer reservations.
... View more
02-28-2022
05:25 AM
Hi , Previous execution means previous workflow execution or previous action execution with in a workflow? I think as i know if you have multiple actions with in a wokflow next action will start only if previous action is succeeded otherwise it fails. If its not working like this can you give the info output for that workflow id? oozie job -oozie http://<oozie-server-host>:11000 -info <workflow-id> Regards, Chethan YM
... View more
02-28-2022
04:50 AM
1 Kudo
Hi, Similar exception can be seen if we do not add credentials block with in the workflow something like below. Do check you have correct credentials if its a kerborized cluster. <credentials> <credential name="hcatauth_creds" type="hcat"> <property> <name>hcat.metastore.uri</name> <value>thrift://<metastore-fqdn>:9083</value> </property> <property> <name>hcat.metastore.principal</name> <value>hive/_HOST@<REALM></value> </property> </credential> Regards, Chethan YM
... View more
02-16-2022
06:21 AM
Hi, As i seen till now there is no such way to rerun the job automatically when it is failed, I think you have to rerun it manually as you do always. If you want this type of behaviour you may need to create custom scripts as per your requirements. Regards, Chethan YM
... View more
02-16-2022
05:46 AM
Hi @mala_etl , It looks like a below known issue: https://issues.cloudera.org/browse/HUE-8717 Do work with Cloudera support to get a patch. Regards, Chethan YM
... View more
02-16-2022
05:32 AM
Hi, To look into the issue we need a complete error stack-trace can you please attach it? Also is this happens every-time after impala restart? Regards, Chethan YM
... View more
02-04-2022
05:00 AM
Hi, Yes, Impala daemons will use the memory during the execution. Your understanding is correct. In the attached screenshot i can see the corrupted stats for the tables involved in the query, We recommend to run "compute stats" on the tables which is having partial stats and rerun the queries otherwise it will generates bad execution plan and uses more memory than expected. Regards, Chethan YM
... View more
01-25-2022
04:08 AM
1 Kudo
Hi, Please try to rerun the sqoop import outside of oozie and confirm whether its working or not. If it is working then we need to check it from oozie side. We need a complete stacktrcae of the error. Regards, Chethan YM
... View more
01-25-2022
02:51 AM
1 Kudo
Hi, You can restrict the amount of memory Impala reserves during query execution by specifying the -mem_limit option. If you set mem_limit=2gb, The query will not use more than 2gb even if it needs. If you cannot set the memory at the time of execution every-time i think you can create a new resource pool under impala admission control. While creating resource pool you can mention Min and Maximum Query Memory Limit and do not use this resource pool for production queries. set request_pool="pool-name" run the test queries. Regards, Chethan YM
... View more
01-03-2022
04:37 AM
Hi, -> Have you tried the same query via Hive? Does it work? -> How this table got created via hive or impala? -> Try "invalidate metadata" in impala and retry Regards, Chethan YM
... View more
12-06-2021
06:15 AM
Hi @HareshAmin , As per the below documentation it looks like Hive UDF's are not supported in Impala. Please refer the documentation: https://docs.cloudera.com/documentation/enterprise/5-8-x/topics/impala_udf.html Regards, Chethan YM
... View more
12-01-2021
08:41 AM
Hi, The Reducer process the output of the mapper. After processing the data, it produces a new set of output. At last HDFS stores this output data. Reducer takes a set of an intermediate key value pair produced by the mapper as the input and runs a Reducer function on each of them. The mappers and reducers depends on the data that is processing. You can manually set the number of reducers with below property but i think it is not recommended. set mapred.reduce.tasks=xx; Regards, Chethan YM
... View more
11-10-2021
04:01 AM
Hi @drgenious , 1. can you attach the full query profile of it to take a look? 2. You mean query is finished in 8 minutes but why Unregister query has took 19 min am i correct ? if not please elaborate the issue. 3. How you are running the queries from impala-shell or Hue? 4. what is the session timeouts you have in the cluster for impala queries? Regards, Chethan YM
... View more
11-10-2021
03:35 AM
Hello @raghu9raghavend , Please provide the error/stack-trace for the failure. Regards, Chethan YM
... View more
11-04-2021
11:38 PM
Hi, As per my previous comment can you destroy it and restart the LLAP and see if this works. Note: llap0 is the default application that will be running if LLAP is installed it will be recreated even if you destroy it and restart the service. # yarn app -destroy llap0 Regards, Chethan YM
... View more
11-04-2021
03:06 AM
Hi, Can you perform these steps and see if this helps to restart LLAP. 1. Check ps -ef | grep -I llapdaemon run it on the node managers and kill the process if queue utilisation was full. check in all the nodemanager ,if any stale llap daemon process is running > ps -ef | grep llap 2. su - hive # yarn app -status llap0 # yarn app -destroy llap0 Regards, Chethan YM
... View more
11-04-2021
02:28 AM
Hello @ighack , You are welcome..!! Was your question answered? Please Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button. Regards, Chethan YM
... View more
11-02-2021
12:19 AM
1 Kudo
Hi @ighack , As like yarn, there is no direct options for impala admission control pool, As per the cloudera documentation you need to select "default settings" under impala admission control to add users and group. Below is the screenshot of it. After you click on Default Settings you need to choose " Allow these users and groups to submit to this pool" to add users and groups. Regards, Chethan YM
... View more
11-01-2021
05:01 AM
Hi @harnu , Could you add these below values to /etc/security/limits.conf file and see if this helpfull? If you already have these values try to increase it and rerun the query. hive soft nofile 65000 hive hard nofile 65000 hive soft proc 80000 hive hard proc 80000 Update limits.conf with above settings and restart hiveserver2 service while stopping all the services on that node. Please su - hive and then ulimit -a to check whether the parameters are updated for no of open files and max user process. Regards, Chethan YM Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
... View more
11-01-2021
04:35 AM
Hi @wert_1311 , Seems to be " load_catalog_in_background" option is set to true in your cluster, This can result in lock contention. The recommended value for this parameter is false. Please change the value to False and see if this helpful. And if you are also seeing any JVM pauses in the catalog logs try to increase the catalog heap size and monitor the cluster. https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/impala_config_options.html Regards, Chethan YM Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
... View more
11-01-2021
04:13 AM
Hi @ighack , When click on Submission Access Control tab to specify which users and groups can submit queries by default, anyone can submit queries. To restrict this permission, select the Allow these users and groups option and provide a comma-delimited list of users and groups in the Users and Groups fields respectively. I am attaching the screenshot for your reference. Please check the below Cloudera documentation for the same. https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/impala_howto_rm.html#enable_admission_control__d3424603e189 Regards, Chethan YM Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
... View more
10-25-2021
11:11 PM
Hi @pauljoshiva , Sqoop uses local hive which does automatic connect so you would need to modify the information in beeline-site.xml as referenced in below article: https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-Usingbeeline-site.xmltoautomaticallyconnecttoHiveServer2 Hence please go to /usr/hdp/<VERSION>/hive/conf then open the beeline-site.xml and modify there. Regards, Chethan YM
... View more
10-16-2021
12:08 AM
Hi, Are you using --password-file option inside your workflow? I have found a bug[1][2] similar to the issue you are facing. If this is the issue can you turn off the uber mode and try the job? <global> <configuration> <property> <name>oozie.launcher.mapreduce.job.ubertask.enable</name> <value>false</value> </property> </configuration> </global> Also are you doing any FileSystem.close() call in your actions? FileSystem objects are automatically cached in Hadoop and thereby if you close the FileSystem object inside from your Java Action Class, which runs in the same JVM as the launcher map task, the singular FileSystem instance itself gets closed. Hence, when the Wrapper/Launcher wishes to exit normally. Regards, Chethan YM Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button. [1]. https://issues.apache.org/jira/browse/SQOOP-2997 [2]. https://jira.cloudera.com/browse/CDH-43107
... View more