Member since
06-06-2016
185
Posts
12
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2695 | 07-20-2016 07:47 AM | |
2256 | 07-12-2016 12:59 PM |
06-10-2017
10:45 AM
Hi@Sagar Morakhia currently no other job are running except this job,actually it was stuck at 97.22% and when i kill the job and rerun again it was stuck same position , last reducer was running forever. up to table creation and insert the values into drome_master_5 it was good but while insert value into .drome_master_6 it was stuck. below is step where the query stuck insert into table dropme_master_6 select * from dropme_master_5 where dropme_master_5.consumer_sequence_id not in ( select consumer_sequence_id from dropme_master_6); and while executing the this step it shows below warning message Warning: Map Join MAPJOIN[30][bigTable=dropme_master_5] in task 'Map 4' is a cross product i was split query and execute it like below and it gave result very fast select count(consumer_sequence_id) from dropme_master_6; result:10352059 select count(*) from dropme_master_5; OK 21287539 i have added below parameter to the job but no use set hive.execution.engine=mr; set hive.vectorized.execution.enabled = true;
set hive.vectorized.execution.reduce.enabled = true;
set hive.tez.container.size=10240;
set hive.tez.java.opts=-Xmx9216m;
set mapreduce.map.memory.mb=8192; set mapreduce.map.java.opts=-Xmx7372m;
set mapreduce.reduce.memory.mb=9216;
set mapreduce.reduce.java.opts=-Xmx8294m;
set yarn.scheduler.minimum-allocation-mb=1024; set yarn.scheduler.maximun-allocation-mb=11264;
set hive.cbo.enable=true; cluster-status.png job-status.png
... View more
06-08-2017
07:46 PM
job-status.pngHi team, I am using hive with tez in HDP and i am running hive query,it was completed 99.27% and it is stuck in reducer phase, attached the screenshot. hive-log.png job-status.pn mem-space.png same job was done in 30min in last run, please suggest why it is happen and solution for this issue? i have below memory space in linux :swap total used free shared buffers cached
Mem: 258041 254721 3320 0 3449 237158 -/+ buffers/cache: 14114 243927
Swap: 49151 660 48491 Please suggest
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
-
Apache Tez
04-10-2017
12:27 PM
Thanks you so much @mqureshi I could not find mapreduce.task.files.preserve.failedtasks , i am using MRv2 HDP 2.1.3 and currently i dont have running jobs..
... View more
04-10-2017
12:05 PM
I have noticed that in dfs file system /user/username/.staging directory reaches 4tb..and directory having old file 15160634 2016-02-09 09:30 /user/userprod/.staging/job_1443521267046_99999/job.jar
/job_1443521267046_99999/job.split
.staging/job_1443521267046_99999/job.splitme
/.staging/job_1443521267046_99999/job.xml
/.staging/job_1443521267046_99999/libjars
/.staging/job_1443521267046_99999/tez-conf.pb
/.staging/job_1443521267046_99999/tez-dag.pb.
/.staging/job_1443521267046_99999/tez.session Can i remove this data?
... View more
Labels:
03-22-2017
04:31 AM
@Jay SenSharma Thanks its working fine warning masg are not coming ..
... View more
03-22-2017
03:51 AM
Hi i want to install Amari 2.4 in Ubuntu 14.2 on EC2 . i got warning msge that transparent_hugepage is enable ..so did below steps sudo vi /etc/default/grub add the below GRUB_CMDLINE_LINUX_DEFAULT=”transparent_hugepage=never” sudo update-grub sudo reboot cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never] but no luck i got same error masge..please help me out of this warning
... View more
Labels:
03-14-2017
02:16 PM
Thank you so much@Jay SenSharma what are step should i follow after restart the ambari? can you suggest any document which help me step by step?
... View more
03-14-2017
02:10 PM
Thanks @Jay SenSharma Yes i have not found files-2.2.2.xxx.jar -rw-r--r-- 1 root root 571873 Jul 12 2016 ambari-admin-2.2.1.12.4.jar
-rw-r--r-- 1 root root 45379969 Jul 12 2016 capacity-scheduler-2.2.1.12.4.jar
-rw-r--r-- 1 root root 98659271 Jul 12 2016 hive-2.2.1.12.4.jar
-rw-r--r-- 1 root root 46912006 Jul 12 2016 pig-2.2.1.12.4.jar
-rw-r--r-- 1 root root 52241298 Jul 12 2016 slider-2.2.1.12.4.jar
-rw-r--r-- 1 root root 48027227 Jul 12 2016 tez-view-2.2.1.12.4.jar
drwxr-xr-x 8 root root 4096 Aug 5 2016 work
how could i add the jar file here?
... View more
03-14-2017
01:59 PM
Thanks for quick response@Ravi Mutyala But here my concern i could not see any FILE View instance to expand,How to enable this?
Browse to the Ambari Administration interface. Click Views, expand the Files View, and click Create Instance.
... View more
03-14-2017
01:27 PM
Hi , I am using HDP 2.4 and ambari 2.2 want to create file view in ambari but when go to manage ambari-->views i could not see any "file" instance ..how can i create file view here..please help me
... View more
Labels:
- Labels:
-
Apache Ambari