About snm1523

snm1523 · ‎09-27-2019

Hello All, Need suggestions / best practices / guidance to deploy below HBase replication scenario: We have 2 running HBase clusters which are in sync with each other (Bi directional HBase replication configured and enabled). With this it ensures that both HBase clusters have up to date data in HBase tables. We have a backup cluster deployed and the idea is to setup replication from both HBase clusters to Backup cluster. So ultimately, we will have 2 Active Masters of which data will be replicated to 1 Slave. If we configure HBase replication jobs for both master and keep them running, it could be the case of duplication of data (as per my understanding and logic). Would be great if anyone could share their experiences or guide further on to achieve this. Thanks Snm1523

snm1523 · ‎06-03-2019

Hello, Is there any way we can configure Cloudera Manager to send an email if utilization of CPU or Memory of a particular daemon / node (for eg. Impala daemon) crosses a specific limit (for eg. 80%)? Please advise. Thanks snm1523

snm1523 · ‎06-03-2019

Hello Harsh, Thank you for the help on this. I was able to identify some information that helped here. Will come back in case need further help. Will accept your reply as Solution. 🙂 Thanks snm1523

snm1523 · ‎03-07-2019

Thank you for the reply Harsh J. Would you be able to please help me with any quick command / script to identify avoidable open files or files stuck in some process using 'lsof' and guide further actions to take? I tried running a generic 'lsof | grep java' but it obviously gave me a huge list of files and became a bit difficult to get relevant information. Thanks snm1523

snm1523 · ‎03-06-2019

Hello All, I am looking for some best practices or recommendations to set a best possible value for rlimit_fds (Maximum Process File Descriptors) property. Currently, it is set to default i.e. 32768 and we are getting File Descriptor Threshold alerts. We would first like to look for a best possible value for rlimit_fds. Is there a formulae or a practice or few checks that can be performed to determine a best value? Thanks snm1523

snm1523 · ‎02-25-2019

Hello gzigldrum, Thank you for the guidance. This certainly helps. I will go through both the KB articles you shared and also review the Navigator Server and CM Agent logs for audit health warnings. Will comeback with the findings. In the meantime, any luck with HIVEMETASTORE_CANARY_HEALTH and NODE_MANAGER_WEB_METRIC_COLLECTION? Thanks snm1523

snm1523 · ‎02-22-2019

Hello Gzigldrum, Thank you for the reply. Below are the exact health messages from CM for each alert: HIVEMETASTORE_CANARY_HEALTH: The health test result for HIVEMETASTORE_CANARY_HEALTH has become bad: The Hive Metastore canary failed to create a database. OR The health test result for HIVEMETASTORE_CANARY_HEALTH has become bad: The Hive Metastore canary failed to create a partition in the table it created. OR The health test result for HIVEMETASTORE_CANARY_HEALTH has become bad: The Hive Metastore canary failed to drop the table it created. REGION_SERVER_AUDIT_HEALTH: The health test result for REGION_SERVER_AUDIT_HEALTH has become bad: There is a problem processing audits for REGIONSERVER. IMPALAD_QUERY_MONITORING_STATUS: he health test result for IMPALAD_QUERY_MONITORING_STATUS has become bad: There are 1 error(s) seen monitoring executing queries, and 0 errors(s) seen monitoring completed queries for this role in the previous 5 minute(s). Critical threshold: any. HIVESERVER2_SCM_HEALTH: The health test result for HIVESERVER2_SCM_HEALTH has become bad: This role s process is starting. This role is supposed to be started. NAME_NODE_AUDIT_HEALTH: The health test result for NAME_NODE_AUDIT_HEALTH has become bad: There is a problem processing audits for NAMENODE. NODE_MANAGER_WEB_METRIC_COLLECTION: The health test result for NODE_MANAGER_WEB_METRIC_COLLECTION has become bad: The Cloudera Manager Agent is not able to communicate with this role s web server. These health alerts occur in every 2-3 days in a time frame of 2-3 hours creating around 5-10 tickets during each interval. We have also checked from network side to verify if there was a network outage or a glitch in those windows, however, no luck. We have tried to diagnose through logs of each alert, but haven't found anything interesting. Hence, looking for some more guidance on what additional could be checked to identify a root cause of these. Thanks snm1523

snm1523 · ‎02-21-2019

Thank you for the quick replies, Gzigldrum. I have created a separate post explaining the alerts we get. Below is the link to the same. Flooded with failed health test alerts For every alert we get an incident in BMC as HP OVO is configured to generate an incident and assign it to us. Every morning we start with close to around 15-20 such incidents followed by 15-20 in rest of the day. Would be great if you could please post some suggestions to troubleshoot those alerts and help in permanently fixing them. Note: I will accept your previous reply as solution to this post. Thanks snm1523

snm1523 · ‎02-21-2019

Hello Gzigldrum, Thank you for the reply. However, the instructions provided in the KB article and values suggested by you are already applied. Please advise. Thanks Snm1523

snm1523 · ‎02-21-2019

Hello All, Need some suggestions on the exact reason of below heath tests getting failed multiple times, almost 3 times every day and generates atleast 3-4 alerts each time. 1. HIVEMETASTORE_CANARY_HEALTH 2. REGION_SERVER_AUDIT_HEALTH 3. IMPALAD_QUERY_MONITORING_STATUS 4. HIVESERVER2_SCM_HEALTH 5. NAME_NODE_AUDIT_HEALTH 6. NODE_MANAGER_WEB_METRIC_COLLECTION Any help or suggestion to permanently fix these alerts would of great help. Also, if anyone could also guide to reach root cause of this would also be helpful Thanks snm1523

snm1523 · ‎02-21-2019

Hello All, We have a HP OVO monitoring tool monitoring all the alerts from Cloudera Manager and raising a BMC Remedy Incident accordingly. However, we are at times flooded with these monitoring alert tickets and when it is immediately checked in the cluster, everything looks green. When we dig in for detailed analysis and check logs of the respective alert, we do not have anything major and also the service is green. It looks like it heals by itself. However, this flood of tickets, raises concerns with Management and questions for a root cause, which we do not have in reality and also unable to find anything as it was auto fixed with no special traces behind. I am looking out for recommendations / best practices on how exactly this should be setup, so we get only the actual / required alerts. Is there any configuration we need to do in CM or there is something that can be configured in HP OVO. Any suggestions would be of a great help. Also, if anyone has a suggestion that should be checked while troubleshooting these alerts would also be welcomed. Thanks snm1523

snm1523 · ‎02-18-2019

Thank you gzigldrum. This certainly helps. Regards, snm1523

snm1523 · ‎02-15-2019

Hello, I would like to know the various activities performed by respective users on Cloudera Manager. For example, which user restarts a service or moves a cluster / service to maintenance mode. Would be great if any one could share some information on this. Thanks snm1523

snm1523 · ‎12-04-2018

Hello TK, Were you able to fix this? Would be great if you could please share the solution as I have the same problem. Thanks Snm1523

snm1523 · ‎09-19-2018

Hello @bgooley, Thank you for the guidance. Will attempt the suggested suggestion and come back. Thanks snm1523

snm1523 · ‎09-11-2018

Hello, I am trying to deploy a POC cluster using Path A of Cloudera Manager installation documentation. However, it is failing with below message in the logs: cat 5.install-cloudera-manager-server-db-2.log Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile Resolving Dependencies --> Running transaction check ---> Package cloudera-manager-server-db-2.x86_64 0:5.15.1-1.cm5151.p0.3.el7 will be installed --> Processing Dependency: postgresql-server >= 8.4 for package: cloudera-manager-server-db-2-5.15.1-1.cm5151.p0.3.el7.x86_64 --> Finished Dependency Resolution Error: Package: cloudera-manager-server-db-2-5.15.1-1.cm5151.p0.3.el7.x86_64 (cloudera-manager) Requires: postgresql-server >= 8.4 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest I also tried to use local repository and point .bin file using --skip_repo_package=1. However, Was unable to find packages for PostgreSQL 8.4 any where, so created repo of 9.6 instead. CM at the first place hunts for PostgreSQL 8.4, and then finds 9.6 in the repo, however, fails at dependency check. It still needs few dependent packages (5-6) to proceed. Before I download each package and put it in repo, seeking suggestions / help from the community on why is the installation with Cloudera repos (installation path A) failing with that message. Thanks snm1523

snm1523 · ‎09-11-2018

Hello, I am trying to deploy a POC cluster using Path A of CM Installation. It is failing with below message from logs: cat 5.install-cloudera-manager-server-db-2.log Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile Resolving Dependencies --> Running transaction check ---> Package cloudera-manager-server-db-2.x86_64 0:5.15.1-1.cm5151.p0.3.el7 will be installed --> Processing Dependency: postgresql-server >= 8.4 for package: cloudera-manager-server-db-2-5.15.1-1.cm5151.p0.3.el7.x86_64 --> Finished Dependency Resolution Error: Package: cloudera-manager-server-db-2-5.15.1-1.cm5151.p0.3.el7.x86_64 (cloudera-manager) Requires: postgresql-server >= 8.4 You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest I also attempted to create a local repository of both CM and PostgreSQL and executed .bin with --skip_repo_package=1. However, I was unable to find PostgreSQL 8.4 package for CentOS / RedHat 7 any where, so downloaded and created repo of PostgreSQL 9.6. CM was further looking for 9.6 dependencies, which was again a list of 5-6 packages. Not sure, how to proceed, please help. Thanks snm1523

snm1523 · ‎07-26-2018

Hi, As per my understanding, CM performs canary tests for each of the enabled and active service in the cluster to ensure if those services are healthy. However, I saw few errors last week related to Hive Metastore Canary test which failed because Hue was unable to create a test database in Hive Metastore. We do not use Hue actively and hence, it has been stopped. After I started Hue service the error was gone and everything reported healthy. Did canary test actually happened on a stopped Hue service or there is something I might have missed? If CM still performs a canary test on stopped services, is there some mis configuration or CM is designed to do this? Please advise. Thanks snm1523

snm1523 · ‎04-05-2018

Hi Jaimie, Just did a little google and found below things to be set. This may resolve the issue: set hive.tez.container.size=2048; set hive.tez.java.opts=-Xmx1700m; --set this 80% of hive.tez.container.size Please give it a try if you feel confident on this. Thanks snm1502

snm1523 · ‎04-05-2018

Hi Brit, One of the log refer to Regex Serde issue: Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.RegexSerDe not found You may also verify if the serde is added and available in the mentioned path in Metastore before this job is executed. Thanks snm1502

snm1523 · ‎03-21-2018

Hello, Do we have a specific guide or a standards document any where (or may be anyone might have created based on their experience) that would provide the best practices / approaches / recommendations for various services that we install in a Cloudera Cluster. For eg. 1. Recommendation on Number of executors should be configured while setting up Spark 2. Best practices or things one should definitely keep in mind while setting up a Cloudera Hadoop cluster with bundled Hadoop. 3. Best practices or recommendations for configurations of Oozie, etc. Any leads on this would be helpful. Thanks snm1529

snm1523 · ‎03-21-2018

Hello All, Replying on an old post. Part A: Did anyone made any success in migrating live Cloudera Management Services from one host to another? Part B: For eg. if I have a Kafka job running that is continuosly streaming the data into HDFS, if I stop Cloudera Management Services, will that job get impacted. My assumption is No, however, would like to have expert comments. Thanks snm1529

snm1523 · ‎09-28-2017

Hello @bgoogley, Thank you for the reply. We have already set the swappiness to 1. Still getting these swappiness warnings. Regards, Snm

snm1523 · ‎09-28-2017

Hello Vogone, Thanks for the reply. Yes, swappiness is set to 1. Regards, snm

snm1523 · ‎09-28-2017

Hello, We have been getting regular warning messages of Swap Memory being utilized more than the threshold limit. Currently, we have set swap memory threshold as below: For, HDFS = 100 MB Impala = 30 MB Yarn = 500 MB (Could be +-20%, I dont remember the exact number). Swap Memory usage for each of the component is crossing the threshold and reaches upto 720 MB (in case of Yarn). Because of this, we usually see warnings on our CM dashboard. I know increasing the Swap Memory Threshold could remove these warnings, however, we would prefer rather reducing the usage. Would be great if anyone could suggestion any Memory Tuning options which would reduce the usage of Swap Memory. Also, if there is a best recommendations of setting memory usage threshold, kindly share that as well. Thanks Snm

snm1523 · ‎09-27-2017

Hello, We are in process of implementing a CDH cluster and are preparing a list of test cases from the administrations / operations point of view to check if the implementation is successful and healthy for all components. Is there a list a of recommended test cases one should execute post implementation is completed? Thanks Snm

snm1523 · ‎01-18-2016

Hello, I was able to find a solution to this issue. You would need to add below property to mapred-site.xml basis on your Hadoop version: Hadoop 1.x: <property> <name>mapred.job.tracker</name> <value>localhost:9101</value> </property> Hadoop 2.x: <property> <name>mapreduce.jobtracker.address</name> <value>localhost:9101</value> </property> Thanks snm1523

snm1523 · ‎01-15-2016

Thank you for the prompt response, Harsh. Output of ls -l /tmp/training: [training@localhost ~]$ ls -l /tmp/training/ total 36 -rw-rw-r-- 1 training training 3175 Jan 15 21:14 hive_job_log_training_201601152114_2081367348.txt -rw-rw-r-- 1 training training 3175 Jan 15 21:15 hive_job_log_training_201601152115_1734130449.txt -rw-rw-r-- 1 training training 9762 Jan 15 21:15 hive.log -rw-rw-r-- 1 training training 15704 Jan 15 21:05 training_20160115210505_a8632532-ad5d-40c8-8265-b0147c38655c.log Also, even after setting the HADOOP_CLIENT_OPTS before running the CLI and setting hadoop.tmp.dir=.; I am getting the same error: [training@localhost ~]$ HADOOP_CLIENT_OPTS="-Djava.io.tmpdir=." [training@localhost ~]$ hive Logging initialized using configuration in file:/etc/hive/conf.dist/hive-log4j.properties Hive history file=/tmp/training/hive_job_log_training_201601152151_801169159.txt hive> use case_ipl; OK Time taken: 1.814 seconds hive> select * from bowling_no_partition limit 5; OK 1 RV Uthappa Kolkata 36.4 299 16 18.68 8.15 13.7 2 GJ Maxwell Punjab 14.2 113 3 37.66 7.88 28.6 4 DR Smith Chennai 13.0 135 4 33.75 10.38 19.5 7 SK Raina Chennai 8.0 43 4 10.75 5.37 12.0 8 JP Duminy Delhi 47.5 334 13 25.69 6.98 22.0 Time taken: 0.806 seconds hive> select name from bowling_no_partition; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. Execution log at: /tmp/training/training_20160115215252_e2cea1ec-fc8e-46e4-ac82-6357d76ce6d1.log Job running in-process (local Hadoop) Hadoop job information for null: number of mappers: 1; number of reducers: 0 2016-01-15 21:52:25,188 null map = 0%, reduce = 0% 2016-01-15 21:52:41,173 null map = 100%, reduce = 0% Ended Job = job_1452912287088_0002 with errors Error during job, obtaining debugging information... Examining task ID: task_1452912287088_0002_m_000000 (and more) from job job_1452912287088_0002 Unable to retrieve URL for Hadoop Task logs. Does not contain a valid host:port authority: local Task with the most failures(4): ----- Task ID: task_1452912287088_0002_m_000000 URL: Unavailable ----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"id":1,"name":"RV Uthappa","team":"Kolkata","overs":36.4,"runs":299,"wickets":16,"avg":18.68,"economy":8.15,"strike_rate":13.7} at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"id":1,"name":"RV Uthappa","team":"Kolkata","overs":36.4,"runs":299,"wickets":16,"avg":18.68,"economy":8.15,"strike_rate":13.7} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:548) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) ... 8 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Mkdirs failed to create file:/tmp/training/hive_2016-01-15_21-52-16_114_6229068466610181659/_task_tmp.-ext-10001 at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:237) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:477) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:525) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529) ... 9 more Caused by: java.io.IOException: Mkdirs failed to create file:/tmp/training/hive_2016-01-15_21-52-16_114_6229068466610181659/_task_tmp.-ext-10001 at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:434) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:420) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:805) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:685) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:674) at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:246) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:234) ... 20 more Execution failed with exit status: 2 Obtaining error information Task failed! Task ID: Stage-1 Logs: /tmp/training/hive.log FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask hive> set hadoop.tmp.dir=.; hive> select name from bowling_no_partition; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. Execution log at: /tmp/training/training_20160115215252_1aa2c785-f1ba-45db-b3c1-4a2f45a26be6.log Job running in-process (local Hadoop) Hadoop job information for null: number of mappers: 1; number of reducers: 0 2016-01-15 21:52:58,406 null map = 0%, reduce = 0% 2016-01-15 21:53:15,462 null map = 100%, reduce = 0% Ended Job = job_1452912287088_0003 with errors Error during job, obtaining debugging information... Examining task ID: task_1452912287088_0003_m_000000 (and more) from job job_1452912287088_0003 Unable to retrieve URL for Hadoop Task logs. Does not contain a valid host:port authority: local Task with the most failures(4): ----- Task ID: task_1452912287088_0003_m_000000 URL: Unavailable ----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"id":1,"name":"RV Uthappa","team":"Kolkata","overs":36.4,"runs":299,"wickets":16,"avg":18.68,"economy":8.15,"strike_rate":13.7} at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"id":1,"name":"RV Uthappa","team":"Kolkata","overs":36.4,"runs":299,"wickets":16,"avg":18.68,"economy":8.15,"strike_rate":13.7} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:548) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) ... 8 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Mkdirs failed to create file:/tmp/training/hive_2016-01-15_21-52-49_840_1514077634964084555/_task_tmp.-ext-10001 at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:237) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:477) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:525) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529) ... 9 more Caused by: java.io.IOException: Mkdirs failed to create file:/tmp/training/hive_2016-01-15_21-52-49_840_1514077634964084555/_task_tmp.-ext-10001 at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:434) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:420) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:805) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:685) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:674) at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:246) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:234) ... 20 more Execution failed with exit status: 2 Obtaining error information Task failed! Task ID: Stage-1 Logs: /tmp/training/hive.log FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask hive> Thanks snm1523

snm1523 · ‎01-15-2016

Hello Harsh, I am back with the same issue. Using a fresh copy of the VM. 1. Have verified that /tmp/training folder exists 2. Have cleaned the folder and tried the same query again its not working 3. Have checked the permissions of tmp and training folder and seems correct: drwxrwxrwt. 34 root root 4096 Jan 15 20:29 tmp drwxrwxrwx 2 training training 4096 Jan 15 20:28 training 4. NodeManager service is also running. Still I get below error while executing select name from batting; hive> select name from batting; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files. Execution log at: /tmp/training/training_20160115203535_f4fdd957-7680-48e3-b7d4-ac26a73b48b8.log Job running in-process (local Hadoop) Hadoop job information for null: number of mappers: 1; number of reducers: 0 2016-01-15 20:35:18,502 null map = 0%, reduce = 0% 2016-01-15 20:35:34,393 null map = 100%, reduce = 0% Ended Job = job_1452671681915_0035 with errors Error during job, obtaining debugging information... Examining task ID: task_1452671681915_0035_m_000000 (and more) from job job_1452671681915_0035 Unable to retrieve URL for Hadoop Task logs. Does not contain a valid host:port authority: local Task with the most failures(4): ----- Task ID: task_1452671681915_0035_m_000000 URL: Unavailable ----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"id":29,"name":"PA Patel","runs":205,"high_score":57,"average":20.5,"strike_rate":110.81,"sixes":1,"team":"Bangalore"} at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"id":29,"name":"PA Patel","runs":205,"high_score":57,"average":20.5,"strike_rate":110.81,"sixes":1,"team":"Bangalore"} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:548) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143) ... 8 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Mkdirs failed to create file:/tmp/training/hive_2016-01-15_20-35-09_549_8437051062805036433/_task_tmp.-ext-10001 at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:237) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:477) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:525) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:531) ... 9 more Caused by: java.io.IOException: Mkdirs failed to create file:/tmp/training/hive_2016-01-15_20-35-09_549_8437051062805036433/_task_tmp.-ext-10001 at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:434) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:420) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:805) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:685) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:674) at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:246) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:234) ... 20 more Execution failed with exit status: 2 Obtaining error information Task failed! Task ID: Stage-1 Logs: /tmp/training/hive.log FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask Request you to please help on this. Thanks snm1523

snm1523 · ‎01-11-2016

Thank you for the help, Harsh. Found NM service was down, restarted the service and its working now.

Online	Offline
Last Visited	‎10-10-2019 07:41 AM

Date Registered	‎10-29-2015 07:36 PM
Last Visited	‎10-10-2019 07:41 AM
Total Messages Posted	40
Total Kudos Received	7

Re: MKDirs failed to create file

HBase replication from Multiple HBase clusters

Report or alert based CPU and Memory usage

Re: Calculate File Descriptor in HBase

Re: Calculate File Descriptor in HBase

Calculate File Descriptor in HBase

Re: Flooded with failed health test alerts

Re: Flooded with failed health test alerts

Re: Best Practice to monitor alerts from Cloudera ...

Re: Best Practice to monitor alerts from Cloudera ...

Flooded with failed health test alerts

Best Practice to monitor alerts from Cloudera Mana...

Re: Cloudera Manager user activity audit

Cloudera Manager user activity audit

Re: Could not get LDAP details for groups in patte...

Re: Cloudera Manager Installation failed with Post...

Cloudera Manager Installation failed with PostgreS...

CM Installation Failed with PostgreSQL 8.4 depende...

Canary Tests in Cloudera Manager

Re: Hive does not run query: error 2

Re: HIVE Query Fails but can preview data

Recommendations for Cloudera Hadoop Installation

Re: Migrate cloudera management service on other h...

Re: Swap Memory Usage for HDFS, Yarn and Impala

Re: Swap Memory Usage for HDFS, Yarn and Impala

Swap Memory Usage for HDFS, Yarn and Impala

Test cases / scenarios for Cloudera Hadoop

Re: MKDirs failed to create file

Re: MKDirs failed to create file

Re: MKDirs failed to create file

Re: MKDirs failed to create file