About Markovich

Markovich · ‎05-11-2017

Have u tried this: hive.server2.parallel.ops.in.session=true https://github.com/cloudera/hue/issues/515 Regards, Andrey

Markovich · ‎12-01-2015

Hi Sai-krish, Of course. <global> <job-xml>${hbase_site}</job-xml> </global> Are your hbase using ACL? Regards, Andrey

Markovich · ‎11-30-2015

Hi everyone, Here is my problem: Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.security.AccessDeniedException): org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions (user=oozie/XXXXXXX, scope=default, params=[namespace=default,table=default:XX_XX.,family=v],action=CREATE) And here is my workflow.xml: <workflow-app name="HBASE_TEST" xmlns="uri:oozie:workflow:0.5"> <global> <job-xml>${hbase_site}</job-xml> </global> <credentials> <credential name="hbase" type="hbase"> </credential> </credentials> <start to="java-2e83"/> <kill name="Kill"> <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <action name="java-2e83" cred="hbase"> <java> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <main-class>test.beginTest</main-class> <arg>${hdfs_home}</arg> <arg>${zookeeper}</arg> <file>${hbase_site}#${hbase_site}</file> </java> <ok to="End"/> <error to="Kill"/> </action> <end name="End"/> </workflow-app> Curent user in hue is not oozie. And this user can explore Hbase using hbase app. Also it can select any information from Hbase using hive. (current user has all privileges in Hbase). So, how can I make oozie to use current user to access HBase from java application? P.S. Hue - 3.7.0, HBase - 1.0.0 and CDH is 5.4.7. Regards, Andrey

Markovich · ‎03-19-2015

Hi, try this in your driver: boolean result = job.waitForCompletion(true); Counters counters = job.getCounters(); long totalErrorsInLines = counters.findCounter(CountersEnum.ERRORS_IN_LINES).getValue(); long lines = counters.findCounter(CountersEnum.RECORDS_READED).getValue(); float percent = 0; if ( lines != 0) percent = ( totalErrorsInLines * 100.0f) /(lines); if (percent >= totalBadLinesPercent) { //Send email, put to Solr, do any thing you want }

Markovich · ‎03-19-2015

Hi, I need to read a lot of RC Files and combine them to 1 big. So I am using RCFileMapReduceInputFormat and RCFileMapReduceOutputFormat. And I set job.setNumReduceTasks(1); Now I need somehow take number of records from any RC File in mapper of reducer and put it in config. context.getConfiguration().setIfUnset(RCFile.COLUMN_NUMBER_CONF_STR,String.valueOf(value.size())); - not working. Just context.getConfiguration().setInt() - not working. Is there any way to do this? Regards, Markovich

Markovich · ‎02-12-2015

Hello everyone, I am using Sqoop (1) from Java. Here is some code: Configuration conf = new Configuration(); SqoopOptions options = new SqoopOptions(); options = new SqoopOptions(conf); options.set()... ... ... ImportTool importTool = new ImportTool(); importTool.run(options) How can I get job_id or even Job object or may be full log of this Job? I need to get some information from job counters. Regards, Markovich

Markovich · ‎08-13-2014

Hi Pavel, I had the same problem. You can try this: 1) Open /opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/hue/apps/search/src/search/views.py 2) Add this before LOG = logging.getLogger(__name__) : import sys reload(sys) # Reload does the trick! sys.setdefaultencoding('UTF8') 3) Restart Hue service. Hope this helps. Update: Here is better solution: https://issues.cloudera.org/browse/HUE-2279 P.S. По аватарке думается, что вы из России. Так ли это? Regrds, Andrey

Markovich · ‎03-24-2014

Hello, I've got table in oracle database which is partitioned by Date field - start_time. I would like to import it into hadoop with sqoop. But if I will split by start_time I will get such error: ${TABLE} where START_TIME >= TO_DATE('2013-09-09', 'YYYY-MM-DD HH24:MI:SS') and START_TIME <= TO_DATE('2013-09-09', 'YYYY-MM-DD HH24:MI:SS') and ( Start_time >= '2013-09-09 00:00:00.0' ) AND ( Start_time <= '2013-09-09 00:00:00.0' ) java.sql.SQLDataException: ORA - 01861 : literal does not match format string That's because sqoop tries to compare string and Date. I have tried with "-D mapDateToTimestamp=false" but didn't help. I can't change source table. Any ideas? Regards, Markovich

Markovich · ‎03-06-2014

Hi, I've got Oracle Exadata (4 inctances) and 20 node Hadoop cluster. (max 400 mappers) I need to export data from Oracle Database to Hadoop. Data is selected by the query. (example: Select * from table where data between start_date and end_date). I am using oraoop connector for sqoop. The data in Oracle is in not normal distribution. So If I say sqoop to import with 40 mappers and split by id column, 35 mappers will get 100 rows and last 5 will get 1 million of rows. So the job will run with only 5 mappers. Also there are problem with parallelizm. If don't say number of mappers, sqoop+oraoop will do the job with 4 mappers. For Oracle it is 4 session, each will have 36 flows. Because of the bad distribution of data, first 3 mappers will complete very fast and 4th will get all data and only 36 flows. I also thied to split-by ROWID, but sqoop don't allow me. What is the best practice to export such data from Oracle to Hadoop? How to make it distibute normally? And small question: Is it possible to use sqoop-export with multiple export-dir folders? Best regards, Andrey

Markovich · ‎12-10-2013

Resolved. I was using WholeFileInputFormat and used value.getBytes() instead of value.copyBytes().

Markovich · ‎12-10-2013

Hello, I have written java Map-only program for sending files from folder on HDFS to Solr. It is working fine for all files, except *.docx. (program v1). I also modified this program to run as simple java program without hadoop. It takes files from ext3 filesystem and sends to Solr. (program v2). Using program v2 I was able to send my *.docx file to solr, but if I put it in HDFS and start MapReduce program (v1) i can't index it. I get such error:: org.apache.solr.common.SolrException: org.apache.tika.exception.TikaException: Unexpected RuntimeException from org.apache.tika.parser.microsoft.ooxml.OOXMLParser@42e20459 at org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:225) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:241) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1909) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:739) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:380) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:169) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.solr.servlet.ProxyUserFilter.doFilter(ProxyUserFilter.java:241) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.solr.servlet.SolrHadoopAuthenticationFilter$2.doFilter(SolrHadoopAuthenticationFilter.java:140) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:384) at org.apache.solr.servlet.SolrHadoopAuthenticationFilter.doFilter(SolrHadoopAuthenticationFilter.java:145) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.solr.servlet.HostnameFilter.doFilter(HostnameFilter.java:86) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.tika.exception.TikaException: Unexpected RuntimeException from org.apache.tika.parser.microsoft.ooxml.OOXMLParser@42e20459 at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:244) at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242) at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120) at org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:219) ... 30 more Caused by: org.apache.poi.openxml4j.exceptions.InvalidOperationException: Can't open the specified file: '/var/lib/solr/apache-tika-242864397475446795.tmp' at org.apache.poi.openxml4j.opc.ZipPackage.<init>(ZipPackage.java:103) at org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:207) at org.apache.tika.parser.microsoft.ooxml.OOXMLExtractorFactory.parse(OOXMLExtractorFactory.java:70) at org.apache.tika.parser.microsoft.ooxml.OOXMLParser.parse(OOXMLParser.java:82) at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242) ... 33 more Also in folder /var/lib/solr/ there are no file apache-tika-242864397475446795.tmp Any suggestions?

Markovich · ‎12-10-2013

@Clint Thank you. I tried hadoop-fuse-dfs, put, copyFromLocal. All this commands are not able to put file in "preserve" mode. I also would like to save owner name of the file. P.S. In version CDH5 all command-line options with "preserve" flag are working. I have tested them, but it is not for production yet. And hadoop-fuse-dfs is not working with CDH5 with "preserve" mode.

Markovich · ‎12-05-2013

I have files and I want to process them by hadoop MapReduce. But one of the atributes I would like to use is last modified date and author. Here is my question, how to put files in HDFS and save modified_date and owner? P.S. I tried to mount HDFS by fuse and to use cp -a (or cp -pR) but it didn't help me. Also I tried hadoop dfs -copyFromLocal and hadoop dfs -put. Nothing helped.

Markovich · ‎10-29-2013

Hi all, Is there some way to synchronize local folder with HDFS folder every day? I used hadoop-fuse-dfs to mount HDFS and then put my files there. Now I want all files in HDFS were up to date, and if user chnages file in local folder it automaticaly chanched in HDFS. I was trying to use fsync but it doesn't work with Hadoop fs very well. rsync: rename "/u01/HDFS/mnt/win.121/e38358/.target.json.KTnJyN" -> "win.121/e38358/target.json": Input/output error (5) Can somebody help me? Regards, Markovich

Markovich · ‎10-28-2013

No, I stoped all jobes before changing configuration in MapReduce service, and restarted all cluster. I also checked folder mapred/local/taskTracker/hdfs/jobcache and i am sure it was empty. Thank you for the link, but I found nothing about jobcache folder. Also, after the job failes or completed job folder are deleted from jobcache folder whith all attemp_xxxx folders.

Markovich · ‎10-28-2013

Hey Chris, I'm sorry, I was wrong. It did not help. I have set MapReduce Service Configuration Safety Valve for mapred-site.xml to <property> <name>mapreduce.tasktracker.local.cache.numberdirectories</name> <value>2000</value> </property> And now I see that I have no space on disk. In folder mapred/local/taskTracker/hdfs/jobcache/job_201310281050_0001 there are more then 6000 files. (On the other host there are more then 5000). P.S. I cheked job.xml and there are value mapreduce.tasktracker.local.cache.numberdirectories and it is set to 2000. Thanks Andrey

Markovich · ‎10-27-2013

Hey Chris, Thank you so much, it helped me! Thanks Andrey

Markovich · ‎10-27-2013

Hey Chris, That's ok:) Thanks Andrey

Markovich · ‎10-26-2013

Hey Chris, I went to Services>MapReduce>Configuration(View and Edit) and past in search mapreduce.tasktracker.local.cache.numberdirectories. I found nothing. Also I typed only cache and only local, and also nothing. Thanks Markovich

Markovich · ‎10-26-2013

Hey Chris, Yes, new workflows work until JT failover. I am runnning my jobs from command line now. I have another strange thing. Here is output of my MapReduce Job: 13/10/25 18:00:46 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 13/10/25 18:00:56 INFO input.FileInputFormat: Total input paths to process : 12451 13/10/25 18:01:06 INFO mapred.JobClient: Running job: job_201310251009_0004 13/10/25 18:01:07 INFO mapred.JobClient: map 0% reduce 0% 13/10/25 18:03:48 INFO mapred.JobClient: map 1% reduce 0% 13/10/25 18:08:25 INFO mapred.JobClient: map 2% reduce 0% 13/10/25 18:12:58 INFO mapred.JobClient: map 3% reduce 0% 13/10/25 18:17:21 INFO mapred.JobClient: map 4% reduce 0% 13/10/25 18:21:53 INFO mapred.JobClient: map 5% reduce 0% 13/10/25 18:26:26 INFO mapred.JobClient: map 6% reduce 0% 13/10/25 18:30:59 INFO mapred.JobClient: map 7% reduce 0% 13/10/25 18:35:26 INFO mapred.JobClient: map 8% reduce 0% 13/10/25 18:39:51 INFO mapred.JobClient: map 9% reduce 0% 13/10/25 18:44:29 INFO mapred.JobClient: map 10% reduce 0% 13/10/25 18:48:53 INFO mapred.JobClient: map 11% reduce 0% 13/10/25 18:53:30 INFO mapred.JobClient: map 12% reduce 0% 13/10/25 18:57:52 INFO mapred.JobClient: map 13% reduce 0% 13/10/25 19:02:11 INFO mapred.JobClient: map 14% reduce 0% 13/10/25 19:06:32 INFO mapred.JobClient: map 15% reduce 0% 13/10/25 19:10:59 INFO mapred.JobClient: map 16% reduce 0% 13/10/25 19:15:19 INFO mapred.JobClient: map 17% reduce 0% 13/10/25 19:19:38 INFO mapred.JobClient: map 18% reduce 0% 13/10/25 19:23:55 INFO mapred.JobClient: map 19% reduce 0% 13/10/25 19:28:27 INFO mapred.JobClient: map 20% reduce 0% 13/10/25 19:32:46 INFO mapred.JobClient: map 21% reduce 0% 13/10/25 19:37:16 INFO mapred.JobClient: map 22% reduce 0% 13/10/25 19:41:44 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10. Trying to fail over immediately. 13/10/25 19:41:44 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 1 fail over attempts. Trying to fail over after sleeping for 1200ms. 13/10/25 19:41:45 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 2 fail over attempts. Trying to fail over after sleeping for 1403ms. 13/10/25 19:41:47 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 3 fail over attempts. Trying to fail over after sleeping for 2140ms. 13/10/25 19:41:49 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 4 fail over attempts. Trying to fail over after sleeping for 1934ms. 13/10/25 19:41:55 INFO mapred.JobClient: map 0% reduce 0% 13/10/25 19:44:50 INFO mapred.JobClient: map 1% reduce 0% 13/10/25 19:49:18 INFO mapred.JobClient: map 2% reduce 0% 13/10/25 19:53:51 INFO mapred.JobClient: map 3% reduce 0% 13/10/25 19:58:23 INFO mapred.JobClient: map 4% reduce 0% 13/10/25 20:02:53 INFO mapred.JobClient: map 5% reduce 0% 13/10/25 20:07:23 INFO mapred.JobClient: map 6% reduce 0% 13/10/25 20:11:54 INFO mapred.JobClient: map 7% reduce 0% 13/10/25 20:16:25 INFO mapred.JobClient: map 8% reduce 0% 13/10/25 20:20:54 INFO mapred.JobClient: map 9% reduce 0% 13/10/25 20:25:16 INFO mapred.JobClient: map 10% reduce 0% 13/10/25 20:29:54 INFO mapred.JobClient: map 11% reduce 0% .... .... ... 13/10/26 03:28:34 INFO mapred.JobClient: map 54% reduce 0% 13/10/26 03:32:51 INFO mapred.JobClient: map 55% reduce 0% 13/10/26 03:34:58 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10. Trying to fail over immediately. 13/10/26 03:34:58 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 1 fail over attempts. Trying to fail over after sleeping for 862ms. 13/10/26 03:34:59 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 2 fail over attempts. Trying to fail over after sleeping for 1076ms. 13/10/26 03:35:00 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 3 fail over attempts. Trying to fail over after sleeping for 1243ms. 13/10/26 03:35:02 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 4 fail over attempts. Trying to fail over after sleeping for 983ms. 13/10/26 03:35:06 INFO mapred.JobClient: map 0% reduce 0% 13/10/26 03:38:08 INFO mapred.JobClient: map 1% reduce 0% 13/10/26 03:42:36 INFO mapred.JobClient: map 2% reduce 0% 13/10/26 03:47:05 INFO mapred.JobClient: map 3% reduce 0% 13/10/26 03:51:34 INFO mapred.JobClient: map 4% reduce 0% 13/10/26 03:56:01 INFO mapred.JobClient: map 5% reduce 0% 13/10/26 04:00:36 INFO mapred.JobClient: map 6% reduce 0% 13/10/26 04:05:01 INFO mapred.JobClient: map 7% reduce 0% 13/10/26 04:09:31 INFO mapred.JobClient: map 8% reduce 0% 13/10/26 04:13:55 INFO mapred.JobClient: map 9% reduce 0% 13/10/26 04:18:27 INFO mapred.JobClient: map 10% reduce 0% 13/10/26 04:23:09 INFO mapred.JobClient: map 11% reduce 0% 13/10/26 04:27:42 INFO mapred.JobClient: map 12% reduce 0% 13/10/26 04:31:59 INFO mapred.JobClient: map 13% reduce 0% 13/10/26 04:36:21 INFO mapred.JobClient: map 14% reduce 0% 13/10/26 04:40:49 INFO mapred.JobClient: map 15% reduce 0% 13/10/26 04:45:20 INFO mapred.JobClient: map 16% reduce 0% 13/10/26 04:49:38 INFO mapred.JobClient: map 17% reduce 0% 13/10/26 04:53:59 INFO mapred.JobClient: map 18% reduce 0% 13/10/26 04:58:21 INFO mapred.JobClient: map 19% reduce 0% 13/10/26 05:02:42 INFO mapred.JobClient: map 20% reduce 0% 13/10/26 05:06:57 INFO mapred.JobClient: map 21% reduce 0% 13/10/26 05:11:20 INFO mapred.JobClient: map 22% reduce 0% 13/10/26 05:15:36 INFO mapred.JobClient: map 23% reduce 0% 13/10/26 05:19:55 INFO mapred.JobClient: map 24% reduce 0% 13/10/26 05:24:26 INFO mapred.JobClient: map 25% reduce 0% 13/10/26 05:28:48 INFO mapred.JobClient: map 26% reduce 0% 13/10/26 05:33:16 INFO mapred.JobClient: map 27% reduce 0% 13/10/26 05:37:43 INFO mapred.JobClient: map 28% reduce 0% 13/10/26 05:42:09 INFO mapred.JobClient: map 29% reduce 0% 13/10/26 05:46:36 INFO mapred.JobClient: map 30% reduce 0% 13/10/26 05:48:53 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10. Trying to fail over immediately. 13/10/26 05:48:53 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 1 fail over attempts. Trying to fail over after sleeping for 1184ms. 13/10/26 05:48:55 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 2 fail over attempts. Trying to fail over after sleeping for 1239ms. 13/10/26 05:48:56 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 3 fail over attempts. Trying to fail over after sleeping for 2134ms. 13/10/26 05:48:58 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 4 fail over attempts. Trying to fail over after sleeping for 1728ms. 13/10/26 05:49:01 INFO mapred.JobClient: map 0% reduce 0% 13/10/26 05:52:05 INFO mapred.JobClient: map 1% reduce 0% 13/10/26 05:56:44 INFO mapred.JobClient: map 2% reduce 0% 13/10/26 06:01:27 INFO mapred.JobClient: map 3% reduce 0% 13/10/26 06:06:02 INFO mapred.JobClient: map 4% reduce 0% 13/10/26 06:10:34 INFO mapred.JobClient: map 5% reduce 0% 13/10/26 06:15:13 INFO mapred.JobClient: map 6% reduce 0% 13/10/26 06:19:49 INFO mapred.JobClient: map 7% reduce 0% 13/10/26 06:21:46 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10. Trying to fail over immediately. 13/10/26 06:21:46 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 1 fail over attempts. Trying to fail over after sleeping for 571ms. 13/10/26 06:21:47 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 2 fail over attempts. Trying to fail over after sleeping for 1910ms. 13/10/26 06:21:49 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 3 fail over attempts. Trying to fail over after sleeping for 1982ms. 13/10/26 06:21:51 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 4 fail over attempts. Trying to fail over after sleeping for 1868ms. 13/10/26 06:21:56 INFO mapred.JobClient: map 0% reduce 0% 13/10/26 06:24:52 INFO mapred.JobClient: map 1% reduce 0% 13/10/26 06:29:42 INFO mapred.JobClient: map 2% reduce 0% 13/10/26 06:34:32 INFO mapred.JobClient: map 3% reduce 0% 13/10/26 06:39:12 INFO mapred.JobClient: map 4% reduce 0% 13/10/26 06:43:48 INFO mapred.JobClient: map 5% reduce 0% The job was running while one JT (active) failed (? or was forced off by the other standby JT) and the second standby JT became active. It's all fine, they just switched roles , but the job restarted. I started from 0% of mappers. And so every time JT switch roles. Maybe I did not properly configured failover controller ? This is very strange message, I can't find anything about it. (WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10. Trying to fail over immediately.) In JT logs here are only WARN: WARN mapreduce.Counters Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead Thanks Andrey

Markovich · ‎10-25-2013

Hi, Chris! I am running the workflow from inside HUE. Thanks, for reply. May be you can also help me with this question (http://community.cloudera.com/t5/Batch-Processing-and-Workflow/How-to-limit-jobcache-foledr-size/m-p/2361#U2361) ? Thanks, Andrey

Markovich · ‎10-24-2013

Hello, I have a hadoop cluster with CDH 4.4.0 and HDFS HA and JobTracker HA. I tried to lunch oozie workflow but got this: JA006: Call From xxxx to xxxx failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused I found this: https://issues.cloudera.org/browse/HUE-1631 But I don't know how to fix my problem. Where to configure oozie to work with JobTracker HA?

Markovich · ‎10-17-2013

Hello hadoop experts, I have new problem and new question for you. I have a lot of ( approximately 70,000 )small files, a total of 40 GB. I've developed Map only java program to analysis this files. I have no reducers and have no output. Only counters. I've a single node setup of CDH 4.4.0. I've got hard disk with 150GB. And all Hadoop environment uses it (logs,libs,hdfs data files and so on). So after I put all the files in HDFS, I was left with a little less than 100GB of free space. I have successfully started my job, and 10 hours later Hadoop fell down due to the fact that it hasn't enough space to write the log. I looked on my disk and found than folder mapred/local/taskTracker/hdfs/jobcache/job_xxxx_xxx occupies all available space. Hadoop has processed approximately 10,000 of files, so that in folder was approximately 10,000 subfolders each of which contains only one file job.xml (weight 8MB). So 8mb * 10,000 ~ 78 Gb. And here is my question: How can I process 70,000 of files? (I will need approximately 550GB of free space to process 40 GB of small files!) Is it possible to configure Hadoop to cleanup after every map? Regards, Markovich

Markovich · ‎10-15-2013

Thanks, this solved my problem.

Markovich · ‎10-11-2013

Hello. I have a lot of different files *.doc, *.pdf and so on. I wanted to process them with mapReduce. I put them in HDFS and then started java MapReduce program using Hue. If files are well formated and doesn't have brackets "(){}[]" in their name all goes fine. But if there is a file OPN_last_[age.PDF I get this errors: Failing Oozie Launcher, Main class [distr.fors.ru.Index], main() threw exception, Illegal file pattern: Unclosed character class near index 17 OPN_last_[age.PDF ^ java.io.IOException: Illegal file pattern: Unclosed character class near index 17 OPN_last_[age.PDF ^ at org.apache.hadoop.fs.GlobFilter.init(GlobFilter.java:70) at org.apache.hadoop.fs.GlobFilter.<init>(GlobFilter.java:49) at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1670) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1627) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:211) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:248) at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1063) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1080) at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:992) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945) at org.apache.hadoop.mapreduce.Job.submit(Job.java:566) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596) at distr.fors.ru.Index.run(Index.java:78) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at distr.fors.ru.Index.main(Index.java:39) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:495) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.util.regex.PatternSyntaxException: Unclosed character class near index 17 OPN_last_[age.PDF ^ at org.apache.hadoop.fs.GlobPattern.error(GlobPattern.java:167) at org.apache.hadoop.fs.GlobPattern.set(GlobPattern.java:151) at org.apache.hadoop.fs.GlobPattern.<init>(GlobPattern.java:42) at org.apache.hadoop.fs.GlobFilter.init(GlobFilter.java:66) ... 32 more If there is a file like this: {2011-01-27} (3769330).pdf I get such error: Input Pattern hdfs://fd-bigdata.distr.fors.ru:8020/{2011-01-27} (3769330).pdf matches 0 files at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:231) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:248) at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1063) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1080) at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:992) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945) at org.apache.hadoop.mapreduce.Job.submit(Job.java:566) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596) at distr.fors.ru.Index.run(Index.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at distr.fors.ru.Index.main(Index.java:37) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:495) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) I realy need to process such files. What can I make to solve such problems? P.S. I am using the latest CDH 4.4.0.

Online	Offline
Last Visited	‎07-08-2017 07:36 AM

Date Registered	‎10-11-2013 10:22 AM
Last Visited	‎07-08-2017 07:36 AM
Total Messages Posted	25
Total Kudos Received	5

Re: Hue 3.11 Hive Query editor - Not retrying thri...

Re: error in searching of non-Latin symbols

Re: Error sending *.docx file from HDFS to SOLR

Re: Hue 3.11 Hive Query editor - Not retrying thri...

Re: Oozie is using oozie user from hue to login in...

Oozie is using oozie user from hue to login in hba...

Re: Reporting on Hadoop counters

How to set column number for RCFile in Mapper or R...

How can I get Sqoop job_id from java?

Re: error in searching of non-Latin symbols

sqoop + Oracle problem while splitting by DATE fie...

Exadata + Sqoop + OraOop best practice?

Re: Error sending *.docx file from HDFS to SOLR

Error sending *.docx file from HDFS to SOLR

Re: How to put files in HDFS leaving last_modified...

How to put files in HDFS leaving last_modified dat...

Synchronize local folder and folder in HDFS

Re: How to limit jobcache foledr size?

Re: How to limit jobcache foledr size?

Re: How to limit jobcache foledr size?

Re: Jobtracker HA and oozie

Re: How to limit jobcache foledr size?

Re: Jobtracker HA and oozie

Re: Jobtracker HA and oozie

Jobtracker HA and oozie

How to limit jobcache foledr size?

Re: Hadoop error while processing file with bracke...

Hadoop error while processing file with brackets.