Member since
10-11-2013
25
Posts
5
Kudos Received
3
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 12878 | 05-11-2017 03:07 PM | |
| 3236 | 08-13-2014 03:46 AM | |
| 2322 | 12-10-2013 07:51 AM |
05-11-2017
03:07 PM
3 Kudos
Have u tried this: hive.server2.parallel.ops.in.session=true https://github.com/cloudera/hue/issues/515 Regards, Andrey
... View more
08-13-2014
03:46 AM
Hi Pavel, I had the same problem. You can try this: 1) Open /opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/hue/apps/search/src/search/views.py 2) Add this before LOG = logging.getLogger(__name__) : import sys reload(sys) # Reload does the trick! sys.setdefaultencoding('UTF8') 3) Restart Hue service. Hope this helps. Update: Here is better solution: https://issues.cloudera.org/browse/HUE-2279 P.S. По аватарке думается, что вы из России. Так ли это? Regrds, Andrey
... View more
12-10-2013
07:51 AM
Resolved. I was using WholeFileInputFormat and used value.getBytes() instead of value.copyBytes().
... View more
12-10-2013
03:23 AM
Hello, I have written java Map-only program for sending files from folder on HDFS to Solr. It is working fine for all files, except *.docx. (program v1). I also modified this program to run as simple java program without hadoop. It takes files from ext3 filesystem and sends to Solr. (program v2). Using program v2 I was able to send my *.docx file to solr, but if I put it in HDFS and start MapReduce program (v1) i can't index it. I get such error:: org.apache.solr.common.SolrException: org.apache.tika.exception.TikaException: Unexpected RuntimeException from org.apache.tika.parser.microsoft.ooxml.OOXMLParser@42e20459 at org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:225) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequest(RequestHandlers.java:241) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1909) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:739) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:380) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:169) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.solr.servlet.ProxyUserFilter.doFilter(ProxyUserFilter.java:241) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.solr.servlet.SolrHadoopAuthenticationFilter$2.doFilter(SolrHadoopAuthenticationFilter.java:140) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:384) at org.apache.solr.servlet.SolrHadoopAuthenticationFilter.doFilter(SolrHadoopAuthenticationFilter.java:145) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.solr.servlet.HostnameFilter.doFilter(HostnameFilter.java:86) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.tika.exception.TikaException: Unexpected RuntimeException from org.apache.tika.parser.microsoft.ooxml.OOXMLParser@42e20459 at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:244) at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242) at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120) at org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(ExtractingDocumentLoader.java:219) ... 30 more Caused by: org.apache.poi.openxml4j.exceptions.InvalidOperationException: Can't open the specified file: '/var/lib/solr/apache-tika-242864397475446795.tmp' at org.apache.poi.openxml4j.opc.ZipPackage.<init>(ZipPackage.java:103) at org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:207) at org.apache.tika.parser.microsoft.ooxml.OOXMLExtractorFactory.parse(OOXMLExtractorFactory.java:70) at org.apache.tika.parser.microsoft.ooxml.OOXMLParser.parse(OOXMLParser.java:82) at org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242) ... 33 more Also in folder /var/lib/solr/ there are no file apache-tika-242864397475446795.tmp Any suggestions?
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Solr
-
HDFS
-
MapReduce
-
Security
10-28-2013
10:44 AM
No, I stoped all jobes before changing configuration in MapReduce service, and restarted all cluster. I also checked folder mapred/local/taskTracker/hdfs/jobcache and i am sure it was empty. Thank you for the link, but I found nothing about jobcache folder. Also, after the job failes or completed job folder are deleted from jobcache folder whith all attemp_xxxx folders.
... View more
10-28-2013
08:36 AM
Hey Chris, I'm sorry, I was wrong. It did not help. I have set MapReduce Service Configuration Safety Valve for mapred-site.xml to <property> <name>mapreduce.tasktracker.local.cache.numberdirectories</name> <value>2000</value> </property> And now I see that I have no space on disk. In folder mapred/local/taskTracker/hdfs/jobcache/job_201310281050_0001 there are more then 6000 files. (On the other host there are more then 5000). P.S. I cheked job.xml and there are value mapreduce.tasktracker.local.cache.numberdirectories and it is set to 2000. Thanks Andrey
... View more
10-26-2013
12:29 AM
Hey Chris, I went to Services>MapReduce>Configuration(View and Edit) and past in search mapreduce.tasktracker.local.cache.numberdirectories. I found nothing. Also I typed only cache and only local, and also nothing. Thanks Markovich
... View more
10-26-2013
12:24 AM
Hey Chris, Yes, new workflows work until JT failover. I am runnning my jobs from command line now. I have another strange thing. Here is output of my MapReduce Job: 13/10/25 18:00:46 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 13/10/25 18:00:56 INFO input.FileInputFormat: Total input paths to process : 12451 13/10/25 18:01:06 INFO mapred.JobClient: Running job: job_201310251009_0004 13/10/25 18:01:07 INFO mapred.JobClient: map 0% reduce 0% 13/10/25 18:03:48 INFO mapred.JobClient: map 1% reduce 0% 13/10/25 18:08:25 INFO mapred.JobClient: map 2% reduce 0% 13/10/25 18:12:58 INFO mapred.JobClient: map 3% reduce 0% 13/10/25 18:17:21 INFO mapred.JobClient: map 4% reduce 0% 13/10/25 18:21:53 INFO mapred.JobClient: map 5% reduce 0% 13/10/25 18:26:26 INFO mapred.JobClient: map 6% reduce 0% 13/10/25 18:30:59 INFO mapred.JobClient: map 7% reduce 0% 13/10/25 18:35:26 INFO mapred.JobClient: map 8% reduce 0% 13/10/25 18:39:51 INFO mapred.JobClient: map 9% reduce 0% 13/10/25 18:44:29 INFO mapred.JobClient: map 10% reduce 0% 13/10/25 18:48:53 INFO mapred.JobClient: map 11% reduce 0% 13/10/25 18:53:30 INFO mapred.JobClient: map 12% reduce 0% 13/10/25 18:57:52 INFO mapred.JobClient: map 13% reduce 0% 13/10/25 19:02:11 INFO mapred.JobClient: map 14% reduce 0% 13/10/25 19:06:32 INFO mapred.JobClient: map 15% reduce 0% 13/10/25 19:10:59 INFO mapred.JobClient: map 16% reduce 0% 13/10/25 19:15:19 INFO mapred.JobClient: map 17% reduce 0% 13/10/25 19:19:38 INFO mapred.JobClient: map 18% reduce 0% 13/10/25 19:23:55 INFO mapred.JobClient: map 19% reduce 0% 13/10/25 19:28:27 INFO mapred.JobClient: map 20% reduce 0% 13/10/25 19:32:46 INFO mapred.JobClient: map 21% reduce 0% 13/10/25 19:37:16 INFO mapred.JobClient: map 22% reduce 0% 13/10/25 19:41:44 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10. Trying to fail over immediately. 13/10/25 19:41:44 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 1 fail over attempts. Trying to fail over after sleeping for 1200ms. 13/10/25 19:41:45 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 2 fail over attempts. Trying to fail over after sleeping for 1403ms. 13/10/25 19:41:47 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 3 fail over attempts. Trying to fail over after sleeping for 2140ms. 13/10/25 19:41:49 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 4 fail over attempts. Trying to fail over after sleeping for 1934ms. 13/10/25 19:41:55 INFO mapred.JobClient: map 0% reduce 0% 13/10/25 19:44:50 INFO mapred.JobClient: map 1% reduce 0% 13/10/25 19:49:18 INFO mapred.JobClient: map 2% reduce 0% 13/10/25 19:53:51 INFO mapred.JobClient: map 3% reduce 0% 13/10/25 19:58:23 INFO mapred.JobClient: map 4% reduce 0% 13/10/25 20:02:53 INFO mapred.JobClient: map 5% reduce 0% 13/10/25 20:07:23 INFO mapred.JobClient: map 6% reduce 0% 13/10/25 20:11:54 INFO mapred.JobClient: map 7% reduce 0% 13/10/25 20:16:25 INFO mapred.JobClient: map 8% reduce 0% 13/10/25 20:20:54 INFO mapred.JobClient: map 9% reduce 0% 13/10/25 20:25:16 INFO mapred.JobClient: map 10% reduce 0% 13/10/25 20:29:54 INFO mapred.JobClient: map 11% reduce 0% .... .... ... 13/10/26 03:28:34 INFO mapred.JobClient: map 54% reduce 0% 13/10/26 03:32:51 INFO mapred.JobClient: map 55% reduce 0% 13/10/26 03:34:58 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10. Trying to fail over immediately. 13/10/26 03:34:58 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 1 fail over attempts. Trying to fail over after sleeping for 862ms. 13/10/26 03:34:59 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 2 fail over attempts. Trying to fail over after sleeping for 1076ms. 13/10/26 03:35:00 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 3 fail over attempts. Trying to fail over after sleeping for 1243ms. 13/10/26 03:35:02 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 4 fail over attempts. Trying to fail over after sleeping for 983ms. 13/10/26 03:35:06 INFO mapred.JobClient: map 0% reduce 0% 13/10/26 03:38:08 INFO mapred.JobClient: map 1% reduce 0% 13/10/26 03:42:36 INFO mapred.JobClient: map 2% reduce 0% 13/10/26 03:47:05 INFO mapred.JobClient: map 3% reduce 0% 13/10/26 03:51:34 INFO mapred.JobClient: map 4% reduce 0% 13/10/26 03:56:01 INFO mapred.JobClient: map 5% reduce 0% 13/10/26 04:00:36 INFO mapred.JobClient: map 6% reduce 0% 13/10/26 04:05:01 INFO mapred.JobClient: map 7% reduce 0% 13/10/26 04:09:31 INFO mapred.JobClient: map 8% reduce 0% 13/10/26 04:13:55 INFO mapred.JobClient: map 9% reduce 0% 13/10/26 04:18:27 INFO mapred.JobClient: map 10% reduce 0% 13/10/26 04:23:09 INFO mapred.JobClient: map 11% reduce 0% 13/10/26 04:27:42 INFO mapred.JobClient: map 12% reduce 0% 13/10/26 04:31:59 INFO mapred.JobClient: map 13% reduce 0% 13/10/26 04:36:21 INFO mapred.JobClient: map 14% reduce 0% 13/10/26 04:40:49 INFO mapred.JobClient: map 15% reduce 0% 13/10/26 04:45:20 INFO mapred.JobClient: map 16% reduce 0% 13/10/26 04:49:38 INFO mapred.JobClient: map 17% reduce 0% 13/10/26 04:53:59 INFO mapred.JobClient: map 18% reduce 0% 13/10/26 04:58:21 INFO mapred.JobClient: map 19% reduce 0% 13/10/26 05:02:42 INFO mapred.JobClient: map 20% reduce 0% 13/10/26 05:06:57 INFO mapred.JobClient: map 21% reduce 0% 13/10/26 05:11:20 INFO mapred.JobClient: map 22% reduce 0% 13/10/26 05:15:36 INFO mapred.JobClient: map 23% reduce 0% 13/10/26 05:19:55 INFO mapred.JobClient: map 24% reduce 0% 13/10/26 05:24:26 INFO mapred.JobClient: map 25% reduce 0% 13/10/26 05:28:48 INFO mapred.JobClient: map 26% reduce 0% 13/10/26 05:33:16 INFO mapred.JobClient: map 27% reduce 0% 13/10/26 05:37:43 INFO mapred.JobClient: map 28% reduce 0% 13/10/26 05:42:09 INFO mapred.JobClient: map 29% reduce 0% 13/10/26 05:46:36 INFO mapred.JobClient: map 30% reduce 0% 13/10/26 05:48:53 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10. Trying to fail over immediately. 13/10/26 05:48:53 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 1 fail over attempts. Trying to fail over after sleeping for 1184ms. 13/10/26 05:48:55 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 2 fail over attempts. Trying to fail over after sleeping for 1239ms. 13/10/26 05:48:56 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 3 fail over attempts. Trying to fail over after sleeping for 2134ms. 13/10/26 05:48:58 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 4 fail over attempts. Trying to fail over after sleeping for 1728ms. 13/10/26 05:49:01 INFO mapred.JobClient: map 0% reduce 0% 13/10/26 05:52:05 INFO mapred.JobClient: map 1% reduce 0% 13/10/26 05:56:44 INFO mapred.JobClient: map 2% reduce 0% 13/10/26 06:01:27 INFO mapred.JobClient: map 3% reduce 0% 13/10/26 06:06:02 INFO mapred.JobClient: map 4% reduce 0% 13/10/26 06:10:34 INFO mapred.JobClient: map 5% reduce 0% 13/10/26 06:15:13 INFO mapred.JobClient: map 6% reduce 0% 13/10/26 06:19:49 INFO mapred.JobClient: map 7% reduce 0% 13/10/26 06:21:46 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10. Trying to fail over immediately. 13/10/26 06:21:46 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 1 fail over attempts. Trying to fail over after sleeping for 571ms. 13/10/26 06:21:47 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 2 fail over attempts. Trying to fail over after sleeping for 1910ms. 13/10/26 06:21:49 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 3 fail over attempts. Trying to fail over after sleeping for 1982ms. 13/10/26 06:21:51 WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10 after 4 fail over attempts. Trying to fail over after sleeping for 1868ms. 13/10/26 06:21:56 INFO mapred.JobClient: map 0% reduce 0% 13/10/26 06:24:52 INFO mapred.JobClient: map 1% reduce 0% 13/10/26 06:29:42 INFO mapred.JobClient: map 2% reduce 0% 13/10/26 06:34:32 INFO mapred.JobClient: map 3% reduce 0% 13/10/26 06:39:12 INFO mapred.JobClient: map 4% reduce 0% 13/10/26 06:43:48 INFO mapred.JobClient: map 5% reduce 0% The job was running while one JT (active) failed (? or was forced off by the other standby JT) and the second standby JT became active. It's all fine, they just switched roles, but the job restarted. I started from 0% of mappers. And so every time JT switch roles. Maybe I did not properly configured failover controller? This is very strange message, I can't find anything about it. (WARN retry.RetryInvocationHandler: Exception while invoking getTaskCompletionEvents of class $Proxy10. Trying to fail over immediately.) In JT logs here are only WARN: WARN mapreduce.Counters Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead Thanks Andrey
... View more