About guillaume_roger

guillaume_roger · ‎09-04-2019

Thanks you nailed it indeed. set hiveconf:tez.am.container.reuse.enabled=false; did the trick.

guillaume_roger · ‎05-03-2019

This query outputs NPE. The tasks with NPEs are retried, and most of the times (but not always) end up succeeding. I could not find a smaller query showing my problem so I give here my full query: select s.ts_utc as sent_dowhour , o.ts_utc as open_dowhour , sum(count(s.ts_utc)) over(partition by s.ts_utc) as sent_count from vault.sent s left join open o on o.id=s.id group by 1, 2 My guess is that the construction sum(count(...)) over(partition by ...) has issues. When it fails, this is the output I get: Vertex failed, vertexName=Reducer 2, vertexId=vertex_1556016846110_42971_7_03, diagnostics= » Task failed, taskId=task_1556016846110_42971_7_03_000221, diagnostics= » TaskAttempt 0 failed, info= » Error: Error while running task ( failure ) : attempt_1556016846110_42971_7_03_000221_0:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:304) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:318) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267) ... 16 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:378) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:294) ... 18 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:795) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:363) ... 19 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.persistence.PTFRowContainer.first(PTFRowContainer.java:115) at org.apache.hadoop.hive.ql.exec.PTFPartition.iterator(PTFPartition.java:114) at org.apache.hadoop.hive.ql.udf.ptf.BasePartitionEvaluator.getPartitionAgg(BasePartitionEvaluator.java:200) at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.evaluateFunctionOnPartition(WindowingTableFunction.java:155) at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.iterator(WindowingTableFunction.java:538) at org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:349) at org.apache.hadoop.hive.ql.exec.PTFOperator.process(PTFOperator.java:123) at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:994) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:940) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:927) at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1050) at org.apache.hadoop.hive.ql.exec.GroupByOperator.processAggr(GroupByOperator.java:850) at org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:724) at org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:790) ... 20 more Semantically my query is valid (and indeed sometimes succeeds) so what is going on? Note: hdp 3.1, hive 3 orc tables, orc intermediate results tez

guillaume_roger · ‎02-15-2019

Short version: How can I get the difference in seconds between 2 timestamps, via the ODBC driver? Long version: Using ODBC for a simple query (not that I use cast (... as timestamp) to have a standalone line, the actual query runs against a table with timestamp data): select unix_timestamp(cast('2019-02-01 01:02:03' as timestamp)) as tto I got the error message: unix_timestamp is not a valid scalar function or procedure call I could not find any configuration option that would change this. Native query is disabled (because I am using prepared statements) and other functions work fine. My guess is that unix_timestamp() (without parameter) is deprecated, and the driver is a bit enthusiastic about preventing using the function. I tried to work around the problem, and I cast the timestamp as bigint instead of using the unix_timestamp function: select cast(cast('2019-02-01 01:02:03' as timestamp) as bigint) This works fine! But when I try to get the diff of 2 timestamps: select cast(cast('2019-02-01 01:02:03' as timestamp) as bigint) - cast(cast('2019-02-01 01:02:03' as timestamp) as bigint) I got the message Operand types SQL_WCHAR and SQL_WCHAR are incompatible for the binary minus operator (but then only for complex queries, not if the query consists only of this select). The driver will accept a diff between 2 timestamps, but then I end up with an interval type, which I cannot convert back to seconds. I would consider that those are bugs in the ODBC drier, but I cannot contact Hortonworks because I am not a paying customer, and I cannot contact Simba either because I am not a paying customer. Any idea how I could get around this?

guillaume_roger · ‎02-15-2019

The ODBC driver does not support all syntax niceties (no CTE) and if there is a syntax error, it will output a completely irrelevant message, which adds a lot to the confusion. To actually see the actual error, you need to add ODBC logging and look at the log files.

guillaume_roger · ‎02-15-2019

@Anika S 2 years later, I have the same issue. Did you manage to fix it? If so, how?

guillaume_roger · ‎01-28-2019

I want to use the new KafkaStorageHandler, which looks awesome. The only thing is that the avro in kafka is not standard (I am looking at you, confluent), I thus need to use my own serde (well, the serde from confluent). I added wihout error to hive the relevant jar which contains io.confluent.kafka.streams.serdes.avro.GenericAvroSerde. add jar hdfs:///tmp/kafka-streams-avro-serde-5.1.0.jar; If I now try to create the external table : CREATE EXTERNAL TABLE click ( ... ) STORED BY 'org.apache.hadoop.hive.kafka.KafkaStorageHandler' TBLPROPERTIES ( "kafka.topic" = "click", "kafka.bootstrap.servers"="kafka:9092" ,"kafka.serde.class"="io.confluent.kafka.streams.serdes.avro.GenericAvroSerde" ); Hive bails out and says: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.RuntimeException: java.lang.ClassNotFoundException: io.confluent.kafka.streams.serdes.avro.GenericAvroSerde (state=08S01,code=1) How can I tell hive where to find this class? Thanks,

guillaume_roger · ‎01-28-2019

Context: Hive3, HDP 3.1. Tests done with Python/odbc (official HDP driver) under Windows and Linux. I ran the following queries: "select ? as lic, ? as cpg" "select * from (select ? as lic, ? as cpg) as t" "with init as (select ? as lic, ? as cpg) select * from init", 1) and 2) work fine, and give me the expected result. 3 gives me a ParseException : Error while compiling statement: FAILED: ParseException line 1:21 cannot recognize input near '?' 'as' 'lic' in select clause (80) (SQLPrepare)") The exact same statements ran with java/jdbc work fine. Note that 2) looks like is a workaround for 3) but it works for this tiny example, not for bigger queries. Is there something I can do to have ODBC working as expected? Alternatively, where can I find the limits of the ODBC driver? For full context, the full test code is as follow: cnxnstr = 'DSN=HiveProd' cnxn = pyodbc.connect(cnxnstr, autocommit=True) cursor = cnxn.cursor() queries = [ "with init as (select ? as lic, ? as cpg) select * from init", "select 2 * ? as lic, ? as cpg", "select * from (select ? as lic, ? as cpg) as t", ] for q in queries: print("\nExecuting " + q) try: cursor.execute(q, '1', '2') except pyodbc.ProgrammingError as e: print(e) continue

guillaume_roger · ‎01-11-2019

In addition to these steps: restart ambari server (we had one instance where it looked like the application was OK but the alert was cached and keep being displayed), check your yarn logs. If there is not enough memory for yarn, the service will not be able to start.

guillaume_roger · ‎01-11-2019

You will lose some job history, but nothing else and certainly no data, so it should not be an issue.

guillaume_roger · ‎01-10-2019

It worked for me eventually after cleaning up *everything*: - destroying the app and cleaning hdfs as explained there: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.1/data-operating-system/content/remove_ats_hbase_before_switching_between_clusters.html - cleaning zookeeper: zookeeper-client rmr /atsv2-hbase-unsecure and finally restarting *all* yarn services from ambari should did the trick.

guillaume_roger · ‎01-07-2019

I got it working by - cleaning up the hdfs directories of hbase-ats - cleaning up the zookeeper nodes related to hbase-hdfs I hope there are better ways, but that's the only one I found out and was working.

guillaume_roger · ‎12-19-2018

In short: I have a working hive on hdp3, which I cannot reach from pyspark, running under yarn (on the same hdp). How do I get pyspark to find my tables? spark.catalog.listDatabases() only show default, any query run will not show in my hive logs. This is my code, with spark 2.3.1 from pyspark.sql import SparkSession from pyspark.conf import SparkConf settings = [] conf = SparkConf().setAppName("Guillaume is here").setAll(settings) spark = ( SparkSession .builder .master('yarn') .config(conf=conf) .enableHiveSupport() .getOrCreate() ) print(spark.catalog.listDatabases()) Note that `settings` is empty. I though it would be sufficient, because in the logs I see loading hive config file: file:/etc/spark2/3.0.1.0-187/0/hive-site.xml and more interestingly Registering function intersectgroups io.x.x.IntersectGroups This is a UDF I wrote and added to hive manually. This means that there is some sort of connection done. The only output I get (except logs) is: [ Database(name=u'default', description=u'default database', locationUri=u'hdfs://HdfsNameService/apps/spark/warehouse')] I understand that I should set `spark.sql.warehouse.dir` in settings. No matter if I set it to the value I find in hive-site, the path to the database I am interested in (it's not in the default location), its parent, nothing changes. I put many other config options in settings (including thrift uris), no changes. I have seen as well that I should copy hive-site.xml into the spark2 conf dir. I did it on all nodes of my cluster, no changes. My command to run is: HDP_VERSION=3.0.1.0-187 PYTHONPATH=.:/usr/hdp/current/spark2-client/python/:/usr/hdp/current/spark2-client/python/lib/py4j-0.10.7-src.zip SPARK_HOME=/usr/hdp/current/spark2-client HADOOP_USER_NAME=hive spark-submit --master yarn --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.1.0-187.jar --py-files /usr/hdp/current/hive_warehouse_connector/pyspark_hwc-1.0.0.3.0.1.0-187.zip --files /etc/hive/conf/hive-site.xml ./subjanal/anal.py

guillaume_roger · ‎11-22-2018

@Aditya Sirna, you are right, hbase runs as a service (is_hbase_system_service_launch is true). I am giving example with nodeN, which are the names of my data nodes, This is based on what I see right now and makes it easier to understand The region server (node5) tries to report for duty but fails. It tries to connect to node1:17020, but port 17020 is only open on node5. On node1 hbase master tried to start, but stopped because it apparently cannot find the active namenode Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null I will look into zookeeper, it seems to ring a bell. I have 2 questions if you don't mind: - how do you start a yarn service on a specic node? - how does the timelinereader know where to connect? In any case thanks, you gave me some ideas to carry on.

guillaume_roger · ‎11-22-2018

Hello, I have a new hdp3.0.1 installation with ats-hbase which runs embedded (with proper queue configured, as per the documentation). At the end of all tasks (seen with the hive compactor, oozie steps), I have hundreds of lines with org.apache.hadoop.yarn.event.AsyncDispatcher: Waiting for AsyncDispatcher to drain. Thread state is :WAITING ending up with : org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Failed to process Event JOB_FINISHED for the job : job_1542872934100_0068 org.apache.hadoop.yarn.exceptions.YarnException: Failed while publishing entity at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher.dispatchEntities(TimelineV2ClientImpl.java:548) at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.putEntities(TimelineV2ClientImpl.java:149) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processEventForNewTimelineService(JobHistoryEventHandler.java:1405) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleTimelineEvent(JobHistoryEventHandler.java:742) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.access$1200(JobHistoryEventHandler.java:93) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$ForwardingEventHandler.handle(JobHistoryEventHandler.java:1795) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$ForwardingEventHandler.handle(JobHistoryEventHandler.java:1791) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126) at java.lang.Thread.run(Thread.java:745) Caused by: com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Call From null to prod-nl-dpnode3.dmdelivery.local:33602 failed on socket timeout exception:t java.lang.Thread.run(Thread.java:745) Caused by: com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Call From null to prod-nl-dpnode3.dmdelivery.local:33602 failed on socket timeout exception Looking at /var/log/hadoop-yarn/yar/hadoop-yarn-nodemanager I have a lot of lines with: Call exception, tries=7, retries=7, started=8194 ms ago, cancelled=false, msg=Call to xxxxx/192.168.x.x:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: prod-nl-dpnode1.dmdelivery.local/192.168.36.161:17020, details=row 'prod.timelineservice.entity,hive!yarn-cluster!xxxx-34-compactor-vault.contact.license_name=lectiva!^?��@@!^?��d��^?��!MAPREDUCE_TASK_ATTEMPT!^?��!attempt_1542205428050_2307_m_000461_0,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740, hostname=xxx,17020,1542294270073, seqNum=-1 Looking at /var/log/hadoop-yarn/yarn/hadoop-yarn-timelinereader, I see Connection refused: dpnode1/192.168.36.161:17020 Indeed, there is no hbase on dpnode1. Hbase does run on dpnode5 (or another one, depending on yarn restart), but in any case, the timelinereader does not know which server to reach, and always goes to one seemingly hardcoded hostname. How can I tell yarn to use the right node to connect to hbase? Thanks,

guillaume_roger · ‎11-13-2018

Eventually, after a restart of everything (not only the services seen as requiring a restart) it went OK.

guillaume_roger · ‎11-01-2018

Hello, I installed a new (not an update) HDP 3.0.1 and seem to have many issues with the timeline server. 1) The first weird thing is that the Yarn tab in ambari keeps showing this error: ATSv2 HBase Application The HBase application reported a 'STARTED' state. Check took 2.125s 2) The second issue seems to be with oozie. Running a job, it starts but stalls with the following log repeated hundreds of times 2018-11-01 11:15:37,842 INFO [Thread-82] org.apache.hadoop.yarn.event.AsyncDispatcher: Waiting for AsyncDispatcher to drain. Thread state is :WAITING Then with: 2018-11-01 11:15:37,888 ERROR [Job ATS Event Dispatcher] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Exception while publishing configs on JOB_SUBMITTED Event for the job : job_1541066376053_0066 org.apache.hadoop.yarn.exceptions.YarnException: Failed while publishing entity at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher.dispatchEntities(TimelineV2ClientImpl.java:548) at org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.putEntities(TimelineV2ClientImpl.java:149) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.publishConfigsOnJobSubmittedEvent(JobHistoryEventHandler.java:1254) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.processEventForNewTimelineService(JobHistoryEventHandler.java:1414) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.handleTimelineEvent(JobHistoryEventHandler.java:742) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.access$1200(JobHistoryEventHandler.java:93) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$ForwardingEventHandler.handle(JobHistoryEventHandler.java:1795) at org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler$ForwardingEventHandler.handle(JobHistoryEventHandler.java:1791) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126) at java.lang.Thread.run(Thread.java:745) Caused by: com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out 3) In hadoop-yarn-timelineserver-${hostname}.log I see, repeated many times: 2018-11-01 11:32:47,715 WARN timeline.EntityGroupFSTimelineStore (LogInfo.java:doParse(208)) - Error putting entity: dag_1541066376053_0144_2 (TEZ_DAG_ID): 6 4) In hadoop-yarn-timelinereader-${hostname}.log I see, repeated many times: Thu Nov 01 11:34:10 CET 2018, RpcRetryingCaller{globalStartTime=1541068444076, pause=1000, maxAttempts=4}, java.net.ConnectException: Call to /192.168.x.x:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.x.x:17020 at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80) ... 3 more Caused by: java.net.ConnectException: Call to /192.168.x.x:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.x.x:17020 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) and indeed, there is nothing listening to port 17020 on 192.168.x.x. 5) I cannot find on any server a process named ats-hbase, this might be the reason for everything else. The queue set is just yarn_hbase_system_service_queue_name=default, which has no limit which would prevent Hbase to start. I am sure that something is very wrong here, and any help would be appreciated.

guillaume_roger · ‎10-29-2018

I was using Oozie 4.2.0 (hdp2.6) and am now trying Oozie 4.3.1 (hdp 3.0). One major difference is that one java action could in the past read its jar from the local filesystem, but it looks like it is not possible anymore. The Java action is basic: <java xmlns="uri:oozie:workflow:0.5"> <job-tracker>http://something.local:8050</job-tracker> <name-node>hdfs://HdfsNameService</name-node> <main-class>io.JsonPoster</main-class> <file>file:///opt/jsonposter/jsonposter.jar</file> </java> The error I get is quite clear: org.apache.oozie.action.ActionExecutorException: UnsupportedOperationException: Accessing local file system is not allowed I know I could put the jar on hdfs put I am trying to avoid that for now (because it used to work and all our deployment are done via rpm). I am ready to take the responsibility of having all jars in sync over all datanodes. I already set oozie.service.HadoopAccessorService.supported.filesystems to * , with no effect. Is there a way to tell Oozie that yes, I am happy for it to read the local FS?

guillaume_roger · ‎04-12-2018

@rtrivedi Thanks for your answer but I believe that's not the issue. I tried a lot of variations with the `hive delete` command, to no avail: delete jar hdfs:///myudfs/myfunc.jar; list jar; --give a localised jar delete jar $localised_jar; CREATE FUNCTION myfunc AS 'io.company.hive.udf.myfunc' USING JAR 'hdfs:///myudfs/myfunc.jar'; And I end up having the same error again.

guillaume_roger · ‎04-12-2018

I created my own (generic) udf, which works very well when added in hive: CREATE FUNCTION myfunc AS 'io.company.hive.udf.myfunc' USING JAR 'hdfs:///myudfs/myfunc.jar'; After a while I wanted to update my udf, so I created a new jar with the same name, and put it in hdfs by overwriting the old jar. Lo and behold, I cannot use my function again! It does not matter if I do first a: drop function if exists myfunc; CREATE FUNCTION myfunc AS 'io.company.hive.udf.myfunc' USING JAR 'hdfs:///myudfs/myfunc.jar'; From beeline, I got one of these error message: java.io.IOException: Previous writer likely failed to write hdfs://ip-10-0-10-xxx.eu-west-1.compute.internal:8020/tmp/hive/hive/_tez_session_dir/0de6055d-190d-41ee-9acb-c6b402969940/hmyfunc.jar Failing because I am unlikely to write too. or org.apache.hadoop.hive.ql.metadata.HiveException: Default queue should always be returned.Hence we should not be here. Looking at the logs, it looks like Hive is localising the jar file (good) but as a session is reused, if the new jar does not match the jar already present in the localised directory hive will complain and will apparently wait indefinitely. If my understanding is correct, is there a way to tell Tez to not reuse any of the current sessions? If my understanding is not correct, is there a way to do what I want? Context: hdp 2.6.0.3, no llap, on aws. Thanks,

guillaume_roger · ‎03-09-2018

I have a query, always failing with the following error: Container exited with a non-zero exit code 1 ]], TaskAttempt 2 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) [...] The query itself is quite a small MERGE (Other much bigger queries work flawlessly): MERGE INTO summary dst USING ( SELECT e.id1 , e.id2 , e.id3 , e.name , e.subject FROM mailing e ) src ON dst.id1 = src.id1 AND dst.id2 = src.id2 AND dst.id3 = src.id3 WHEN MATCHED THEN UPDATE SET name = src.name , subject=src.subject The source table has 1.7M rows (50M on disk), the destination has 75M rows, (1.5GB on disk). Both are ACID tables, ORC. On the image, map 1 is the one with the issue, and I cannot understand why it has only one task. Naively I would think that more tasks would each have a smaller load and would work better, but I did not manage to do that. Note that I maxed out already all memory parameters, I cannot do more on those: yarn-site/yarn.nodemanager.resource.memory-mb = 24064 yarn-site/yarn.scheduler.minimum-allocation-mb = 1024 yarn-site/yarn.scheduler.maximum-allocation-mb = 24064 mapred-site/mapreduce.map.memory.mb = 4096 mapred-site/mapreduce.reduce.memory.mb = 8192 mapred-site/mapreduce.map.java.opts = 3276 mapred-site/mapreduce.reduce.java.opts = 6553 hive-site/hive.tez.container.size = 4096 Is there a way to increase the number of tasks in the mapper, or another way to not get this out of memory error?

guillaume_roger · ‎01-16-2018

@Jordan Moore Not really relevant to the question but no this is not the point. The use case here is data export, where some clients have their own BI tools, processes and so on. They just need the data, csv in a zip file. Other clients do not have this in place and have a different access to this data.

guillaume_roger · ‎01-15-2018

The zip file is the output of the process, not to be read in hdfs anymore - it will just end up being downloaded and sent to a user. In this context using zip makes sense, as I am only looking at *compressing* multiple csv together, not reading them afterwards. Using beeline with formatted output is what I do currently, but I end up downloading locally multiple gigs, compress and re-upload. This is a waste and could actually fill my local disks up. Using coalesce in Spark is the best option I found, but the compression step is still not easy. Thanks!

guillaume_roger · ‎01-12-2018

Hello, I am running on hdp 2.6 on QWS with 8 nodes. Here are my relevant settings: yarn.nodemanager.resource.memory-mb=24gb yarn.scheduler.maximum-allocation-mb=24gb yarn.scheduler.minimum-allocation-mb=1gb tez container size=4gb I have 32GB per DN, I see in my monitoring that only 13GB (max) is used, but I still receive when running a big hive query: received a TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) Could you help me make sense of this? Thanks

guillaume_roger · ‎01-09-2018

My end goal is to run a few hive queries, get 1 csv file (with headers) per query, compress all those files together in one zip (not gzip or bzip, unfortunately, needs to open natively under windows) and hopefully get the zip back in hdfs. My current solution (CTAS) ends up creating one directory per table, with possibly multiple files under it (depending on number of reducers and presence/absence of UNION). I can easily generate as well a header file per table with only one line in it. Now how to put all that together? The only option I could find implies to do all the processing locally (hdfs dfs -getmerge followed by a an actual zip command). This adds a lot of overhead and could technically fill up the local disk. So my questions are: is there a way to concatenate files inside hdfs without getting them locally? is there a way to compress a bunch of files together (not individually) in zip, inside hdfs? Thanks

guillaume_roger · ‎12-11-2017

hdp 2.6.0, HMS has 6GB memory, the metastore itself is mysql. After a few days the server hosting the HMS will have its CPU 100% used, hive queries are slow, and looking at the GC logs the HMS is constantly having stop the world events. Restarting the metastore 'fixes' the problem for a few days. I have found a few JIRA link related to memory leaks https://issues.apache.org/jira/browse/HIVE-15551 or https://issues.apache.org/jira/browse/HIVE-13749 . Is it a known issue in hdp 2.6.0? Is it known if the latest hdp version is fixed? Thanks,

guillaume_roger · ‎12-11-2017

hdp 2.6.0, HMS has 6GB memory, the metastore itself is mysql. After a few days the server hosting the HMS will have its CPU 100% used, hive queries are slow, and looking at the GC logs the HMS is constantly having stop the world events. Restarting the metastore 'fixes' the problem for a few days. I have found a few JIRA link related to memory leaks https://issues.apache.org/jira/browse/HIVE-15551 or https://issues.apache.org/jira/browse/HIVE-13749 . Is it a known issue in hdp 2.6.0? Is it known if the latest hdp version is fixed? Thanks,

guillaume_roger · ‎11-22-2017

TL;DR; how to properly set up hive.tez.container.size for a job with wildly different steps? I have a 8 data node hdp2.6 cluster, all data nodes are identical, with 32GB ram. yarn.scheduler.maximum-allocation-mb is set up to the total server ram minus what is used by other services (OS, nodemanager...), ie. 20GB in my case, yarn.scheduler.minimum-allocation-mb is set up to 1GB, I am running only one hive MERGE statement, once per day, which has about 100k mappers. If I set up hive.tez.container.size to 1GB, many mappers can run in parallel (faster query), but I will end up with one of those errors: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Reducer 3, vertexId=vertex_1510697553800_0993_2_03, diagnostics=[Task failed, taskId=task_1510697553800_0993_2_03_000150, diagnostics=[TaskAttempt 0 failed, info=[Container container_e102_1510697553800_0993_01_000042 finished with diagnostics set to [Container failed, exitCode=-104. Container [pid=32295,containerID=container_e102_1510697553800_0993_01_000042] is running beyond physical memory limits. Current usage: 5.4 GB of 5.3 GB physical memory used; 7.4 GB of 11.0 GB virtual memory used. Killing container. Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Reducer 3, vertexId=vertex_1511269090751_0011_2_03, diagnostics=[Exception in VertexManager, vertex:vertex_1511269090751_0011_2_03 [Reducer 3],org.apache.tez.dag.api.TezUncheckedException: Atleast 1 bipartite source should exist. If I set up hive.tez.container.size to a bigger value I will run a lot less queries in parallel (longer query time) but eventually the query will succeed. The thing is that I do not know in advance how big the data will be so even if I by trial and error find a good hive.tez.container.size it might not be good enough tomorrow, and maybe eventually my server memory will be too small . Further more, sizing for the worst case scenario feels like a waste of resource. Is there any way to have a sort of dynamic tez container size to get a fast and succeeding query? Cheers,

guillaume_roger · ‎10-24-2017

Then I interpret this setting as "if there is too much data let's use it all instead of pruning it" and am very confused 🙂 I suppose it's due to internal hive implementation as you said.

guillaume_roger · ‎10-23-2017

I indeed see INFO [HiveServer2-Handler-Pool: Thread-107]: optimizer.RemoveDynamicPruningBySize (RemoveDynamicPruningBySize.java:process(61)) - Disabling dynamic pruning for: TS. Expected data size is too big: 1119008712 So if I understand well, this has to do with event size and not data size? I did try to get the value very high to enable pruning, pruning did indeed occur but locking all partitions timed out. Will post an explain asap.

guillaume_roger · ‎10-23-2017

I added this explain.

Online	Offline
Last Visited	‎09-05-2019 01:53 AM

Date Registered	‎10-13-2016 09:10 AM
Last Visited	‎09-05-2019 01:53 AM
Total Messages Posted	63
Total Kudos Received	7

Re: Hive odbc with prepared statements: ParseExcep...

Re: Fix under replicated blocks very slow

Re: Adding a host to ambari from another DC behind...

Re: NullPointerException (but not always) in Group...

NullPointerException (but not always) in GroupBy i...

Hive ODBC cannot use unix_timestamp

Re: Hive odbc with prepared statements: ParseExcep...

Re: Getting the following error while running sql ...

Use my own serde with KafkaStorageHandler

Hive odbc with prepared statements: ParseException

Re: ATS hbase does not seem to start

Re: ATS hbase does not seem to start

Re: ATS hbase does not seem to start

Re: ATS2-hbase starts but on the wrong node

Pyspark cannot reach hive

Re: ATS2-hbase starts but on the wrong node

ATS2-hbase starts but on the wrong node

Re: ATS hbase does not seem to start

ATS hbase does not seem to start

Oozie cannot access local filesystem

Re: How to update the jar of a udf

How to update the jar of a udf

java.lang.OutOfMemoryError: in tez

Re: Concatenate and zip files in hdfs

Re: Concatenate and zip files in hdfs

Yarn can use 24GB of ram, but gives oom with only ...

Concatenate and zip files in hdfs

Hive metastore going oom after a few days

Hive metastore leaking memory?

yarn & tez container memory

Re: Why not set hive.tez.dynamic.partition.pruning...

Re: Why not set hive.tez.dynamic.partition.pruning...

Re: MERGE statement and partition pruning