Hi, recently i noticed a lot of failures for jobs submitted to Yarn, and all are giving same error below: 2019-07-05 03:10:12,513 INFO [main] org.apache.hadoop.mapreduce.Job: Job job_1561377786220_19198 failed with state FAILED due to: Application application_1561377786220_19198 failed 2 times due to Error launching appattempt_1561377786220_19198_000002. Got exception: java.net.SocketTimeoutException: Call From NAME_NODE/xx.xx.xx.xx to DATA_NODE:45454 failed on socket timeout exception: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/xx.xx.xx.xx:33870 remote=DATA_NODE/xx.xx.xx.xx:45454]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout at sun.reflect.GeneratedConstructorAccessor250.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:775) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1501) at org.apache.hadoop.ipc.Client.call(Client.java:1443) at org.apache.hadoop.ipc.Client.call(Client.java:1353) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) at com.sun.proxy.$Proxy111.startContainers(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:128) at sun.reflect.GeneratedMethodAccessor290.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359) at com.sun.proxy.$Proxy112.startContainers(Unknown Source) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:123) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:308) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/xx.xx.xx.xx:33870 remote=DATA_NODE/xx.xx.xx.xx:45454] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) at java.io.FilterInputStream.read(FilterInputStream.java:83) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:554) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1802) at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1167) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1063) . Failing the application. also checked Ambari and getting "operation timed out after 7000 miliseconds" for Yarn node managers which come and go randomly Connection failed to http://NODE:8042/ws/v1/node/info (Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/alerts/alert_nodemanager_health.py", line 166, in execute connection_timeout=curl_connection_timeout, kinit_timer_ms = kinit_timer_ms) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/curl_krb_request.py", line 201, in curl_krb_request _, curl_stdout, curl_stderr = get_user_call_output(curl_command, user=user, env=kerberos_env) File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/get_user_call_output.py", line 62, in get_user_call_output raise ExecutionFailed(err_msg, code, files_output[0], files_output[1]) ExecutionFailed: Execution of 'curl --location-trusted -k --negotiate -u : -b /var/lib/ambari-agent/tmp/cookies/xxx-xxx-xx-xx -c /var/lib/ambari-agent/tmp/cookies/xxx-xxx-xx-xx http://NODE:8042/ws/v1/node/info --connect-timeout 5 --max-time 7 1>/tmp/tmpuFMG22 2>/tmp/tmpD6RtPF' returned 28. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0 curl: (28) Operation timed out after 7000 milliseconds with 0 out of -1 bytes received ) in the beginning it was only one data node, then its occurring for most of the data nodes. Types of jobs that are failing : 1- oozie - shell action executing spark 2- sqoop i have 2 name nodes for HA and 7 data nodes anyone faced same issue before ? thanks in advance. Best Regards, ... View more

Hi, we recently upgraded from HDP 2.6.2 to HDP 3.1 , i am trying to run hive query on beeline, (select count(*) from big_table) where big_table is a table containing millions of records, i got the below error ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1557754551780_0008_4_00, diagnostics=[Task failed, taskId=task_1557754551780_0008_4_00_000000, diagnostics=[TaskAttempt 0 failed, info=[attempt_1557754551780_0008_4_00_000000_0 being failed for too many output errors. failureFraction=1.0, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 1 failed, info=[attempt_1557754551780_0008_4_00_000000_1 being failed for too many output errors. failureFraction=1.0, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 2 failed, info=[attempt_1557754551780_0008_4_00_000000_2 being failed for too many output errors. failureFraction=1.0, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 3 failed, info=[attempt_1557754551780_0008_4_00_000000_3 being failed for too many output errors. failureFraction=1.0, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1557754551780_0008_4_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1557754551780_0008_4_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1557754551780_0008_4_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1 INFO : Completed executing command(queryId=hive_20190513174303_d7607f92-baaa-4fb2-825a-1af9a0287910); Time taken: 15.3 seconds Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1557754551780_0008_4_00, diagnostics=[Task failed, taskId=task_1557754551780_0008_4_00_000000, diagnostics=[TaskAttempt 0 failed, info=[attempt_1557754551780_0008_4_00_000000_0 being failed for too many output errors. failureFraction=1.0, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 1 failed, info=[attempt_1557754551780_0008_4_00_000000_1 being failed for too many output errors. failureFraction=1.0, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 2 failed, info=[attempt_1557754551780_0008_4_00_000000_2 being failed for too many output errors. failureFraction=1.0, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0], TaskAttempt 3 failed, info=[attempt_1557754551780_0008_4_00_000000_3 being failed for too many output errors. failureFraction=1.0, MAX_ALLOWED_OUTPUT_FAILURES_FRACTION=0.1, uniquefailedOutputReports=1, MAX_ALLOWED_OUTPUT_FAILURES=10, MAX_ALLOWED_TIME_FOR_TASK_READ_ERROR_SEC=300, readErrorTimespan=0]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1557754551780_0008_4_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1557754551780_0008_4_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1557754551780_0008_4_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1 (state=08S01,code=2) but when i execute (select count(*) from small_table) where this table contains only about 3000 records, it runs fine, before the upgrade both queries were running fine, so do i have to make any tuning for hive 3.1 ? N.B: Both tables are external tables Thanks ... View more

Hi, during upgrading from HDP 2.6.2 to HDP 3.1, we faced some issues "starting 2 data nodes was giving OOM exception, so we had to increase the java heap size manually and start them manually as well" that required manual steps , now we are at "Finalize Upgrade Pre-Check", and everything looks fine in the cluster, so we need to bypass this step in the upgrade, as its giving us error below Upgrade did not succeed on 2 hosts Your options: Pause Upgrade, delete the unhealthy hosts and return to the Upgrade Wizard to Proceed. Perform a Downgrade, which will revert all hosts to the previous stack version. thanks ... View more

Hi, I am upgrading to to HDP 3.1, we have Ambari 2.7.3 and HDP 2.6.2 - Kerberos Enabled. ** Ambari Server is installed, but i dont have ambari-agent on that server, so it doesn't appear on the (api/v1/clusters/clustername/hosts?fields=Hosts/ip,Hosts/host_name)** In step (Regenerate Missing Keytabs) create principle its failing, giving an error (Host not found), i also checked the ambari-server log, and found the below exception : Task #3611 failed to complete execution due to thrown exception: org.apache.ambari.server.HostNotFoundException:Host not found, hostname=myhost.mydomain.com org.apache.ambari.server.HostNotFoundException: Host not found, hostname=myhost.mydomain.com at org.apache.ambari.server.state.cluster.ClustersImpl.getHost(ClustersImpl.java:456) at org.apache.ambari.server.state.ConfigHelper.getEffectiveDesiredTags(ConfigHelper.java:189) at org.apache.ambari.server.state.ConfigHelper.getEffectiveDesiredTags(ConfigHelper.java:173) at org.apache.ambari.server.controller.AmbariManagementControllerImpl.findConfigurationTagsWithOverrides(AmbariManagementControllerImpl.java:2371) at sun.reflect.GeneratedMethodAccessor424.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50) at com.sun.proxy.$Proxy131.findConfigurationTagsWithOverrides(Unknown Source) at org.apache.ambari.server.state.ConfigHelper.calculateExistingConfigurations(ConfigHelper.java:2158) at org.apache.ambari.server.controller.KerberosHelperImpl.calculateConfigurations(KerberosHelperImpl.java:1725) at org.apache.ambari.server.controller.KerberosHelperImpl.getActiveIdentities(KerberosHelperImpl.java:1800) at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.calculateServiceIdentities(KerberosServerAction.java:511) at org.apache.ambari.server.serveraction.kerberos.KerberosServerAction.processIdentities(KerberosServerAction.java:455) at org.apache.ambari.server.serveraction.kerberos.CreatePrincipalsServerAction.execute(CreatePrincipalsServerAction.java:92) at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:550) at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:466) at java.lang.Thread.run(Thread.java:745) i removed the short name from the /etc/hosts, also the DNS server its pointing to the correct IPS, so when i try nslookup from any of the data nodes, its able to resolve the ambari server host name, any suggestions please ? ... View more

Hi guys,   i am using Flume 1.7 , and Kafka 0.10.0 , i have 1 topic with 2 partitions , and 2 flume sources i want each source to send its messages to certain partition id , also assign client.id to each source, below my configuration    a1.sources = r1 r2 a1.channels = c1 c2 a1.sinks = k1 k2 a1.channels.c1.type = memory a1.channels.c1.capacity = 10000 a1.channels.c1.transactionCapacity = 10000 a1.channels.c1.byteCapacityBufferPercentage = 20 a1.channels.c1.byteCapacity = 800000 a1.channels.c2.type = memory a1.channels.c2.capacity = 10000 a1.channels.c2.transactionCapacity = 10000 a1.channels.c2.byteCapacityBufferPercentage = 20 a1.channels.c2.byteCapacity = 800000 a1.sources.r1.type = taildir a1.sources.r1.channels = c1 a1.sources.r1.positionFile = /home/cloudera/FlumeDir/taildir_position2.json a1.sources.r1.filegroups = f1 a1.sources.r1.filegroups.f1 = /home/cloudera/Public/another.* a1.sources.r2.type = taildir a1.sources.r2.channels = c2 a1.sources.r2.positionFile = /home/cloudera/FlumeDir/taildir_position3.json a1.sources.r2.filegroups = f2 a1.sources.r2.filegroups.f2 = /home/cloudera/Public/file_log.log a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink a1.sinks.k1.channel = c1 a1.sinks.k1.kafka.topic = multiple_partitions a1.sinks.k1.kafka.client.id = agent0 a1.sinks.k1.kafka.producer.acks = 1 a1.sinks.k1.kafka.bootstrap.servers = 192.168.184.132:9092 a1.sinks.k1.kafka.flumeBatchSize = 1000 a1.sinks.k1.kafka.defaultPartitionId = 0 a1.sinks.k1.kafka.producer.linger.ms = 1 a1.sinks.ki.kafka.producer.compression.type = snappy a1.sinks.k2.type = org.apache.flume.sink.kafka.KafkaSink a1.sinks.k2.channel = c2 a1.sinks.k2.kafka.topic = multiple_partitions a1.sinks.k2.kafka.defaultPartitionId = 1 a1.sinks.k2.kafka.client.id = agent1 a1.sinks.k2.kafka.bootstrap.servers = 192.168.184.132:9092 a1.sinks.k2.kafka.flumeBatchSize = 1000 a1.sinks.k2.kafka.producer.acks = 1 a1.sinks.k2.kafka.producer.linger.ms = 1 a1.sinks.k2.kafka.producer.compression.type = snappy when i start flume agent it , it doesn't read either client.id or partitionid    INFO producer.ProducerConfig: ProducerConfig values: compression.type = snappy metric.reporters = [] metadata.max.age.ms = 300000 metadata.fetch.timeout.ms = 60000 reconnect.backoff.ms = 50 sasl.kerberos.ticket.renew.window.factor = 0.8 bootstrap.servers = [192.168.184.132:9092] retry.backoff.ms = 100 sasl.kerberos.kinit.cmd = /usr/bin/kinit buffer.memory = 33554432 timeout.ms = 30000 key.serializer = class org.apache.kafka.common.serialization.StringSerializer sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 ssl.keystore.type = JKS ssl.trustmanager.algorithm = PKIX block.on.buffer.full = false ssl.key.password = null max.block.ms = 60000 sasl.kerberos.min.time.before.relogin = 60000 connections.max.idle.ms = 540000 ssl.truststore.password = null max.in.flight.requests.per.connection = 5 metrics.num.samples = 2 client.id = ssl.endpoint.identification.algorithm = null ssl.protocol = TLS request.timeout.ms = 30000 ssl.provider = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] acks = 1 batch.size = 16384 ssl.keystore.location = null receive.buffer.bytes = 32768 ssl.cipher.suites = null ssl.truststore.type = JKS security.protocol = PLAINTEXT retries = 0 max.request.size = 1048576 value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer ssl.truststore.location = null ssl.keystore.password = null ssl.keymanager.algorithm = SunX509 metrics.sample.window.ms = 30000 partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner send.buffer.bytes = 131072 linger.ms = 1 from configuration file , i assigned "agent 1" to send data to partiton id "1" , and "agent 0" to send data to partion id "0" , when i write to "agent 1" i got half the data in partition 1 and other half in "agent 0 " which means configuration "kafka.defaultPartitionId = 1" is overriden to be the default partitioning "roundrobin" . ... View more