Member since
06-15-2015
13
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 7363 | 07-01-2015 01:25 AM |
01-05-2018
10:30 PM
1 Kudo
It's worth reading this https://community.hortonworks.com/questions/146229/hive-llap-has-conflict-with-hive-sql-standadard-au.html
... View more
12-22-2017
08:26 AM
I have Service Auto Start configured for a cluster and it works. However, there does not appear to be a way to monitor the activities of Service Auto Start via Ambari. This creates a situation where it can appear that a service has failed to start, but the start is still in progress. LLAP is a particularly relevant example of this as start is significantly slower that for other components and (in my experience) is not as reliable. Have I missed a way to monitor via Ambari? Thanks in advance.
... View more
Labels:
12-07-2017
12:15 PM
Thank you for your response. Indeed, 129s is less than 2 x 78s. I've looked into cluster utilisation and your point about the cluster being fully utilised when executing the single query is well made.
... View more
12-05-2017
07:08 PM
Hi anarasimham, Thanks for your response. To answer your questions: # concurrent queries: 3 Memory per node: 62G Number of nodes: 3 Memory allocated to LLAP: "Memory per Daemon" is 18432; llap0 shows 56320MB allocated in YARN ResourceManager UI Please let me know if there is anything else you would like to know. Thanks, Martin
... View more
12-05-2017
01:03 PM
I'm investigating running concurrent queries in LLAP (HDP 2.6.1).
When running more than 1 query I am seeing non-zero "Killed Task Attempts" in the DAG page in Ambari (Tez View -> DAG ID link). In some cases the number of "Killed Task Attempts" is into the hundreds. The logs show messages of the form: 2017-12-05T12:13:42,003 INFO [IPC Server handler 2 on 33976 (1512469369970_0013_2_00_000022_1)] impl.TaskRunnerCallable: Kill task requested for id=attempt_1512469369970_0013_2_00_000022_1, taskRunnerSetup=false
2017-12-05T12:13:42,003 INFO [IPC Server handler 2 on 33976 (1512469369970_0013_2_00_000022_1)] impl.ContainerRunnerImpl: SubmissionState for attempt_1512469369970_0013_2_00_000022_1 : REJECTED
2017-12-05T12:13:42,009 INFO [IPC Server handler 4 on 33976 (1512469369970_0013_2_00_000004_1)] impl.TaskExecutorService: wait queue full, size=10. numSlotsAvailable=0, runningFragmentCount=3. attempt_1512469369970_0013_2_00_000004_1 not added
In some cases REJECTED is EVICTED_OTHER.
The query in isolation executed in 78s. When two executions of the same query run concurrently the execution times were 125s and 129s. As the number of concurrent executions is increased the mean elapsed time continues to rise. There is variation in elapsed time between runs whenever there is more than 1 query execution at a time. There is also a correlation between the elapsed time and the number of "Killed Task Attempts". The following is from executing the query 3 times concurrently: Execution # Elapsed Time/s Killed Task Attempts
----------- -------------- --------------------
1 90 1
2 159 110
3 190 182
I'd be very grateful for guidance on what is happening and why.
Am I hitting https://issues.apache.org/jira/browse/HIVE-15779?
Thanks,
Martin
... View more
- Tags:
- concurrency
- llap
08-17-2015
10:02 AM
In the process of investigating performance of data retrieval from Impala we have encountered unexpected behaviour (it is unexpected by us at least). We have a C++ application that connects to Impala’s HiveServer2 using Thrift directly. We know that we will be retrieving a large number of rows (> 1 million) from Impala and therefore we want to retrieve as much data as possible in each network round trip. In an attempt to achieve this we increase the Thrift buffer size from the default of 512 bytes. Using strace we can see that the non-default buffer size is being requested, but the number of bytes returned is less than the request in many cases as the request size is increased. In order to quantify this I ran simple tests with the following buffer sizes: 512 (matching the default) 1024 8192 1048576 (expected optimium for throughput) For each buffer size I executed the same query and ran strace on the client process (using a script). Extracting only the recvfrom calls from each of the strace output files showed the following - histograms available here 512 bytes requested resulted in 98.55% of the responses matching the requested size 1024 bytes requested resulted in 96.00% of the responses matching the requested size 8192 bytes requested resulted in 55.73% of the responses matching the requested size 1048576 bytes requested resulted in none of the responses returning 1M and a broad range of sizes being returned. My questions are: Is this expected behaviour? Are we missing some configuration option that will allow us to minimise network round trips when retrieving a significant amount of data? Comparing the query profile for the 512 and 8192 byte requests reveals the scale of the performance benefit: 512 byte requests Query Timeline: 34s612ms - Start execution: 64.337us (64.337us) - Planning finished: 4.871ms (4.807ms) - Ready to start remote fragments: 155.137ms (150.266ms) - Remote fragments started: 156.321ms (1.184ms) - Rows available: 368.767ms (212.445ms) - First row fetched: 368.962ms (195.516us) - Unregister query: 34s610ms (34s241ms) ImpalaServer: - ClientFetchWaitTimer: 30s435ms - RowMaterializationTimer: 3s497ms 1M 8192 byte requests Query Timeline: 14s755ms - Start execution: 110.916us (110.916us) - Planning finished: 4.852ms (4.741ms) - Ready to start remote fragments: 154.449ms (149.597ms) - Remote fragments started: 156.79ms (1.629ms) - Rows available: 363.807ms (207.727ms) - First row fetched: 365.63ms (1.255ms) - Unregister query: 14s754ms (14s389ms) ImpalaServer: - ClientFetchWaitTimer: 10s603ms - RowMaterializationTimer: 3s495ms Note that as shown in the histograms, the 8k requests are only satisfied 55.73% of the time, with 14.94% still using 512 bytes, so there is scope for a greater performance increase. The following shows an excerpt from the strace output for 1M requests: recvfrom(5, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\036028\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576, 0, NULL, NULL) = 12288 recvfrom(5, "\036028\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576, 0, NULL, NULL) = 6144 recvfrom(5, "\0\0\0\0\0\0\0\0\0\0\036028\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576, 0, NULL, NULL) = 8704 recvfrom(5, "\0\0\0\0\0\0\0\0\0\0\036028\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 1048576, 0, NULL, NULL) = 6879 Thanks in advance for any guidance.
... View more
08-10-2015
08:42 AM
Please excuse the delayed response... Cluster { "name": "cdh-01", "instances": [ { "ipAddress": "10.45.0.66", "virtualInstance": { "id": "69fb7304-f4b4-4559-b9a8-e8d8845e95e6", "template": { "name": "cdh-node", "type": "r3.large", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } }, "capabilities": { "operatingSystemType": "REDHAT_COMPATIBLE", "operatingSystemVersion": "REDHAT_COMPATIBLE_6", "virtualizationType": "HARDWARE_ASSISTED", "pythonVersion": "2.6.6", "packageManager": "YUM", "passwordlessSudoEnabled": true, "selinuxEnabled": true, "iptablesEnabled": true }, "health": { "status": "GOOD", "lastReported": 1438807900432 } }, { "ipAddress": "10.45.0.167", "virtualInstance": { "id": "7951a428-ef3a-420a-8224-27f9941d23e7", "template": { "name": "cdh-node", "type": "r3.large", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } }, "capabilities": { "operatingSystemType": "REDHAT_COMPATIBLE", "operatingSystemVersion": "REDHAT_COMPATIBLE_6", "virtualizationType": "HARDWARE_ASSISTED", "pythonVersion": "2.6.6", "packageManager": "YUM", "passwordlessSudoEnabled": true, "selinuxEnabled": true, "iptablesEnabled": true }, "health": { "status": "GOOD", "lastReported": 1438807900432 } }, { "ipAddress": "10.45.0.166", "virtualInstance": { "id": "e4fe0ebf-def5-4a60-bad9-97bd07ddd908", "template": { "name": "cdh-node", "type": "r3.large", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } }, "capabilities": { "operatingSystemType": "REDHAT_COMPATIBLE", "operatingSystemVersion": "REDHAT_COMPATIBLE_6", "virtualizationType": "HARDWARE_ASSISTED", "pythonVersion": "2.6.6", "packageManager": "YUM", "passwordlessSudoEnabled": true, "selinuxEnabled": true, "iptablesEnabled": true }, "health": { "status": "GOOD", "lastReported": 1438807900432 } }, { "ipAddress": "10.45.0.165", "virtualInstance": { "id": "d333982f-07f5-463e-88d8-1b3292dcee05", "template": { "name": "cdh-node", "type": "r3.large", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } }, "capabilities": { "operatingSystemType": "REDHAT_COMPATIBLE", "operatingSystemVersion": "REDHAT_COMPATIBLE_6", "virtualizationType": "HARDWARE_ASSISTED", "pythonVersion": "2.6.6", "packageManager": "YUM", "passwordlessSudoEnabled": true, "selinuxEnabled": true, "iptablesEnabled": true }, "health": { "status": "GOOD", "lastReported": 1438807900432 } }, { "ipAddress": "10.45.0.125", "virtualInstance": { "id": "2ce13830-9493-40c3-80fd-cc155c9bdb93", "template": { "name": "cdh-node", "type": "r3.large", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } }, "capabilities": { "operatingSystemType": "REDHAT_COMPATIBLE", "operatingSystemVersion": "REDHAT_COMPATIBLE_6", "virtualizationType": "HARDWARE_ASSISTED", "pythonVersion": "2.6.6", "packageManager": "YUM", "passwordlessSudoEnabled": true, "selinuxEnabled": true, "iptablesEnabled": true }, "health": { "status": "NOT_AVAILABLE", "lastReported": 1438360119927 } }, { "ipAddress": "10.45.0.127", "virtualInstance": { "id": "f21b3ac5-f02e-42a5-be90-8bcd1396901b", "template": { "name": "cdh-node", "type": "r3.large", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } }, "capabilities": { "operatingSystemType": "REDHAT_COMPATIBLE", "operatingSystemVersion": "REDHAT_COMPATIBLE_6", "virtualizationType": "HARDWARE_ASSISTED", "pythonVersion": "2.6.6", "packageManager": "YUM", "passwordlessSudoEnabled": true, "selinuxEnabled": true, "iptablesEnabled": true }, "health": { "status": "NOT_AVAILABLE", "lastReported": 1438360119927 } }, { "ipAddress": "10.45.0.126", "virtualInstance": { "id": "6aaeddcd-24df-4f96-aff9-9801c6f423eb", "template": { "name": "cdh-node", "type": "r3.large", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } }, "capabilities": { "operatingSystemType": "REDHAT_COMPATIBLE", "operatingSystemVersion": "REDHAT_COMPATIBLE_6", "virtualizationType": "HARDWARE_ASSISTED", "pythonVersion": "2.6.6", "packageManager": "YUM", "passwordlessSudoEnabled": true, "selinuxEnabled": true, "iptablesEnabled": true }, "health": { "status": "NOT_AVAILABLE", "lastReported": 1438360119927 } }, { "ipAddress": "10.45.0.124", "virtualInstance": { "id": "0a39600c-657c-46cf-82a6-cb3cbb861b82", "template": { "name": "cdh-gateway", "type": "t2.medium", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } }, "capabilities": { "operatingSystemType": "REDHAT_COMPATIBLE", "operatingSystemVersion": "REDHAT_COMPATIBLE_6", "virtualizationType": "HARDWARE_ASSISTED", "pythonVersion": "2.6.6", "packageManager": "YUM", "passwordlessSudoEnabled": true, "selinuxEnabled": true, "iptablesEnabled": true }, "health": { "status": "GOOD", "lastReported": 1438807900432 } }, { "ipAddress": "10.45.0.98", "virtualInstance": { "id": "9234d300-81d1-419c-b73b-93e5db9f896e", "template": { "name": "cdh-node", "type": "r3.large", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } }, "capabilities": { "operatingSystemType": "REDHAT_COMPATIBLE", "operatingSystemVersion": "REDHAT_COMPATIBLE_6", "virtualizationType": "HARDWARE_ASSISTED", "pythonVersion": "2.6.6", "packageManager": "YUM", "passwordlessSudoEnabled": true, "selinuxEnabled": true, "iptablesEnabled": true }, "health": { "status": "GOOD", "lastReported": 1438807900432 } } ], "services": [ { "serviceName": "HDFS", "health": { "status": "GOOD", "lastReported": 1438807900417 } }, { "serviceName": "HIVE", "health": { "status": "GOOD", "lastReported": 1438807900417 } }, { "serviceName": "HUE", "health": { "status": "GOOD", "lastReported": 1438807900417 } }, { "serviceName": "IMPALA", "health": { "status": "GOOD", "lastReported": 1438807900417 } }, { "serviceName": "OOZIE", "health": { "status": "GOOD", "lastReported": 1438807900417 } }, { "serviceName": "SQOOP", "health": { "status": "GOOD", "lastReported": 1438807900417 } }, { "serviceName": "YARN", "health": { "status": "GOOD", "lastReported": 1438807900417 } }, { "serviceName": "ZOOKEEPER", "health": { "status": "GOOD", "lastReported": 1438807900417 } } ] } Cluster Status { "stage": "READY", "description": "Done", "descriptionDetails": [], "remainingSteps": 0, "completedSteps": 133, "health": { "status": "GOOD", "lastReported": 1438808067089 } } Cluster Template { "name": "cdh-01", "productVersions": { "CDH": "5" }, "services": [ "HDFS", "HIVE", "HUE", "IMPALA", "OOZIE", "SQOOP", "YARN", "ZOOKEEPER" ], "servicesConfigs": {}, "virtualInstanceGroups": { "masters": { "name": "masters", "virtualInstances": [ { "id": "69fb7304-f4b4-4559-b9a8-e8d8845e95e6", "template": { "name": "cdh-node", "type": "r3.large", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } } ], "minCount": 1, "serviceTypeToRoleTypes": { "HDFS": [ "NAMENODE", "SECONDARYNAMENODE", "BALANCER" ], "HIVE": [ "HIVEMETASTORE", "HIVESERVER2" ], "HUE": [ "HUE_SERVER" ], "IMPALA": [ "CATALOGSERVER", "STATESTORE" ], "OOZIE": [ "OOZIE_SERVER" ], "SQOOP": [ "SQOOP_SERVER" ], "YARN": [ "RESOURCEMANAGER", "JOBHISTORY" ], "ZOOKEEPER": [ "SERVER" ] }, "roleTypesConfigs": {} }, "workers": { "name": "workers", "virtualInstances": [ { "id": "e4fe0ebf-def5-4a60-bad9-97bd07ddd908", "template": { "name": "cdh-node", "type": "r3.large", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } }, { "id": "d333982f-07f5-463e-88d8-1b3292dcee05", "template": { "name": "cdh-node", "type": "r3.large", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } }, { "id": "7951a428-ef3a-420a-8224-27f9941d23e7", "template": { "name": "cdh-node", "type": "r3.large", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } }, { "id": "9234d300-81d1-419c-b73b-93e5db9f896e", "template": { "name": "cdh-node", "type": "r3.large", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } } ], "minCount": 4, "serviceTypeToRoleTypes": { "HDFS": [ "DATANODE" ], "IMPALA": [ "IMPALAD" ], "YARN": [ "NODEMANAGER" ] }, "roleTypesConfigs": {} }, "gateway": { "name": "gateway", "virtualInstances": [ { "id": "0a39600c-657c-46cf-82a6-cb3cbb861b82", "template": { "name": "cdh-gateway", "type": "t2.medium", "image": "ami-c2a818aa", "bootstrapScript": "", "config": { "subnetId": "subnet-69cc8253", "rootVolumeSizeGB": "30", "instanceNamePrefix": "cdh", "securityGroupsIds": "sg-f197ff95" }, "tags": {}, "normalizeInstance": true } } ], "minCount": 1, "serviceTypeToRoleTypes": { "HDFS": [ "GATEWAY" ], "HIVE": [ "GATEWAY" ], "YARN": [ "GATEWAY" ] }, "roleTypesConfigs": {} } }, "externalDatabaseTemplates": {}, "externalDatabases": {}, "parcelRepositories": [ "http://archive.cloudera.com/cdh5/parcels/5.3.3/", "http://archive.cloudera.com/cdh4/parcels/latest/", "http://archive.cloudera.com/search/parcels/latest/", "http://archive.cloudera.com/impala/parcels/latest/" ], "restartClusterOnUpdate": false, "redeployClientConfigsOnUpdate": false } I got a message about invalid HTML and the message being updated when using preview before posting, so hopefully nothing in the output changed significantly.
... View more
08-05-2015
01:25 PM
Yes, I can confirm that. It may not have been clear from my original description that it appeared the only parts of the removing cluster operation which failed were the "Restart Cluster" and "Redeploy Client Configuration" actions. That is to say that the removal of the cluster nodes worked without issue, but the post removal actions of redeploying the client configuration and restarting the cluster failed (for a reason I have not identified).
... View more
08-05-2015
12:42 AM
I believe my situation is different to that discussed in the other thread, but maybe I'm wrong. After updating STATUS and restarting Cloudera Director Server I have been able to add new nodes to the cluster via Cloudera Director and have not been able to find any indication that there is a problem. Your comment about needing to recover the cluster object that contains the list of instances implies that I should expect there to be a problem with respect to the instances in the cluster, but I do not see that. Please would you clarify what problem you would expect me to observe?
... View more
08-03-2015
03:58 AM
Thanks for the response. I didn't want rebuild the cluster from scratch so had a look at other ways around this. I found this post which covers how to connect to the H2 database, so used that to poke around. I found the following in the CLUSTERS table for the problem cluster: sql> select status from clusters where name = 'cdh-01'; STATUS {"stage":"UPDATING","description":"Starting","descriptionDetails":[],"remainingSteps":0,"completedSteps":0} (1 row, 2 ms) sql> Based on values in CLUSTERS.STATUS for other clusters in the same Cloudera Director I ran the following: sql> update clusters set status='{"stage":"READY","description":"Done","descriptionDetails":[],"remainingSteps":0,"completedSteps":0}' where name = 'cdh-01'; (Update count: 1, 32 ms) sql> commit; (Update count: 0, 0 ms) sql> I'd stopped Cloudera Director Server before making the above database change and on restarting it the state of the cluster was back to the desired "Good health". I appreciate that the approach I have taken is likely not supported or recommended. The environment is test/evalution so I'm not overly concerned if it turns out something has been broken by my actions. My worse case is rebuild Cloudera Director Server and the clusters in it. I am interested in any comments on the approach I've taken.
... View more
07-31-2015
05:52 AM
While removing nodes from a cluster using Cloudera Director I checked the following options on the "Modify Cluster" screen:
Restart Cluster
Redeploy Client Configuration
The deletion of the nodes from the cluster worked as expected, but the cluster was not restarted and the client configuration was not redeployed. Right now this cluster shows as on step 19/23 and has been stuck there. I resorted to redeploying the client configuration and restarting the cluster via Cloudera Manager, which worked fine. However, there is no sign of Cloudera Director moving out of its current status for this cluster. I attempted to resolve the problem by restarting Cloudera Director, but that has made no difference.
Is there a way that a can clear the status for the cluster within Cloudera Director?
... View more
Labels:
07-01-2015
01:25 AM
Thanks for the response. I have been meaning to come back and add a comment here. I found the root of the problem as well as the solution here: http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/kerberos-authentication-failure-GSSAPI-Failure-gss-accept-sec/td-p/23333
... View more
