Member since
06-27-2017
14
Posts
2
Kudos Received
0
Solutions
06-25-2019
09:39 AM
2 Kudos
We use Percona distro of MySQL for a MySQL cluster. We use Haproxy to connect to the cluster. Our actual connect is to localhost:port that Haproxy proxies to mysql:port All of our DB connection configurations specify localhost:3306
... View more
06-07-2019
02:54 PM
Is a parent's instatanious fair share ever used in preemption calculations? Can a child/leaf queue process be preempted if the child/leaf queue is over its instatanious fair share even though the parent is not? This would make sense to preempt within a parent to give to another child, but not give to another non child queue. In attached picture, parent not over but chhild is. Since parent not over I would think that its resources are only available to its own children, not others.
... View more
Labels:
03-04-2019
03:03 PM
spark.dynamicAllocation.executorIdleTimeout = 60 Can this be lowered? Is there any benefit/harm? My question is regarding preemption. Are executors of a preempted container "idle". or is this idle step skipped during preemption?
... View more
03-04-2019
02:49 PM
In Hive configuration we have spark.dynamicAllocation.enabled = true, actually checked spark.executor.cores = 6 Does dynamic override the set number of 6? It does not appear dynamic cores are being allocated.
... View more
02-28-2019
01:54 PM
This is not the same a previously explained, but shows two child queues with same weight so I would think would use same resources. Not sure I have a picture of the scenario I described. {"defaultFairSharePreemptionThreshold":0.8,"defaultFairSharePreemptionTimeout":30,"defaultMinSharePreemptionTimeout":null,"defaultQueueSchedulingPolicy":"drf","queueMaxAMShareDefault":null,"queueMaxAppsDefault":null,"queuePlacementRules":[{"create":false,"name":"specified","queue":null,"rules":null},{"create":null,"name":"secondaryGroupExistingQueue","queue":null,"rules":null},{"create":null,"name":"reject","queue":null,"rules":null}],"queues":[{"aclAdministerApps":" cis_us_edfhdpadmin","aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":0.8,"fairSharePreemptionTimeout":30,"minSharePreemptionTimeout":null,"name":"root","queues":[{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"default","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":{"cpuPercent":null,"memory":16384,"memoryPercent":null,"vcores":1},"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" cis_us_edfprototype,cis_us_edfdtadmin,cis_us_edfhdpadmin","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"cis_us_edfprototype","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" cis_us_edfeag","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"cis_us_edfeag","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" cis_us_edfcolombiabatch","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"cis_us_edfcolombiabatch","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":100.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":"edfhdpadmin cis_us_edfhdpadmin","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"cis_us_edfhdpadmin","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":5.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" cis_us_edfmpq","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"cis_us_edfmpq","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":6.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" da_us_edfienretro,da_us_edfienretro_dbadmin","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"da_us_edfienretro","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":12.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" df_us_colana,df_us_colana_ext","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"df_us_colana","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":16.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" edf_adad,edf_adad_ext","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_adad","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":6.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" edf_cdr,edf_cdr_ext,edf_cdr_ro","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_cdr","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":6.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" edf_ukpin,edf_ukpin_ro,edf_ukpin_ext","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_ukpin","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":6.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" cis_us_edfbissvcs,cis_us_edfbissvcs_dbaadmin,edf_bis_sbfe_prod,edf_bis_sbfe_prod_ext,edf_bis_sbfe_prod_ro","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"bis","queues":[{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_sbfe","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":2.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"cis_us_edfbissvcs","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":2.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_dae","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_pir","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_cir","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_os","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":2.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_sbfe_cds","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_sbfe_dd","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_sbfe_qa","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_da","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_bf_edq","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null}],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":500.0}],"schedulingPolicy":"drf","type":"parent"},{"aclAdministerApps":null,"aclSubmitApps":" edf_ode,edf_ode_ro,edf_ode_ext,cis_us_edfhdpadmin","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_ode","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":120.0}],"schedulingPolicy":"drf","type":null}],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null}],"userMaxAppsDefault":null,"users":[]}
... View more
02-28-2019
12:33 PM
We have our queues set for weights only. We have a parent queue with a weight of 500 and several child queues in it. We have another queue with a weight of 100. A job is submitted in the 100 weight queue and it uses up all resources. A job is submitted is a child queue of the 500 parent queue. It did not appear that preemption was occurring. We had to add a max resource to the 100 queue to force preemption. Per link https://blog.cloudera.com/blog/2018/06/yarn-fairscheduler-preemption-deep-dive/ Resources are preempted only if the resulting free space matches a starving application’s request. This ensures none of the preempted containers go unused. Please help me understand this. So preemption is container by container? It will not batch several containers to meet the starved queue's demand? If a pending container needs 10G of RAM, but each containers in a queue over its fair share is less that 10G of RAM, none will be preempted. Is this a correct statement based on the above bullet
... View more
Labels:
02-28-2019
12:23 PM
Based on the link provided Per link https://blog.cloudera.com/blog/2018/06/yarn-fairscheduler-preemption-deep-dive/ Resources are preempted only if the resulting free space matches a starving application’s request. This ensures none of the preempted containers go unused. Please help me understand this. If a pending container needs 10G of RAM, but each containers in a queue over its fair share is less that 10G of RAM, none will be preempted. Is this a correct statement based on the above bullet
... View more
03-20-2018
02:45 PM
#!/usr/bin/env python import ssl,sys,time from cm_api.api_client import ApiResource from cm_api.endpoints.types import ApiClusterTemplate from cm_api.endpoints.cms import ClouderaManager from cm_api.endpoints import clusters, events, hosts, external_accounts, tools from cm_api.endpoints import types, users, timeseries, roles, services ssl._create_default_https_context = ssl._create_unverified_context try: cm = ApiResource("CM_SERVER","7183","admin","CM_PASS","true","15") cluster = cm.get_cluster("CLUSTER_NAME") except: print "Failed log into cluster %s" % (" CLUSTER_NAME ") sys.exit(0) servers = [ "server1.company.com", " server2. company .com", " server3. company .com"] s = cluster.get_service("solr") ra = [] for r in s.get_roles_by_type("SOLR_SERVER"): hostname = cm.get_host(r.hostRef.hostId).hostname if hostname in servers: ra.append([hostname,r]) ra.sort() print "\nWill restart %s SOLR instances" % len(ra) for hostname,r in ra: print "\nRestarting SOLR on %s" % (hostname) s.restart_roles(r.name) r = s.get_role(r.name) wait = time.time() + 180 # three minutes while r.roleState != "STARTED": print "Role State = %s" % (r.roleState) print "Waiting for role state to be STARTED" print time.strftime("%H:%M:%S") if time.time() > wait: print "SOLR failed to restart on %s" % (hostname) sys.exit(1) time.sleep(10) r = s.get_role(r.name) print "SOLR restarted on %s" % (hostname) print "\nAll SOLR roles restarted" sys.exit(0)
... View more
- Tags:
- solr
12-18-2017
01:42 PM
#--------------------------------------------------------------------- # Example configuration for a possible web application. See the # full configuration options online. # # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt # #--------------------------------------------------------------------- #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1d timeout server 1d timeout tunnel 1d timeout http-keep-alive 10s timeout check 10s maxconn 3000 #--------------------------------------------------------------------- # MySQL #--------------------------------------------------------------------- listen mysql-cluster bind 127.0.0.1:3306 mode tcp option tcpka stick-table type ip size 1 nopurge stick on dst server mysql-1 mysql-1:3308 check server mysql-2 mysql-2:3308 check backup server mysql-3 mysql-3:3308 check backup
... View more
11-16-2017
09:33 AM
We are trying to use HAProxy. Do you care to share you haproxy.conf? Thanks, Gene
... View more
11-15-2017
10:08 AM
Cloudera documentation identifies the steps for MySQL replication. I cannot find any information of how you set up high availabilty should the DB crash. Yes the data is replicated to another server but we are not pointing to that server. I was told that Cloudera only supprorts the Db being Active/Passive. So how do you reference the "passive" if needed? There are nine configurations for database hostname. 1. Cloudera Manager com.cloudera.cmf.db.host 2. Hue Database Hostname 3. Activity Monitor Database Hostname 4. Navigator Audit Server Database Hostname 5. Navigator Metadata Server Database Hostname 6. Reports Manager Database Hostname 7. Oozie Server Database Host 8. Hive Metastore Database Host 9. Sentry Server Database Host We have tried to use various load balancers (single IP) to multiple DB servers, but have found this inconsistant. Sentry fails to start. We get DB timeouts as if connection are stale and not resubmitted when the DBs failover. Is it expected that we manually have to change nine configuration points and do service restarts? Is this the concept of high availability?
... View more
Labels:
06-28-2017
08:00 AM
The mgmt service roles are running on their own host (not the one being patched) and are fine. The issue is that if you restart a host with stopped roles, you are guaranteed to have services with health issues after the reboot. If the roles are running before the reboot, then the services recover. CM cannot keep track that roles are stopped. Stopping roles does not cause health issues, but rebooting with stopped roles does. For now I am forgoing the health check after the reboot and just starting all roles on a host no matter the state. My method is working on a small test cluster but I am leary to start using the process on our production cluster and doing OS patching in a rolling manner without taking any downtime. To veer a little from the topic, I have not found a way via the API to access the mgmt service. The mgmt service roles return with a hosts's host.roleRefs but are not accessible as they have no handle (that I can figure out via the API) 'cluster.get_service(rref.serviceName).stop_roles(rref.roleName)' does not work as the mgmt service is not part of 'cluster'. I can get a ref to the role but I cannot actually access the role. Maybe someone knows what I am missing and can point to a link or share the secret. Service mgmt Roles mgmt-HOSTMONITOR-a9de710bc2672ee1ba304933bdf0b946 mgmt-REPORTSMANAGER-a9de710bc2672ee1ba304933bdf0b946 mgmt-EVENTSERVER-a9de710bc2672ee1ba304933bdf0b946 mgmt-ACTIVITYMONITOR-a9de710bc2672ee1ba304933bdf0b946 mgmt-SERVICEMONITOR-a9de710bc2672ee1ba304933bdf0b946
... View more
06-27-2017
11:49 AM
I can find very little on OS patching methodology. We are required to apply patches monthly. I have been trying via CM API to start and stop roles on a server before patching and rebooting. Put host into maintenance Stop all roles Apply patches Reboot Start roles Take out of maint The issue occurs after the reboot. CM labels all stopped roles as FATAL after a reboot. If the roles are running prior to reboot, they restart, but if they are stopped first, they do not come back after the reboot as stopped. The main issue is testing health reports poor health due to FATAL compared to stopped/exited. What are others doing to patch on a regular basis?
... View more
Labels:
