part of Split or Merge (Verifiable by HMaster Logs)   what log and what in the log lets you know its part of a split or merge?   ... View more

found https://my.cloudera.com/knowledge/ERROR-quotUpgrade-not-allowed-in-Express-or-Trial-modequot?id=296899 ... View more

Each time you roll a key you get a new key. The short name is a pointer to the latest, but when deleting a key, it refers to all keys prefaced by the name. If the name is in use, there is no way to delete a key version, even though the key version is not in use. If you roll your keys daily, you end up with 364 keys you cannot delete. ... View more

solution, change  #$admin = HBaseAdmin.new(connection) $admin = connection.getAdmin();   leads to other changes new script   # # hbase org.jruby.Main list_small_regions.rb min_siz <namespace.tablename> # # Note: Please replace namespace.tablename with your namespace and table, eg NS1.MyTable. This value is case sensitive. require 'digest' require 'java' java_import org.apache.hadoop.hbase.HBaseConfiguration java_import org.apache.hadoop.hbase.client.HBaseAdmin java_import org.apache.hadoop.hbase.TableName java_import org.apache.hadoop.hbase.HRegionInfo; java_import org.apache.hadoop.hbase.client.Connection java_import org.apache.hadoop.hbase.client.ConnectionFactory java_import org.apache.hadoop.hbase.client.Table java_import org.apache.hadoop.hbase.util.Bytes def list_bigger_regions(table_name) cluster_status = $admin.getClusterStatus() master = cluster_status.getMaster() biggers = [] cluster_status.getServers.each do |s| cluster_status.getLoad(s).getRegionsLoad.each do |r| # getRegionsLoad returns an array of arrays, where each array # is 2 elements # Filter out any regions that don't match the requested # tablename next unless r[1].get_name_as_string =~ /#{table_name}\,/ if r[1].getStorefileSizeMB() > $low_size if r[1].get_name_as_string =~ /\.([^\.]+)\.$/ biggers.push $1 else raise "Failed to get the encoded name for #{r[1].get_name_as_string}" end end end end biggers end def regions_to_merge?(table_name) bigger_regions = list_bigger_regions(table_name) #regions = $admin.getTableRegions(Bytes.toBytes(table_name)); regions = $admin.getTableRegions(table_name); filtered_regions = regions.reject do |r| bigger_regions.include?(r.get_encoded_name) end puts "#{table_name},#{regions.length},#{bigger_regions.length},#{filtered_regions.length-1}" filtered_regions.length end limit_batch = 1000 do_merge = false config = HBaseConfiguration.create() connection = ConnectionFactory.createConnection(config) #$admin = HBaseAdmin.new(connection) $admin = connection.getAdmin(); # Handle command line parameters $low_size = 1 if ARGV[0].to_i >= $low_size $low_size=ARGV[0].to_i end all_tables=$admin.listTableNames if ARGV.length > 1 p "ARGV - #{ARGV[1]}" Table table = connection.getTable(TableName.valueOf("#{ARGV[1]}")); tables.push table else tables=all_tables end puts "TABLE_NAME,TOT_REGIONS,REGIONS_OK,REGIONS_NEED_MERGED" tables.each do |table| #table_name = table.getName() regions_to_merge?(table) end $admin.close   ... View more

Thanks, What I am experiencing is that the complete file, if 300GB, has to be assembled before upload to S3. This requires either 300GB of memory or disk. Distcp does not create a part file per block. I have not witnessed any file split being done. Multi part uploads require you get an upload ID and upload many part files with a numeric extension and in the end ask S3 to put them back together. I do not see any of this being done. I admit I do not know much about all this and it could be happening out of my sight. ... View more

Is it possible that large HDFS file not have to be extracted from HDFS in full to either disk or memory? And uploaded in parts as HDFS blocks are read? Would this take a change to distcp to not be one map per file, but one map per block? ... View more

We use Percona distro of MySQL for a MySQL cluster. We use Haproxy to connect to the cluster. Our actual connect is to localhost:port that Haproxy proxies to mysql:port   All of our DB connection configurations specify localhost:3306 ... View more

This is not the same a previously explained, but shows two child queues with same weight so I would think would use same resources. Not sure I have a picture of the scenario I described.     {"defaultFairSharePreemptionThreshold":0.8,"defaultFairSharePreemptionTimeout":30,"defaultMinSharePreemptionTimeout":null,"defaultQueueSchedulingPolicy":"drf","queueMaxAMShareDefault":null,"queueMaxAppsDefault":null,"queuePlacementRules":[{"create":false,"name":"specified","queue":null,"rules":null},{"create":null,"name":"secondaryGroupExistingQueue","queue":null,"rules":null},{"create":null,"name":"reject","queue":null,"rules":null}],"queues":[{"aclAdministerApps":" cis_us_edfhdpadmin","aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":0.8,"fairSharePreemptionTimeout":30,"minSharePreemptionTimeout":null,"name":"root","queues":[{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"default","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":{"cpuPercent":null,"memory":16384,"memoryPercent":null,"vcores":1},"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" cis_us_edfprototype,cis_us_edfdtadmin,cis_us_edfhdpadmin","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"cis_us_edfprototype","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" cis_us_edfeag","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"cis_us_edfeag","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" cis_us_edfcolombiabatch","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"cis_us_edfcolombiabatch","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":100.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":"edfhdpadmin cis_us_edfhdpadmin","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"cis_us_edfhdpadmin","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":5.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" cis_us_edfmpq","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"cis_us_edfmpq","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":6.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" da_us_edfienretro,da_us_edfienretro_dbadmin","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"da_us_edfienretro","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":12.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" df_us_colana,df_us_colana_ext","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"df_us_colana","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":16.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" edf_adad,edf_adad_ext","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_adad","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":6.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" edf_cdr,edf_cdr_ext,edf_cdr_ro","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_cdr","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":6.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" edf_ukpin,edf_ukpin_ro,edf_ukpin_ext","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_ukpin","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":6.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":" cis_us_edfbissvcs,cis_us_edfbissvcs_dbaadmin,edf_bis_sbfe_prod,edf_bis_sbfe_prod_ext,edf_bis_sbfe_prod_ro","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"bis","queues":[{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_sbfe","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":2.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"cis_us_edfbissvcs","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":2.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_dae","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_pir","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_cir","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_os","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":2.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_sbfe_cds","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_sbfe_dd","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_sbfe_qa","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_da","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null},{"aclAdministerApps":null,"aclSubmitApps":null,"allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_bis_bf_edq","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null}],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":500.0}],"schedulingPolicy":"drf","type":"parent"},{"aclAdministerApps":null,"aclSubmitApps":" edf_ode,edf_ode_ro,edf_ode_ext,cis_us_edfhdpadmin","allowPreemptionFrom":null,"fairSharePreemptionThreshold":null,"fairSharePreemptionTimeout":null,"minSharePreemptionTimeout":null,"name":"edf_ode","queues":[],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":120.0}],"schedulingPolicy":"drf","type":null}],"schedulablePropertiesList":[{"impalaDefaultQueryMemLimit":null,"impalaDefaultQueryOptions":null,"impalaMaxMemory":null,"impalaMaxQueuedQueries":null,"impalaMaxRunningQueries":null,"impalaQueueTimeout":null,"maxAMShare":null,"maxChildResources":null,"maxResources":null,"maxRunningApps":null,"minResources":null,"scheduleName":"default","weight":1.0}],"schedulingPolicy":"drf","type":null}],"userMaxAppsDefault":null,"users":[]} ... View more

Based on the link provided Per link https://blog.cloudera.com/blog/2018/06/yarn-fairscheduler-preemption-deep-dive/   Resources are preempted only if the resulting free space matches a starving application’s request. This ensures none of the preempted containers go unused.   Please help me understand this.   If a pending container needs 10G of RAM, but each containers in a queue over its fair share is less that 10G of RAM, none will be preempted. Is this a correct statement based on the above bullet ... View more

#!/usr/bin/env python import ssl,sys,time from cm_api.api_client import ApiResource from cm_api.endpoints.types import ApiClusterTemplate from cm_api.endpoints.cms import ClouderaManager from cm_api.endpoints import clusters, events, hosts, external_accounts, tools from cm_api.endpoints import types, users, timeseries, roles, services ssl._create_default_https_context = ssl._create_unverified_context try:   cm = ApiResource("CM_SERVER","7183","admin","CM_PASS","true","15")   cluster = cm.get_cluster("CLUSTER_NAME") except:   print "Failed log into cluster %s" % (" CLUSTER_NAME ")   sys.exit(0) servers = [ "server1.company.com", " server2. company .com", " server3. company .com"] s = cluster.get_service("solr") ra = [] for r in s.get_roles_by_type("SOLR_SERVER"):   hostname = cm.get_host(r.hostRef.hostId).hostname   if hostname in servers:     ra.append([hostname,r]) ra.sort() print "\nWill restart %s SOLR instances" % len(ra) for hostname,r in ra:   print "\nRestarting SOLR on %s" % (hostname)   s.restart_roles(r.name)   r = s.get_role(r.name)   wait = time.time() + 180 # three minutes   while r.roleState != "STARTED":     print "Role State = %s" % (r.roleState)     print "Waiting for role state to be STARTED"     print time.strftime("%H:%M:%S")     if time.time() > wait:       print "SOLR failed to restart on %s" % (hostname)       sys.exit(1)     time.sleep(10)     r = s.get_role(r.name)   print "SOLR restarted on %s" % (hostname) print "\nAll SOLR roles restarted" sys.exit(0) ... View more

#--------------------------------------------------------------------- # Example configuration for a possible web application. See the # full configuration options online. # # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt # #--------------------------------------------------------------------- #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1d timeout server 1d timeout tunnel 1d timeout http-keep-alive 10s timeout check 10s maxconn 3000 #--------------------------------------------------------------------- # MySQL #--------------------------------------------------------------------- listen mysql-cluster bind 127.0.0.1:3306 mode tcp option tcpka stick-table type ip size 1 nopurge stick on dst server mysql-1 mysql-1:3308 check server mysql-2 mysql-2:3308 check backup server mysql-3 mysql-3:3308 check backup ... View more

We are trying to use HAProxy. Do you care to share you haproxy.conf? Thanks, Gene ... View more

The mgmt service roles are running on their own host (not the one being patched) and are fine. The issue is that if you restart a host with stopped roles, you are guaranteed to have services with health issues after the reboot. If the roles are running before the reboot, then the services recover. CM cannot keep track that roles are stopped. Stopping roles does not cause health issues, but rebooting with stopped roles does. For now I am forgoing the health check after the reboot and just starting all roles on a host no matter the state.   My method is working on a small test cluster but I am leary to start using the process on our production cluster and doing OS patching in a rolling manner without taking any downtime.   To veer a little from the topic, I have not found a way via the API to access the mgmt service. The mgmt service roles return with a hosts's host.roleRefs but are not accessible as they have no handle (that I can figure out via the API)   'cluster.get_service(rref.serviceName).stop_roles(rref.roleName)' does not work as the mgmt service is not part of 'cluster'. I can get a ref to the role but I cannot actually access the role. Maybe someone knows what I am missing and can point to a link or share the secret.   Service mgmt Roles mgmt-HOSTMONITOR-a9de710bc2672ee1ba304933bdf0b946 mgmt-REPORTSMANAGER-a9de710bc2672ee1ba304933bdf0b946 mgmt-EVENTSERVER-a9de710bc2672ee1ba304933bdf0b946 mgmt-ACTIVITYMONITOR-a9de710bc2672ee1ba304933bdf0b946 mgmt-SERVICEMONITOR-a9de710bc2672ee1ba304933bdf0b946 ... View more