Hi, I am trying to implement this paper : ATH: Auto-Tuning HBase’s Configuration via Ensemble Learning (https://ieeexplore.ieee.org/document/7950900/) For this I have to collect a training set. In the section III, It show the structure of the training set. And in the begin au page 5, it explains how to generate the data for the training set. To build the models, we need to construct a training set S.
S is a matrix, with each row being the following vector:
vj = [perfj, cij, . . . , cij, . . . , cnj ], j = 1, . . . , m, (1)
with vj the jth observation, perfj the throughput or latency,
and cij the ith HBase configuration parameter of the jth observation. n is the total number of HBase configuration
parameters, and m is the total number of vectors in matrix S (observations or training examples). I need to generate random value for parameters showed in the table in the page 8. I used API of Cloudera manager. I go through the hbase service to try to update the parameters with python code. But they are some parameters that they aren't int the Hbase service like : "hbase.rs.cacheblocksonwrite"
"hbase.storescanner.parallel.seek.threads"
"hbase.storescanner.parallel.seek.enable"
"hbase.regionserver.hlog.blocksize"
"hbase.client.max.perregion.tasks"
"hbase.client.max.perserver.tasks"
"hbase.ipc.server.callqueue.handler.factor"
"hbase.ipc.server.callqueue.read.ratio"
"hbase.ipc.server.callqueue.scan.ratio"
"io.storefile.bloom.block.size"
"ycsb.client.threads"
"hfile.index.block.max.size"
"hfile.block.bloom.cacheonwrite"
"hfile.block.index.cacheonwrite" Maybe not all of them but a majority of them. So I wonder if it's the fact that my architecture doesn't use this parameters or my code doesn't work. below my code : # Get a list of all clusters
cdh5 = None
for c in api.get_all_clusters():
# print c.name
if c.version == "CDH5":
cdh5 = c
for s in cdh5.get_all_services():
# print s
if s.type == "HBASE":
hbase = s
nn = None
for r in hbase.get_all_services():
print r
if r.type == 'HBASE':
nn = r
print "Role name: %s\nState: %s\nHealth: %s\nHost: %s" % (
nn.name, nn.roleState, nn.healthSummary, nn.hostRef.hostId)
t = r.get_config(view='full')
for key, value in t.iteritems():
print(key)
print(value)
nn.update_config(
{
'io_storefile_bloom_block_size':True
#the others parameters with the value
}
) Thank you I hope i was clear
... View more
