you property was removed, removed from what/where? I cannot find it. ... View more

part of Split or Merge (Verifiable by HMaster Logs)   what log and what in the log lets you know its part of a split or merge?   ... View more

found https://my.cloudera.com/knowledge/ERROR-quotUpgrade-not-allowed-in-Express-or-Trial-modequot?id=296899 ... View more

Each time you roll a key you get a new key. The short name is a pointer to the latest, but when deleting a key, it refers to all keys prefaced by the name. If the name is in use, there is no way to delete a key version, even though the key version is not in use. If you roll your keys daily, you end up with 364 keys you cannot delete. ... View more

solution, change  #$admin = HBaseAdmin.new(connection) $admin = connection.getAdmin();   leads to other changes new script   # # hbase org.jruby.Main list_small_regions.rb min_siz <namespace.tablename> # # Note: Please replace namespace.tablename with your namespace and table, eg NS1.MyTable. This value is case sensitive. require 'digest' require 'java' java_import org.apache.hadoop.hbase.HBaseConfiguration java_import org.apache.hadoop.hbase.client.HBaseAdmin java_import org.apache.hadoop.hbase.TableName java_import org.apache.hadoop.hbase.HRegionInfo; java_import org.apache.hadoop.hbase.client.Connection java_import org.apache.hadoop.hbase.client.ConnectionFactory java_import org.apache.hadoop.hbase.client.Table java_import org.apache.hadoop.hbase.util.Bytes def list_bigger_regions(table_name) cluster_status = $admin.getClusterStatus() master = cluster_status.getMaster() biggers = [] cluster_status.getServers.each do |s| cluster_status.getLoad(s).getRegionsLoad.each do |r| # getRegionsLoad returns an array of arrays, where each array # is 2 elements # Filter out any regions that don't match the requested # tablename next unless r[1].get_name_as_string =~ /#{table_name}\,/ if r[1].getStorefileSizeMB() > $low_size if r[1].get_name_as_string =~ /\.([^\.]+)\.$/ biggers.push $1 else raise "Failed to get the encoded name for #{r[1].get_name_as_string}" end end end end biggers end def regions_to_merge?(table_name) bigger_regions = list_bigger_regions(table_name) #regions = $admin.getTableRegions(Bytes.toBytes(table_name)); regions = $admin.getTableRegions(table_name); filtered_regions = regions.reject do |r| bigger_regions.include?(r.get_encoded_name) end puts "#{table_name},#{regions.length},#{bigger_regions.length},#{filtered_regions.length-1}" filtered_regions.length end limit_batch = 1000 do_merge = false config = HBaseConfiguration.create() connection = ConnectionFactory.createConnection(config) #$admin = HBaseAdmin.new(connection) $admin = connection.getAdmin(); # Handle command line parameters $low_size = 1 if ARGV[0].to_i >= $low_size $low_size=ARGV[0].to_i end all_tables=$admin.listTableNames if ARGV.length > 1 p "ARGV - #{ARGV[1]}" Table table = connection.getTable(TableName.valueOf("#{ARGV[1]}")); tables.push table else tables=all_tables end puts "TABLE_NAME,TOT_REGIONS,REGIONS_OK,REGIONS_NEED_MERGED" tables.each do |table| #table_name = table.getName() regions_to_merge?(table) end $admin.close   ... View more

Thanks, What I am experiencing is that the complete file, if 300GB, has to be assembled before upload to S3. This requires either 300GB of memory or disk. Distcp does not create a part file per block. I have not witnessed any file split being done. Multi part uploads require you get an upload ID and upload many part files with a numeric extension and in the end ask S3 to put them back together. I do not see any of this being done. I admit I do not know much about all this and it could be happening out of my sight. ... View more

Is it possible that large HDFS file not have to be extracted from HDFS in full to either disk or memory? And uploaded in parts as HDFS blocks are read? Would this take a change to distcp to not be one map per file, but one map per block? ... View more