hi experts! in HDFS serivices there is tool called balancer, which purposed to ensure even distribution of blocks across cluster. my question is how frequently it kicks in to check is cluster imbalanced or not? is there any way to change this frequency?   thanks! ... View more

hi dear experts!   i do have a challenge.  i do have a dynamuc service pool, let's say root.marketing. Many users, who belong to this pool is submitting jobs on it (Bob, Alice, Tom). i want to know resource consumption for each of the users. like for the last day Bob used in average 33 cores, Alice 12, Tom 118... or something like this. in other words want to know who consume what within the same pool   thanks! ... View more

Hi dear experts!   i'm trying to load data with ImportTSV tool , like this: hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dmapreduce.job.reduces=1000 -Dimporttsv.columns="data:SS_SOLD_DATE_SK, HBASE_ROW_KEY" -Dimporttsv.separator="|" -Dimporttsv.bulk.output=/tmp/store_sales_hbase store_sales /user/root/benchmarks/bigbench/data/store_sales/*, but have only one reducer (despite on -Dmapreduce.job.reduces=1000 setting ). i even set  mapreduce.job.reduces=1000 on the Cluster wide, but still have only one reducer.   Could anybody hint how to resolve this? thank you in advance for any input! ... View more

Hi dear experts! i'm trying to load data from CSV format on HDFS to HBase with ImportTSV (importtsv). it works perfectly fine in case when HBASE_ROW_KEY is the single CSV column. but i don't know how to create composite HBASE_ROW_KEY (from two columns). for example, i have CSV with 3 columns:   row1, 1, abc row1, 2, dd row2, 1, iop row3, 1, kk and row could be uniqly identified by first two columns.   any inputs will be highly appreciated! ... View more

Hi dear expert! i'm trying to export data with sqoop.export.records.per.statement parameter. But for some reasons sqoop doesn't recognize it:   sqoop export --direct --connect jdbc:oracle:thin:@scaj43bda01:1521:orcl --username bds --password bds --table orcl_dpi --export-dir /tmp/dpi --input-fields-terminated-by ',' --lines-terminated-by '\n' -m 70 --batch -Dsqoop.export.records.per.statement=10000 -Dsqoop.export.statements.per.transaction=100 Warning: /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p1168.923/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 16/01/22 19:59:38 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.1 16/01/22 19:59:38 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 16/01/22 19:59:38 ERROR tool.BaseSqoopTool: Error parsing arguments for export: 16/01/22 19:59:38 ERROR tool.BaseSqoopTool: Unrecognized argument: -Dsqoop.export.records.per.statement=10000 16/01/22 19:59:38 ERROR tool.BaseSqoopTool: Unrecognized argument: -Dsqoop.export.statements.per.transaction=100  i've tried to remove --direct key (target DB is Oracle), but it also doesn't help: sqoop export --connect jdbc:oracle:thin:@host:1521:orcl --username user --password pass --table orcl_dpi --export-dir /tmp/dpi --input-fields-terminated-by ',' --lines-terminated-by '\n' -m 70 --batch -Dsqoop.export.records.per.statement=10000 -Dsqoop.export.statements.per.transaction=100 Warning: /opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p1168.923/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 16/01/22 20:00:29 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.1 16/01/22 20:00:29 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 16/01/22 20:00:29 ERROR tool.BaseSqoopTool: Error parsing arguments for export: 16/01/22 20:00:29 ERROR tool.BaseSqoopTool: Unrecognized argument: -Dsqoop.export.records.per.statement=10000 16/01/22 20:00:29 ERROR tool.BaseSqoopTool: Unrecognized argument: -Dsqoop.export.statements.per.transaction=100 thank you! ... View more

Hi experts!   seems that LZMA algorithm could be pretty siutable for some Hadoop cases (like storing historical inmutable data). Does someone know is it possible to implement it somehow or reuse some library?    any ideas are very welcome!   thanks! ... View more

Hi dear experts!   i'm wondering is there any way to force block redistridution for some particular file/directory. my case is: 1) load file from node that have DataNode process with replication factor 1 2) increace replication factor by executing: hdfs dfs -setrep 3 /tmp/path/to/my/file 3) check distribution with some specific Java tool: hadoop jar FileDistribution.jar /tmp/ path/to/my/file   and got: ----------------------------------- ----------------------------------- Files distribution in directory across cluster is : {scaj31bda05.us.oracle.com=400, scaj31bda03.us.oracle.com=183, scaj31bda04.us.oracle.com=156, scaj31bda01.us.oracle.com=151, scaj31bda02.us.oracle.com=154, scaj31bda06.us.oracle.com=156}   it's obvious that first node contain 400 blocks. other 400*2=800 blocks evenly distributed across other nodes. it there any way for force block redistribution for make it even?   thanks!   ... View more

Hi dear expert! i'm wondering is there any way to check file distribution amond nodes in HDFS?   some way that allow to check on which nodes place some particular file of dirrectory?   thanks! ... View more

Hi dear experts! i'm strugling with configuring sqoop2+hue(3.7)+Oracle DB. i'm trying to create connection in HUE, but getting error:     i have  ojdbc6.jar in /var/lib/sqoop2/ dirrectory (as hinted some forums): [root@sqoop2server ~]# ll /var/lib/sqoop2/ total 7684 -rw-r--r-- 1 sqoop2 sqoop 2677451 Nov 6 12:46 derby-10.8.2.2.jar -rw-r--r-- 1 sqoop2 sqoop2 960396 Nov 6 13:39 mysql-connector-java.jar -rw-r--r-- 1 root root 3670975 Nov 6 13:52 ojdbc6.jar -rw-r--r-- 1 sqoop2 sqoop2 539705 Nov 6 13:39 postgresql-9.0-801.jdbc4.jar drwxr-xr-x 3 sqoop2 sqoop2 4096 Nov 5 21:17 repository drwxr-xr-x 3 sqoop2 sqoop2 4096 Nov 2 19:06 repositoy drwxr-xr-x 5 sqoop2 sqoop 4096 Nov 6 13:39 tomcat-deployment   + another one question is the any way to config Oraoop sqoop with Hue?   thanks! ... View more

Hi dear experts!   i'm curious how it possible to handle read IO size in my MR jobs. for exampe, i have some file in HDFS, under the hood it's files in Linux filesystem /disk1/hadoop/.../.../blkXXX. in ideal case this file size should be equal block size (128-256MB). my question is how it possible to set IO size for reading operation?   thank you! ... View more

Hi dear experts!   i discovering Spark's persist capabilities and noted interesting behaivour of DISK_ONLY persistance. as far as i understand the main goal - to store reusable and intermediate RDDs, that were produced from permanent data (that lays on HDFS).   import org.apache.spark.storage.StorageLevel val input = sc.textFile("/user/hive/warehouse/big_table"); val result = input.coalesce(600).persist(StorageLevel.DISK_ONLY) scala> result.count() …… // and repeat command …….. scala> result.count() so, i was surprised when saw that second iteration was significantly faster... could anybody describe why?   thanks!   ... View more

Hi everyone!   i trying to understand Sort shuffle in spark and will very appreciate if someone could answer on simple question, let's imagine: 1) i have 600 partitions (HDFS blocks, for simplicity) 2) it place in 6 node cluster 3) i run spark with follow parameters:  --executor-memory 13G --executor-cores 6 --num-executors 12 --driver-memory 1G --properties-file my-config.conf that's  mean that on each server i will have 2 executor with 6 core each.   4) according my config reduce phase has only 3 reducers.   so, ny question is how many files on each servers will be after Sort Shuffle: - 12 like a active map task  - 2 like a number of executors on each server - 100 like a number of partitions that place on this server (for simplicity i just devide 600 on 6)   and the second question is how names buffer for storing intermediate data before spill it on disk on the map stage?   thanks! ... View more