Member since
02-08-2017
8
Posts
1
Kudos Received
0
Solutions
10-10-2017
03:59 PM
Could you share how you generated a blueprint to install Spark on top of a running cluster with other existing services? It's not obvious how to do it.
... View more
04-13-2017
03:35 PM
Changed "fs.defaultFS" just to check if it was picking it up from there and indeed it somehow thinks that appending the filesystem to "fs.defaultFS" is the way to find the file. Is this a bug?
... View more
04-13-2017
03:15 PM
When running bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -Dfs.s3a.buffer.dir=/tmp/hbase_snap_tmp -snapshot TestTable-20170413-143020 -copy-to s3a://bucket-backup/hbase/snapshots/ The operation fails with: 2017-04-13 15:03:24,947 ERROR [main] snapshot.ExportSnapshot: Snapshot export failed
java.io.FileNotFoundException: File does not exist: hdfs://clusterID/home/ubuntu/hbase-1.2.4/hbase-prefix-tree/target/hbase-prefix-tree-1.2.4.jar
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
I assume it's looking for the "/home/ubuntu/hbase-1.2.4/hbase-prefix-tree/target/hbase-prefix-tree-1.2.4.jar" which does exist but obviously not under hdfs://. I don't know why this file is needed or why it's being looked for in the hdfs:// instead of the local filesystem, where it does exist. Any ideas?
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache HBase
04-12-2017
07:41 AM
From that link I see that having open files could be an issue, does this mean I can't backup with distcp (since I'm running Hbase on top and that can never be stopped)? I can't run a copytable to the local filesystem since the data is just too large for that. Are there any other sensible alternatives for backing up to S3?
... View more
04-11-2017
04:08 PM
Hello everyone, When performing distcp from hdfs:// to s3a:// after a while I get an error stating something like: Caused by: java.io.IOException: Mismatch in length of source:hdfs://clustername/hbase/WALs/hdp-data1.example.com,16020,1491913605286/hdp-data1.example.com%2C16020%2C1491913605286..meta.1491922613008.meta and target:s3a://bucket-backup/hbase/.distcp.tmp.attempt_local1903592397_0001_m_000000_0 It then quickly fails with: 17/04/11 15:50:54 INFO mapreduce.Job: Job job_local1903592397_0001 failed with state FAILED due to: NA
17/04/11 15:50:54 INFO mapreduce.Job: Counters: 28
File System Counters
FILE: Number of bytes read=723868
FILE: Number of bytes written=764685
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=2169097700
HDFS: Number of bytes written=0
HDFS: Number of read operations=469
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
S3A: Number of bytes read=0
S3A: Number of bytes written=2169097700
S3A: Number of read operations=471
S3A: Number of large read operations=0
S3A: Number of write operations=97
Map-Reduce Framework
Map input records=40
Map output records=0
Input split bytes=156
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=1376
Total committed heap usage (bytes)=521142272
File Input Format Counters
Bytes Read=13228
File Output Format Counters
Bytes Written=8
org.apache.hadoop.tools.mapred.CopyMapper$Counter
BYTESCOPIED=2169020252
BYTESEXPECTED=2169020252
at org.apache.hadoop.tools.DistCp.waitForJobCompletion(DistCp.java:205)
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:156)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:126)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:430)
Any ideas? We have HBase running on top of this HDFS setup which is performing writes. Is that a problem for distcp?
... View more
Labels:
- Labels:
-
Apache Hadoop
02-08-2017
03:57 PM
That's what I was thinking. Was just wondering if there was a way to define a custom package somehow. Thanks!
... View more
02-08-2017
01:54 PM
@Artem we do need to rebuild HBase since we need to run custom co-processors from Thrift. We want to do something like is described here: https://issues.apache.org/jira/browse/HBASE-5600 "We could create a thrift method to take the name of the class, method, and an array of params and then call coprocessorExec".
If there's a better way to run a custom co-processor and access it from Thrift I'm all ears. But this brings me back to a good way to run a custom build of HBase with Ambari.
... View more
02-08-2017
11:25 AM
1 Kudo
I'd like to use Ambari but be able to build HBase from source as we need a custom co-processor. Is there a documented way of doing this or do I have to turn into a full-custom install?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache HBase