question Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data in Support Questions

HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

allen_chu — Tue, 25 Mar 2025 03:58:56 GMT

I am using HBase 2.1.6.
Exception happened As following :
hbase(main):001:0> scan 'FPCA_ITEMS_TY_NEW',{STARTROW => '08', LIMIT => 2}
ROW COLUMN+CELL
org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive/data/default/FPCA_ITEMS_TY_NEW/bf92b15900f190730a5482b53d350df0/cf/ab741ac0919480a47353778bda55d142202502239bf346dbbfc6475c8967734c2edfaaf4
at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:440)
at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:354)
at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:73)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:153)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:6581)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:6745)
at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:6518)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3155)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3404)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42190)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive/data/default/FPCA_ITEMS_TY_NEW/bf92b15900f190730a5482b53d350df0/cf/ab741ac0919480a47353778bda55d142202502239bf346dbbfc6475c8967734c2edfaaf4
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1581)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1574)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1589)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
at org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:139)
at org.apache.hadoop.hbase.regionserver.HStoreFile.<init>(HStoreFile.java:214)
at org.apache.hadoop.hbase.mob.CachedMobFile.create(CachedMobFile.java:49)
at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:220)
at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:401)
... 13 more

I've checked HDFS file path, and the file is not exist truly.

How can I resolve this problem? How HBase know what to find file path?

Thanks in advance.

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

Shelton — Fri, 28 Mar 2025 11:23:51 GMT

@allen_chu
As you correctly suspected the error indicates that HBase is trying to access a MOB (Medium Object) file in HDFS that no longer exists at the expected location

hdfs://ha:8020/apps/hbase/data/archive/data/default/FPCA_ITEMS_TY_NEW/bf92b15900f190730a5482b53d350df0/cf/ab741ac0919480a47353778bda55d142202502239bf346dbbfc6475c8967734c2edfaaf4

Potential Root Causes:

The MOB file was manually deleted from HDFS
HBase's MOB cleanup process didn't properly update metadata
The HDFS path was changed without proper HBase configuration updates
Corrupted HBase metadata
Incomplete data migration
Filesystem inconsistency
Incomplete compaction or archiving process

1. Verify HDFS Integrity

# Check HDFS file system health
hdfs fsck /apps/hbase/data/archive/data/default/FPCA_ITEMS_TY_NEW

2. HBase Data Consistency Check

# Start HBase shell
hbase shell

# Major compact the table to rebuild metadata
major_compact 'FPCA_ITEMS_TY_NEW'

# Verify table status
status 'detailed'

3. Immediate Recovery Options
Option A: Repair the table

hbase hbck -j <path_to_hbase_classpath> -repair FPCA_ITEMS_TY_NEW

4. Advanced Recovery Options

# If previous methods fail, consider:
# a) Snapshot and restore
hbase shell
snapshot 'FPCA_ITEMS_TY_NEW', 'FPCA_ITEMS_TY_NEW_SNAPSHOT'

# b) Clone the snapshot
clone_snapshot 'FPCA_ITEMS_TY_NEW_SNAPSHOT', 'FPCA_ITEMS_TY_NEW_RECOVERED'

Option B: Disable and re-enable the table

disable 'FPCA_ITEMS_TY_NEW'
enable 'FPCA_ITEMS_TY_NEW'

Option C: Run MOB compaction

hbase org.apache.hadoop.hbase.mob.mapreduce.Sweeper FPCA_ITEMS_TY_NEW cf

Replace the 'cf' with your actual column family name
Option 😧 If data is not critical, you can reset MOB references

alter 'FPCA_ITEMS_TY_NEW', {NAME => 'cf', MOB_COMPACT_PARTITION_POLICY => 'daily'}
major_compact 'FPCA_ITEMS_TY_NEW'

3. Preventive Measures

<property>
<name>hbase.mob.file.expired.period</name>
<value>86400</value> 
</property>

Additional Checks

1.Verify HDFS permissions for the HBase user
2.Check HDFS health (namenode logs, datanode availability)
3.Review HBase MOB configuration for the table

describe 'FPCA_ITEMS_TY_NEW'

With the above steps you should be able to resolve your hbase issue

Happy hadooping

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

allen_chu — Mon, 31 Mar 2025 06:55:22 GMT

Thanks for your detailed answer.

I'll try these options you provided in a Test Env.

For now, because I am sure that the MOB HFile is missing, I put an empty HFile in that path instead. And the Exception is not showing again when I scan data.

By the way, I am curious about why set hbase.mob.file.expired.period can be a Preventive Measures?

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

shubham_sharma — Mon, 31 Mar 2025 14:30:52 GMT

Hi @allen_chu @Shelton

There is no configuration named hbase.mob.file.expired.period in any of the hbase versions. @Shelton Could you please give reference for the property you have shared?

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

Shelton — Tue, 01 Apr 2025 12:17:29 GMT

@shubham_sharma

The hbase.mob.file.expired property in HBase is found in the hbase-site.xml configuration file.

This property is related to HBase's MOB (Medium-sized Objects) feature, which is designed to efficiently store objects that are larger than typical HBase cells but not large enough to warrant HDFS storage.

<property>
<name>hbase.mob.file.expired</name>
<value>30</value>
<description>The number of days to keep a mob file before deleting it. Default value is 30 days.</description>
</property>

Happy hadooping

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

shubham_sharma — Wed, 02 Apr 2025 11:28:29 GMT

@Shelton

Have you tested this property in your environment? Neither hbase.mob.file.expired nor hbase.mob.file.expired.period properties are present in hbase.

A MOB HFile will be subject to archiving under any of the following conditions:

Any MOB HFile older than the column family’s TTL
Any MOB HFile older than a "too recent" threshold with no references to it from the regular hfiles for all regions in a column family

MOB files are cleaned when MOB cleaner chore runs. This runs every 24 hours. Cleaner chore identifies the files whose TTL's are expired and removes them. TTL needs to be set on a table schema level. Default TTL is forever.

hbase.master.mob.cleaner.period

Description

The period that MobFileCleanerChore runs. The unit is second. The default value is one day. The MOB file name uses only the date part of the file creation time in it. We use this time for deciding TTL expiry of the files. So the removal of TTL expired files might be delayed. The max delay might be 24 hrs.

Default

86400

Always verify the answers from AI generated sources before posting it to community.

Reference -

https://hbase.apache.org/book.html#_mob_architecture

cc: @cjervis

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

Shelton — Wed, 02 Apr 2025 14:09:56 GMT

@shubham_sharma
my bad its hbase.mob.file.expired.period and not hbase.mob.file.expired
Happy hadooping

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

shubham_sharma — Mon, 28 Apr 2025 14:05:08 GMT

@Shelton Please read my previous answer carefully. None of the properties provided by you are in hbase codebase

question Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data in Support Questions

HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

3. Immediate Recovery OptionsOption A: Repair the table

4. Advanced Recovery Options

Option B: Disable and re-enable the table

Option C: Run MOB compaction

Replace the 'cf' with your actual column family nameOption 😧 If data is not critical, you can reset MOB references

3. Preventive Measures

Additional Checks

1.Verify HDFS permissions for the HBase user2.Check HDFS health (namenode logs, datanode availability)3.Review HBase MOB configuration for the table

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

Re: HBase - java.io.FileNotFoundException: File does not exist: hdfs://ha:8020/apps/hbase/data/archive when scan mob data

3. Immediate Recovery Options
Option A: Repair the table

Replace the 'cf' with your actual column family name
Option 😧 If data is not critical, you can reset MOB references

1.Verify HDFS permissions for the HBase user
2.Check HDFS health (namenode logs, datanode availability)
3.Review HBase MOB configuration for the table