question Re: What does bucket_00003_flush_length indicate while writing to Hive in Archives of Support Questions (Read Only)

What does bucket_00003_flush_length indicate while writing to Hive

tspardhu — Tue, 18 Apr 2017 08:58:20 GMT

We have been attempting to Stream data in to Hive Tables using Storm Trident topology. There are cases where we see multiple files of type bucket_XXXX_flush_length in hive file explorer. what does these files Indicate ..? and when will they occur...?

Re: What does bucket_00003_flush_length indicate while writing to Hive

balavignesh_nag — Tue, 18 Apr 2017 13:07:33 GMT

@Pardhu T

You might have to check this Link. Its related to the ticket here that you might want to look at.

Re: What does bucket_00003_flush_length indicate while writing to Hive

tspardhu — Tue, 18 Apr 2017 20:19:40 GMT

I believe my question is more related to why these files occur in first place and the significance of them .i see many of them when we are trying to stream the data in to hive. My a question is not Not related to how they can be compacted as mentioned in the link you provided.

Re: What does bucket_00003_flush_length indicate while writing to Hive

ekoifman — Thu, 20 Apr 2017 02:43:53 GMT

The short answer is you can ignore these.

When you are using Hive Streaming Ingest (https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest) which is used by Storm/Flume/NiFi, Hive creates these file for it's internal housekeeping for maintaining transactional consistency. They should normally be removed as soon as the TransactionBatch is closed (usually once transaction Y of delta_X_Y/ finishes). The flush_legth file may remain around if the Writer process crashes before TransactionBatch is closed. They will eventually be cleaned by the Compactor process.