HDP 2.3 Index Life Cycle(Client events are in orange)- Columns Names in SYSTEM.CATALOG table:- Query:- “select TABLE_NAME,DATA_TABLE_NAME,INDEX_TYPE,INDEX_STATE,INDEX_DISABLE_TIMESTAMP from system.catalog where INDEX_TYPE is not null;” TABLE_NAME:- Name of the index DATA_TABLE_NAME:- Name of the parent data table of index INDEX_TYPE:- GLOBAL(1) LOCAL(2) INDEX_STATE:- Index States (Abbreviation stored in SYSTEM.CATALOG, states in bold are available to client too):- BUILDING("b") USABLE("e") UNUSABLE("d") ACTIVE("a") INACTIVE("i") DISABLE("x") REBUILD("r") (Below paragraph is from phoenix site) DISABLE will cause the no further index maintenance to be performed on the index and it will no longer be considered for use in queries. REBUILD will completely rebuild the index and upon completion will enable the index to be used in queries again. BUILDING will partially rebuild the index from the last disabled timestamp and upon completion. INACTIVE/UNUSABLE will cause the index to no longer be considered for use in queries, however index maintenance will continue to be performed. ACTIVE/USABLE will cause the index to again be considered for use in queries. Note that a disabled index must be rebuild and cannot be set as USABLE INDEX_DISABLE_TIMESTAMP:- It is the timestamp at which index is disabled. It will be 0 , if the index is active or disabled by client manually and will non-zero when index is disabled during write failures. Automatic Rebuild process:- MetaDataRegionObserver is responsible of running rebuild thread, so upsert select query to update the disabled index is executed from the regionserver , which is hosting SYSTEM.CATALOG table. INACTIVE and DISABLE indexes are chosen for rebuild( provided all the regions of index table are online) All Indexes of all tables are built serially. We build the index from disabled timestamp - phoenix.index.failure.handling.rebuild.overlap.time(default 5 minutes) to SCN Upsert select query used is "UPSERT /*+ NO_INDEX */ INTO index_table_name(indexedCols) select dataCols from data_table” Index re-build lifecycle:- Properties to control automatic Rebuilding :- "phoenix.index.failure.handling.rebuild"(default true) "phoenix.index.failure.handling.rebuild.interval" (default 10 seconds) Consistency/Write Failures https://phoenix.apache.org/secondary_indexing.html#Mutable Tables ... View more