phoneix seconday index blocking the updates of hbase table in phoenix client

[#1]

We are using Phoenix indexes in both the cases ,only Method of Data loading into Hbase is different(1- put hbase command & 2- phoenix upsert command), why we are able to see new/updated records while loading the data through phoenix , why are not able to see new ones while loading the records into hbase directly ??

The issue is - We are not getting updated/real time records while querying Phoenix index tables as well as parent hbase tables through phoenix client . where as if I delete the phoenix indexes on hbase parent tables, we are able to see updated records in hbase tables through phoenix client . May I know what will be the issue ??

But if I use the phoenix indexes on hbase tables which is getting updated with new records through phoenix client(phoenix upsert query) , I am able to see new/updated records in habse table as well as phoenix index table through phoenix client .

[#2 · Accepted solution]

First off: you should not use HBase APIs to write data into Phoenix tables. Use the Phoenix API.

The reason you should not do this is the reason that you are not seeing the data you've written. Using Phoenix API's (the UPSERT command) is what triggers the update to the secondary index table. When you add data via the HBase APIs, Phoenix has no idea that you did this and thus cannot ensure referential integrity with your secondary indices. If you want to use Phoenix, use the Phoenix APIs to read and write data.

[#3]

Thanks for your quick and valuable response .

Could you please let me know what is the way to do secondary indexing on hbase data with out using phoenix . i found some hindex using coprocessor , but that is on hbase 0.9.4 . it will be very helpful if you provide any reference link or guidence . FYI - hbase 1.1.2 is my current version . thank you so much .

[#4]

I would not recommend anything other than Phoenix 🙂

[#5]

If writing data table by using hbase API and want that secondary indexes created with Phoenix are also updated, you need to include following attributes in mutations(put/delete) while inserting data to data table so that Phoenix knows what all indexes needs to be updated. for eg:- Put mutation.

PTable dataPTable = PhoenixRuntime.getTableNoCache(conn, dataTableFullName);

List<PTable> indexes = dataPTable.getIndexes();
List<IndexMaintainer> maintainers = Lists.newArrayListWithExpectedSize(indexes.size());
for (PTable index : indexes) {
maintainers.add(index.getIndexMaintainer(dataPTable, conn));
}
ImmutableBytesWritable indexMetaDataPtr = new ImmutableBytesWritable(ByteUtil.EMPTY_BYTE_ARRAY);IndexMaintainer.serializeAdditional(dataPTable, indexMetaDataPtr, indexList,conn);

byte[] indexMetaData = ByteUtil.copyKeyBytesIfNecessary(indexMetaDataPtr);
Put put = new Put(CellUtil.cloneRow(cell));
put.setAttribute(PhoenixIndexCodec.INDEX_MD, indexMetaData);
put.setAttribute(PhoenixIndexCodec.INDEX_UUID, uuidValue);dataHtable.put(put)

Adjust some variables accordingly in above snippet.

[#6]

Thanks Ankit , looks like when ever we want to update indexes, we need to run the above put mutation program manually . but the data is coming in real time , performing queries on index table or also in real time. is it work?. correct me if am wrong . if it is work , please let me know what is the package i have to use and where and how to deploy the code . thank you .

[#7]

Is it a hbase coprocessor ?? Could you please provide full example for the above code and how to run . thanks

[#8]

you just need to add following two attributes in PUT which you are already creating for inserting data in HBase.

• put.setAttribute(PhoenixIndexCodec.INDEX_MD, indexMetaData);
• put.setAttribute(PhoenixIndexCodec.INDEX_UUID, uuidValue);

To get indexMetaData,you can refer above snippet.

[#9]

Can you provide the full code?I cant know how to get the 'indexList' and so on,Thank you.