Hive LLAP (in HDP 2.6) is not supported with OneFS

[#1]

Hortonworks had released HDP 2.6 in Q2 2017. Isilon engineering team worked closely with Hortonworks engineering team to validate HDP 2.6 with OneFS.

One of the features of HDP 2.6 was LLAP (Long Live And Process). LLAP enables as fast as sub-second SQL analytics on Hadoop by intelligently caching data in memory with persistent servers that instantly process SQL queries.

However, please note that LLAP is officially NOT SUPPORTED with OneFS. Customers are advised NOT to use LLAP (even though it may seem to work after config changes) because it could potentially result in data inconsistency issue

Additional Details

• By default, LLAP uses “inode paths” to access data from file system. OneFS currently does not support referencing files by inode. So queries coming through LLAP will fail on OneFS.
• However, it is possible to configure Hive LLAP to NOT use inode path by setting hive.llap.io.use.fileid.path = false

With this configuration change, the queries will be executed and this will make LLAP technically “usable” with Isilon but will also cause potential data inconsistency issue. This is because with this capability turned off, LLAP will try to access data by filename (and not by inode path) in the filesystem, which means that the LLAP cache can potentially has an outdated file with the same filename but not knowing that there is a newer one to fetch.

Recommendation:

LLAP support with Isilon is currently planned for future release of OneFS. For more details on timing of release, contact your Isilon/Hortonworks representative

[#2]

@Vivek Somani : Can you please let me know if Hive LLAP is supported by OneFS in new release? If yes please share the version .

Thanks in advance for your help .