Hortonworks had released HDP 2.6 in Q2 2017. Isilon engineering team worked closely with Hortonworks engineering team to validate HDP 2.6 with OneFS. One of the features of HDP 2.6 was LLAP (Long Live And Process). LLAP enables as fast as sub-second SQL analytics on Hadoop by intelligently caching data in memory with persistent servers that instantly process SQL queries. However, please note that LLAP is officially NOT SUPPORTED with OneFS. Customers are advised NOT to use LLAP (even though it may seem to work after config changes) because it could potentially result in data inconsistency issue Additional Details By default, LLAP uses “inode paths” to access data from file system. OneFS currently does not support referencing files by inode. So queries coming through LLAP will fail on OneFS. However, it is possible to configure Hive LLAP to NOT use inode path by setting hive.llap.io.use.fileid.path = false With this configuration change, the queries will be executed and this will make LLAP technically “usable” with Isilon but will also cause potential data inconsistency issue. This is because with this capability turned off, LLAP will try to access data by filename (and not by inode path) in the filesystem, which means that the LLAP cache can potentially has an outdated file with the same filename but not knowing that there is a newer one to fetch. Recommendation: LLAP support with Isilon is currently planned for future release of OneFS. For more details on timing of release, contact your Isilon/Hortonworks representative ... View more

The latest and greatest version of Hortonworks HDB i.e version 2.2.0.0 addresses many of the concerns from customers/prospects who have been considering Hortonworks HDB for their production data analytics workloads. Below is the list of major improvements available in Hortonworks HDB 2.2.0.0 over its predecessor. Ranger support (in BETA mode) Lack of support for Ranger has been a deal breaker in field in past. Hortonworks HDB 2.2.0.0 is step in right direction and includs Beta support for Apache Ranger. Hortonworks HDB 2.2.0.0 now include a RESTful service that provides integration between HDB and Ranger policy management. Ranger integration enables you to use Apache Ranger to authorize user access to Hortonworks HDB resources. Using Ranger enables you to manage all of your Hadoop components’ authorization policies using the same user interface, policy store, and auditing stores. For more details on how to use Ranger for authorization in Hortonworks HDB 2.2.0.0, please refer to http://hdb.docs.pivotal.io/220/hawq/ranger/ranger-overview.html PXF ORC with Pivotal HDB ORC is a columnar file format that provides a highly efficient way to both store and access HDFS data. ORC files offer improvements over text and RCFile formats in terms of both compression and performance.ORC support is provided by a new profile “HiveORC “ exposed in the PXF Hive plug-in. The “HiveORC” profile supports access to data stored in ORC format only through a Hive mapped table Optimized Row Columnar file format, which had been Beta in prior versions, is now fully supported by Hortonworks HDB 2.2.0.0 Support for CentOS 7 / RHEL 7 Hortonworks HDB 2.2 now has downloads for Red Hat Enterprise Linux 7 and CentOS 7 operating systems Compatibility with Hortonworks Data Platform ecosystem Hortonworks HDB 2.2 is certified to work with Hortonworks Data Platform (DHP) 2.5.3 and Ambari 2.4.2 The support for HDP 2.6 is currently undergoing testing and is tentatively planned to be generally available in Q3 2017 ... View more

Thanks to deep collaboration between Hortonworks and Isilon engineering teams, we have come up with the below table that shows the status of Ambari Views Through Ambari 2.2.2 Ambari Version Kerberos Enabled WebHDFS* Enabled Ambari View Works? Remarks Through Ambari 2.2.2 Y Y No Does not work with Kerberos enabled because OneFS implementation of WebHDFS does not support chunked encoding. However we believe that if we use HTTPFS in front , it should work. This has not been officially tested though. Through Ambari 2.2.2 Y N No Does not work in this combination . This is default behavior of Ambari views which does not support Kerberos without WebHDFS. Through Ambari 2.2.2 N Y Yes If the default fs.defaultfs is set to "hdfs" in core-site.xml, then it works. The default value in Ambari is: hdfs:// <rpc address, so it works by default. if you need to use custom Hive View Cluster Configuration, be sure to use hdfs:// protocol and port 8020 for the WebHDFS FileSystem URI. Through Ambari 2.2.2 N N Yes This should work without any change if you need to use custom Hive View Cluster Configuration, be sure to use hdfs:// protocol and port 8020 for the WebHDFS FileSystem URI. * - OneFS implementation of WebHDFS. With Ambari 2.4 Ambari Version Kerberos Enabled WebHDFS* Enabled Ambari View Works? Remarks Ambari 2.4 Y Y No Does not work with Kerberos enabled because OneFS implementation of WebHDFS does not support chunked encoding. However we believe that if we use HTTPFS in front, it should work. This has not been officially tested though Ambari 2.4 Y N No Does not work in this combination . This is default behavior of Ambari views which does not support Kerberos without WebHDFS. Ambari 2.4 N Y Default Hive does not work (use a workaround) Hive Views do not work with default settings. File View, Tez View, Zeppelin works Workaround : Only way to use Hive View is to use custom Hive View Cluster Configuration, be sure to use hdfs:// protocol and port 8020 for the WebHDFS FileSystem URI. Ambari 2.4 N N Yes We believe this should work without any change if you need to use custom Hive View Cluster Configuration, be sure to use hdfs:// protocol and port 8020 for the WebHDFS FileSystem URI. * - OneFS implementation of WebHDFS. ... View more

On Oct 25th, EMC Isilon announced general availability of OneFS 8.0.1. The official announcement can be found here - https://isiblog.emc.com/category/onefs-8-0-1/ OneFS 8.0.1 is the first major upgrade to OneFS 8.0 platform that was released in Feb 2016. While this upgrade has many generic enhancements to the storage side of OneFS, there are four major functionalities related to Hadoop Supports Ambari 2.4 This is the first major release of OneFS that will have support for Ambari 2.4. Hortonworks had released HDP 2.5 and Ambari 2.4 at end of Aug 2016. Since then Isilon team had been working to integrate Ambari 2.4 with OneFS. With the release one OneFS 801, customers can now use Ambari 2.4 and HDP 2.5 For more instructions on how to configure, see the below link : https://community.emc.com/community/products/isilon/blog/2016/10/24/onefs-and-ambari-24-hdp-25 Ambari Metrics and Alerts integration Though OneFS has been supporting Ambari for a few releases, it was not leveraging the metrics capabilities as yet. So customers were not able to make use of Ambari metrics server With OneFS 801, OneFS host metrics and alerts will be integrated in Ambari. This will help customers to proactively monitor key performance metrics and alerts for OneFS through single management window i.e. Ambari Use the below link to learn on how to configure Ambari metrics system to pull OneFS Host metrics and set up Thresholds and Alerts. https://community.emc.com/community/products/isilon/blog/2016/10/25/ambari-metrics-and-alerts-with-emc-isilon Integration with Ranger deny first policy framework Another major enhancement in OneFS 801 is related to access permissions to the file system. OneFS 801 is now integrated with Apache Ranger to provide additional layer of security to provide stricter control and deny access to the files and directories to users and groups. This has been made possible by using Deny policy that was added in Ranger 0.6.0 release and is available in HDP 2.5. By default, the deny policy in Ranger Admin UI is turned off. Customers need to follow the information in the below link to enable Deny Policy and use it with OneFS https://community.emc.com/community/products/isilon/blog/2016/10/25/apache-ranger-deny-policies-with-onefs-8010 Wire Encryption OneFS 801 also have support for data in-flight encryption between Isilon nodes and the HDFS client. Details on how to enable Wire Encryption will be available soon. ... View more