Member since
05-02-2019
19
Posts
15
Kudos Received
0
Solutions
09-29-2017
12:38 AM
4 Kudos
Hortonworks had released HDP 2.6 in Q2 2017. Isilon engineering team worked closely with Hortonworks engineering team to validate HDP 2.6 with OneFS. One of the features of HDP 2.6 was LLAP (Long Live And
Process). LLAP enables as fast as sub-second SQL
analytics on Hadoop by intelligently caching data in memory with persistent
servers that instantly process SQL queries. However, please note that LLAP is officially NOT SUPPORTED with
OneFS. Customers are advised NOT to use LLAP (even though it may seem to
work after config changes) because it could potentially result in data
inconsistency issue Additional Details By
default, LLAP uses “inode paths” to access data from file system. OneFS currently
does not support referencing files by inode. So queries coming through LLAP
will fail on OneFS. However,
it is possible to configure Hive LLAP to NOT use inode path by setting hive.llap.io.use.fileid.path = false With this configuration change, the queries will be executed and this will make LLAP technically “usable”
with Isilon but will also cause potential data inconsistency issue. This
is because with this capability turned off, LLAP will try to access data by
filename (and not by inode path) in the filesystem, which means that the LLAP cache can potentially has
an outdated file with the same filename but not knowing that there is a newer
one to fetch. Recommendation: LLAP support with Isilon is currently
planned for future release of OneFS. For more details on timing of release, contact your Isilon/Hortonworks representative
... View more
Labels:
06-30-2017
03:24 AM
5 Kudos
The latest and greatest version of Hortonworks
HDB i.e version 2.2.0.0 addresses many of the concerns from customers/prospects who have been
considering Hortonworks HDB for their production data analytics workloads. Below is the list of major improvements available
in Hortonworks HDB 2.2.0.0 over its predecessor.
Ranger support (in BETA mode) Lack of support
for Ranger has been a deal breaker in field in past. Hortonworks HDB 2.2.0.0 is step in right direction and includs Beta support for Apache Ranger. Hortonworks
HDB 2.2.0.0 now include a RESTful service that provides integration between HDB and
Ranger policy management. Ranger
integration enables you to use Apache Ranger to authorize user access to Hortonworks
HDB resources. Using Ranger enables you to manage all of your Hadoop
components’ authorization policies using the same user interface, policy store,
and auditing stores. For more details on how to use Ranger for authorization in Hortonworks
HDB 2.2.0.0, please refer to http://hdb.docs.pivotal.io/220/hawq/ranger/ranger-overview.html PXF ORC with Pivotal HDB
ORC is a columnar
file format that provides a highly efficient way to both store and access HDFS
data. ORC files offer improvements over text and RCFile formats in terms of
both compression and performance.ORC support is
provided by a new profile “HiveORC “ exposed in the PXF Hive plug-in. The “HiveORC” profile supports access to data
stored in ORC format only through a Hive mapped table Optimized Row Columnar file format, which had been Beta in prior versions, is now fully supported by Hortonworks HDB 2.2.0.0
Support for CentOS 7 / RHEL 7 Hortonworks
HDB 2.2 now has downloads for Red Hat Enterprise Linux 7 and CentOS 7 operating
systems Compatibility with Hortonworks Data Platform ecosystem
Hortonworks HDB 2.2 is certified to work
with Hortonworks Data Platform (DHP) 2.5.3 and Ambari 2.4.2 The support for HDP 2.6 is currently
undergoing testing and is tentatively planned to be generally available in Q3
2017
... View more
01-12-2017
07:49 PM
3 Kudos
Thanks to deep collaboration between Hortonworks and Isilon engineering teams, we have come up with the below table that shows the status of Ambari Views Through Ambari 2.2.2
Ambari
Version
Kerberos
Enabled
WebHDFS*
Enabled
Ambari View
Works?
Remarks
Through Ambari
2.2.2
Y
Y
No
Does
not work with Kerberos enabled because OneFS implementation of WebHDFS does not support chunked encoding.
However we believe that if we use HTTPFS in front ,
it should work. This has not been officially tested though.
Through Ambari
2.2.2
Y
N
No
Does not work in this combination . This is default behavior of Ambari views which does not support Kerberos without WebHDFS.
Through
Ambari 2.2.2
N
Y
Yes
If
the default fs.defaultfs is set to "hdfs" in core-site.xml, then it
works.
The default value in Ambari is: hdfs:// <rpc
address, so it works by default.
if you need to use custom Hive View Cluster Configuration,
be sure to use hdfs:// protocol and port 8020 for the WebHDFS FileSystem URI.
Through
Ambari 2.2.2
N
N
Yes
This should work without any change
if you need to use custom Hive View Cluster Configuration,
be sure to use hdfs:// protocol and port 8020 for the WebHDFS FileSystem URI.
* - OneFS implementation of WebHDFS. With Ambari 2.4
Ambari
Version
Kerberos
Enabled
WebHDFS*
Enabled
Ambari View
Works?
Remarks
Ambari 2.4
Y
Y
No
Does not work with Kerberos enabled because OneFS implementation of WebHDFS does not support chunked encoding.
However we believe that if we use HTTPFS in front, it should work. This has not been officially tested though
Ambari 2.4
Y
N
No
Does not work in this combination . This is default behavior of Ambari views which does not support Kerberos without WebHDFS.
Ambari 2.4
N
Y
Default Hive does
not work
(use a workaround)
Hive
Views do not work with default settings.
File View, Tez View, Zeppelin works
Workaround : Only way to use Hive
View is to use custom Hive View Cluster Configuration, be sure to use hdfs://
protocol and port 8020 for the WebHDFS FileSystem URI.
Ambari 2.4
N
N
Yes
We
believe this should work without any change
if you need to use custom Hive View Cluster Configuration,
be sure to use hdfs:// protocol and port 8020 for the WebHDFS FileSystem URI.
* - OneFS implementation of WebHDFS.
... View more
Labels:
11-04-2016
10:22 PM
With 801 FS, there is support for encryption of data-in-flight as well as data-at-rest . Data-at-rest encryption is achieved by using Self-Encrypting-Drives (SED)s . In-flight data encryption is through new features that leverages Kerberos infrastructure. To know details on how to set up encryption on OneFS 801, visit https://community.emc.com/community/products/isilon/blog/2016/11/04/enhanced-hadoop-security-with-onefs-801-and-hortonworks-hdp
... View more
10-29-2016
02:06 AM
1 Kudo
On Oct 25th, EMC Isilon announced general availability of OneFS 8.0.1. The official announcement can be found here - https://isiblog.emc.com/category/onefs-8-0-1/ OneFS 8.0.1 is the
first major upgrade to OneFS 8.0 platform that was released in Feb 2016. While
this upgrade has many generic enhancements to the storage side of OneFS, there
are four major functionalities related to Hadoop
Supports Ambari 2.4 This is the first major release of OneFS that will have support for Ambari 2.4. Hortonworks
had released HDP 2.5 and Ambari 2.4 at end of Aug 2016. Since then Isilon team had been working to
integrate Ambari 2.4 with OneFS. With
the release one OneFS 801, customers can now use Ambari 2.4 and HDP 2.5 For
more instructions on how to configure, see the below link : https://community.emc.com/community/products/isilon/blog/2016/10/24/onefs-and-ambari-24-hdp-25
Ambari
Metrics and Alerts integration Though OneFS has been supporting Ambari for a few
releases, it was not leveraging the metrics capabilities as yet. So customers were not able to make use of Ambari metrics server With OneFS 801, OneFS host
metrics and alerts will be integrated in Ambari. This will help customers to proactively monitor key
performance metrics and alerts for OneFS through single management window i.e.
Ambari Use
the below link to learn on how to configure Ambari metrics system to pull OneFS
Host metrics and set up Thresholds and Alerts. https://community.emc.com/community/products/isilon/blog/2016/10/25/ambari-metrics-and-alerts-with-emc-isilon Integration with Ranger deny first policy framework Another
major enhancement in OneFS 801 is related to access permissions to the
file system. OneFS
801 is now integrated with Apache Ranger to
provide additional layer of security to provide stricter control and deny
access to the files and directories to users and groups. This has been made possible by using Deny policy that was added
in Ranger 0.6.0 release and is available in HDP 2.5. By default, the deny policy in Ranger Admin UI is turned off. Customers need to follow the information in the below link to
enable Deny Policy and use it with OneFS https://community.emc.com/community/products/isilon/blog/2016/10/25/apache-ranger-deny-policies-with-onefs-8010 Wire Encryption OneFS 801
also have support for data in-flight
encryption between Isilon nodes and the
HDFS client. Details on how to enable Wire Encryption will be available soon.
... View more
04-11-2016
06:48 PM
The best practice to download Isilon patches is to go to support.emc.com and search for the patch number 159065.
... View more
12-02-2015
06:42 PM
Thank you Ali. This document will definitely be useful for partner. Will share it with them
... View more
12-02-2015
06:41 PM
Thanks Neeraj
... View more
12-01-2015
11:39 PM
What are the guidelines around allocating storage space for AMS service? A partner is looking for ballpark number for a small / medium and large cluster so they can ensure that they don't run out of storage space for AMS
... View more
Labels:
- Labels:
-
Apache Ambari
11-09-2015
05:58 PM
Thanks Neeraj. In this case, the partner has HA, but no RAID. So they'll just need to failover to to the Passive NN
... View more