@Shikhar Agarwal Have you tried reading the Hive table using SparkSession instead? The following code works for me using Spark 2 on HDP 2.5.3: /usr/hdp/current/spark2-client/bin/spark-shell --num-executors 2 --executor-memory 512m --master yarn import org.apache.spark.sql.Row import org.apache.spark.sql.SparkSession val hiveSession = SparkSession .builder() .appName("Spark Hive Example") .enableHiveSupport() .getOrCreate() val hiveResult = sql("SELECT * FROM timesheet LIMIT 10") hiveResult.show() Details for this can be found in the Apache Spark docs: http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables ... View more

@Michał Kabocik You can assign access control filters to the shiro urls to restrict on certain roles, in your case it would look something like this: [urls] # authentication method and access control filters /api/version = anon /** = authc, roles[role1] You can even take it a step further limit access to a subset of configuration pages within Zeppelin (i.e, lock down the 'Interpreter' or 'Configurations' page to be used by admins only): [urls] # authentication method and access control filters /api/version = anon /api/interpreter/** = authc, roles[admin] /api/configurations/** = authc, roles[admin] /api/credential/** = authc, roles[admin] #/** = anon /** = authc See the following documentation for an example Zeppelin shiro config that does this: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_zeppelin-component-guide/content/zepp-shiro-ref.html ... View more

@sprakash That is correct - NiFi LDAP authentication only works over HTTPS. If you want to secure user access via LDAP in NiFi, you need to configure NiFi in HTTPS mode and remove HTTP access. Check out the following article for details of configuring LDAP auth via NiFi: https://community.hortonworks.com/articles/7341/nifi-user-authentication-with-ldap.html ... View more

@jbarnett To answer your question - no, if you have SSSD configured you do not need to also configure core-site mapping with LDAP. Regarding your issue, it could be related to the space in your group name - could you try remove the space in the 'domain users' or test with a group that doesn't contain any spaces. ... View more

@Vijay Kumar J NiFi is definitely feasible for production use, and it is perfectly suited for your MongoDB to HBase data movement use case. NiFi is a tool used for managing dataflow and integration between systems in an automated and configurable way. It allows you to stream, transform, and sort data and uses a drag-and-drop UI. Dealing with failures - NiFi is configurable - when you build your flow within NiFi you can determine how you want to handle failures. In your case, you could build a flow in NiFi that retries on failure, and sends out an email on failure (this is an example, how you want to handle failures for fetch and storing data can be configured however you need) Execute NiFi on an Hourly Basis - NiFi isn't like traditional data movement schedulers, and flows built using NiFi are treated as 'always-on' where data can be constantly streamed as it is received. That being said, NiFi provides the ability to schedule each processor if needed, so in your case you could have your GetMongo processor set to run once every hour, and your PutHBaseJSON processor to push data to HBase as soon as it is received from the GetMongo processor. Check out this tutorial for getting started and building your first flow: http://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi/ ... View more

@Predrag Minovic - Slight update on storm, we can run multiple Nimbus servers: https://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_Ambari_Users_Guide/content/ch05s05.html This is to deal with cases where Nimbus can't be automatically restarted (e.g disk failure on the node). Details of Nimbus HA is outlined here: http://hortonworks.com/blog/fault-tolerant-nimbus-in-apache-storm/ ... View more

@sankar rao To elaborate on the answer provided by @Artem Ervits The edge node is typically used to install client tools, so it will make sense to install the AWS S3 CLI on the edge. For adding new users to the cluster, you need to ensure that the new users exist in ALL nodes. The reason is that Hadoop by default takes the user/group mappings from the UNIX users by default. So for Hadoop to 'know' about the new user you've created on the edge node, that same userid should exist on all nodes. ... View more

@KC Your 'InferAvroSchema' is likely capturing the schema as an attribute called 'inferred.avro.schema' (assuming you followed the tutorial here: https://community.hortonworks.com/articles/28341/converting-csv-to-avro-with-apache-nifi.html ) If that's the case, you can view its output by looking at one of the flowfiles in queue after 'InferAvroSchema' (List queue > select a flowfile > view attributes > view inferred.avro.schema property). If you want to manually define the schema without changing too much of your flow, you can directly replace your 'InferAvroSchema' processor with an 'UpdateAttribute' processor - within the 'UpdateAttribute' define a new property called inferred.avro.schema and paste in your avro schema as the value (json format). ... View more

@Neeraj The docs are conflicting. It is recommending "use HiveContext" in secure mode (with kerberos): http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_spark-guide/content/ch_installing-kerb-spark.html#spark-kerb-access-hive and under best practices it states that it is not supported with Kerberos: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_spark-guide/content/ch_practices-spark.html ... View more

Here are some details on setting up WASB as the default filesystem (fs.defaultFS) using HDP 2.3.x on Azure IaaS: Architecture Considerations Before using WASB as the defaultFS, it is important to understand the architectural impact this will have on the cluster. What is WASB? + Pros Windows Azure Storage Blob (WASB) is an extension built on top of the HDFS APIs that interfaces with data stored within an Azure Blob Storage account. One of the key advantages of using WASB is that it creates a layer of abstraction that enables separation of storage from compute. You can also add/remove/modify files in the Azure blob store without regard to the Hadoop cluster, as directory and file references are checked at runtime for each job. This allows you to better utilise the flexibility of a cloud deployment, as data can persist without the need for a cluster / compute nodes. You have the following deployment options when using WASB: HDFS as defaultFS - Deploy a HDFS cluster and configure an external WASB filesystem to interface with a separate Blob Storage account. In this scenario your default Hadoop data would still be stored on HDFS, and you would have some data stored in a Blob Storage account that can be accessed when needed. WASB as defaultFS - Deploy the cluster with WASB as the default filesystem to completely replace HDFS with WASB. In this scenario HDFS still technically exists but will be empty as all files / folders deployed with the cluster are stored in WASB (blob storage). Note, the steps below describe how to configure a cluster for this approach. In many ways you can interface with the WASB filesystem as you would with HDFS, by specifying the WASB URLs: wasb://<containername>@<accountname>.blob.core.windows.net/<path> For example, the following hadoop FileSystem Shell command to WASB would behave similar to the same call made to HDFS: hadoop fs -ls wasb://<containername>@<accountname>.blob.core.windows.net/ However, it is important to note that WASB and HDFS are separate filesystems and WASB currently has some limitations that should be considered before implementing. Current Limitations - As at October 2015 The following limitations currently exist within the WASB filesystem: WebHDFS is not compatible with WASB This limits the functionality within both Hue and Ambari Views (and any application that interfaces with the WebHDFS APIs) Security permissions are persisted in WASB, but are not enforced. File owner and group are persisted for all directories and files, but the permission model is not enforced at the filesystem level and all authorisation occurs at the level of the entire Azure Blob Storage account. When deciding to use WASB as the defaultFS, it is recommended that you first review and assess your security and data access requirements and ensure you can meet them against the above limitations. Installation + Configuration Steps Important: There is a known bug in HDP 2.3.0 that prevents non HDFS filesystems (S3, WASB etc) being set as the default filesystem [HADOOP-11618, HADOOP-12304]. For the following steps to work, you will need to use either a patched version of HDP 2.3 or use HDP 2.3.2 or later. To configure WASB as the default filesystem in-place of HDFS, the following Azure information is required: Azure Storage Account URL Container Name Storage Access Key This information is used along with the Hadoop configurations within core-site.xml to configure WASB. For a fresh install, follow the standard documentation for installing HDP via Ambari found here until you get to Customize Services, where you will need to configure your WASB properties. Also note that when using WASB as the defaultFS, you do not need to mount any additional data drives to the servers (as you would for a HDFS cluster). Customize Services The following is a list of configurations that should be modified to configure WASB: fs.defaultFS wasb://<containername>@<accountname>.blob.core.windows.net fs.AbstractFileSystem.wasb.impl org.apache.hadoop.fs.azure.Wasb fs.azure.account.key.<accountname>.blob.core.windows.net <storage_access_key> Even though WASB will be set as the fs.defaultFS, you still need to define DataNode directories for HDFS. As the intent here is to use WASB as the primary FS, you can set the HDFS datanode directories to the temporary /mnt/resource mount point that is provided with Azure compute servers if you only plan to use HDFS for temporary job files. DataNode Directories /mnt/resource/Hadoop/hdfs/data Outside of these core-site.xml configurations, hive has the following requirements when working with blob storage on Azure: Point archive and jar files to 'wasb://' instead of 'hdfs://' templeton.hive.archive templeton.pig.archive templeton.sqoop.archive templeton.streaming.jar wasb:///hdp/apps/${hdp.version}/hive/hive.tar.gz wasb:///hdp/apps/${hdp.version}/pig/pig.tar.gz wasb:///hdp/apps/${hdp.version}/sqoop/sqoop.tar.gz wasb:///hdp/apps/${hdp.version}/mapreduce/hadoop-streaming.jar Skip azure metrics in custom webhcat-site When WASB is used, metrics collection is enabled by default. For webhcat server, this causes an unnecessary overhead that we can disable. fs.azure.skip.metrics true Further Reading WASB - Apache Docs MSDN Blog - Understanding WASB MSDN Blog - Why WASB ... View more