Community Articles

Quick command to find the total number of partitions in a Kafka cluster, it could help for example in Mirror Maker sizing.Please replace ZK_SERVER values with your cluster details.cd /tmpzookeeper="ZK_SERVER1:2181,ZK_SERVER2:2181,ZK_SERVER3:2181"sum=0for i in $(/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --list --zookeeper $zookeeper ); do count=$(/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper $zookeeper --topic $i |grep Leader | wc -l); sum=`expr $sum + $count` ; echo 'total partitions is ' $sum; doneIf you want to count partitions with specific filter on the name for Topicszookeeper="ZK_SERVER1:2181,ZK_SERVER2:2181,ZK_SERVER3:2181"sum=0for i in $(/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --list --zookeeper $zookeeper | grep 'FILTER'); do count=$(/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --describe --zookeeper $zookeeper --topic $i |grep Leader | wc -l); sum=`expr $sum + $count` ; echo 'total partitions is ' $sum; done

Create a Falcon ClusterIntroductionApache Falcon is a framework to simplify data pipeline processing and management on Hadoop clusters.It makes it much simpler to onboard new workflows/pipelines, with support for late data handling and retry policies. It allows you to easily define relationship between various data and processing elements and integrate with metastore/catalog such as Hive/HCatalog. Finally it also lets you capture lineage information for feeds and processes. In this tutorial, we are going to create a Falcon cluster by : Preparing up HDFS directories Creating two cluster entities (primaryCluster and backupCluster)Prerequisite Download Hortonworks Sandbox 2.5 Complete the Learning the Ropes of the Hortonworks Sandbox tutorial, you will need it for logging into Ambari as an administrator user.Once you have downloaded the Hortonworks Sandbox and run the VM, navigate to the Ambari interface on the port 8080 of the IP address of your Sandbox VM. Login with the usern

What: Executing Scala Apache Spark Code in JARS from Apache NiFiWhy: You don't want all of your scala code in a continuous block like Apache ZeppelinTools: Apache NiFi - Apache Livy - Apache Spark - ScalaFlows:Option 1: Inline Scala CodeApache Zeppelin Running the Same Scala Job (have to add the jar to interpreter for Spark and restart)Grafana Charts of Apache NiFi RunLog Search Helps You Find ErrorsRun Code For Your Spark ClassSetting up Your ExecuteSparkInteractive ProcessorSetting Up Your Spark Service for ScalaTracking the Job in Livy UITracking the Job in Spark UII was looking at doing this: Pull code from Git and put into a NiFi attribute, run directly. For bigger projects, you will have many classes and dependencies that may require a full IDE and SBT build cycle. Once I build a Scala jar I want to run against that.Example Codepackage com.dataflowdeveloper.exampleimport org.apache.spark.sql.SparkSessionclass Example () { def run( spark: SparkSession) { try {

Background:When installing HDF only, there used to be a default Grafana dashboard for NiFi with the general metrics of NiFi available. When installing HDF on top of an existing HDP cluster (with latest versions of stacks), there is no default dashboard for NiFi even though there are some for Storm and Kafka. This should be fixed for the next version.Problem Summary:Nifi dashboard not visible in Grafana after installing HDF over HDP.Root cause: json file is missing in the grafana directoryResolution: 2 Options 1. Import Json files using grafana dashboard (HDP) or 2. Copy json file as below cp /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/package/files/grafana-dashboards/HDF/grafana-nifi-ho* /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/package/files/grafana-dashboards/HDP/

1. GoalThis article is in continuation of this HCC article https://community.hortonworks.com/content/kbentry/101181/rowcolumn-level-security-in-sql-for-apache-spark-2.html.One can take advantage of Row/Column level security of Spark via various Zeppelin interpreters as explained in the following table:Interpreter nameRow/Column security feature supported?Reason for no support% jdbc (with spark1.x STS)Yes% jdbc (with spark2 STS)Yes% livy.sqlNoZeppelin’s livy interpreter won’t support Row/Column level security because it uses yarn-cluster mode and it needs delegation tokens to access HiveServer2 in yarn-cluster mode. This support is not present in Spark1.x% livy2.sqlYes% spark.sqlNoZeppelin’s Spark interpreter group does not support user impersonation% spark2.sqlNoZeppelin’s Spark interpreter group does not support user impersonationIn this article, we will show how to configure Zeppelin’s livy2 and jdbc interpreters to take advantage of Row/Column level security feature provided by Spar

The article describes how to install and configure Sparkmagic to run in HDP2.5 against Livy Server and Spark 1.6.2Reference:Using Jupyter with Sparkmagic and Livy Server on HDP 2.5 in HCC 1) Install Jupyter http://jupyter.org/install ###### Init venv## Just first timesudo yum install python-pip python-dev python-virtualenv -ymkdir ~/jupyter_env## After 2ndvirtualenv --system-site-packages ~/jupyter_envsource ~/jupyter_env/bin/activatecurl -O https://bootstrap.pypa.io/get-pip.pysudo python get-pip.pysudo easy_install -U pippython -m pip install --upgrade pip# for jupyter, it need gccsudo yum install gcc -ysudo pip install jupyter notebook ipython http://zzeng-hdp-ambari:8888/tree?token=17dfdcb7525ff7470a637752450bbd586f607eddccc86a7f 2) Use Jupyter to connect Spark Livy https://community.hortonworks.com/articles/70501/using-jupyter-with-sparkmagic-and-livy-server-on-h.html # Failed building wheel for pykerberos sudo yum install krb5-devel -ysudo -H pip install sparkmagicsudo pi

prerequisites: 1) You must have an MySQL Server database instance running to be used by Ranger. 2) Execute the following command on the Ambari Server host. Replace database-type with mysql|oracle|postgres|mssql|sqlanywhere and /jdbc/driver/path based on the location of corresponding JDBC driver: ambari-server setup --jdbc-db={database-type} --jdbc-driver={/jdbc/driver/path} 3) make sure root user have access to the db from ranger host and ranger kms host. eg: your ranger host is ranger.host.com & ranger kms host is ranger.kms.host.com then you should run following command on the mysql database: GRANT ALL ON *.* TO 'root'@'ranger.host.com' IDENTIFIED BY '<root_password>' WITH GRANT OPTION; GRANT ALL ON *.* TO 'root'@'ranger.kms.host.com' IDENTIFIED BY '<root_password>' WITH GRANT OPTION; flush privileges;Steps:1) go to the ambari and go to the add service wizard 2) select the ranger and ranger kms host 3) make sure you fill following properties carefully in Ranger &am

Another of people have asked at the Future of Data Meetup (https://www.meetup.com/futureofdata-princeton/) how they can automate the deployment, versioning and other operations around Apache NiFi. Fortunately, there is a project that is current, very easy to use and now available in version 0.8. That tool that you need is here: https://github.com/Chaffelson/nipyapi The nipyapi does it all. As you can see in the screenshot above, you can even run it in Apache Zeppelin if it's installed on your HDP cluster.Installation and SetupThis is very simple. I have installed in on HDP and HDF nodes in Centos 7 and on my Mac in both version 2.7 and 3.x. You will need Python and PiP to get started.For most people you can quickly install via:pip install nipyapiIf not, you can clone the repo and then:pip install -r requirements.txt && pip install -r requirements_dev.txtTo quote, @Dan Chaffelson NiPyApi v0.8.0 released!The#Python Client for@apachenifi now supports secured environm

Goals Get basic understanding on how variables are set at HiveCLI and Beeline utilitiesRetrieving values from table based on variabelsCreating views based on variablesExecuting multiple statements using session variables via HiveCLI and Beeline Manipulating variables at runtime within one session (possible only at hiveCLI)HiveCLIVariables can be set at the session level or while launching the hive shell.Setting session variable while launching hive shell[hive@sandbox ~]$ hive -hiveconf myvar=2016-01-01Logging initialized using configuration in file:/etc/hive/2.3.2.0-2950/0/hive-log4j.propertieshive> set myvar;myvar=2016-01-01hive>OR Setting the variable at the runtime within Hive CLIhive> set myvar=2061-01-20 > ;hive> set myvar;myvar=2061-01-20hive>Performing SELECTs based on variable to retrieve datahive> select * from t_t1 limit 2;OKstring_1 2016-01-01string_2 2016-01-02Time taken: 4.904 seconds, Fetched: 2 row(s)hive> describe t_t1;OKf1

Objective This tutorial walks you through a NiFi flow that utilizes the PublishKafkaRecord_0_10 processor to easily convert a CVS file into JSON and then publish to Kafka. The tutorial is based on the blog "Integrating Apache Nifi with Apache Kafka", updated with the more recent record-oriented processors and controller services available in NiFi. Note: The record-oriented processors and controller services were introduced in NiFi 1.2.0. As such, the tutorial needs to be done running Version 1.2.0 or later.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.3.0 Apache Kafka 0.10.2.1PublishKafkaRecord_0_10 (CSV to JSON)Support Files Here is a template of the flow discussed in this tutorial: publishkafkarecord.xmlDemo ConfigurationKafka Download & Install The flow in this demo utilizes the PublishKafkaRecord_0_10 processor, which as the name implies, utilizes the Kafka 0.10.x Producer API. As a result, a

While enabling SSL for Ranger (I am writing for Ranger 0.4 from HDP2.2.4, configured by Ambari) I ran into issues loosing some days. The critical points areverify that property "Common Name For Certificate" in Ranger policy definition AND in Plugin configuration is matching the DN of your used certificateif you use HA enabled services like Namenode-HA, you have to use the same cert (at least) throughout both Namenodes to match the "Common Name for Certificate". !! Do not use the servers FQDN for the DN property at certification creation time !!This is the step-by-step instruction what I did to make it work. Please keep in mind that I will use ONE certificate throughout all nodes. If that does not match your security criteria, you have to adapt the steps accordinglycreate Ranger-admin keystore cd /etc/ranger/admin/conf/ sudo /usr/java/jdk1.7.0_79/bin/keytool -genkey -keyalg RSA -alias ranger-admin -keystore ranger-admin-keystore.jks -validity 360 -keysize 2048 -storepass <password

BackgroundStarting with Ambari-2.6.1 release, a new repository has been introduced called HDP-GPL. This article covers FAQs related to the purpose of this repository, when it takes effect and the impact it can have on your cluster esp. when dealing with LZO compression libraries. FAQsWhat is the need for a new GPL repo?LZO compression libraries use a GPL license, which differs from rest of the packages in Ambari/HDP which are Apache licensed. Hence, the need for a separate repo which is named ‘HDP-<version_number>-GPL’. The Version Definition File (VDF) starting HDP-2.6.4.0 release has been updated to include the new ‘HDP-2.6-GPL’ repo. Refer the sample VDF file for CentOS7 here. Here is a snippet from Ambari web UI for the new 'HDP-2.6-GPL' field:What are the exact contents of HDP-GPL repo?As of HDP-2.6.4.0 release, the repo contains hadooplzo package files as follows:hadooplzo_2_6_4_0_91-0.6.0.2.6.4.0-91.x86_64.rpmhadooplzo_2_6_4_0_91-native-0.6.0.2.6.4.0-91.x86_64.rpmHow is Am

Extracting Text or HTML from PDF, Excel and Word Documents via Apache NiFiThis version has been tested with HDF 3.1 and Apache NiFi 1.5. This processor is using Apache Tika 1.17 and is a non-supported Open Source Community processor that I have written.A user posted asking about HTML output, I took a look and it was easy so I added an option for that.Apache NiFi FlowYou must download or build the nifi-extracttextprocessor nar and put in your lib, then you can add the processor.Select html or textHere's is the autogenerate documentation:You can see we set the output mime.type to text/html.Apache NiFi Example Flow to Read a File and Convert to HTMLSource and Junit in EclipseExample Output HTML<html xmlns="http://www.w3.org/1999/xhtml"><head><meta name="pdf:PDFVersion" content="1.3"/><meta name="X-Parsed-By" content="org.apache.tika.parser.DefaultParser"/><meta name="X-Parsed-By" content="org.apache.tika.parser.pdf.PDFParser"/><meta name="xmp:CreatorTool"

Big Data DevOps: Part 2: Schemas, Schemas, Schemas. Know Your Records, Know Your DataTypes, Know Your Fields, Know Your Data.Since we can do process records in Apache NiFi, Streaming Analytics Manager, Apache Kafka and any tool that can work with a schema, we have a real need to use a Schema Registry. I have mentioned them before. One thing that is important is to be able to automate the management of schemas. Today we will be listing and exporting them for backup and migration purposes. We will also cover how to upload new schemas and version of schemas.The steps to backup schemas with Apache NiFi 1.5+ is easy.Backup All Schemas GetHTTP: Get the List of Schemas for SR via GET SplitJson to turn list into individual recordsEvaluateJsonPath: get the schema name.InvokeHTTP: get the schema bodyEvaluateJsonPath: turn the schema text into a separate flow fileRename and save both the full JSON record from the registry and the schema only.NiFi FlowInitial Call to List All Sche

This article discusses prerequisites required for Cloudbreak credential and addresses common issues. This article applies to Cloudbreak 2.4.0. When getting started with Cloudbreak, the most common pain point is creating the Cloudbreak credential, or - more precisely - meeting the prerequisites for the Cloudbreak credential. Why do you even need this Cloudbreak credential? Cloudbreak not only runs on your cloud provider account, but once you start using, it provisions resources (such as VMs) that run as part of your account and that you or your organization must pay for. Cloudbreak credential provides the means for Cloudbreak to authenticate with your cloud provider account and provision such resources. The exact details vary depending on the cloud provider: On Azure this is typically done by creating an app registration in Azure Active Directory and providing its details to Cloudbreak. On AWS this typically happens by assigning a proper IAM role to Cloudbreak. Cloudbreak can assu

In order to check the status and stability of your cluster it makes sense to run the service checks that are included in Ambari. Usually each Ambari Service provides its own service check, but their might be services that wont include any service check at all. To run a service check you have to select the service (e.g. HDFS) in Ambari and click "Run Service Check" in the "Actions" dropdown menu. Service Checks can be started via the Ambari API and it is also possible to start all available service checks with a single API command. To bulk run these checks it is necessary to use the same API/method that is used to trigger a rolling restart of Datanodes (request_schedules). The "request_schedules" API starts all defined commands in the specified order, its even possible to specify a pause between the commands. Available Service Checks: Service Name service_name Command HDFS HDFS HDFS_SERVICE_CHECK YARN YARN YARN_SERVICE_CHE

Goal Create and Setup IPA Keytabs for HDPTested on CentOS 7 with IPA 4, HDP 2.3.0 and Ambari 2.1.1Assumptions Ambari does NOT currently (2.1.x) support the automatic generation of keytabs against IPAIPA Server is already installed and IPA clients have been installed/configured on all HDP cluster nodesIf you are using Accumulo, you need to create the user 'tracer' in IPA. A keytab for this user will be requested for Accumulo Tracer.We'll use Ambari's 'resources' directory to simplify the distribution of scripts and keys.Enable kerberos using wizardIn Ambari, start security wizard by clicking Admin -> Kerberos and click Enable Kerberos. Then select "Manage Kerberos principals and key tabs manually" option Enter your realm (Optional, Read on before doing this...) Remove clustername from smoke/hdfs principals to remove the -${cluster_name} references to look like below smoke user principal: ${cluster-env/smokeuser}@${realm}HDFS user principal: ${hadoop-env/hdfs_user}@${realm}HBase user

How to get the number of documents indexed : curl -o /tmp/result.txt --negotiate -u : -X GET "SOLR_SERVER:8886/solr/ranger_audits_shard1_replica1/select?q=*:*&distrib=false"How to run a delete command via curl ( delete data older than 24h 😞curl --negotiate -u: "SOLR_SERVER:8886/solr/ranger_audits/update?commit=true" -H "Content-Type: text/xml" --data-binary "<delete><query>evtTime:[* TO NOW-24HOURS]</query></delete>"How to run optimize command via curl :curl --negotiate -u: "SOLR_SERVER:8886/solr/ranger_audits/update?optimize=true"

##install Luna client, Unzip Luna client for example under /opt/LUNAHSMunder /opt/LUNAHSM/linux/64/run sh install.sh allFollow the the instructions below, for the questions asked, please answer as below :Accept conditions(y/n) yProductsChoose Luna Products to be installed [1]: Luna SA [2]: Luna PCI-E [3]: Luna G5 [4]: Luna Remote Backup HSM [N|n]: Next [Q|q]: QuitEnter selection: 1ProductsChoose Luna Products to be installed*[1]: Luna SA [2]: Luna PCI-E [3]: Luna G5 [4]: Luna Remote Backup HSM [N|n]: Next [Q|q]: QuitEnter selection: nAdvancedChoose Luna Components to be installed [1]: Luna Software Development Kit (SDK)*[2]: Luna JSP (Java)*[3]: Luna JCProv (Java) [B|b]: Back to Products selection [I|i]: Install [Q|q]: QuitEnter selection: iList of Luna Products to be installed:- Luna SAList of Luna Components to be installed:- Luna JSP (Java)- Luna JCProv (Java)... installation complete<br>#now to swap the certificate : copy SERVER.pem from LUNA server to your

Quick post to add an auto fix for Solr infra lock issue.Ranger server under : /usr/hdp/current/ranger-admin/contrib/solr_for_audit_setup/confEdit the file solrconfig.xmlUncomment and change <unlockOnStartup>false</unlockOnStartup> to <unlockOnStartup>true</unlockOnStartup>Submit the new xml: /usr/lib/ambari-infra-solr-client/solrCloudCli.sh --zookeeper-connect-string XXXX:2181/infra-solr --upload-config --config-dir /usr/hdp/current/ranger-admin/contrib/solr_for_audit_setup/conf --config-set ranger_audits --jaas-fileIncrease the sleep time from 5 to 30 seconds in /opt/lucidworks-hdpsearch/solr/bin/solrsed -i 's/(sleep 5)/(sleep 30)/g'/opt/lucidworks-hdpsearch/solr/bin/solrOr in the following :sed -i 's/(sleep 5)/(sleep 30)/g' /usr/lib/ambari-infra-solr/bin/solr you can also add in the script the following command :hadoop fs -rm /user/infra-solr/ranger_audits/core_node1/data/index/write.lock

Quick tips to optimise your infra Solr for Ranger audits using SolrCloud.1) Change the SolrCloud retention period of the audit.On the server of ranger under : /usr/hdp/current/ranger-admin/contrib/solr_for_audit_setup/conf #### Edit the file or use sed to replace the 90 Days in the solrconfig.xml , choose the right retention period, here is 6 hourssed -i 's/+90DAYS/+6HOURS/g' solrconfig.xml sed -i 's/86400/7200/g' solrconfig.xml2) Change ZK config, by submitting the xml again/usr/lib/ambari-infra-solr-client/solrCloudCli.sh --zookeeper-connect-string XXXXXX:2181/infra-solr --upload-config --config-dir /usr/hdp/current/ranger-admin/contrib/solr_for_audit_setup/conf --config-set ranger_audits --jaas-file /usr/hdp/current/ranger-admin/conf/ranger_solr_jaas.confCheck that we loaded it correctly, in the Solr UI or with the following command#Download the solrconfig.xml from Zookeeper /usr/lib/ambari-infra-solr/server/scripts/cloud-scripts/zkcli.sh --zkhost XXXXXX:2181 -cmd getfile /infra-so

A Hive Database can contain both transactional and non transactional table. Hence, if we are doing some quick checks to determine if the table is ACID enabled, please run the following command.# hive -e "describe extended <Database>.<tablename>;" | grep "transactional=true"If you get an output with the string that you grep for, then the table is transactional.Example: #hive -e "describe extended default.hello_acid;" | grep "transactional=true"

This is for people preparing to attend my talk on Deep Learning at DataWorks Summit Berling 2018 (https://dataworkssummit.com/berlin-2018/#agenda) on Thursday April 19, 2018 at 11:50AM Berlin time.In this example we required Apache NiFi 1.5 or newer.This is part 2 of https://community.hortonworks.com/articles/155435/using-the-new-mxnet-model-server.htmlOur flow that receives the JSON files from the server does some minimal processing. We add some meta data fields, infer an AVRO schema from the JSON file (we only need to do this once in development and then you can delete that box from your flow). As you can see I can easily push that data to HDFS as a parquet file.This is if you wish to not install Apache MXNet on your HDF, HDP or related nodes. You can now install Apache MXNet plus MMS on a cloud or edge server and call it via HTTP from Apache NiFi for processing.Local Apache NiFi Flow To Call Our SSD Predict and Squeeze Net Predict REST ServicesCluster Receiving The Two Remote

The preferred method to manage network security with Cloudbreak is to use existing security groups in the target cloud. You manage the security groups in the target cloud's native tools and Cloudbreak adds the nodes to those existing groups. This allows the access rules in the security group to be centrally managed eliminating the need to propagate changes to existing clusters. Each cluster that gets created will have the same standard access rules, which can be very granular to provide only the access that is required. This also allows separation of the security management duties from the cluster administration duties if an organization requires it.Network security groups - Microsoft Azure Each cloud has its own interface for managing security groups. In Azure, for instance, the access rules are in network security groups and each group is available in an individual location. Here we see four network security groups that I've created in Azure's South Central US location for u

Ingest All The Things Series: Flight Data Via RadioI am using the FlightAware Pro Stick Plus ADS-B USB Receiver with Built-in Filter on a Mac, I should hook this up to one of my raspberry pis and add a longer antenna outside. You need a good antenna, a good location and nothing blocking your signal. It also depends on what air traffic is nearby.For a proof of concept it's pretty cool to see air data going through a cheap USB stick into a computer stored in a file and loaded into Apache NiFi to send on for data processing.There is a web server you can run to see the planes on a map which is pretty cool, but I want to just ingest the data for processing.My Equipment If you wish to watch the data flash by in a command-line interface, you can run with interactive flag and watch all the updates.We are dumping the data as it streams as raw text into a file. A snippet of it tailed in Apache NiFi is shown below:We are also ingesting ADS-B data that is provided by an open data REST API