Resources

Manufacturing IoT/Process Monitoring DemoSummaryYou will be provided with background on what type of usage cases Manufacturing Shop Floors are implementing to maintain a competitive advantage in their respective marketplaces. And then we will walk through, step by step, instructions on how to build your own Manufacturing IoT - Process Monitoring Dashboard environment. This dashboard will be isolated in this demo to two fans on a single Manufacturing Shop Floor and a single cluster for demo purposes.What’s IncludedInstructions and pointers on how to:Obtain and deploy HDP, HDF, and KEPServerEXSetup KEPServerEX to access an OPC TCP UA ServerSetup KEPServerEX - IoT Gateway as a REST API EndpointA pre-build Nifi Flow (fans_demo_v7.xml)Use either the Nifi “InvokeHTTP” or “GetFile” processorInbound Source can be an IoT Gateway REST API or Fan Event File Dump dataA Fan Events Data Tar file containing a dump of individual fan events (fans.tar.gz)A Hive Create Table query (create_table_fans_demo

IoT Series: Sensors: Utilizing Breakout Garden Hat: Part 2 - Integrating MQTT, TensorFlow and Kafka StreamsSee Part 1: https://community.hortonworks.com/content/kbentry/229522/iot-series-sensors-utilizing-breakout-garden-hat-p.htmlIn this second part, I have incremented the functionality in the Python capture, MiniFi, NiFi and post-NiFi processing. I have added a Kafka Streams Java application.With this NiFi flow we are consuming the MQTT and Kafka messages send by the Kafka Streams application.In one flow, we received MQTT messages, pull out the entire flow as a message and send to a Slack channel.In another flow we ingest two types of Kafka messages and store the JSON ones that have a schema in an HBase table via the record processor. In this flow we receive from the local NiFi router that was called by MiniFi over S2S/HTTP(s). We build two types of messages and send them to Kafka 2.0 brokers. One is the full JSON message with a schema, the other is just the temperature. We

Implementing Streaming Machine Learning and Deep Learning In Production Part 1After we have done our data exploration with Apache Zeppelin, Hortonworks Data Analytics Studio and other Data Science Notebooks and Tools, we will start building iterations of ever improving models that need to be used in live environments. These will need to run at scale and score millions of records in real-time streams.These can be in various frameworks, versions,types and many options of data required.There are a number of things we need to think about when doing this.Model Deployment OptionsApache SparkApache Storm (Hortonworks Streaming Analytics Manager - SAM)Apache Kafka StreamsApache NiFiYARN 3.1YARN SubmarineTensorFlow Serving on YARNCloudera Data Science WorkbenchRequirementsClassificationREST APISecurityAutomationData LineageSchema Versioning, REST API and ManagementData ProvenanceScriptingIntegration with KafkaContainerized Services Support Docker Containers running on YARN Support Dockerized

Problem statement : i am upgrading my ambari-2.6.2.2 to ambari-2.7.3 and the ambari-server upgrade commands fails with exception : java.sql.SQLSyntaxErrorException: Unknown table 'ambari_configuration' in information_schema[root@slambe-1 java]# ambari-server upgradeUsing python /usr/bin/pythonUpgrading ambari-serverINFO: Upgrade Ambari ServerINFO: Updating Ambari Server properties in ambari.properties ...INFO: Updating Ambari Server properties in ambari-env.sh ...INFO: Original file ambari-env.sh keptINFO: Fixing database objects ownerAmbari Server configured for MySQL. Confirm you have made a backup of the Ambari Server database [y/n] (n)? yINFO: Upgrading database schemaERROR: Unexpected ValueError: No JSON object could be decodedFor more info run ambari-server with -v or --verbose optionwhen i see ambari-server logs i see the following exception:2018-12-28 13:59:07,062 INFO [main] DBAccessorImpl:869 - Executing query: CREATE TABLE ambari_configuration (category_name VARCHAR(100)

Beeline is a JDBC client tool which is used to connect to HiveServer2 or HiveServer2Interactive(LLAP). Beeline, which connects to HiveServer2 and requires access to only one .jar file: hive-jdbc-<version>-standalone.jar Hortonworks recommends using HiveServer2 and a JDBC client (such as Beeline) as the primary way to access Hive. This approach uses SQL standard-based authorization or Ranger-based authorization. However, some users may wish to access Hive data from other applications, such as Pig. For these use cases, use the Hive CLI and storage-based authorization. Connecting to Hive with Beeline The following examples demonstrate how to use Beeline to connect to Hive for all possible variations of these modes. Embedded Client Use the following syntax to connect to Hive from Beeline in embedded mode: !connect jdbc:hive2:// Remote Client with HiveServer2 TCP Transport Mode and SASL Authentication Use the following syntax to connect to HiveServer2 in TCP mode from a remot

IntroductionIn Customizing Atlas (Part1): Model governance, traceability and registry I provided a brief overview of Atlas types and entities and showed how to customize them to fit your needs. I showed the specific example of a Model type used to govern your deployed data science models and complex Spark code.In Customizing Atlas (Part2): Deep source metadata and embedded entities I showed how to customize Atlas to hold knowledge of ingested data that goes deep beyond the data itself, e.g. detailes of the device that generated the data. I also showed how to implement the pattern of embedding an entity (not string) as an attribute value in your custom type. The result is a clickable hyperlink in the UI that opens that entity and its metadata.In this post I will:show how to extend your Atlas lineage to include processing and outputs on non-Hadoop systemsrepresent the above as a single lineage that connects data in Hadoop to a reporting system which generates a report and sends an ema

Here is the steps to access HBase UI through Knox. This is tried and tested with Ambari 2.4 and HDP2.51. Make sure Knox is configured properly and it works fine.2. ssh to Knox gateway host and go to /var/lib/knox/data-2.5.3.0-37/services3. download the configurations from https://github.com/apache/knox/tree/v0.11.0/gateway-service-definitions/src/main/resources/services/hbaseui/1.1.0/ URL.4. make sure your folder structure should look alike /var/lib/knox/data-2.5.3.0-37/services/hbaseui/1.1.0 and should have rewrite.xml and service.xml files5. change the owner/Group permissions to Knox for /var/lib/knox/data-2.5.3.0-37/services/hbaseui/ and subdirectory6. Go to Knox configurations Modify "Advanced topology" with below service tag <service> <role>HBASEUI</role> <url>http://HBASEMANTERHOST:16010</url> </service> <service>7. Restart Knox service.8. You should be able to access HBase UI from

IntroductionIn the previous post Customizing Atlas (Part1): Model governance, traceability and registry we: provided a brief overview of Atlas types and entitiesshowed how to customize Atlas types and entities to fit your own needs and appear in Atlas search and lineagecustomized a special type called model, which inherited from Process and empowered Atlas to govern the deployment of data science modelscommented on operationalizing custom entitiesIn this post we will: customize an Atlas type to represent deep source metadata (metadata beyond source data itself)customize an Atlas type to represent devices (metadata about the actual device that generates data)embed the device entity in the deep source entity (make the device entity an attribute value in the deep source metadata)show how device as an attribute value is a clickable link in the Atlas UI that opens to the full device entityConcepts and ExampleMain ConceptsThe main concepts are summarized in the table below.Example: Gene sequ

Disclaimer:This article is based on my personal experience and knowledge.Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case. Always contact Hortonworks support if its production clusterProblem Description: I am currently using ambari-2.6.2 version. I am trying to update some configs via configs.py and my python version is python-2.7.9 and my configs.py is failing with below error :[root@asnaik1 certs]# /var/lib/ambari-server/resources/scripts/configs.py --port=8443 --action=set --host=asnaik1.openstacklocal --cluster=asnaik --config-type=kafka-env --user=admin --password=admin --key=kafka_log_dir --value=/tmp --protocol=https2018-12-05 10:24:57,615 INFO ### Performing "set":2018-12-05 10:24:57,615 INFO ### new property - "kafka_log_dir":"/tmp"Traceback (most recent call last):File "/var/lib/ambari-server/resources/scripts/configs.py", line 364, in <module>sys.exit(main())File "/var/lib/ambari-server/reso

Introduction Continuing my series on Beast Mode Quotient, let's automate the creation and termination of data science ready clusters. As always, since this is step 2 of a series of article, this tutorial depends on my previous article. Pre-Requisites In order to run this tutorial you will need to have a CB instance available. There are plenty of good tutorial out there, I recommend this one Agenda This tutorial is divided in the following sections: Section 1: Create a blueprints & recipes to run a minimal data science ephemeral cluster Section 2: Add blueprint and recipes via Cloudbreak interface Section 3: Automate cluster launch and terminate clusters Section 1: Create a blueprints & recipes to run a minimal data science ephemeral cluster A Cloudbreak blueprint has 3 parts: Part 1: Blueprint details Part 2: Services Configuration Part 3: Host Components configuration Here are the details of each part for our blueprint. Blueprint details This part i

With Stochastic Streaming Algorithms it's possible to analyze highly efficient and performant massive amounts of data. The main characteristics of sketches are very small size (sub-lineare space consumption), single-pass, mergeable and deliver approximate results in proven error bounds. If an approximate result is acceptable, then it is possible that data sketches are orders-of-magnitude faster compared to classic indexing or aggregation techniques. (I don't go into the mathematical discussion, more details here DataSketches)The following example shows how to use Data Sketches within Hive (also possible with Druid):Download and install Hortonworks Sandbox (it comes with demo database: foodmart)Download Java artifact sketches-hive-0.11.0-with-shaded-core.jar (provided by Yahoo) here jar repo and copy the jar file into HDFS /apps/hive/warehouse/lib directory#Login as hive user hive@sandbox-hdpwget https://search.maven.org/remotecontent?filepath=com/yahoo/datasketches/sketches-hive/0.11.0

Install AWS CLI from the steps provided in the below linkhttps://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.htmlVerify AWS CLI installation$aws --versionConfigure AWS credentials$aws configure Download Ozone 0.3.0-alpha tarball from here, untar it.Go to the $PWD/ozone-0.3.0-aplha/compose/ozones3 directory, and start the server:docker-compose up -dCreate alias commandalias ozones3api='aws s3api --endpoint http://localhost:9878'Create bucket:$ ozones3api create-bucket --bucket documentsPut objects to bucket:$ ozones3api put-object --bucket documents --key S3Doc --body ./S3.md$ ozones3api put-object --bucket documents --key hddsDoc --body ./Hdds.md$ ozones3api put-object --bucket documents --key javaDoc --body ./JavaApi.mdList objects in a bucket:$ ozones3api list-objects --bucket documents{"Contents": [{"LastModified": "2018-11-02T21:57:40.875Z","ETag": "1541195860875","StorageClass": "STANDARD","Key": "hddsDoc","Size": 2845},{"LastModified": "2018-11-02T22:36:23.358Z","ET

This article is designed to the steps need to convert an existing Spark pipeline model into a MLeap bundle. To follow the steps required to build the pipeline model used in the tutorial, please download the complete Apache Zeppelin notebook here on my GitHub site. Tools Used In Tutorial Spark 2.3 MLeap Spark 0.13 Step 1 - Load MLeap Spark DependenciesThis dependency is required to convert Spark models into MLeap bundles. in the following section of code, decency files are added to my Zeppelin notebook. Keep in mind this document needs to be present on Maven. %dep//In Zeppelin//Run this before initializing Spark Contextz.load("ml.combust.mleap:mleap-spark_2.11:0.13.0")Step 2 - Build Pipeline Model The Spark pipeline model needs to be built before it can be converted. In this section of code, a Spark pipeline model is built from previously defined stages. To view the complete code list, please download the notebook from my Github site. A pipeline model will be built once it

Post HDP upgrade you will observe all previous version HDP directories will be exist under /usr/hdp/ directory (Including current active). Once upgrade is finalized you can clear those older version directories using Ambari. use following "curl" command. Ssh to ambari server and run following command. (You can use any linux box in network which has curl installed and able to access ambari console URL).curl 'http://<$AMBARI_SERVER>:8080/api/v1/clusters/<$MyClusterName>/requests' -u admin:<Password> -H "X-Requested-By: ambari" -X POST -d'{"RequestInfo":{"context":"remove_previous_stacks", "action" : "remove_previous_stacks", "parameters" : {"version":"<$Current_Active_version>"}}, "Requests/resource_filters": [{"hosts":"<$ClusterNode1>, <$ClusterNode2>"}]}' $MBARI_SERVER => Ambari server name and port. If you are using SSL use the SSL port no. $MyClusterName => Enter cluster name you want to remove directories.$ Current_Active_version =&gt

in the documentation, you can find on the Sqoop partYou can use Sqoop to import data into HDFS or directly into Hive. However, Sqoop can only import data into Hive as a text file or as a SequenceFile. To use the ORC file format, you must use a two-phase approach: first use Sqoop to move the data into HDFS, and then use Hive to convert the data into the ORC file format[...]However, we can use the Sqoop-HCatalog integration feature, which is a table abstraction.Let's use the HDP 2.3.2 sandbox, starting with creating our Hive table, stored as ORC:[root@sandbox ~]# hive hive> CREATE TABLE cds (id int, artist string, album string) STORED AS ORCFILE; hive> INSERT INTO TABLE cds values (1,"The Shins","Port of Morrow"); hive> select * from cds; 1 The Shins Port of MorrowNow let's build our MySQL table[root@sandbox ~]# mysqlWelcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 85Server version: 5.1.73 Source distributionCopyright (c) 2000, 2013, O

This Article will show how to read csv file which do not have header information as the first row. We will then specify the schema for both DataFrames and then join them together. import org.apache.spark.sql.types._val pathA = "hdfs:/tpc-ds/data/store_sales"val pathB = "hdfs:/tpc-ds/data/store/"// For Spark 2.x use -> val df = spark.read.option("header", true).csv(path)val A_df = sqlContext.read.format("com.databricks.spark.csv") .option("header","false") .option("inferSchema","false") .option("delimiter","|") .load(pathA)// Assign column names to the Store Sales dataframeval storeSalesDF = A_df.select( A_df("_c0").cast(IntegerType).as("SS_SOLD_DATE_SK"), A_df("_c1").cast(IntegerType).as("SS_SOLD_TIME_SK"), A_df("_c2").cast(IntegerType).as("SS_ITEM_SK"), A_df("_c7").cast(IntegerType).as("SS_STORE_SK") )val B_df = sqlContext.read.format("com.databricks.spark.csv") .option("header","false") .option("inferSchema","false") .option("delimiter","|") .load(pathB)// A

Disclaimer:This article is based on my personal experience and knowledge.Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case. Always contact Hortonworks support if its production cluster Problem Statement: I have installed HDF-3.1 and ambari-2.6.2.2 and i am upgrading my ambari to 2.7.0.0 to upgrade my HDF to 3.2+ versions and its failing with below exception : INFO: about to run command: /usr/java/jdk1.8.0_162/bin/java -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/java/latest/mysql-connector-java.jar:/usr/share/java/mysql-connector-java.jar' org.apache.ambari.server.upgrade.SchemaUpgradeHelper > /var/log/ambari-server/ambari-server.out 2>&1INFO:process_pid=16599Traceback (most recent call last):File "/usr/sbin/ambari-server.py", line 1060, in <module>mainBody()File "/usr/sbin/ambari-server.py", line 1030, in mainBodymain(options, args, parser)File "/usr/sbin/ambari-server.py", line

PROBLEMWhen setting Bind Anonymous in Ambari's Ranger usersync tab it complains that ranger.usersync.ldap.binddn can not be empty in ambari usersync service logs, with the below stack trace :File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 44, in ranger setup_usersync(upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 319, in setup_usersync password_validation(params.ranger_usersync_ldap_ldapbindpassword) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 249, in password_validation raise Fail("Blank password is not allowed for Bind user. Please enter valid password.") resource_management.core.exceptions.Fail: Blank password is not allowed for Bind user. Please enter valid password. ROOT CAUSE : Currently 'Bind Anonymous' option is not supported in Ranger. Hortonworks Internal BUG-68578 has be

Consider my cluster is having SSO configured and my SSO certificate contains character larger than 2048 and i am upgrading to ambari-2.7.0 using the command : ambari-server upgrade There is a high chance your upgrade will get stuck at schema upgrade phase and in the ambari-server log you will find the below exception : Internal Exception: java.sql.BatchUpdateException: Batch entry 2 INSERT INTO ambari_configuration (property_name, category_name, property_value) VALUES ('ambari.sso.provider.certificate','sso-configuration',<LONG VERTIFICATE VALUE>') was aborted: ERROR: value too long for type character varying(2048) Call getNextException to see other errors in the batch. Root Cause : While upgrade to ambari-2.7.x + versions ambari will move the sensitive data from ambari.properties to ambari database table : ambari_configuration . if any of this sensitive data contains more than 2048 characters it will fail. usually certificates issues wont be more than 2048 characters but

Disclaimer:This article is based on my personal experience and knowledge.Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case. Always contact Hortonworks support if its production cluster Issue Description: Ambari Showing Wrong NIFI Version After Upgrading to HDF 3.3 , as per release notes , the NIFI version in HDF-3.3 is 1.8 but ambari shows it as 1.7 in Stacks and Versions page. Root cause : This is a known bug in hdf-ambari-mpack-3.3.0.0-165 ( the mpack used by ambari to manage HDF-3.3 ) This issue is fixedin HDF-3.3.1 mpack. Solution : we can ignore the version shown in the stacks and services as its just ambari shows its wrong. we can verify the NIFI version in nifi ui or from hdf-select command in Host where nifi is installed. Workaround : Navigate to : /var/lib/ambari-server/resources/mpacks/hdf-ambari-mpack-3.3.0.0-165/hdp-addon-services/HDF/3.3/NIFI/1.7.0 Edit the File : metainfo.xml Change t

How to setup a cluster in AWS? What type of storage is supported for HDFS? EBS? EMR? EBS is supported and recommended mainly for mission critical, that is for data that must be (mostly) available. You can do ephemeral storage, which will be faster, but if the node goes down you won’t be able to restore that data and since AWS (and other cloud providers) are known to have entire regions disappear, you can and will lose your whole clusterEBS volumes will be available again when the region comes back online, ephemeral won’t. However EBS is also very pricy and you may not want to pay for that option.However another option is using ephemeral storage, but setting up backup routines to S3, so you can restore back to a point in time. (If you want you can use EBS and back up with S3). I guess the main reason EBS is not recommended for HDFS also is that it is very expensive, but it is supported.For HBASE workloads you should use i2. Only use d2 nodes for a storage density workload t

Currently there is no Atlas hook for HBase, HDFS, or Kafka. For these components, you must manually create entities in Atlas. You can then associate tags with these entities and control access using Ranger tag-based policies. On the Atlas web UI Search page, click the create new entity link at the top of the page.On the Create Entity pop-up, select an entity type.Enter the required information for the new entity. Click All to display both required and non-required information. Click Create to create the new entity.The entity is created and returned in search results for the applicable entity type. You can now associate tags with the new entity and control access to the entity with Ranger tag-based policies.The above example illustrates manually creating an entity for HDFS_PATH only. This can be extended to KAFKA and HBASE entities.

IoT Series: Sensors: Utilizing Breakout Garden Hat: Part 1 - IntroductionAn easy option for adding, removing and prototype sensor reads from a standard Raspberry Pi with no special wiring.Hardware Component List:Raspberry PiUSB Power CablePimoroni Breakout Garden Hat1.12" Mono OLED Breakout 128x128 White/Black ScreenBME680 Air Quality, Temperature, Pressure, Humidity SensorLWM303D 6D0F Motion Sensor (X, Y, Z Axes)BH1745 Luminance and Color SensorLTR-559 Light and Proximity Sensor 0.01 lux to 64,000 luxVL53L1X Time of Flight (TOF) Sensor Pew Pew Lasers!Software Component List:RaspianPython 2.7JDK 8 JavaApache NiFiMiniFiSource Code:https://github.com/tspannhw/minifi-breakoutgarden Shell Script (https://github.com/tspannhw/minifi-breakoutgarden/blob/master/runbrk.sh)Python (https://github.com/tspannhw/minifi-breakoutgarden/blob/master/brk.py)SummaryOur Raspberry Pi has a Breakout Garden Hat with 5 sensors and one small display. The display is showing the last reading and is constan

Problem Statement: Deploying and Governing ModelsMachine Learning and Artificial Intelligence are in the process of exploding in importance and prevalence in the enterprise. With this explosive growth comes fundamental challenges in governing model deployments ... and doing this at scale. These challenges revolve around answering the following fundamental questions: Which models were deployed when? and to where? Was this deployment to a microservice, a Spark context on Hadoop, or other?What was the serialized object deployed? How can I find it?What version was deployed? Who is the owner? What is the larger context around the projectHow do I know the details of the model, ie. how do I trace the model in Production to its actual code, its training data, owner, etc?Previous article: Why and how you should use Atlas to govern your models Article: Customizing Atlas (Part1): Model governance, traceability and registryIn the previous article I showed how Atlas is a powerful and natural f

Sometimes it is desired to see what kind of Http request ambari server makes to the outside works for troubleshooting purpose and to know the value of various http headers (like: User-Agent, Accept, Last-Modified, Content-Type, Content-Length, Connection, ETag, Server, X-Cache ...etc) which ambari-server uses to make calls to Ambari Public repo ...etc.So in order to achieve the Http Logging of ambari server made http requests we can do the following:Step-1). Create a file "/etc/ambari-server/conf/http.logging.properties" with the following content:.level=INFOhandlers=java.util.logging.ConsoleHandlerjava.util.logging.ConsoleHandler.level=ALLsun.net.www.protocol.level=ALLsun.net.www.protocol.http.HttpURLConnection.level=ALLdebug.level=ALL.Step-2). Inside the "/var/lib/ambari-server/ambari-env.sh" file variable "AMBARI_JVM_ARGS" make sure to add the following parameter "-Djava.util.logging.config.file=/etc/ambari-server/conf/http.logging.properties" to enable the http logging.Example:exp