Resources

OSQueryOSQuery is a cool tool that lets you query your servers via SQL. It supports Windows, OSX and most Linux variants.Installing osquerywget https://osquery-packages.s3.amazonaws.com/centos7/osquery-2.2.1-1.el7.x86_64.rpmrpm -ivh osquery-2.2.1-1.el7.x86_64.rpmsudo cp /usr/share/osquery/osquery.example.conf/etc/osquery/osquery.confsudo service osqueryd startRedirecting to /bin/systemctl start osqueryd.servicesudo service osqueryd statusRedirecting to /bin/systemctl status osqueryd.service● osqueryd.service - The osquery Daemon Loaded: loaded(/usr/lib/systemd/system/osqueryd.service; disabled; vendor preset: disabled) Active: active(running) since Wed 2017-01-25 20:32:06 UTC; 4s ago Process: 21531ExecStartPre=/bin/sh -c if [ ! -f $FLAG_FILE ]; then touch $FLAG_FILE; fi(code=exited, status=0/SUCCESS) Main PID: 21534(osqueryd) CGroup:/system.slice/osqueryd.service ├─21534 /usr/bin/osqueryd --flagfile/etc/osquery/osquery.flags --config_path /etc/osquery/osquery.conf └─21537osqu

This is for people preparing to attend my talk on Deep Learning at DataWorks Summit Berling 2018 (https://dataworkssummit.com/berlin-2018/#agenda) on Thursday April 19, 2018 at 11:50AM Berlin time.This is for running Apache MXNet on a Raspberry Pi.Let's get this installed!git clone https://github.com/apache/incubator-mxnet.gitThe installation instructions at Apache MXNet's website (http://mxnet.incubator.apache.org/install/index.html) are amazing. Pick your platform and your style. I am doing this the simplest way on Linux path.Installation:This builds on previous builds, so see those articles. We installed the drivers for Sense Hat, Intel Movidius and the USB Web Cam previously. Please note that versions for Raspberry Pi, Apache MXNet, Python and other drivers are updated every few months so if you are reading this post DWS 2018 you should check the relevant libraries and update to the latest versions.You need Python, Python Devel and PIP installed and you may need to run as root.

Raspberry Pi Killer? Nope, but this device has twice the RAM and a bit more performance. It's mostly compatible with Pi, but not fully. It is very new and has little ecosystem but can get the job done.Device SetupIt is easy to install Python and all the libraries required for IoT and some deep learning. I found most instructions worked for Raspberry Pi on this device. It has more RAM which helps on some of these activities.I downloaded and burned with Etcher a MicroSD image of TinkerOS_Debian V1.8 (Beta version). It's a Debian variant close enough to Raspian for most IoT developers and users to be comfortable. An Android OS is now available for download as well and that may be worth trying, I am wondering if Google will add this device to the AndroidThings supported devices? Perhaps. One quirk, make sure you remember this: TinkerOS default username is “linaro”, password is “linaro”. Connect to the device via ssh linaro@SOMEIP. Python Setupsudo apt-get updatesudo apt-ge

Keeping track of where a cat is can be a tricky task. In this article, we'll design and prototype a smart IoT cat sensor which detects when a cat is in proximity. This sensor is meant to be part of a larger network of cat sensors covering a target space.Flow DesignSensor inputWe start by polling an image sensor (camera) for image data. We use the GetUSBCamera processor configured for the USB camera device attached to our sensor controller.On our system, we had to open up permissions on the USB camera device, otherwise MiNiFi's GetUSBCamera processor would record an access denied error in the logs:chown user /dev/bus/usb/001/014We then configure the sensor processor as such:Processors: - name: Get class: GetUSBCamera Properties: FPS: .5 Format: RAW USB Vendor ID: 0x045e USB Product ID: 0x0779 Machine Learning Inference on the EdgeWe use TensorFlow to perform class inference on the image data. We do this at the sensor rather than in a centralized system in orde

This article helps in estimating the total data change on hdfs for a given time period and what will be replicated by Data Lifecycle Manager (DLM)Step 1) On each of the data nodes, capture the jmx response by simply retrieving a json file from the node. Example something like this,curl “http://<datanodeip/fqdn>:<port>/jmx”(you can check the port in Ambari for the property - dfs.datanode.http.address)Step 2) In the json response, look for the value in the field “BytesWritten” which will be under a subsection, name" : "Hadoop:service=DataNode,name=DataNodeActivity-name of your datanode-50010". There should only be one entry/value for this.Step 3) The above value reflects the actual data change value (additions, deletions, modifications). Essentially, if you add and drop a hdfs file, the value won’t reduce and will reflect the actual data change rate.Step 4) Summing up these values for all the data nodes will give the overall data change in the entire system. Step 5) Finally,

This is a basic Ambari blueprint for clusters that implement high availability for HDFS, Yarn and Hive for HDP 2.6. This includes Namenode HA, Resourcemanager HA, Hive Metastore server HA and Hive Server HA. It does not implement a high-availability database for the Hive metastore, though, as that is done within the relational database backed itself. It has been developed and tested using Cloudbreak, but should work with Ambari as well.Hostgroup Layout Raw Blueprint JSON { "Blueprints": { "blueprint_name": "cb24-hdp26-ha", "stack_name": "HDP", "stack_version": "2.6" }, "settings": [ { "recovery_settings": [] }, { "service_settings": [ { "name": "HIVE", "credential_store_enabled": "false" } ] }, { "component_settings": [] } ], "host_groups": [ { "name": "master_mgmt", "components": [ { "name": "METRICS_COLLECTOR" }, { "name": "METRICS_GRAFANA"

This is for people preparing to attend my talk on Deep Learning at DataWorks Summit Berling 2018 (https://dataworkssummit.com/berlin-2018/#agenda) on Thursday April 19, 2018 at 11:50AM Berlin time.Another way to work with Apache MXNet is by using your Apache Zeppelin notebook to run your Python deep learning scripts.Apache Zeppelin NotebookAs you can see we can format that data as a table using Apache Zeppelin display technology.Use this print statement:print("%table top1pct\ttop1\top2\ttop2pct\ttop3pct\ttop3\ttop4pct\ttop4\ttop5pct\ttop5\timagefilename\truntime\tuuid\n" + top1pct + "\t" + top1 + "\t" + top2pct + "\t" + top2 + "\t" + top3pct + "\t" + top3 + "\t" + top4pct + "\t" + top4 + "\t" + top5pct + "\t" + top5 + "\t" + filename + "\t" + str(round(end - start)) + "\t" + uniqueid + "\n" )We use the pyspark interpreter to run this Python script, but there's no Spark in here yet.This data also gets loaded in Apache Hive via Apache NiFi as shown here: Deep Learning ModelsYou w

This is for people preparing to attend my talk on Deep Learning at DataWorks Summit Berling 2018 (https://dataworkssummit.com/berlin-2018/#agenda) on Thursday April 19, 2018 at 11:50AM Berlin time.Many people are using IBM's excellent DSX platform which uses Jupyter Notebooks and the ever popular Kubernetes.I wanted to try out Apache MXNet in this environment.It's great.Create or reuse an existing notebook. For Python, the default is Jupyter. Zeppelin is now also supported. I am using Python 2.7 with DSX Desktop on an OSX workstation.This supports Apache MXNet.My local Apache MXNet installation and MXNet python installation worked well with DSX. I needed OpenCV for this example, so I was able to install right inside IBM DSX via !pip install --user opencv-python.Very easy to start a notebook and add your code, you get nice syntax coloring.I uploaded the precompiled modelHere we can check our list of Python libraries with !pip list --isolated --format=columns. Very easy to run y

Hortonworks Data Flow v3.1.1 Documentation (link)Thank you to the Hortonworks Data Flow Development, Product Management, Quality Engineering, Partner Certification, Documentation, and Release Engineering teams.

Building Apache Tez with Apache Hadoop 2.8.0 or later fails due to client/server jar separation in Hadoop [1]. The build fails with the following error:[ERROR] COMPILATION ERROR :[INFO] -------------------------------------------------------------[ERROR] /src/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[48,30] cannot find symbol symbol: class DistributedFileSystem location: package org.apache.hadoop.hdfs[ERROR] /src/tez/tez-api/src/test/java/org/apache/tez/client/TestTezClientUtils.java:[680,50] cannot find symbol symbol: class DistributedFileSystem location: class org.apache.tez.client.TestTezClientUtils[ERROR] /src/tez/ecosystem/tez/tez-api/src/test/java/org/apache/tez/common/TestTezCommonUtils.java:[62,42] cannot access org.apache.hadoop.hdfs.DistributedFileSystemTo get Tez to compile successfully, you will need to use the new hadoop28 profile introduced by TEZ-3690 [2]. E.g. here is how you compile Tez with Apache Hadoop 3.0.0:mvn clean package -

I find those who install NiFi via ambari using local repos are generally required for security proposes to call out ports to be opened. Typical on cloud environments. I plan update this list this with community feedback to keep this list fresh. These ports are not set in stone. NiFi ports are configure by simply changing the port in the properties file. Lets get to it: Ambari 8080Zookeeper 2181Protocol port9088HTTP port (ssl) 9091HTTP port (non-ssl) 9090Certificate Authority10443nifi.remote.input.socket.port 8022 nifi.cluster.node.protocol.port 8021 Remote Process Group Raw 8022HTTP 8070nifi.remote.input.socket.port 9999

This is for people preparing to attend my talk on Deep Learning at DataWorks Summit Berling 2018 (https://dataworkssummit.com/berlin-2018/#agenda) on Thursday April 19, 2018 at 11:50AM Berlin time.See: https://community.hortonworks.com/content/kbentry/174399/apache-deep-learning-101-using-apache-mxnet-on-apa.htmlTo do proper analytics and provide fast SQL access to our inception data generated by Apache MXNet from our images, we need to land it into Apache Hive Transactional tables. We will use the Apache NiFi PutHiveStreaming processor to insert data into our ACID table at a rapid rate. This only works if you create a transactional table with Apache ORC, see the DDL below. You must also be running a new version of HDP 2.6+ that has ACID turned on.Tip: In HDP 2.6.4, you will need to create and work with Apache Hive ACID tables with Hive. Not sql in Apache Zeppelin, since that is Apache Spark. jdbc(hive) is Apache Hive. See the configuration below to hive CBO and TEZ ena

Cloudbreak 2.4.0 is now live!What is Cloudbreak: Cloudbreak simplifies cluster provisioning on public cloud infrastructure platforms Microsoft Azure, Amazon Web Services (AWS), and Google Cloud Platform (GCP), and on the private cloud infrastructure platform OpenStack.Here are the highlights of what’s new in Cloudbreak 2.4.0 compared with Cloudbreak 1.16.x: New UI/UXCloudbreak 2.4.0 introduces a new user interface. All major options are now easily accessible from the collapsible navigation menu. All UI options and wizards have been redesigned in order to make cluster creation and management more intuitive. The following screenshot shows the cluster dashboard:The following screenshot shows cluster details page:> Note that autoscaling is available from the Autoscaling tab.> Note the Show CLI Command available from Actions menu. This allows you to generate a CLI template of the cluster. A similar option is available on the last page of the create cluster wizard.The following screens

Alfredo Sauce - Hadoop HTTP, Kerberos and SPNEGOKerberos SPNEGO authentication for HTTP has been part of hadoop for some time now. On secure cluster many services use it to authenticate HTTP APIs and WEB UIs.Setup and configuration can become a challenge as it involves many aspects, including: kerberos principals, keytabs, network and load balancers, remote users accessing via different browsers, different operative systems, etc.In this article I will share how Kerberos SPNEGO authentication for HTTP works in hadoop.IntroductionKerberos SPNEGO authentication for HTTP was introduced to hadoop via HADOOP-7119. The implementation is based on a servlet filter that is configured to front all incoming HTTP requests to the application. If not valid hadoop.auth cookie is found, the servlet filter calls the KerberosAuthenticationHandler to perform kerberos authentication for the UserAgent request. Upon a successful kerberos authentication, servlet filter adds a signed cookie to the response so

This is for people preparing to attend my talk on Deep Learning at DataWorks Summit Berling 2018 (https://dataworkssummit.com/berlin-2018/#agenda) on Thursday April 19, 2018 at 11:50AM Berlin time.Motivation: Use the Power of Our Massive HDP Hadoop Cluster and YARN to Run My Apache MXNetFor a real-world Deep Learning problem, you would want to have some GPUs in your cluster and in the near term YARN 3.0 will allow you to manage those GPU resources. For now, install a few GPUs in special HDP cluster just for training Data Science jobs. You can make this cluster compute and RAM heavy for use by TensorFlow, Apache MXNet, Apache Spark and other YARN workloads.In my example, we are running inception against an image, this could be from a security camera, a drone or other industrial purposes.We will dive deeper into industrial IIOT use cases here:If you are in Philadelphia, please join me. If not, all of the content will be shared on slideshare, github and here.To set this up, we wil

If you'd like to generate some data to test out the HDP/HDF platforms at a larger scale, you can use the following GitHub repository:https://github.com/anarasimham/data-genThis will allow you to generate two types of data:Point-of-sale (POS) transactions, containing data such as transaction amount, time stamp, store ID, employee ID, part SKU, and quantity of product. These are transactions you make at a store when you are checking out. For simplicity's sake, this assumes each shopper only buys one product (potentially greater than 1 in quantity)Automotive manufacturing parts production records, simulating the completion of parts in an assembly line. Imagine a warehouse completing different components of a car, such as the hood, front bumper, etc. at different points in time and those parts being tested for heat and vibration thresholds. This data will contain a timestamp of when the part was produced, thresholds for heat & vibration, values as tested for heat & vibration, quani

resource_management.core.exceptions.ExecutionFailed: Execution of '/usr/bin/yum -d 0 -e 0 -y install mysql-community-release' returned 1. Error: Nothing to doAmbari got the error above.Try install mysql directly:[centos@zzeng-hdp-2 ~]$ sudo yum install mysql-community-releaseLoaded plugins: fastestmirrorLoading mirror speeds from cached hostfile * base: mirror.web-ster.com * extras: mirror.hostduplex.com * updates: mirrors.sonic.netNo package mysql-community-release available.Error: Nothing to dohttps://bugs.mysql.com/bug.php?id=81037Try MySQL 5.7[centos@zzeng-hdp-2 ~]$ sudo rpm -ivh http://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpmRetrieving http://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpmwarning: /var/tmp/rpm-tmp.hGWJu5: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEYPreparing... ################################# [100%]Updating / installing... 1:mysql57-community-release-el7-8 ################################# [100

How HDFS Apply Ranger Policies Apache Ranger. Apache Ranger™ is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform. The vision with Ranger is to provide comprehensive security across the Apache Hadoop ecosystem. With the advent of Apache YARN, the Hadoop platform can now support a true data lake architecture. Enterprises can potentially run multiple workloads, in a multi tenant environment. Data security within Hadoop needs to evolve to support multiple use cases for data access, while also providing a framework for central administration of security policies and monitoring of user access.Ranger Goals OverviewApache Ranger has the following goals:Centralized security administration to manage all security related tasks in a central UI or using REST APIs.Fine grained authorization to do a specific action and/or operation with Hadoop component/tool and managed through a central administration toolStandardize authorization method across all H

Moving to more advanced application of our basic monitoring flow. Picking up where we left off in the first part of this series: Monitor Temperature & Humidity Sensors with Apache NiFiNow that we have the ability to collect the sensor data using NiFi, we will want to be able to store and process the data using our HDP cluster. The resources at the sensor level would not be enough to provide deep analytics or long term storage. For this reason we can leverage NiFi's site-2-site protocol to send the sensor events directly to our HDF cluster for processing. Start of by adding a remote process group to your NiFi flow at the sensor end and feed the output of ExecuteProcess into this remote processor. Supply the URL of your HDF NiFi instance (same as you would type into browser to get to the design gui).Now move to your remote instance of NiFi on your HDF cluster and create a new input port. replace the execute process processor with the input port. Now when you start the processors on b

hive.aux.jars.path doesn't work if you set it as a Custom property on Ambari Hive configs like hive.aux.jars.path.Please set the property on command line using Hive CLI and beeline. Please specify the jar files on hive-env.sh file from Ambari UI. Example : hive-env tab: HIVE_AUX_JARS_PATH=/usr/hdp/lib/elasticsearch-hadoop-2.2.0-rc1.jarand restart Hive services.

Note - I did receive few complaints about this script that it's failing on new Ambari versions, It happened because of change in kerberos payload. I have modified the script and it works like a charm again for new versions. Please continue to provide me the feedback! 🙂Disclaimer - "Please do not use unless you know what u r doing! Test on Dev. cluster before using on production systems" 🙂Automated Kerberos Installation and Configuration – For this post, I have written a shell script which uses Ambari APIs to configure Kerberos on HDP Single or Multinode clusters. You just need to clone our github repository and modify property file according to your cluster environment, execute setup script and phew!! Within 5-10 minutes you should have your cluster completely secured by Kerberos! Cool isn’t it? Detailed Steps(Demo on HDP Sandbox 2.4):1. Clone our github repository Ambari server in your HDP ClusterNote - This script will install and configure KDC on your Ambari Server.git clone https

Use commodity hardware (Arduino Boards) to monitor temperature and humidity with NiFi.get the source to follow along on git: REPOIn this tutorial we will need some hardware: Arduino BoardDHT11 Sensor (DHT22 will work as well)Jumper wiresIn Arduino IDE select your board and port and enter the following code: #include "DHT.h"#define DHTPIN 2 // what digital pin we're connected to// Uncomment whatever type you're using!#define DHTTYPE DHT11 // DHT 11//#define DHTTYPE DHT22 // DHT 22 (AM2302), AM2321//#define DHTTYPE DHT21 // DHT 21 (AM2301)// Connect pin 1 (on the left) of the sensor to +5V// NOTE: If using a board with 3.3V logic like an Arduino Due connect pin 1// to 3.3V instead of 5V!// Connect pin 2 of the sensor to whatever your DHTPIN is// Connect pin 4 (on the right) of the sensor to GROUND// Connect a 10K resistor from pin 2 (data) to pin 1 (power) of the sensor// Initialize DHT sensor.// Note that older versions of this library took an optional third parameter to// tw

How HiveServer2 Apply Ranger Policies Apache Ranger. Apache Ranger™ is a framework to enable, monitor and manage comprehensive data security across the Hadoop platform. The vision with Ranger is to provide comprehensive security across the Apache Hadoop ecosystem. With the advent of Apache YARN, the Hadoop platform can now support a true data lake architecture. Enterprises can potentially run multiple workloads, in a multi tenant environment. Data security within Hadoop needs to evolve to support multiple use cases for data access, while also providing a framework for central administration of security policies and monitoring of user access.Ranger Goals OverviewApache Ranger has the following goals: Centralized security administration to manage all security related tasks in a central UI or using REST APIs.Fine grained authorization to do a specific action and/or operation with Hadoop component/tool and managed through a central administration toolStandardize authorization method acr

Version: HDF 3.1, HDP 2.6.4, PySpark 2.2.0, Python 2.7, Apache NiFi 1.5.I push my power data from a local Apache NiFi 1.5 server over Site-to-Site HTTP to a cloud hosted HDF 3.1 cluster. This cluster has a Remote Input that passes the information on to a version controlled Process Group called "Spark-Kafka-Streaming". Once inside, I set a schema name and data type then push the data to Kafka 1.0 hosted in HDF 3.1. The PublishKafkaRecord_1.0 settings are super easy. We use the JsonTreeReader and the supplied schema to read the JSON file into records. I chose to use the JsonRecordSetWriter to push JSON out. I could have easily done Apache Avro or CSV or another format. I chose JSON as it is easy to work with in Apache Spark and good for debug display.This method and code will work for several versions forward, but I cannot confirm for previous versions.This article is how to connect Apache NiFi with Apache Spark via Kafka using Spark Streaming. The example code is in

User and Groups significance in HDFSBefore we even start lets take a look back on how user and groups are handled in Linux. The key take away from the previous articles werelets Look at the various relationship that exists1. Every group has a group id.2. Every user has a user id3. In linux its not possible to have user without a group id.(by default when a user is created , it has a group with same name)4. A user can have one primary group and multiple secondary groups.5. A group can have multiple users.6. Authentication is done based on username and password.7. Authorization is done based on groups as unix follow POSIX permission for user : group : others8.A user cannot exist without a group.9. A group can exist without a user.10. A file can only have usernames and groups which are part of the Linux OS (Local or Remote service)11. A file ownership can never be changed to a non existent user ( Create a file and try chown XXXXXX fileName ).12. Linux is applying authorization policy