Resources

Many a time, there are instances where one wants to deploy the Atlas UI changes (like static htmls, images, css and js files) which are built locally while remotely connecting to an Atlas server backend. This way, you don't really have to install Atlas (and other dependent services) on your local machineIn order to achieve this, one can set up a local grunt server with the below-mentioned stepsInstall npm (npm is a node package manager, used for nodejs projects)Checkout the Atlas codebaseNavigate to <workspace_directory>/dashboardv2 folderTunnel (add Local Port Forwarding) to Atlas instance deployed on a remote cluster using the following commandssh -i <private_key> -L <local_port>:<cluster_host>:<cluster_port> <user_name>@<cluster_host>Once you execute the above command, all the requests being sent to http://localhost:21000 will be redirected to our remote Atlas instance running at http://<cluster_host>:<cluster_port>;Now hit, http

1. In your Ambari UI, click on the Manage Ambari button.2. Then on the left of the screen, click on views.3. Click on Create Instance on the top right of the screen.4. Fill in the boxes as shown below. For select view, choose FILES from the drop down. The blocked out field below is your personal S3 key. 5. Click Save6. Once saved, you can go back to the views UI and see your new S3 view, click the URL to get through to the files view of your S3 bucket. It will look something like this example below.7. At this point, you can manage all of your directories and files in S3 from this Ambari view. You can delete, add or move things around with ease.

What is CDC Change Data Capture (CDC) in Data Warehousing typically refers to the process of capturing changes over time in Dimension tables. Dimension tables qualify Fact tables (measures) by containing information to answer questions around Who (e.g. Customer) , What (e.g. product), Where (e.g. Location), and When (e.g. Time). This article steps will demonstrate how to implement a very basic and rudimentary solution to CDC in Hadoop using MySQL, Sqoop, Spark, and Hive. It includes basic PySpark code to get you started with using Spark Data Frames. In a real world example you would include audit tables to store information for each run.   How to do CDC in Hadoop In the past, I have worked at a company in which CDC on Hadoop was a big challenge for us. Initially, we were using Hive to merge source changes (Sqoop extracts) with the existing Dimension tables and then building new tables. This worked well until our data volumes starting getting in the hundreds of millions of records.

My Example Mesh NetworkArgon: Wi-Fi + Mesh Gateway/RepeaterThis is to connect to the outside network via WiFi. We connect to Particle Cloud.Espressif ESP32-D0WD 2.4G Wi-FiNordic Semiconductor nRF52840 SoC for bluetooth and NFC A tagARM TrustZone CryptoCell-310 Cryptographic and security module Thread/BLE, another for Wi-Fi)Xenon: Mesh + BLENordic Semiconductor nRF52840 SoC for bluetooth and NFC A tagARM TrustZone CryptoCell-310 Cryptographic and security moduleReferences https://github.com/Seeed-Studio/Grove_Starter_Kit_for_Photon_Demos?files=1https://docs.particle.io/datasheets/accessories/mesh-accessories/https://community.particle.io/c/meshhttps://www.particle.io/meshhttps://docs.particle.io/datasheets/mesh/xenon-datasheet/https://docs.particle.io/datasheets/wi-fi/argon-datasheet/https://blog.particle.io/2018/04/28/how-to-build-a-wireless-mesh-network/WiFi Mesh = https://www.threadgroup.org/https://github.com/Seeed-Studio/Grove_Starter_Kit_for_Photon_Demos/blob/master/Example%20-%

I came across an article on how to setup NiFi to write into ADLS which required cobbling together various integration pieces and launching HDI. Since then there have been many updates in NiFi enabling a much easier integration. Combined with CloudBreak's rapid deployment of a HDF clusters provides an incredible ease of user experience. ADLS is Azure's native cloud storage (Look and feel of HDFS) and the capability to read/write via NiFi is key. This article will demonstrate how use use a CloudBreak Recipe to rapidly deploy a HDF NiFI "ADLS Enabled" cluster.AssumptionsA CloudBreak instance is availableAzure Credentials availableModerate familiarity with AzureUsing HDF 3.2+From Azure you will need:ADLS urlApplication IDApplication PasswordDirectory IDNiFi requires ADLS jars, core-site.xml, and hdfs-site.xml. The recipe I built will fetch these resources for you. Simply download the recipe/script from:https://s3-us-west-2.amazonaws.com/sunileman1/scripts/setAdlsEnv.shOpen it and scroll

I came across an article on how to setup NiFi to write into ADLS which required users to cobble together various integration pieces and launching HDI. Since then there have been many updates in NiFi enabling a much easier integration. Combine with CloudBreak's rapid deployment of a HDF cluster provides incredible ease of use. ADLS is native cloud storage provided by Azure (Look and feel of HDFS) and the capabilities to read/write via NiFi is key. This article will demonstrate how use use CloudBreak to rapidly deploy a HDF NiFI "ADLS Enabled" cluster.

Article This is the first of the two articles (see Part 2) where we will at look at Kubernetes (K8s) as the container orchestration and scheduling platform for a grid like architectural variation of the distributed pricing engine discussed here. Kubernetes @ Hortonworks But wait … before we dive in … why are we talking about K8s here? Well, if you missed KubeCon + CloudNativeCon, find out more! Hortonworks joined CNCF with an increased commitment to cloud native solutions for big data management. Learn more about the Open Hybrid Architecture Initiative led by Hortonworks to enable consistent data architectures on-prem and/or across cloud providers and stay tuned to discover more about how K8s is used across the portfolio of Hortonworks products – DPS, HDF and HDP to propel big data’s cloud native journey! The Application Architecture With that said, this prototype is a stateless compute application – a variation of the Dockerized Spark on YARN version described here

Problem:Of late, there are many HCC questions on how to create hive table and lineage using REST APIs in Atlas. This article will be act a step by step guide to create hive tables and lineage using REST API.Solution:As part of the solution to this FAQ, I will create two hive tables and lineage(CTAS) between them. I have tested these changes on HDP-2.5 release, so make sure you have HDP version >= 2.5.Step1: JSON for creating table1:[{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"-11893021824425525", "version":0, "typeName":"hive_db", "state":"ACTIVE" }, "typeName":"hive_db", "values":{ "name":"default", "location":"hdfs://mycluster/apps/hive/warehouse", "description":"Default Hive database", "ownerType":2, "qualifiedName":"default@cl1", "owner":"public", "clusterName":"cl1", "parameters":{ } }, "traitNames":[

Deep Speech with Apache NiFi 1.8 Tools: Python 3.6, PyAudio, TensorFlow, Deep Speech, Shell, Apache NiFi Why: Speech-to-Text Use Case: Voice control and recognition. Series: Holiday Use Case: Turn on Holiday Lights and Music on command.Cool Factor: Ever want to run a query on Live Ingested Voice Commands? Other Options: https://community.hortonworks.com/articles/155519/voice-controlled-data-flows-with-google-aiy-voice.html We are using Python 3.6 to write some code around pyaudio, tensorflow and Deep Speech to capture audio, store it in a wave file and then process it with Deep Speech to extract some text. This example is running in OSX without a GPU on Tensorflow v1.11. The Mozilla Github repo for their Deep Speech implementation has nice getting started information that I used to integrate our flow with Apache NiFi. Installation as per https://github.com/mozilla/DeepSpeech pip3 install deepspeechwget -O - https://github.com/mozilla/DeepSpeech/releases/downloa

Introduction Cloudbreak documentation is fairly thorough and can be found on Hortonworks' documentation portal (e.g. here). However, if you are as lazy as I am, you'll appreciate the few quick examples I'm listing in this article. First and foremost, you can access the documentation of available APIs on your CB instance by using the following URL https://[YOUR_CB_URL]/cb/apidocs; you should see something like this: Authenticating Authentication in cloudbreak is OAuth 2, and getting a detailed in the Cloudbreak API. Here is simple curl call to get a token: curl -k -iX POST -H "accept: application/x-www-form-urlencoded" -d 'credentials={"username":"[YOUR_USER]","password":"[YOUR_PWD]"}' "https://[YOUR_CB_URL]/identity/oauth/authorize?response_type=token&client_id=cloudbreak_shell&scope.0=openid&source=login&redirect_uri=http://cloudbreak.shell" | grep location | cut -d'=' -f 3 | cut -d'&' -f 1 GET Example: Listing all blueprints The biggest

When an ambari agent starts, it bootstraps with the ambari server via registration. The server sends information to the agent about the components that have been enabled for auto start along with the other auto start properties in ambari.properties. The agent compares the current state of these components against the desired state, to determine if these components are to be installed, started, restarted or stopped.These are the values ambari-server will send to ambari-agent by default unless configured in cluster-env.xml file. "recovery_max_count": "6","recovery_lifetime_max_count": "1024","recovery_type": "AUTO_START","recovery_window_in_minutes": "60",recovery_lifetime_max_count ---- The maximum number of recovery attempts of a failed component during the lifetime of an Ambari Agent instance. This is reset when the Ambari Agent is restarted. recovery_window_in_minutes -- The length of a recovery window, in minutes, in which recovery attempts can be retried. recovery_max_count --- The

Phoenix secondary indexPhoenix secondary Indexes are useful for point lookups or scans directed against non primary key of Phoenix or non row key columns of HBase. This saves the “full scan” of data table you would otherwise do if you intend to retrieve data based non rowkey.You create secondary indexes by choosing existing non primary key column from data table and making it as primary or a covered column. By covered column, we mean making exact copy of the covered column’s data from data table to index table.Types of secondary Index:Functional Index: Built on functions rather than just columnsGlobal secondary Index: This is the one where we make a exact copy of covered columns and call it index table. In simple terms, its an upsert select on all chosen columns from data table to Index table. Since a lot of write is involved during initial stages of index creation, this type of index would work best for read heavy use cases where data is written only occasionally and read more frequen

Apache Spark 2.2.0 with Scala 2.11.8 with Java 1.8.0_112 on HDP 2.6.4 called from HDF 3.1 with Apache NiFi 1.5.This is a follow up to: https://community.hortonworks.com/articles/173818/hdp-264-hdf-31-apache-spark-streaming-integration.html and https://community.hortonworks.com/articles/155326/monitoring-energy-usage-utilizing-apache-nifi-pyth.html We are using the same Apache NiFi flow to send messages to Apache Kafka. What is nice you could have the Structured Streaming version, non-structured version and others listening to the same Topic and same messages sent by Apache NiFi.When we start, no data yet.We quickly get a ton of dataBy Default A Kafka Cluster is 3 Nodes. Replication Factor of 3 is good then. I have one node. I had to change this. Tons of warnings in the /usr/hdf/current/kafka-broker/logs directory.The simplest Apache Spark client is one run in the shell:/usr/hdp/current/spark2-client/bin/spark-shell --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.0,o

AbstractAdding known clarification tags during data Ingestion using Atlas and NiFi.IntroductionMost of the times we know some metadata about the data we are ingesting into bigdata platform and this metadata could play an important role in data governance and security as the data moves through downstream pipeline, metadata like terms and conditions, country, consent agreements, validity period of the data etc, these are very useful defining GDRP like requirements and if we don’t add these classification during ingestion we loose the context, as you all know Apache Atlas is used as open metadata governance tool defining these classifications and tags etc, one way is to use NiFi to ingest the data which captures the data lineage in Atlas , but you have to add additional classification tags manually based on the nature of the data at later point of time in the process which is error prone . ProblemIf you do not add metadata classifications automatically part of the data ingestion you wi

Apache NiFi Processor for Apache MXNet SSD: Single Shot MultiBox Object Detector (Deep Learning)The news is out, Apache MXNet has added a Java API. So as soon as I could I got my hands on the maven repo and an example program and got to work writing a new Apache NiFi processor for it.I have run this on standalone Apache NiFi 1.8.0 and on HDF 3.3 - Apache NiFi 1.8.0 and both work. So anyone who wants to be an alpha tester, please download it and give it a try.Apache MXNet SSD is a good example of a pretrained deep learning model that works pretty well for general images in a use cases especially around people and cars. You can fine-tune this with some more images and runs: https://mxnet.incubator.apache.org/faq/finetune.htmlThe nice thing is now we can start including Apache MXNet as part of Java applications such as Kafka Streams, Apache Storm, Apache Spark, Spring Boot and other use cases using Java. I could potentially inject this into a Hive UDF (https://community.hortonwor

IntroductionPerformance of a cluster running on Hadoop can be impacted by the OS partitioning. This document is intended to understand the best practices to setup the “/var” folder/partition with optimum size.Lets try to approach this problem by asking some important questions. What is “/var” used for? How can the “/var” folder run out of disk space? Common issue to expect on a Hadoop cluster if “/var” is out of disk space. How is the current setup of “/var” in my cluster ?Question 1 - What is /var used for?From OS perspective, “/var” is commonly used for constantly changing files i.e. variable. The short form of which is “var”.Example of such files could be the log file, mail, transient file, the printer spool, temporary files, cached data, etc.For example - “/var/tmp” holds the temporary files between system reboots.On any node (Hadoop or non-Hadoop), /var directory holds content for a number of applications. It also is used to store downloaded update packages

The latest Docker version available form official Centos and RedHat repository has an issue preventing to exec the container.Issue when starting Cloudbreak is the following:Starting Cloudbreakgenerating docker-compose.ymlgenerating uaa.ymlInitialize and migrate databasesCreating cbreak_commondb_1 ...Creating cbreak_commondb_1 ... done[ERROR] The database hasn't started within 30 seconds.Docker version: docker-1.13.1-84.git07f3374.el7.centos.x86_64There are the following solutions, you can choose either:Downgrade to an older version:yum downgrade docker-1.13.1-75.git8633870.el7.centos.x86_64 docker-client-1.13.1-75.git8633870.el7.centos.x86_64 docker-common-1.13.1-75.git8633870.el7.centos.x86_64Upgrade to a newer version, using Docker repository:https://docs.docker.com/install/linux/docker-ce/centos/#set-up-the-repositoryAfter choosing either option docker restart should work.

Issue:Trying to run "msck repair table <tablename>" gives the below error Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. nullResolution:The above error occurs when hive.mv.files.thread=0, increasing the value of the parameter to 15 fixes the issueThis is a known bug

Decrypting OpenSSL Data in Apache NiFi Who Am I? My name is Andy LoPresto and I am a software engineer on the HDF team at Hortonworks. I previously worked in information security (encryption in financial systems & device drivers, PKI, etc.) at Apple and was a security architect at TigerText. The Issue A question I've been asked many times over my career thus far is, "When should we encrypt something?" Now, the most correct first answer to any information security question is, "It depends," but people get tired of hearing that very quickly. So my default response to this question is, "Whenever you don't have a good reason not to." Using encryption in high-level languages is usually straightforward in theory at this point, thanks to many helpful libraries such as BouncyCastle, but can be more difficult in practice. Implementation details matter. Unless you have a Ph.D. in cryptography or discrete mathematics and an extensive background in software engineering (in which case y

This article has been set on a HDP 2.5.3 version, you may consider adjusting some parameters to reflect your actual version.We'll here set Kafka loglevel through the Logging MBean with jConsole.For that, the first step is to enable JMX access:add in Kafka configs/kafka-env templateexport KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false"export JMX_PORT="9999" For avoiding JMX port conflicts like mentioned in https://community.hortonworks.com/articles/73750/kafka-jmx-tool-is-failing-with-port-already-in-use.html, let’s modify /usr/hdp/current/kafka/bin/kafka-run-class.sh on all broker nodes: replace# JMX port to useif [ $JMX_PORT ]; then KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"fiwith# JMX port to useif [ $ISKAFKASERVER = "true" ]; then JMX_REMOTE_PORT=$JMX_PORTelse JMX_REMOTE_PORT=$CLIENT_JMX_PORTfiif [ $JMX_REMO

Linkedin PostPresto is a tool designed to efficiently query vast amounts of data using distributed queries.We will be installing Presto in single server mode, Access Hive and then add worker node.Cross query - RBDMS, Hive, NoSqlTutorial**Java 8 must **Install - link (for the latest versions)wget https://repo1.maven.org/maven2/com/facebook/presto/presto-server/0.122/presto-server-0.122.tar.gztar xvfz presto-server-0.122.tar.gzLet's start with Single node setup (master and worker on the same node)cd presto-server-0.122mkdir etc[root@ns2 presto-server-0.122]# cd etc/mkdir catalog and we will create 3 files as shown below[root@ns2 etc]# lscatalog config.properties jvm.config log.properties node.properties[root@ns etc]# cat config.propertiescoordinator=truenode-scheduler.include-coordinator=truehttp-server.http.port=9080query.max-memory=10GBquery.max-memory-per-node=1GBdiscovery-server.enabled=truediscovery.uri=http://ns2:9080[root@ns2 etc]# cat log.propertiescom.facebook.presto=INFO[roo

Steps: When configuring a Kafka broker to use only SSL, you can have authentication and encryption by enabling 2-ways SSL by using parameter ssl.client.auth=required.Go to Ambari > Kafka > configs > custom Kafka-broker, add ssl.client.auth=required parameter.In this case certificate owner name (subject) works as username & authentication happens with this username. You have to add server certificate owner name from each broker as superuser (usernames separated by semicolon) so that each broker can access resources from other brokers. Example: In the case of two brokers you need to add two server certificate owner names to superuser.Go to Ambari > kafka > configs > custom kafka-broker, configure super.users parametersuper.users = User:CN=user1,OU=kafka,O=kafka,L=kafka,ST=kafka,C=xx;User:CN=user2,OU=kafka,O=kafka,L=kafka,ST=kafka,C=xxYou can create different client certificates for different users, if client certificate owner name is "CN=kafka,OU=kafka,O=kafka,L=k

Hi everyone! Is there a way where I can download all the videos on the Cloudera OnDemand app so that I can view it while commuting, instead having to download 1 by 1? Thanks!

MiniFi Java Agent 0.5Copy over necessary NARs from Apache NiFi 1.7 lib: nifi-ssl-context-service-nar-1.7.0.narnifi-standard-services-api-nar-1.7.0.narnifi-kafka-1-0-nar-1.7.0.narThis will support PublishKafka_1_0 and ConsumeKafka_1_0.Then create a consume and/or publish flow. You can combine the two based on your needs. In my simple example I consume the Kafka messages in MiniFi and write to a file. I also write the metadata to a JSON file.Consume Kafka Publish Electric Monitoring Data To KafkaLet's monitor the messages going through our topic, smartPlug.Publish Messages to KafkaConsume Any Messages From the smartPlug topicLogsProvenance Event file containing 377 records. In the past 5 minutes, 1512 events have been written to the Provenance Repository, totaling 839.32 KB2018-11-26 19:42:32,473 INFO [main] o.a.n.c.s.StandardProcessScheduler Starting PutFile[id=25a86505-031a-37d9-0000-000000000000]2018-11-26 19:42:32,474 INFO [main] o.a.n.c.s.StandardProcessScheduler Starting Update

HBase along with Phoenix is one of the most powerful NoSQL combinations. HBase/Phoenix capabilities allow users to host OLTPish workloads natively on Hadoop using HBase/Phoenix with all the goodness of HA and analytic benefits on a single platform (Ie Spark-hbase connector or Phoenix Hive storage handler). Often a requirement for HA implementations is a need for DR environment. Here I will describe a few common patterns and in no way is this the exhaustive HBase DR patterns. In my opinion, pattern 5 is the simplest to implement and provides operational ease & efficiency. Here are some of the high level replication and availability strategies with HBase/Phoenix HBASE provides High Availability within a cluster by managing region server failures transparently. HBASE provides various cross DC asynchronous replication schemes Master/Master replication topology Two clusters replicating all edits, bi-directionally to each other Master/Slave topology replic