Community Articles

in the documentation, you can find on the Sqoop partYou can use Sqoop to import data into HDFS or directly into Hive. However, Sqoop can only import data into Hive as a text file or as a SequenceFile. To use the ORC file format, you must use a two-phase approach: first use Sqoop to move the data into HDFS, and then use Hive to convert the data into the ORC file format[...]However, we can use the Sqoop-HCatalog integration feature, which is a table abstraction.Let's use the HDP 2.3.2 sandbox, starting with creating our Hive table, stored as ORC:[root@sandbox ~]# hive hive> CREATE TABLE cds (id int, artist string, album string) STORED AS ORCFILE; hive> INSERT INTO TABLE cds values (1,"The Shins","Port of Morrow"); hive> select * from cds; 1 The Shins Port of MorrowNow let's build our MySQL table[root@sandbox ~]# mysqlWelcome to the MySQL monitor. Commands end with ; or \g.Your MySQL connection id is 85Server version: 5.1.73 Source distributionCopyright (c) 2000, 2013, O

This Article will show how to read csv file which do not have header information as the first row. We will then specify the schema for both DataFrames and then join them together. import org.apache.spark.sql.types._val pathA = "hdfs:/tpc-ds/data/store_sales"val pathB = "hdfs:/tpc-ds/data/store/"// For Spark 2.x use -> val df = spark.read.option("header", true).csv(path)val A_df = sqlContext.read.format("com.databricks.spark.csv") .option("header","false") .option("inferSchema","false") .option("delimiter","|") .load(pathA)// Assign column names to the Store Sales dataframeval storeSalesDF = A_df.select( A_df("_c0").cast(IntegerType).as("SS_SOLD_DATE_SK"), A_df("_c1").cast(IntegerType).as("SS_SOLD_TIME_SK"), A_df("_c2").cast(IntegerType).as("SS_ITEM_SK"), A_df("_c7").cast(IntegerType).as("SS_STORE_SK") )val B_df = sqlContext.read.format("com.databricks.spark.csv") .option("header","false") .option("inferSchema","false") .option("delimiter","|") .load(pathB)// A

Disclaimer:This article is based on my personal experience and knowledge.Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case. Always contact Hortonworks support if its production cluster Problem Statement: I have installed HDF-3.1 and ambari-2.6.2.2 and i am upgrading my ambari to 2.7.0.0 to upgrade my HDF to 3.2+ versions and its failing with below exception : INFO: about to run command: /usr/java/jdk1.8.0_162/bin/java -cp '/etc/ambari-server/conf:/usr/lib/ambari-server/*:/usr/java/latest/mysql-connector-java.jar:/usr/share/java/mysql-connector-java.jar' org.apache.ambari.server.upgrade.SchemaUpgradeHelper > /var/log/ambari-server/ambari-server.out 2>&1INFO:process_pid=16599Traceback (most recent call last):File "/usr/sbin/ambari-server.py", line 1060, in <module>mainBody()File "/usr/sbin/ambari-server.py", line 1030, in mainBodymain(options, args, parser)File "/usr/sbin/ambari-server.py", line

PROBLEMWhen setting Bind Anonymous in Ambari's Ranger usersync tab it complains that ranger.usersync.ldap.binddn can not be empty in ambari usersync service logs, with the below stack trace :File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 44, in ranger setup_usersync(upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 319, in setup_usersync password_validation(params.ranger_usersync_ldap_ldapbindpassword) File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py", line 249, in password_validation raise Fail("Blank password is not allowed for Bind user. Please enter valid password.") resource_management.core.exceptions.Fail: Blank password is not allowed for Bind user. Please enter valid password. ROOT CAUSE : Currently 'Bind Anonymous' option is not supported in Ranger. Hortonworks Internal BUG-68578 has be

Consider my cluster is having SSO configured and my SSO certificate contains character larger than 2048 and i am upgrading to ambari-2.7.0 using the command : ambari-server upgrade There is a high chance your upgrade will get stuck at schema upgrade phase and in the ambari-server log you will find the below exception : Internal Exception: java.sql.BatchUpdateException: Batch entry 2 INSERT INTO ambari_configuration (property_name, category_name, property_value) VALUES ('ambari.sso.provider.certificate','sso-configuration',<LONG VERTIFICATE VALUE>') was aborted: ERROR: value too long for type character varying(2048) Call getNextException to see other errors in the batch. Root Cause : While upgrade to ambari-2.7.x + versions ambari will move the sensitive data from ambari.properties to ambari database table : ambari_configuration . if any of this sensitive data contains more than 2048 characters it will fail. usually certificates issues wont be more than 2048 characters but

Disclaimer:This article is based on my personal experience and knowledge.Don't take it as a standard guidelines, understand the concept and modify it for your environmental best practices and use case. Always contact Hortonworks support if its production cluster Issue Description: Ambari Showing Wrong NIFI Version After Upgrading to HDF 3.3 , as per release notes , the NIFI version in HDF-3.3 is 1.8 but ambari shows it as 1.7 in Stacks and Versions page. Root cause : This is a known bug in hdf-ambari-mpack-3.3.0.0-165 ( the mpack used by ambari to manage HDF-3.3 ) This issue is fixedin HDF-3.3.1 mpack. Solution : we can ignore the version shown in the stacks and services as its just ambari shows its wrong. we can verify the NIFI version in nifi ui or from hdf-select command in Host where nifi is installed. Workaround : Navigate to : /var/lib/ambari-server/resources/mpacks/hdf-ambari-mpack-3.3.0.0-165/hdp-addon-services/HDF/3.3/NIFI/1.7.0 Edit the File : metainfo.xml Change t

How to setup a cluster in AWS? What type of storage is supported for HDFS? EBS? EMR? EBS is supported and recommended mainly for mission critical, that is for data that must be (mostly) available. You can do ephemeral storage, which will be faster, but if the node goes down you won’t be able to restore that data and since AWS (and other cloud providers) are known to have entire regions disappear, you can and will lose your whole clusterEBS volumes will be available again when the region comes back online, ephemeral won’t. However EBS is also very pricy and you may not want to pay for that option.However another option is using ephemeral storage, but setting up backup routines to S3, so you can restore back to a point in time. (If you want you can use EBS and back up with S3). I guess the main reason EBS is not recommended for HDFS also is that it is very expensive, but it is supported.For HBASE workloads you should use i2. Only use d2 nodes for a storage density workload t

Currently there is no Atlas hook for HBase, HDFS, or Kafka. For these components, you must manually create entities in Atlas. You can then associate tags with these entities and control access using Ranger tag-based policies. On the Atlas web UI Search page, click the create new entity link at the top of the page.On the Create Entity pop-up, select an entity type.Enter the required information for the new entity. Click All to display both required and non-required information. Click Create to create the new entity.The entity is created and returned in search results for the applicable entity type. You can now associate tags with the new entity and control access to the entity with Ranger tag-based policies.The above example illustrates manually creating an entity for HDFS_PATH only. This can be extended to KAFKA and HBASE entities.

IoT Series: Sensors: Utilizing Breakout Garden Hat: Part 1 - IntroductionAn easy option for adding, removing and prototype sensor reads from a standard Raspberry Pi with no special wiring.Hardware Component List:Raspberry PiUSB Power CablePimoroni Breakout Garden Hat1.12" Mono OLED Breakout 128x128 White/Black ScreenBME680 Air Quality, Temperature, Pressure, Humidity SensorLWM303D 6D0F Motion Sensor (X, Y, Z Axes)BH1745 Luminance and Color SensorLTR-559 Light and Proximity Sensor 0.01 lux to 64,000 luxVL53L1X Time of Flight (TOF) Sensor Pew Pew Lasers!Software Component List:RaspianPython 2.7JDK 8 JavaApache NiFiMiniFiSource Code:https://github.com/tspannhw/minifi-breakoutgarden Shell Script (https://github.com/tspannhw/minifi-breakoutgarden/blob/master/runbrk.sh)Python (https://github.com/tspannhw/minifi-breakoutgarden/blob/master/brk.py)SummaryOur Raspberry Pi has a Breakout Garden Hat with 5 sensors and one small display. The display is showing the last reading and is constan

Problem Statement: Deploying and Governing ModelsMachine Learning and Artificial Intelligence are in the process of exploding in importance and prevalence in the enterprise. With this explosive growth comes fundamental challenges in governing model deployments ... and doing this at scale. These challenges revolve around answering the following fundamental questions: Which models were deployed when? and to where? Was this deployment to a microservice, a Spark context on Hadoop, or other?What was the serialized object deployed? How can I find it?What version was deployed? Who is the owner? What is the larger context around the projectHow do I know the details of the model, ie. how do I trace the model in Production to its actual code, its training data, owner, etc?Previous article: Why and how you should use Atlas to govern your models Article: Customizing Atlas (Part1): Model governance, traceability and registryIn the previous article I showed how Atlas is a powerful and natural f

Sometimes it is desired to see what kind of Http request ambari server makes to the outside works for troubleshooting purpose and to know the value of various http headers (like: User-Agent, Accept, Last-Modified, Content-Type, Content-Length, Connection, ETag, Server, X-Cache ...etc) which ambari-server uses to make calls to Ambari Public repo ...etc.So in order to achieve the Http Logging of ambari server made http requests we can do the following:Step-1). Create a file "/etc/ambari-server/conf/http.logging.properties" with the following content:.level=INFOhandlers=java.util.logging.ConsoleHandlerjava.util.logging.ConsoleHandler.level=ALLsun.net.www.protocol.level=ALLsun.net.www.protocol.http.HttpURLConnection.level=ALLdebug.level=ALL.Step-2). Inside the "/var/lib/ambari-server/ambari-env.sh" file variable "AMBARI_JVM_ARGS" make sure to add the following parameter "-Djava.util.logging.config.file=/etc/ambari-server/conf/http.logging.properties" to enable the http logging.Example:exp

Many a time, there are instances where one wants to deploy the Atlas UI changes (like static htmls, images, css and js files) which are built locally while remotely connecting to an Atlas server backend. This way, you don't really have to install Atlas (and other dependent services) on your local machineIn order to achieve this, one can set up a local grunt server with the below-mentioned stepsInstall npm (npm is a node package manager, used for nodejs projects)Checkout the Atlas codebaseNavigate to <workspace_directory>/dashboardv2 folderTunnel (add Local Port Forwarding) to Atlas instance deployed on a remote cluster using the following commandssh -i <private_key> -L <local_port>:<cluster_host>:<cluster_port> <user_name>@<cluster_host>Once you execute the above command, all the requests being sent to http://localhost:21000 will be redirected to our remote Atlas instance running at http://<cluster_host>:<cluster_port>;Now hit, http

1. In your Ambari UI, click on the Manage Ambari button.2. Then on the left of the screen, click on views.3. Click on Create Instance on the top right of the screen.4. Fill in the boxes as shown below. For select view, choose FILES from the drop down. The blocked out field below is your personal S3 key. 5. Click Save6. Once saved, you can go back to the views UI and see your new S3 view, click the URL to get through to the files view of your S3 bucket. It will look something like this example below.7. At this point, you can manage all of your directories and files in S3 from this Ambari view. You can delete, add or move things around with ease.

What is CDC Change Data Capture (CDC) in Data Warehousing typically refers to the process of capturing changes over time in Dimension tables. Dimension tables qualify Fact tables (measures) by containing information to answer questions around Who (e.g. Customer) , What (e.g. product), Where (e.g. Location), and When (e.g. Time). This article steps will demonstrate how to implement a very basic and rudimentary solution to CDC in Hadoop using MySQL, Sqoop, Spark, and Hive. It includes basic PySpark code to get you started with using Spark Data Frames. In a real world example you would include audit tables to store information for each run.   How to do CDC in Hadoop In the past, I have worked at a company in which CDC on Hadoop was a big challenge for us. Initially, we were using Hive to merge source changes (Sqoop extracts) with the existing Dimension tables and then building new tables. This worked well until our data volumes starting getting in the hundreds of millions of records.

My Example Mesh NetworkArgon: Wi-Fi + Mesh Gateway/RepeaterThis is to connect to the outside network via WiFi. We connect to Particle Cloud.Espressif ESP32-D0WD 2.4G Wi-FiNordic Semiconductor nRF52840 SoC for bluetooth and NFC A tagARM TrustZone CryptoCell-310 Cryptographic and security module Thread/BLE, another for Wi-Fi)Xenon: Mesh + BLENordic Semiconductor nRF52840 SoC for bluetooth and NFC A tagARM TrustZone CryptoCell-310 Cryptographic and security moduleReferences https://github.com/Seeed-Studio/Grove_Starter_Kit_for_Photon_Demos?files=1https://docs.particle.io/datasheets/accessories/mesh-accessories/https://community.particle.io/c/meshhttps://www.particle.io/meshhttps://docs.particle.io/datasheets/mesh/xenon-datasheet/https://docs.particle.io/datasheets/wi-fi/argon-datasheet/https://blog.particle.io/2018/04/28/how-to-build-a-wireless-mesh-network/WiFi Mesh = https://www.threadgroup.org/https://github.com/Seeed-Studio/Grove_Starter_Kit_for_Photon_Demos/blob/master/Example%20-%

I came across an article on how to setup NiFi to write into ADLS which required cobbling together various integration pieces and launching HDI. Since then there have been many updates in NiFi enabling a much easier integration. Combined with CloudBreak's rapid deployment of a HDF clusters provides an incredible ease of user experience. ADLS is Azure's native cloud storage (Look and feel of HDFS) and the capability to read/write via NiFi is key. This article will demonstrate how use use a CloudBreak Recipe to rapidly deploy a HDF NiFI "ADLS Enabled" cluster.AssumptionsA CloudBreak instance is availableAzure Credentials availableModerate familiarity with AzureUsing HDF 3.2+From Azure you will need:ADLS urlApplication IDApplication PasswordDirectory IDNiFi requires ADLS jars, core-site.xml, and hdfs-site.xml. The recipe I built will fetch these resources for you. Simply download the recipe/script from:https://s3-us-west-2.amazonaws.com/sunileman1/scripts/setAdlsEnv.shOpen it and scroll

I came across an article on how to setup NiFi to write into ADLS which required users to cobble together various integration pieces and launching HDI. Since then there have been many updates in NiFi enabling a much easier integration. Combine with CloudBreak's rapid deployment of a HDF cluster provides incredible ease of use. ADLS is native cloud storage provided by Azure (Look and feel of HDFS) and the capabilities to read/write via NiFi is key. This article will demonstrate how use use CloudBreak to rapidly deploy a HDF NiFI "ADLS Enabled" cluster.

Article This is the first of the two articles (see Part 2) where we will at look at Kubernetes (K8s) as the container orchestration and scheduling platform for a grid like architectural variation of the distributed pricing engine discussed here. Kubernetes @ Hortonworks But wait … before we dive in … why are we talking about K8s here? Well, if you missed KubeCon + CloudNativeCon, find out more! Hortonworks joined CNCF with an increased commitment to cloud native solutions for big data management. Learn more about the Open Hybrid Architecture Initiative led by Hortonworks to enable consistent data architectures on-prem and/or across cloud providers and stay tuned to discover more about how K8s is used across the portfolio of Hortonworks products – DPS, HDF and HDP to propel big data’s cloud native journey! The Application Architecture With that said, this prototype is a stateless compute application – a variation of the Dockerized Spark on YARN version described here

Problem:Of late, there are many HCC questions on how to create hive table and lineage using REST APIs in Atlas. This article will be act a step by step guide to create hive tables and lineage using REST API.Solution:As part of the solution to this FAQ, I will create two hive tables and lineage(CTAS) between them. I have tested these changes on HDP-2.5 release, so make sure you have HDP version >= 2.5.Step1: JSON for creating table1:[{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Reference", "id":{ "jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id", "id":"-11893021824425525", "version":0, "typeName":"hive_db", "state":"ACTIVE" }, "typeName":"hive_db", "values":{ "name":"default", "location":"hdfs://mycluster/apps/hive/warehouse", "description":"Default Hive database", "ownerType":2, "qualifiedName":"default@cl1", "owner":"public", "clusterName":"cl1", "parameters":{ } }, "traitNames":[

Deep Speech with Apache NiFi 1.8 Tools: Python 3.6, PyAudio, TensorFlow, Deep Speech, Shell, Apache NiFi Why: Speech-to-Text Use Case: Voice control and recognition. Series: Holiday Use Case: Turn on Holiday Lights and Music on command.Cool Factor: Ever want to run a query on Live Ingested Voice Commands? Other Options: https://community.hortonworks.com/articles/155519/voice-controlled-data-flows-with-google-aiy-voice.html We are using Python 3.6 to write some code around pyaudio, tensorflow and Deep Speech to capture audio, store it in a wave file and then process it with Deep Speech to extract some text. This example is running in OSX without a GPU on Tensorflow v1.11. The Mozilla Github repo for their Deep Speech implementation has nice getting started information that I used to integrate our flow with Apache NiFi. Installation as per https://github.com/mozilla/DeepSpeech pip3 install deepspeechwget -O - https://github.com/mozilla/DeepSpeech/releases/downloa

Introduction Cloudbreak documentation is fairly thorough and can be found on Hortonworks' documentation portal (e.g. here). However, if you are as lazy as I am, you'll appreciate the few quick examples I'm listing in this article. First and foremost, you can access the documentation of available APIs on your CB instance by using the following URL https://[YOUR_CB_URL]/cb/apidocs; you should see something like this: Authenticating Authentication in cloudbreak is OAuth 2, and getting a detailed in the Cloudbreak API. Here is simple curl call to get a token: curl -k -iX POST -H "accept: application/x-www-form-urlencoded" -d 'credentials={"username":"[YOUR_USER]","password":"[YOUR_PWD]"}' "https://[YOUR_CB_URL]/identity/oauth/authorize?response_type=token&client_id=cloudbreak_shell&scope.0=openid&source=login&redirect_uri=http://cloudbreak.shell" | grep location | cut -d'=' -f 3 | cut -d'&' -f 1 GET Example: Listing all blueprints The biggest

When an ambari agent starts, it bootstraps with the ambari server via registration. The server sends information to the agent about the components that have been enabled for auto start along with the other auto start properties in ambari.properties. The agent compares the current state of these components against the desired state, to determine if these components are to be installed, started, restarted or stopped.These are the values ambari-server will send to ambari-agent by default unless configured in cluster-env.xml file. "recovery_max_count": "6","recovery_lifetime_max_count": "1024","recovery_type": "AUTO_START","recovery_window_in_minutes": "60",recovery_lifetime_max_count ---- The maximum number of recovery attempts of a failed component during the lifetime of an Ambari Agent instance. This is reset when the Ambari Agent is restarted. recovery_window_in_minutes -- The length of a recovery window, in minutes, in which recovery attempts can be retried. recovery_max_count --- The

Phoenix secondary indexPhoenix secondary Indexes are useful for point lookups or scans directed against non primary key of Phoenix or non row key columns of HBase. This saves the “full scan” of data table you would otherwise do if you intend to retrieve data based non rowkey.You create secondary indexes by choosing existing non primary key column from data table and making it as primary or a covered column. By covered column, we mean making exact copy of the covered column’s data from data table to index table.Types of secondary Index:Functional Index: Built on functions rather than just columnsGlobal secondary Index: This is the one where we make a exact copy of covered columns and call it index table. In simple terms, its an upsert select on all chosen columns from data table to Index table. Since a lot of write is involved during initial stages of index creation, this type of index would work best for read heavy use cases where data is written only occasionally and read more frequen

Apache Spark 2.2.0 with Scala 2.11.8 with Java 1.8.0_112 on HDP 2.6.4 called from HDF 3.1 with Apache NiFi 1.5.This is a follow up to: https://community.hortonworks.com/articles/173818/hdp-264-hdf-31-apache-spark-streaming-integration.html and https://community.hortonworks.com/articles/155326/monitoring-energy-usage-utilizing-apache-nifi-pyth.html We are using the same Apache NiFi flow to send messages to Apache Kafka. What is nice you could have the Structured Streaming version, non-structured version and others listening to the same Topic and same messages sent by Apache NiFi.When we start, no data yet.We quickly get a ton of dataBy Default A Kafka Cluster is 3 Nodes. Replication Factor of 3 is good then. I have one node. I had to change this. Tons of warnings in the /usr/hdf/current/kafka-broker/logs directory.The simplest Apache Spark client is one run in the shell:/usr/hdp/current/spark2-client/bin/spark-shell --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.0,o

AbstractAdding known clarification tags during data Ingestion using Atlas and NiFi.IntroductionMost of the times we know some metadata about the data we are ingesting into bigdata platform and this metadata could play an important role in data governance and security as the data moves through downstream pipeline, metadata like terms and conditions, country, consent agreements, validity period of the data etc, these are very useful defining GDRP like requirements and if we don’t add these classification during ingestion we loose the context, as you all know Apache Atlas is used as open metadata governance tool defining these classifications and tags etc, one way is to use NiFi to ingest the data which captures the data lineage in Atlas , but you have to add additional classification tags manually based on the nature of the data at later point of time in the process which is error prone . ProblemIf you do not add metadata classifications automatically part of the data ingestion you wi