Community Articles

Have recently run into multiple issues where ORC files on hive are not getting compacted.There are a couple of parameters required to enable concat on ORC.SET hive.merge.tezfiles=true; SET hive.execution.engine=tez;SET hive.merge.mapredfiles=true;SET hive.merge.size.per.task=256000000;SET hive.merge.smallfiles.avgsize=256000000;SET hive.merge.mapfiles=true;SET hive.merge.orcfile.stripe.level=true;SET mapreduce.input.fileinputformat.split.minsize=256000000;SET mapreduce.input.fileinputformat.split.maxsize=256000000;SET mapreduce.input.fileinputformat.split.minsize.per.node=256000000;SET mapreduce.input.fileinputformat.split.minsize.per.rack=256000000;ALTER TABLE <table_name> SET TBLPROPERTIES('EXTERNAL'='FALSE');alter table <table_name> partition ( file_date_partition='<partition_info>') concatenate;ALTER TABLE <table_name> SET TBLPROPERTIES('EXTERNAL'='TRUE');mapreduce.input.fileinputformat.split.minsize.per.nodeSpecifies the minimum number of bytes that each in

This is the third in the series of articles on WFM.Part 1: https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.html Part 2: https://community.hortonworks.com/articles/82967/apache-ambari-workflow-designer-view-for-apache-oo.html Part 4: https://community.hortonworks.com/articles/83051/apache-ambari-workflow-designer-view-for-apache-oo-2.htmlPart 5: https://community.hortonworks.com/articles/83361/apache-ambari-workflow-manager-view-for-apache-ooz.htmlPart 6: https://community.hortonworks.com/articles/83787/apache-ambari-workflow-manager-view-for-apache-ooz-1.htmlPart 7: https://community.hortonworks.com/articles/84071/apache-ambari-workflow-manager-view-for-apache-ooz-2.htmlPart 8: https://community.hortonworks.com/articles/84394/apache-ambari-workflow-manager-view-for-apache-ooz-3.htmlPart 9: https://community.hortonworks.com/articles/85091/apache-ambari-workflow-manager-view-for-apache-ooz-4.htmlPart 10: https://community.hortonworks.co

Following steps will help you install miniconda, conda or anaconda using wget.1. Locate the version you need to install by opening the following link on your browserhttps://repo.continuum.io/miniconda/for example lets say you decided to use https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh2. Download the installeryum install wget bzip2wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh3. Run the installer and follow the stepschmod +x Miniconda3-latest-Linux-x86_64.sh./Miniconda3-latest-Linux-x86_64.shNote: I recommend to select an install location like /opt/miniconda3 or similar that can be shared and used by different users

1. As a first step, enable HTTPS for HDFS, you can follow the article https://community.hortonworks.com/articles/52875/enable-https-for-hdfs.html2. Add/Update below configurations in "Custom mapred-site" mapred-site.xmlmapreduce.jobhistory.http.policy=HTTPS_ONLYmapreduce.jobhistory.webapp.https.address=<JHS>:<JHS_HTTPS_PORT> mapreduce.ssl.enabled=truemapreduce.shuffle.ssl.enabled=trueEx: mapreduce.jobhistory.webapp.https.address=apappu-hdp234-2.openstacklocal:198893. Add/update below configurations under "Advanced yarn-site" ( yarn-site.xml)yarn.http.policy=HTTPS_ONLYyarn.log.server.url=https://JHS:JHS_HTTPS_PORT/jobhistory/logsyarn.resourcemanager.webapp.https.address=RM:RM_HTTPS_PORTyarn.nodemanager.webapp.https.address=0.0.0.0:NM_HTTPS_PORTEx: yarn.log.server.url=https://apappu-hdp234-2.ambari.org:19889/jobhistory/logsyarn.resourcemanager.webapp.https.address=apappu-hdp234-2.ambari.org:8090yarn.nodemanager.webapp.https.address=0.0.0.0:80424. Add/update below property in

Good design goes mostly unnoticed. If a user interface works well, you simply complete a task and move on to the next. That time when something doesn't work as expected, or you think why do I have to do it this way, is when the design becomes much more noticeable.The following is a look back over recent months at some notable UI/UX improvements that have recently made or will be soon making their way to Apache NiFi. They vary a lot in size and complexity, but overall they contribute equally to make NiFi better by design.Enable/Disable Support for Multiple ComponentsMuch like starting/stopping multiple components on the canvas, now you can enable or disable more than one component at the same time. You can also perform the action on a process group, therefore effecting all applicable components within. If your flow has a large number of stopped components, disabling them can help reduce the load NiFi takes on during validation.NIFI-5066Automatically Reconnect NiFi Node to ClusterIf a mu

In order to use a non operative system default version of python with Zeppelin Pyspark interpreter you need to add the following configuration in the interpreter configuration page:Previous to this configuration you need to install the python version you like to use along side the operative system one in separate folder. Important: Changing the default os python could have an impact on other services and applications running on the machine. Hence we don't recommend doing so. Once you have installed the python version you wish to use set the above configuration pointing to the path of the python binary. After adding the above, save and restart the interpreter. To test changes:\Important: The new python version installed needs to be present on zeppelin machine and all cluster worker nodes. Use same directory and make sure access permissions to the python dir is correctly set.

About Apache Spark: Apache Spark is an open-source distributed general-purpose cluster computing framework with in-memory data processing engine. We can do ETL analysis, Machine learning, Graph processing on large volumes of data at rest as well as data in motion with rich consise high level APIs in programming languages such as Python, Java, Scala, R and SQL. Spark execution engine is capable of processing batch data and real time streaming data from different data sources via Spark streaming. Spark supports an interactive shell for performing data exploration and for ad hoc analysis. We can run Spark applications locally or distributed across a cluster, either by using an interactive shell or by submitting an application. spark-shell - scala interactive shell pyspark - python interactive shell sparkR - R interactive shell To run applications distributed across a cluster, Spark requires a cluster manager. Spark environment can be deployed and managed through the following cluster mana

The Key Distribution Center (KDC) is available as part of the domain controller and performs two key functions which are: Authentication Service (AS) and Ticket-Granting Service (TGS)By default the KDC requires all accounts to use pre-authentication. This is a security feature which offers protection against password-guessing attacks. The AS request identifies the client to the KDC in plain text. If pre-authentication is enabled, a time stamp will be encrypted using the user's password hash as an encryption key. If the KDC reads a valid time when using the user's password hash, which is available in the Active Directory, to decrypt the time stamp, the KDC knows that request isn't a replay of a previous request.When you do not enforce pre-authentication, a malicious attacker can directly send a dummy request for authentication. The KDC will return an encrypted TGT and the attacker can brute force it offline. Upon checking the KDC logs, nothing will be seen except a single request for a

Detecting Language with Apache NiFiThis is a work-in-progress as I am still experimenting with some libraries and techniques to improve this. I originally looked at Apache OpenNLP, Apache Tika, optimaize, Deep Learning and some older libraries. It turns out that most of them are defunct or use optimaize. So I am using the Apache Tika wrapper of optimaize for this pass. I am testing with a few experts to see if we can parse the main languages we need. JUnit TestAdd The Processor (First copy to LIB directory and restart the NiFi server) Do not run this in production. Download the NAR from the github.A Quick FlowIt produces an attribute: langdetectTikaSo far I have tested with Spanish (es) and English (en).Source: https://github.com/optimaize/language-detectorJUNit Results18:30:37.386 [pool-1-thread-1] DEBUG com.optimaize.langdetect.LanguageDetectorImpl - ==> [DetectedLanguage[es:0.9999962696603875]]18:30:37.389 [pool-1-thread-1] DEBUG com.optimaize.langdetect.Language

IssueSoon after starting Druid services, coordinator keeps going down. This is mostly experienced on a new setup. The exception in the coordinator log looks like this2018-02-02T00:37:24,123 ERROR [main] io.druid.cli.CliCoordinator - Error when starting up. Failing.org.skife.jdbi.v2.exceptions.CallbackFailedException: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'druid.druid_rules' doesn't exist [statement:"SELECT id from druid_rules where datasource=:dataSource", located:"SELECT id from druid_rules where datasource=:dataSource", rewritten:"SELECT id from druid_rules where datasource=?", arguments:{ positional:{}, named:{dataSource:'_default'}, finder:[]}] at org.skife.jdbi.v2.DBI.withHandle(DBI.java:284) ~[jdbi-2.63.1.jar:2.63.1] at io.druid.metadata.SQLMetadataRuleManager.createDefaultRule(SQLMetadataRuleManager.java:83) ~[druid-server-0.10.1.2.6.4.0-91.jar:0.10.1.2.6.4.0-91] at

How do you know if your customers (and potential customers) are talking about you on social media?The key to making the most of social media is listening to what your audience has to say about you, your competitors, and the market in general. Once you have the data you can undertake analysis, and finally, reach social business intelligence; using all these insights to know your customers better and improve your marketing strategy. This is the third part of the series of articles on how to ingest social media data like streaming using the integration of HDP and HDF tools. To implement this article, you first need to implement the previous two. https://community.hortonworks.com/articles/177561/streaming-tweets-with-nifi-kafka-tranquility-druid.htmlhttps://community.hortonworks.com/content/kbentry/182122/integrating-nifi-to-druid-with-a-custom-processor.html Let’s get started! In this new article, we will address two main points: 1 - How to collect data from various social networks

Original PostHive and Google Cloud StorageGoogle’s Cloud Platform provides the infrastructure to perform MapReduce data analysis using open source software such as Hadoop with Hive and Pig. Google's Compute Engine provides the compute power and Cloud Storage is used to store the input and output of the MapReduce jobs.HDP deployment using CloudBreak.Before we deploy HDP in GCE, we need to setup account in GCE and CloudBreak.Signup for free trial account on https://cloud.google.com/free-trial/Step1) Login into your google dashboard and then click Create a project. For example: I created a project called hadoop-installStep 2) Create credentials.Click Create new Client ID and then choose Service account.Click Okay got it and it will download JSON key (We won’t be using this file). You will see Client ID, Email address and Certificate fingerprints in the same window after downloading JSON key. There will be an option to Generate new P12 key.Step 3) Enable APIDefault API when you loginSearch

Integrating Keras (TensorFlow) YOLOv3 Into Apache NiFi WorkflowsFor this article I wanted to try the new YOLOv3 that's running in Keras.Out of the box with video streaming, pretty cool:git clone https://github.com/qqwweee/keras-yolo3 wget https://pjreddie.com/media/files/yolov3.weights python convert.py yolov3.cfg yolov3.weights model_data/yolo.h5 python yolo.pySee:https://github.com/qqwweee/keras-yolo3My article on Darknet original YOLO v3 is here: https://community.hortonworks.com/articles/191259/integrating-darknet-yolov3-into-apache-nifi-workfl.htmlExample JSON Output{ "boxes" : "Found 8 boxes for img", "class7" : "diningtable", "score7" : "0.7484486", "left7" : "636", "right7" : "1096", "top7" : "210", "bottom7" : "693", "class6" : "sofa", "score6" : "0.31372178", "left6" : "1114", "right6" : "1276", "top6" : "172", "bottom6" : "381", "class5" : "chair", "score5" : "0.3455438", "left5" : "990", "right5" : "1048", "top5" : "183", "bottom5" : "246", "class4" : "

Updating The Apache OpenNLP Community Apache NiFi Processor to Support Flow FilesIn this new release, we add the ability to read content from the FlowFile and analyze that for Locations, Dates, Organizations and Names. We are using the Apache OpenNLP 1.5 Models that are available for download. These do a decent job. You can build new models as needed. I also changed it to output one attribute per type with a String list of locations, organizations, dates and names.I put out a new release, built around Apache NiFi 1.6.0.Source and NAR Downloadhttps://github.com/tspannhw/nifi-nlp-processor/releases/tag/1.6Download the Pre-trained Models for Your Language Here:http://opennlp.sourceforge.net/models-1.5/I chose English (en).In a future release I made add Organization, Money, Time and Percentage to the lists we extract if there is interest.A Final JSON File ProducedExample OutputThe Main Flow For Trying Out The NLP ProcessorSet Your ModelsNew NLP Processor DocumentationHere is the sch

PROBLEM Users able to drop table on hive though they are not the table owners. Need to enable metastore server security to start using the storage based auth. SOLUTION To enable metastore security we need to enable the following parameter hive.metastore.pre.event.listeners [This turns on metastore-side security.] Set to org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListenerhive.security.metastore.authorization.manager [This tells Hive which metastore-side authorization provider to use. The default setting uses DefaultHiveMetastoreAuthorizationProvider, which implements the standard Hive grant/revoke model. To use an HDFS permission-based model (recommended) to do your authorization, use StorageBasedAuthorizationProvider as instructed above]Set to org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProviderhive.security.metastore.authenticator.managerSet to org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticatorhive.security.m

https://roaringelephant.org/2018/05/22/episode-89/With the San Jose edition of the DataWorks Summit only a month away, we go over the sessions that are available in the agenda today and offer our top picks. If you’re going, or if you will be watching the replays online, we hope to guide you on your selection of sessions. DataWorks Summit San Jose 2018 Play in new window | Download (Duration: 1:12:20 — 49.9MB)And here is the dashboard we created with statistics on the San Jose sessions, for your enjoyment: https://aka.ms/DWS2018SJThe agenda is still in flux so we will be updating the dashboard regularly. Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

About this article The Metron tutorial article for adding Squid telemetry walks through the process of creating the parser from scratch for Elasticsearch as the Indexing service. This article gives details of extending the tutorial for getting Squid telemetry working with Solr as the backend Indexing service. In other words, these steps are an equivalent of "Installing Squid parser template" for the Elasticsearch. Pre-requisites HCP >= 1.5.0.0 HDP search >= 3.0.0 It is assumed that you have deployed a HCP stack with Solr by following the HCP documentation The Solr node is co-located with the Metron node. In the event that these nodes are on different hosts, ensure that you copy the Metron schema files located at $METRON_HOME/config/schema to the Solr node. It is also assumed that you have followed the Metron tutorial for Squid telemtry by installing the squid sensor, creating the kafka topic and have started the storm topology Steps 1. SSH to the Metron host and run the fol

When you are working with machine learning algorithms, you must be mindful with how the algorithm treats the data that you input into them. Often, there is a required transformation process to conform the desired values into a usable form. In the case of categorical data, ML algorithms will misinterpret raw values, which will lead to improper results. The solution to this using categorical data is known as One Hot Encoding. This process entails creating an additional column for each of the values represented in the categorical data set. This followed by only setting one of the newly columns to value true or 1, while setting the rest of them to false or 0. In our example below, imagine these new columns as booleans with titles: Is_apple, Is_banana, and Is_coconut. Continuing our example, the value "banana" would have the values for columns Is_apple=0, Is_banana=1. and Is_coconut=0. Using the tools below, we will create a vector that can be used as input into ML algorithms. Step

This article is designed to extend the great work by @Ali Bajwa: Sample HDF/NiFi flow to Push Tweets into Solr/Banana, HDFS/Hive I have included the complete notebook on my Github site, which can be found here Step 1 - Follow Ali's tutorial to establish an Apache Solr collection called "tweets" Step 2 - Verify the version of Apache Spark being used, and visit the Solr-Spark connector site. The key is to match the version of Spark the version of the Solr-Spark connector. In the example below, the version of Spark is 2.2.0, and the connector version is 3.4.4 %spark2scsc.version res0: org.apache.spark.SparkContext = org.apache.spark.SparkContext@617d134ares1: String = 2.2.0.2.6.4.0-91 Step 3 - Include the Solr-Spark dependency in Zeppelin. Important note: This needs to be run before the Spark Context has been initialized. %depz.load("com.lucidworks.spark:spark-solr:jar:3.4.4")//Must be used before SparkInterpreter (%spark2) initialized//Hint: put this paragraph before any Spark co

This article discuss the process related to Oozie Manual Sharelib update and the prerequisites for Spark Oozie SharelibCopy the sharelib to a local directory # mkdir oozie_share_lib # hadoop fs -copyToLocal <current-share-lib-directory> oozie_share_lib/libTo update oozie sharelib once the existing oozie sharelib copied from HDFS to local as above:/usr/hdp/current/oozie-client/bin/oozie-setup.sh sharelib create -fs /user/oozie/share/lib/ -locallib oozie_share_lib/This will create a new sharelib including SPARK Oozie sharelib:the destination path for sharelib is: /user/oozie/share/lib/lib_20180502070613Fixing oozie spark sharelibSpark is locally installed at /usr/hdp/2.6.3.0-235/oozie/../sparkRenaming spark to spark_orig in /user/oozie/share/lib/lib_20180502070613Creating new spark directory in /user/oozie/share/lib/lib_20180502070613Copying Oozie spark sharelib jar to /user/oozie/share/lib/lib_20180502070613/sparkCopying oozie_share_lib/lib/spark/oozie-sharelib-spark-4.2.0.2.6.3.

Accessing public social data from Facebook for a company's page is easy.Find your Facebook page, say Hortonworks.Run http://findmyfbid.com/ and get the Page ID (289994161078999) for your Page.Create a Facebook Application and Add a New Application.Create a Facebook Access Token in Graph API Explorer using your Application.Create your Facebook Graph API URL. https://graph.facebook.com/v2.7/289994161078999/tagged?access_token=ACCESSTOKENFROMFACEBOOK&limit=100. Because we are using a Facebook App Token, we need to use HTTPS/SSL. To access an SSL site in GetHTTP, we need a SSL Service with a Trust Store.Facebook Graph API ExplorerFacebook Graph API Explorer TestAdd the URL to the GetHTTP Processor.Create a Standard SSL Context Service (Controller Service), for the Sandbox, use the Java SSL Trust Store.Add the SSL Context Service to the GetHTTP Processor.Save to HDFS (PutHDFS).Download[root@sandbox demo]#hdfs dfs -cat /social/facebook1469644415053.json{ "data": [ { "message": "S

DevOps Tips: Using the Apache NiFi Toolkit with Apache NiFI 1.6.0 Apache NiFi now has a cool command-line tool that can run interactive or one off. You can see it here: https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli You will need to fork and build the current version of Apache NiFi from github. If you have the Java JDK 8+ and Maven you are good to guy. It's pretty quick. After you build, navigate to the nifi-toolkit target directory and you will have a zip to copy and unzip. You can then use this helpful tool. This is a separate tool from https://github.com/Chaffelson/nipyapi. Both are awesome though. I ran this on my local Mac. nifi-toolkit-1.6.0 List Registry Buckets ./bin/cli.sh registry list-buckets -u http://localhost:18080 # Name Id Description- ------------------------ ------------------------------------ -----------1 Development dd323482-c62e-4b18-9f9

https://roaringelephant.org/2018/05/15/episode-88-roaring-news/Returning to our more regular schedule, we have a Roaring News episode today. Dave has articles on multi-cloud readiness, Big Data being a pariah, and Google Duplex and Jhon came up with Synthetic data, data engineers and scientists and a Neural Network sharing cake recipes. Breaking NewsPlay in new window | Download (Duration: 35:07 — 24.4MB)Dave Less than 10% ready for multi cloud http://www.cloudpro.co.uk/cloud-essentials/hybrid-cloud/7451/idc-less-than-10-of-organisations-are-ready-for-multi-cloudTech companies distancing themselves from Big Data https://qz.com/1262102/tech-companies-are-distancing-themselves-from-big-data/Google Duplex https://ai.googleblog.com/2018/05/duplex-ai-system-for-natural-conversation.htmlJhon The Rise of Synthetic Data to Help Developers Create and Train AI Algorithms Quickly and Affordably https://insidebigdata.com/2018/05/08/rise-synthetic-data-help-developers-create-train-ai-algorithms-

https://roaringelephant.org/2018/05/08/episode-87-druid-high-performance-column-oriented-distributed-data-store-part-2/This is the second part of an interview with Fangjin Yang, co-founder and CEO at Imply and committer/PMC member for the Druid project. Druid: a high-performance, column-oriented, distributed data store which has entered the Hadoop environment with the recent integration with Apache and we since Druid has been around for a while, we are grateful to FJ for spending some time with our listeners. Play in new window | Download (Duration: 31:53 — 22.1MB) Fangjin YangCofounder and CEO at Imply (linkedin) Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

https://roaringelephant.org/2018/05/01/episode-86-druid-a-high-performance-column-oriented-distributed-data-store-part-1/This is the first part of an interview with Fangjin Yang, co-founder and CEO at Imply and committer/PMC member for the Druid project. Druid: a high-performance, column-oriented, distributed data store which has entered the Hadoop environment with the recent integration with Apache and we since Druid has been around for a while, we are grateful to FJ for spending some time with our listeners. Play in new window | Download (Duration: 31:57 — 22.2MB) Fangjin YangCofounder and CEO at Imply (linkedin) Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.