Community Articles

DESCRIPTION: Received frequent alerts for connection timeout with journal node. Upon checking the connectivity, it results in below output.curl -v http://123.example.com:8480--max-time 4 | tail -4 * About to connect() to 123.example.com:8480 port 8480 (#0) * Trying 10.24.16.11... connected * Connected to 123.example.com (10.24.16.11) port 8480 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.18 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: 123.example.com:8480 > Accept: */* > % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0* Operation timed out after 4000 milliseconds with 0 bytes received 0 0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0* Closing connection #0 curl: (28) Operation timed out after 4000 milliseconds with 0 bytes received Checking netstat command further for port 8480 gives us huge number of CLOSE_WAIT messages.

This article with cover step by step how to configure HDP Search Solr with Ranger PluginStep 1Download and install the 2.2.9+ mpack - FYI: Previous mpack versions don't support integration of HDP Search Solr with Ranger. Mpack 2.2.9 includes configurable section for the solr-security on ambari which allows to add the authorization information.wget 'http://public-repo-1.hortonworks.com/HDP-SOLR/hdp-solr-ambari-mp/solr-service-mpack-2.2.9.tar.gz'; -O /tmp/solr-service-mpack-2.2.9.tar.gzambari-server install-mpack --mpack=/tmp/solr-service-mpack-2.2.9.tar.gzStep 2On HDP solr host runyum install ranger-solr-plugin.noarchcd /usr/hdp/2.6.2.0-205/ranger-solr-pluginEdit install.properties and make sure the following settings at least are properly configured:POLICY_MGR_URL=http://<ranger-host>:6080 SQL_CONNECTOR_JAR=/usr/share/java/mysql-connector-java.jarEdit solr-plugin-install.properties and set correct value for install dir:COMPONENT_INSTALL_DIR_NAME=/opt/lucidworks-hdpsearch/solr/ser

1. Create User [root@chupa1 ~]# curl -iv -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{"Users/user_name":"dav","Users/password":"pass","Users/active":"true","Users/admin":"false"}' http://localhost:8080/api/v1/users * About to connect() to localhost port 8080 (#0) * Trying ::1... connected * Connected to localhost (::1) port 8080 (#0) * Server auth using Basic with user 'admin' > POST /api/v1/users HTTP/1.1 > Authorization: Basic YWRtaW46YWRtaW4= > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > Host: localhost:8080 > Accept: */* > X-Requested-By: ambari > Content-Length: 93 > Content-Type: application/x-www-form-urlencoded > < HTTP/1.1 201 Created HTTP/1.1 201 Created 2. Create Group [root@chupa1 ~]# curl -iv -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{"Groups/group_name":"davgroup"}' http://localhost:8080/api/v1/groups * About to connect() to localhost port

The concept of time is at the core of all Big Data processing technologies but is particularly important in the world of data stream processing. Indeed, it is reasonable to say that the way in which different systems handle time-based processing is what differentiates the wheat from the chaff as it were, at least in the world of real-time stream processing.The demand for stream processing is increasing a lot these days. A common need across Hadoop projects is to build up-to-date indicators from streaming data.Social Media analysis is a great use case for show how we can build a dashboard showing streaming analytics with NiFi, Kafka, Tranquility, Druid, and SupersetThis processing flow has these steps:1) Tweets ingestion using Apache NiFi2) Stream processing using Apache Kafka3) Integrating data with Tranquility4) OLAP database storage using Druid5) Visualization using Apache SupersetBefore putting our hands on coding, take a look on each component:Nifi: https://br.hortonworks.com/apach

Using Apache MXNet GluonCV with Apache NiFi for Deep Learning Computer VisionSource: https://github.com/tspannhw/OpenSourceComputerVision/Gluon and Apache MXNet have been great for deep learning especially for newbies like me. It got even better! They added a Deep Learning Toolkit that is easy to use and has a number of great pre-trained models that you can easily use to do some general use cases around computer vision. So I have used a simple well-documented example that I tweaked to save the final image and send some JSON details via MQTT to Apache NiFi. This may sound familiar: https://community.hortonworks.com/articles/198912/ingesting-apache-mxnet-gluon-deep-learning-results.html GluonCV makes this even easier! Let's check it out. Again let's take a simpmle Python example tweak it, run it via a shell script and send the results over MQTT.See: https://gluon-cv.mxnet.io/build/examples_detection/demo_ssd.html#sphx-glr-build-examples-detection-demo-ssd-pyPython Code: https://github

Ingesting Apache MXNet Gluon Deep Learning Results Via MQTT and Apache NiFiSummary:Using a Pre-Trained Model in Apache MXNet Gluon Python 3 code to classify a webcam image captured and processed with OpenCV. In our Python script, we capture the image to disk and capture JSON metadata about the percentage, probabilities and device information. This JSON data is then sent via MQTT to a broker. Apache NiFi processes the JSON data.Example ImageSource CodeSchema: https://github.com/tspannhw/OpenSourceComputerVision/blob/master/gluon2.avscPython Source: https://github.com/tspannhw/OpenSourceComputerVision/blob/master/nifigluon2.pyShell Script: https://github.com/tspannhw/OpenSourceComputerVision/blob/master/rungluon2.shSQL Table DDLCREATE EXTERNAL TABLE IF NOT EXISTS gluon2 (top1pct STRING, top2pct STRING, top3pct STRING, top4pct STRING, top5pct STRING, top1 STRING, top2 STRING, top3 STRING, top4 STRING, top5 STRING, imgname STRING, host STRING, end STRING, te STRING, battery INT

Executing TensorFlow Classifications from Apache NiFi Using Apache Spark 2.3 and Apache LivyTechnology: Apache Spark 2.3 + Apache Livy + Apache NiFi 1.5 + TensorFlow + PythonPython Code: https://github.com/tspannhw/DWS-DeepLearning-CrashCourse/blob/master/tensorflowsparknifi.pyTIP: In this version of Apache NiFi, you need to use double quotes (") instead of single quotes (') in your Python code.Python Code for NiFi ExecuteSparkInteractiveSee GithubSimple Apache NiFi Flow To Execute TensorFlow Python Applications via Apache LivyI am just using Apache Livy as the transport call from Apache NiFi to Apache Spark. My Apache Spark 2.3 cluster is not doing any Spark specific processing. PySpark is just running a vanilla TensorFlow python application in this version. We could also call TensorFlow on Spark code in this way. My goal was to run TensorFlow on my Spark cluster trigger from Apache NiFi and get back results.Results Returned in Success From ExecuteSparkInteractive Call{ "

I was recently working with a customer to deploy a Storm topology in their dev environment. I had built and tested the topology in my environment but when we went to deploy it at the customer site, nothing happened - literally.We could see the topology was in fact running but it wasn’t consuming any data from the Kafka topic to which it was subscribing. We verified that data was in fact being written there using the kafka-console-consumer.sh utility but still the topology wasn’t doing anything and even more confusing, the storm log for the topology didn’t show anything beyond the deployment steps.Going into the Storm UI we wanted to drill down to get the specifics of the topology and potentially turn on debug logging to see what was up, however when we clicked on the topology the Storm UI is unresponsive trying to load the topology summary. Even more confusing was the fact that the Storm UI didn’t show any error messages.Turns out that at some point in the past in this development envi

In DPS-1.1.0 We can't remove cluster from DPS UI. We can use the curl to remove the cluster. Note : User with Dataplane Admin Role can perform the below. screen-shot-2018-06-14-at-102253-am.pngTo delete smayani-hdp cluster Step1:- Find the cluster ID, which you want to remove. You can use the developers tools from the browser to find the cluster ID.screen-shot-2018-06-14-at-101629-am.pngFrom above example smayani-hdp cluster ID is: 3 ( https://172.26.125.109/api/lakes/3/servicesDetails ) Step 2 :- From the console use the below curl to remove the cluster.curl -k -u <username>:<Password> -X DELETE https://<DPS_HOST>/api/lakes/<cluster_ID>;Example :curl -k -u admin:kjncsadasdcsdc -X DELETE https://172.26.125.109/api/lakes/3Once the above is executed you should no longer see the cluster in UI.screen-shot-2018-06-14-at-102509-am.pngAlternative You can also use rm_dp_cluster.sh in /usr/dp/current/core/bin on DPS installed server.Usage: ./rm_dp_cluster.sh DP_JWT HAD

Short Description Migrate Hive saved queries from Hue 3.9.0 of CDH 5.7.2 into Hive Views of Ambari 2.6.2.2 on HDP 2.6.5 Environment: CentOS 7, CDH 5.7.2, Hue 3.9.0, Ambari 2.6.2, HDP 2.6.2, Hue 2.6.2, mySQL5.6 Problem: There are over 40,000 saved / history queries in CDH-Hue. During CDH to HDP migration, Hive saved queries from Hue needs to be migrated to Ambari Hive View saved queries. Assumptions: You have Hue of CDH configured with MySQL and CDH to HDP migration is successful.Solution Step 1: Install Hue 2.6.2 on HDP cluster pointing to a new database called hue_new262 follow the link HDP-2.6.5 - installing_hue Step 2: Insert records from hue_cdh (database of Hue CDH) to hue_new262 (database of Hue HDP) Impacted tables auth_user, auth_user_groups, beeswax_savedquery, beeswax_queryhistory insert into hue_new262.auth_user (id, username, first_name, last_name, email, password, is_staff, is_active, is_superuser, last_login, date_joined) select id, username, first_name, last_nam

Roaring Elephant Podcast - Episode 90 – Roaring newshttps://roaringelephant.org/2018/05/29/episode-90-roaring-news/In this weeks Roaring News episode, Dave brings up the resilience of Apache Community open source projects and plays some Doom. Jhon has some practical Apache NIFI guides and the emergence of multi modal NoSQL databases. Play in new window | Download (Duration: 38:09 — 26.4MB)DataWorks Summit Berlin video recordings are up: https://www.youtube.com/user/HadoopSummit/playlistsFind Dave on his Australian road-trip: http://bit.ly/aus-nz-ibm-hwx-tourDave DataTorrent, Stream Processing Startup, Folds (Apache Apex) https://www.datanami.com/2018/05/08/datatorrent-stream-processing-startup-folds/DOOM! https://arxiv.org/abs/1804.09154https://www.technologyreview.com/s/611072/ai-generates-new-doom-levels-for-humans-to-play/https://www.youtube.com/watch?v=K32FZ-tjQP4Bonus doom news:https://www.rockpapershotgun.com/2018/03/28/dodge-fireballs-forever-in-a-neural-nets-doom-nightm

Roaring Elephant Podcast - Episode 91 – ODPi is back and better than ever!https://roaringelephant.org/2018/06/05/episode-91-odpi-is-back-and-better-than-ever/In this episode, we welcome back John Mertic, director of Program Management for ODPi, R Consortium, and the Open Mainframe Project. It’s been almost two years since we checked in with John and the ODPi initiative and as John mentions in the interview, a lot has changed in Hadoop… Play in new window | Download (Duration: 1:08:00 — 46.9MB)John MerticDirector of Program Management for ODPi, R Consortium, and Open Mainframe Projecthttps://www.linkedin.com/in/jmertic/ODPi website links: https://www.odpi.org/https://www.odpi.org/blog/2018/04/04/the-state-of-open-source-and-big-data-three-years-laterhttps://www.odpi.org/projects/data-governance-pmchttps://www.odpi.org/events Please use the Contact Form on this blog or our twitter feed to send us your questions, or to suggest future episode topics you would like us to cover.

In DPS-1.1.0 we can't edit all the LDAP configuration properties after the initial setup. If we have to correct LDAP configs, you need to re-initialize the DPS setup to change it, which can be a painful task. To avoid re-initializing the DPS setup, we can make the changes directly in the postgres database of dataplane. Step1:- Find the container id of dp-database on DPS machinedocker psStep2: - connect to the docker machinedocker exec -it cf3f4a31e146 /bin/bashStep3:- Login to the postgres database (dataplane)su - postgrespsql -d dataplaneTake backup of the table, create table dataplane.ldap_configs_bkp as select * from dataplane.ldap_configs;To view the existing configuration:select * from dataplane.ldap_configs;Sample Output: id | url | bind_dn | user_searchbase | usersearch_attributename | group_searchbase | groupsearch_attr ibutename | group_objectclass | g

In a non-Kerberized environment, for Kafka console consumer command displays all messages, perform the following:Note:Here, <broker_host> is Kafka broker hostname, <zk_host> is zookeeper hostname and Kafka port will be 6667: Using Old consumer API (prior to HDP 2.5.x), run the following:bin/kafka-console-consumer.sh --zookeeper <zk_host>:2181 --topic test --from-beginning Using New consumer API ( From HDP 2.5.x onwards), run the following: bin/kafka-console-consumer.sh --bootstrap-server <broker_host>:6667 --topic test --from-beginnin To view a specific number of message in a Kafka topic, use the --max-messages option. To view the oldest message, run the console consumer with --from-beginning and --max-messages 1: Using Old consumer API ( Prior to HDP 2.5.x):bin/kafka-console-consumer.sh --zookeeper <zk_host>:2181 --topic test --from-beginning --max-messages 1 Using New consumer API ( From HDP 2.5.x onwards): bin/kafka-console-consumer.sh --bootstrap-ser

Due to conflict in Jackson jar versions, Oozie job with spark2 action(spark action with spark2 sharelib) may get failed with below error: 2018-06-05 16:53:04,567 [Thread-20] INFO org.apache.spark.SparkContext - Created broadcast 0 from showString at NativeMethodAccessorImpl.java:0 Traceback (most recent call last): File "/grid/9/hadoop/yarn/local/usercache/XXXX/appcache/application_1528131553123_0280/container_e81_1528131553123_0280_01_000002/stg_gl_account_classification_master.py", line 9, in <module> gacm.show() File "/grid/9/hadoop/yarn/local/usercache/XXXX/appcache/application_1528131553123_0280/container_e81_1528131553123_0280_01_000002/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 318, in show File "/grid/9/hadoop/yarn/local/usercache/XXXX/appcache/application_1528131553123_0280/container_e81_1528131553123_0280_01_000002/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__ File "/grid/9/hadoop/yarn/local/usercache/XXXX/

Microservice applications prefer OpenID SSO by existing IDP, for example Keycloak. When these application access Hadoop Services with JWT, Knox should be able to verify and proxy to internal services. This article describe how to configure Knox JWT Provider and customize for your IDP requirement.There are two options.KnoxSSO direct integrate with IDPKnoxSSO, Knox Single Sign On service, integrates with IDP, for example Keycloak. After authenticated by IDP, Knox signs a JWT token for all microservices and Hadoop RESTful services. This solution is complex to configure and Knox could become the performance bottleneck.Knox JWTProvider accept jwt signed by other IDPMicroservices still use current IDP for single sign on, and configure an extra Knox topology to accept other IDP signed JWT token. The second solution has less impact on existing architecture and Knox is only used when access Hadoop services. Here we will focus on this solution, and discuss how to configure Knox and what is the l

Please follow below steps to run spark2 action via Oozie on HDP clusters.https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_spark-component-guide/content/ch_oozie-spark-action.htmlYour Oozie job may get failed with below error because of jar conflicts between 'oozie' sharelib and 'spark2' sharelib.Error:2018-06-04 13:27:32,652 WARN SparkActionExecutor:523 - SERVER[XXXX] USER[XXXX] GROUP[-] TOKEN[] APP[XXXX] JOB[0000000-<XXXXX>-oozie-oozi-W] ACTION[0000000-<XXXXXX>-oozie-oozi-W@spark2] Launcher exception: Attempt to add (hdfs://XXXX/user/oozie/share/lib/lib_XXXXX/oozie/aws-java-sdk-kms-1.10.6.jar) multiple times to the distributed cache. java.lang.IllegalArgumentException: Attempt to add (hdfs://XXXXX/user/oozie/share/lib/lib_20170727191559/oozie/aws-java-sdk-kms-1.10.6.jar) multiple times to the distributed cache. at org.apache.spark.deploy.yarn.Client$anonfun$prepareLocalResources$13$anonfun$apply$8.apply(Client.scala:632) at org.apache.spark.deploy.yarn.Client$an

We are pleased to announce the immediate availability of Hortonworks Data Flow(HDF) version 3.1.2 for both x86 and IBM Power Systems. This version is a maintenance release and includes critical bug fixes for NiFi, MiNiFi, Ranger, Streams Analytics Manager(SAM), Schema Registry, plus more. HDF 3.1.2 includes the following components: Apache Ambari 2.6.2Apache Kafka 1.0.0Apache NiFi 1.5.0NiFi Registry 0.1.0Apache Ranger 0.7.0Apache Storm 1.1.1Apache ZooKeeper 3.4.6Apache MiNiFi Java Agent 0.4.0Apache MiNiFi C++ 0.4.0Hortonworks Schema Registry 0.5.0Hortonworks Streaming Analytics Manager 0.6.0The release and documentation are available at: Hortonworks Data Flow v3.1.2 Download: (link) Hortonworks Data Flow v3.1.2 Documentation: (link)

OBJECTIVE:Updating the log configs of DPS App. Example default log file is set to logs/application.log which can be changed or Updating the log level to DEBUG for troubleshooting. Since DP App will be running in docker we can use docker commands to update them.STEPS: 1. Find the docker container running DP App on the host running DPS. Use "docker ps"[root@dps-node ~]#docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESabd412417907 hortonworks/dlm-app:1.1.0.0-41 "runsvdir /etc/sv" 28 hours ago Up 2 hours 9011/tcp dlm-app62620e578e31 hortonworks/dp-app:1.1.0.0-390 "/bootstrap.sh" 2 days ago Up 16 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 9000/tcp

Prerequisite: Create an Account in S3 and get the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEYAWS Command Line: For the AWS command line to work have AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY configured in ~/.aws/credentials. Something like: [default] aws_access_key_id=$AWS_ACCESS_KEY_ID aws_secret_access_key=$AWS_SECRET_ACCESS_KEYYou might also want to set the region and output in ~/.aws/config. Something like: [default]region=us-west-2output=json Steps: Create a bucket in S3. You can create it online on Amazon Console ( CreatingABucket.html ) or using the command line like: aws s3 mb $BUCKET_NAME Modify the below properties in core-site.xml: fs.defaultFS to s3a://$BUCKET_NAME fs.s3a.access.key to $AWS_ACCESS_KEY_ID fs.s3a.secret.key to $AWS_SECRET_ACCESS_KEY fs.AbstractFileSystem.s3a.imp to org.apache.hadoop.fs.s3a.S3A (HADOOP-11262) You might also want to set the below property in tez-site.xml if you need to run some Example jobs: tez.staging-dir to hdfs://$NN_HOS

OBJECTIVE:Set logging level within DLM to "DEBUG" for all components to provide more detail when troubleshooting DLM/DSS issues such as pairing problems.OVERVIEW:Using some Docker and bash sleight of hand, update the logback configuration in the DLM Docker container to direct the application to log at a more verbose DEBUG level.PREREQUISITES:Root access or sudo on the DLM host.STEPS:1. Find the running Docker container for DLM on the host running DPS/DLM.sudo docker ps Note the docker container ID running dlm-app.2. Get the current log configuration file from the containersudo docker exec -it <container ID> /bin/cat /usr/dlm-app/conf/logback.xml | tee logback.xml.local3. Using vi or your favorite editor, update the logging configuration section below in the new local copy logback.xml.local, setting to "DEBUG" where desired:...<logger name="play" level="DEBUG" /><logger name="application" level="DEBUG" /><root level="DEBUG"> <appender-ref ref="ASYNCFILE" /&

Hive HBase Generate HFilesDemo scripts available at: https://github.com/sakserv/hive-hbase-generatehfilesBelow contains an example of leveraging the Hive HBaseStorageHandler for HFile generation. This pattern provides a means of taking data already stored in Hive, exporting it as HFiles, and bulk loading the HBase table from those HFiles.OverviewThe HFile generation feature was added in HIVE-6473.It adds the following properties that are then leveraged by the Hive HBaseStorageHandler. hive.hbase.generatehfiles - true to generate HFileshfile.family.path - path in HDFS to put the HFiles.Note that for hfile.family.path, the final sudirectory MUST MATCH the column family name.The scripts in the repo called out above can be used with the Hortonworks Sandbox to test and demo this feature.ExampleThe following is an example of how to use this feature. The scripts in the repo above implement the steps below.It is assumed that the user already has data stored in a hive table, for the sake of thi

Cloudbreak 2.4.2 has been released. In addition to a few bug fixes, the release comes with new base and prewarmed images including Ambari 2.6.2 and HDP 2.6.5 (Cloudbreak 2.4.0 and 2.4.1 came with earlier Ambari and HDP versions).For documentation, see https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.4.2/content/index.htmlFor release notes, see https://docs.hortonworks.com/HDPDocuments/Cloudbreak/Cloudbreak-2.4.2/content/releasenotes/index.html

Rack Awareness:Rack awareness is having the knowledge of Cluster topology or more specifically how the different data nodes are distributed across the racks of a Hadoop cluster. The importance of this knowledge relies on this assumption that collocated data nodes inside a specific rack will have more bandwidth and less latency whereas two data nodes in separate racks will have comparatively less bandwidth and higher latency.The main purpose of Rack awareness is:Increasing the availability of data block Better cluster performanceLet us assume the cluster has 9 Data Nodes with replication factor 3.Let us also assume that there are 3 physical racks where these machines are placed:Rack1: DN1;DN2;DN3Rack2: DN4;DN5;DN6Rack3: DN7:DN8;DN9The following diagram depicts an example block placement when HDFS and Yarn are not rack aware:What happens if Rack1 goes down? -> Potentially data in Block1 might be lost Not being Rack aware the entire cluster is thought of placed in default-rack The foll