Resources

Symptoms:Users may see an incorrect host domain name that may also include unexpected markers e.g. hadoopnode1.kjtijasdh.com , hadoopnode1.??kjtijasdh??.comDiagnosis:All data collected by HST Agents is anonymized via a process that uses a set of configurations along with anonymization rules to mask potentially sensitive data. The configurations include definitions for anonymization markers that are used to identify the start and end of a piece of data that has been anonymized via the rules. Certain anonymized content, such as host domain names, can later be deanonymized at the time of sending SmartSense recomendations so that the host domain is recognizable to the user. Typically the configured markers are unicode characters such as “¶” and “‡” which can can be incorrectly identified if the locale on the host running the HST Agent is not setup correctly which results in ?? as a default replacement.Solution:Making sure that you are using the same user that runs the HST agent on the pr

This issue is caused due to the size of "~/.beeline/history" file in user's home directory. While we run big queries, it will flood the history file and slow down beeline on start up and shutdown. To resolve this issue, please move "~/.beeline/history" file and retry beeline. In the lower versions (below 2.1.0), we can have a cron job to delete "history" file periodically. This issue has been fixed in latest Hive versions by adding "--maxHistoryRows". Please check the below JIRA for more details. https://issues.apache.org/jira/browse/HIVE-15166

The standard solutionLet's say you want to collect log messages from an edge cluster with NIFI, and push it to a central NIFI cluster via the Site To Site (S2S) protocol. This is exactly what NIFI is designed for, and results in a simple flow setup like this: A processor that tails the log filewhich sends it's flowfiles to a remote process group which is configured with the FQDN URL of the central NIFI cluster on the central NIFI cluster an INPUT port is definedand from that input port the rest of the flow is doing it's thing with the incoming flow files, like filtering, transformations and eventually sinking it into kafka, HDFS or SOLR.The NIFI S2S protocol is used for the connection between the edge NIFI cluster and the central nifi cluster.which PUSHES the flowfiles from the edge cluster to the central NIFI cluster. And now with a firewall blocking incoming connections in betweenThis standard setup however assumes the central NIFI cluster has a public FQDN and isn't behind a firewal

Error Stack is below.Package Manager failed to install packages. Error: (4, 'Interrupted system call')Traceback (most recent call last): File "/var/lib/ambari-agent/cache/custom_actions/scripts/install_packages.py", line 376, in install_packages retry_count=agent_stack_retry_count File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 58, in action_upgrade self.upgrade_package(package_name, self.resource.use_repos, self.resource.skip_repos) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 56, in upgrade_package

Prior to release 1.2, Apache NiFi supports the notion of pluggable "Schema Registry”, and ConvertXToY processors including but not limited to handling format conversions between CSV, JSON and Avro. Starting from 1.2, NiFi supports a new concept - an abstraction of “record”. Any FlowFile can contain one or more record objects of a given format and schema. NiFi then offers a series of controller services which provide a ‘RecordReader' interface and will have immediate support for Avro, JSON Tree, JSON Path, Grok, and CSV as provided source formats. in the same 1.2 release, NiFi will support a series of controller services which provide a “RecordWriter" interface and will have immediate support for Avro, JSON, Text, CSV.Each of the Reader and Writer objects will support interaction with another Controller Service which will be referencing a schema registry for retrieving schemas when necessary. An Avro schema registry and an HWX schema registry will be immediately available in Apache NiFi

Below is a short Scala program and SBT build script to generate a Spark application to submit to YARN that will run on the HDP Sandbox with Spark 1.6.We consume KAFKA messages in microbatches of 2 seconds. We set a number of parameters for performance of the SparkSQL write to HDFS including enabling Tungsten, using Snappy compression, Backpressure and the KryoSerializer.We setup the Spark Context and the Spark Streaming Context, then use the KafkaUtils to prepare a Stream of generic Avro data records from a byte array. From there we then convert to a Scala Case Class that models the Twitter tweet from our source. We convert to a DataFrame, register it as a Temp Table and then save to ORC File, AVRO, Parquet and JSON.To Build this Application:sbt clean assembly Create Kafka Topiccd /usr/hdp/current/kafka-brokerbin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic meetupTo Prepare Hadoop Environment# Create HDFS Directories (also avro,

Where in the world is Tim Spann, The NiFi Guy?I recommend the BU-353-S4 USB GPS, it works well with Raspberry PIs and is very affordable. Connecting this to a RPIWZ, I can run this on a small battery and bring this everywhere for location tracking. Put it on your delivery truck, chasis, train, plane, drone, robot and more. I'll track myself so I can own the data. What do these GPS Fields Mean?EPS - Error Estimate in Meter/SecondEPX = Estimated Longitude Error in MetersEPV = Estimated Vertical Error in MetersEPT = Estimated Timestamp Error Speed = Speed !!!Climb = Climb (Positive) or Sink (Negative) rate in meters per second of upwards or downwards movement.Track = Course over ground in degrees from True NorthMode = NMEA mode; values are 0 - NA, 1 - No Fix, 2D and 3D.My question is if you already have an estimate of the error, do something about it!!!Python Walk ThroughFirst install the utilities you need for GPS and Python. We also install NTP to get as accurate time as possib

Here are the steps to get pyspark working on SHCa) add the following into Ambari -> Spark -> Configs -> Advanced spark-env -> spark-env template export SPARK_CLASSPATH=/usr/hdp/current/hbase-client/lib/hbase-common.jar:/usr/hdp/current/hbase-client/lib/hbase-client.jar:/usr/hdp/current/hbase-client/lib/hbase-server.jar:/usr/hdp/current/hbase-client/lib/hbase-protocol.jar:/usr/hdp/current/hbase-client/lib/guava-12.0.1.jar:/usr/hdp/current/hbase-client/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.5.3.0-37/spark/conf/b) kinit as e.g. hbase userc) run$ pyspark --packages com.hortonworks:shc:1.0.0-1.6-s_2.10 --repositories http://repo.hortonworks.com/content/groups/public/d) call each line separately:from pyspark.sql import Rowdata = range(0,255)rdd = sc.parallelize(data).map(lambda i : Row(name=i,age=i))import jsoncat = json.dumps({"table":{"namespace":"default", "name":"dk", "tableCoder":"PrimitiveType"},"rowkey":"key","columns":{"name":{"cf":"rowkey", "col":"key", "type":"

One of the search use cases that I’ve been introduced to would require the ability to index text such as scanned text in png files. I set out to figure out how to do this with SOLR. I came across a couple pretty good blog posts, but as usual, you have to put together what you learn from multiple sources before you can get things to work correctly (or at least that’s what usually happens for me). So I thought I would put together the steps I took to get it to work.I used HDP Sandbox 2.3.Step-by-step guideInstall dependencies - this will provide you support for processing pngs, jpegs, and tiffsyum install autoconf automake libtoolyum install libpng-develyum install libjpeg-develyum install libtiff-develyum install zlib-develDownload Leptonica, an image processing librarywget http://www.leptonica.org/source/leptonica-1.69.tar.gzDownload Tesseract, an Optical Character Recognition enginewget http://tesseract-ocr.googlecode.com/files/tesseract-ocr-3.02.02.tar.gzEnsure proper variables and p

To get an idea of the write performance of a Spark cluster i've created a Spark version of the standard TestDFSIO tool, which measures the I/O performance of HDFS in your cluster. Lies, damn lies and benchmarks, so the goal of this tool is providing a sanity check of your Spark setup, focusing on the HDFS writing performance, not on the compute performance. Think the tool can be improved? Feel free to submit a pull request or raise a Github issueGetting the Spark Jar Download the Spark Jar from here: https://github.com/wardbekker/benchmark/releases/download/v0.1/benchmark-1.0-SNAPSHOT-jar-with-dependencies.jar It's build for Spark 1.6.2 / Scala 2.10.5Or build from from source $ git clone https://github.com/wardbekker/benchmark $ cd benchmark && mvn clean packageSubmit args explains <file/partitions> : should ideally be equal to recommended spark.default.parallelism (cores x instances). <bytes_per_file> : should fit in memory: for example: 90000000. &l

IntroductionRecently I was asked how to Monitor and alert while MonitorDiskUsage or MonitorMemory Reporting Task WARNINGS are generated when it exceeds a predefined threshold, I am trying to capture the steps to implement the same.Prerequisites To test this, Make sure HDF-2.x version of NiFi is up an running.You Already have a MonitorDiskUsage/MonitorMemory Reporting task created and configured, set a lower threshold to force generate the alert while testing. 3. Make a note of the Reporting Task uuid to be monitored:Creating a Flow to Monitor Reporting Task. Drop a GenerateFlowFile to execute every 5mins, with 1 byte file to trigger the flow. 2. Drop an UpdateAttribute processor to the canvas with below configuration:NIFI_HOST : <Your NiFi FQDN>NIFI_PORT : <Your NiFi http port>REPORTING-TASK-UUID : <Your MonitorDiskUsage or MonitorMemory controller service uuid>3. Connect Success relationship of GenerateFlowFile to UpdateAttribute.4. Drop a InvokeHTTP proce

What kind of text or unstructured data are you collecting within your company? Are you able to fully utilize this data to enhance predictive models, build reports/visualizations, and detect emerging trends found within the text? Hadoop enables distributed, low-cost storage for this growing amount of unstructured data. In this post, I'll show one way to analyze unstructured data using Apache Spark. Spark is advantageous for text analytics because it provides a platform for scalable, distributed computing. When it comes to text analytics, you have a few option for analyzing text. I like to categorize these techniques like this: Text Mining (i.e. Text clustering, data-driven topics) Categorization (i.e. Tagging unstructured data into categories and sub-categories; hierarchies; taxonomies) Entity Extraction (i.e. Extracting patterns such as phrases, addresses, product codes, phone numbers, etc.) Sentiment Analysis (i.e. Tagging positive, negative, or neutral with varying levels of sentime

Scenario: Trying to add new columns to an already partitioned Hive table. Problem: The newly added columns will show up as null values on the data present in existing partitions. Solution: One of the workaround can be copying/moving the data in a temporary location,dropping the partition, adding back the data and then adding back the partition. It works and the new column picks up the values. But for big tables this is not a viable solution. Best approach: We need to construct the alter statement to add columns with CASCADE option as follows: ALTER TABLE default.test_table ADD columns (column1 string,column2 string) CASCADE; From the Hive documentation: “ALTER TABLE CHANGE COLUMN with CASCADE command changes the columns of a table's metadata, and cascades the same change to all the partition metadata. RESTRICT is the default, limiting column change only to table metadata.” I found out that this option is not at all in wide use and can help who face this situation. Thanks

Issue:User has a valid kerberos ticket, when he's trying to connect to Hive CLI - the session throws the below error[karthick@mrt1 ~]$ hive WARNING: Use "yarn jar" to launch YARN applications. Logging initialized using configuration in file:/etc/hive/2.4.3.0-227/0/hive-log4j.properties Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No cluster diagnostics found. at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:544) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org

Ambari version 2.5 provides a new Upgrade History page. This is useful in many ways such as:Determine the history of upgrades that your cluster has undergone (and any issues).Collect any specific logs from within the web UI without the need to login to the respective host.Determine time taken for upgrade, which gives an indication of the time required for the next upgrade.In short, Upgrade planning and troubleshooting are two aspects addressed through this pageHere is how the ‘Upgrade History’ page looks:Besides the above UI, you could also retrieve the entire Upgrade output through a couple of Ambari API calls

In this article I will be explaining about creating oozie custom action node to clone a git repository to a required file path.Git repo location and file path will be taken as input while running the workflow.And we will use ambari's workflow manager view to create a workflow using newly created action node. If you are new to workflow manager, then this would be a good starting point.Details about custom action nodes could be found in oozie docs here.We need to follow some prerequisites before we jump to workflow manager view and start using custom action nodes.Step 1 : Implementing Oozie custom action handler : For this article, I will be creating a custom action which clones a git repository . This implementation extends ActionExecutor class (provided by Oozie) and overrides the required methods. This implementation follows Oozie documentation and implements all required methods.import java.io.File;import org.apache.oozie.ErrorCode;import org.apache.oozie.action.ActionExecutor;import

Sometimes when running Hive on Tez queries, such as "select * from table" large output files are created that may swamp your local disk. Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1444941691373_0130_1_00, diagnostics=[Task failed, taskId=task_1444941691373_0130_1_00_000007, diagnostics=[TaskAttempt 1 failed, info=[Error: Failure while running task:org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1444941691373_0130_1_00_000007_1_10003_0/file.outat org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:402)......], ....Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:106, Vertex vertex_1444941691373_0130_1_01 [Map 3] killed/failed due to:null]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVerti

HORTONWORKS : SCRIPT HOW TO DISABLE THP REDHAT 7 ?#!/bin/bash### BEGIN INIT INFO# Provides: disable-transparent-hugepages# Required-Start: $local_fs# Required-Stop: # Author: Amine Hallam# Default-Start: 2 3 4 5# Default-Stop: 0 1 6# Short-Description: Disable Linux transparent huge pages# Description: Disable Linux transparent huge pages, to improve # database performance.### END INIT INFOcase $1 in start) if [ -d /sys/kernel/mm/transparent_hugepage ]; then thp_path=/sys/kernel/mm/transparent_hugepage elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then thp_path=/sys/kernel/mm/redhat_transparent_hugepage else return 0 fi echo 'never' > ${thp_path}/enabled echo 'never' > ${thp_path}/defrag re='^[0-1]+$' if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]] then # RHEL 7 echo 0 > ${thp_path}/khugepaged/defrag else # RHEL 6 echo 'no' > ${thp_path}/khugepaged/defrag fi unset re unset thp_path

Mindwave NeuroskyThe Mindwave Neurosky is a headset that allows you to record your brainwaves using EEG technology. In this article we show you how to ingest these brainwaves with NIFIMindwave Neurosky driver installation for OSX Sierra Download and install the latest driver from http://download.neurosky.com/public/Products/MindWave%20headset/RF%20driver%20for%20Mac/MindWaveDriver5.1.pkg After the driver is installed, download and install the latest MindWave Manager from http://download.neurosky.com/public/Products/MindWave%20headset/RF%20driver%20for%20Mac/MindWave%20Manager4.0.4.zip Launch the MindWave Manager, navigate to "Pairing" section and click the "Search for MindWave", then follow the instructions to pair the headset.Install NIFI on OSX Sierra with Homebrew Install Homebrew from the terminal: /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" Install NIFI (time of writing v1.1.2): brew install nifiImport NIFI Flow Template An

The Spark Component Guide and Command Line Installation Guide were updated to reflect new Spark features. Here are links to several of the latest features:Support for Spark 2, documented in several topics including: Installing Spark Using AmbariInstalling and Configuring Apache Spark 2 (manual installation)Running SparkConfiguring Spark2 for Wire EncryptionAutomating Spark Jobs with Oozie Spark ActionUsing Livy with Spark Versions 1 and 2Livy API information, in Submitting Spark Applications Through LivyEnabling Spark SQL user impersonation for the Spark Thrift Server (doAs support), in Configuring the Spark Thrift ServerThe Zeppelin Component Guide was updated with additional details and examples for configuring Zeppelin with LDAP/AD and Kerberos security; see Configuring Zeppelin Security. In addition, the documentation for interpreters and user impersonation was extended. Portions of this information that apply to HDP 2.5 were also added to the Security chapter in the HDP 2.

Please consider the following for this install;IBM Power servers are on centos 7The install is performed by using a non-root userthere is no access to internet or proxy to remote repos, we installed a local repos------------------------------------------------------------prerequisite-------------------------------------------------------------#Check the Maximum Open File Descriptors#The recommended maximum number of open file descriptors is 10000, or more. To check the current value set for the maximum number of open file descriptors, execute the following shell commands on each host:ulimit -Snulimit -Hn#If the output is not greater than 10000, run the following command to set it to a suitable default:ulimit -n 10000#SElinuxsudo setenforce 0sudo sh -c 'sudo sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config'sudo sh -c 'sudo sed -i 's/SELINUX=permissive/SELINUX=disabled/g' /etc/selinux/config'sudo umask 0022sudo echo umask 0022 >> /etc/profile---------------------

GOALTo change the default log location for HUE from /var/log/hue to elsewhere.StepsDefault HUE service writes the log files to /var/log/hue directory. This is somehow enforced in the code which states this The ``log_dir`` will replace the %LOG_DIR% in log.conf. If not specified, we look for the DESTKOP_LOG_DIR environment variable, and then default to the DEFAULT_LOG_DIR.However, trying to set either variables i.e. DESKTOP_LOG_DIR and DEFAULT_LOG_DIR don't seem to work. We can however change the %LOG_DIR% with an absolute path within /etc/hue/conf/log.conf. Here is an output of the log.conf replaced with absolute path for the log directory.#args=('%LOG_DIR%/access.log', 'a', 1000000, 3)args=('/opt/log/hue/access.log', 'a', 1000000, 3)--#args=('%LOG_DIR%/error.log', 'a', 1000000, 3)args=('/opt/log/hue/error.log', 'a', 1000000, 3)--#args=('%LOG_DIR%/%PROC_NAME%.log', 'a', 1000000, 3)args=('/opt/log/hue/%PROC_NAME%.log', 'a', 1000000, 3)--#args=('%LOG_DIR%/shell_output.log', 'a', 1000000

ErrorException in thread "main" java.lang.ArrayIndexOutOfBoundsException: 110 at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.planReadPartialDataStreams(RecordReaderImpl.java:914) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readPartialDataStreams(RecordReaderImpl.java:958) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:793) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:986) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1019) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:205) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:598) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rows(ReaderImpl.java:585) at org.apache.hadoop.hive.ql.io.orc.FileDump.printMetaDataImpl(FileDump.java:291) at org.apache.hadoop.hive.ql.io.orc.FileDump.printMetaData(FileDump.java:261) at org.apache.hadoop.hi

When running a distcp process from HDFS to AWS S3, credentials are required to authenticate to the S3 bucket. Passing these into the S3A URI would leak secret values into application logs. Storing these secrets in core-site.xml is also not ideal because this means any user with hdfs CLI access can access the S3 bucket to which these AWS credentials are tied.The Hadoop Credential API can be used to manage access to S3 in a more fine-grained way.The first step is to create a local JCEKS file in which to store the AWS Access Key and AWS Secret Key values:hadoop credential create fs.s3a.access.key -provider localjceks://file/path/to/aws.jceks<enter Access Key value at prompt>hadoop credential create fs.s3a.secret.key -provider localjceks://file/path/to/aws.jceks<enter Secret Key value at prompt>We'll then copy this JCEKS file to HDFS with the appropriate permissions.hdfs dfs -put /path/to/aws.jceks /user/admin/hdfs dfs -chown admin:admin /user/admin/aws.jcekshdfs dfs -chmod 400

Problem: There's a known bug in Ambari 2.4 and 2.5 that causes "ambari-server upgrade" to fail if the agent RPM is not upgraded first.E.g. stack trace:Using python /usr/bin/python Setup ambari-server Traceback (most recent call last): File "/usr/sbin/ambari-server.py", line 33, in from ambari_server.dbConfiguration import DATABASE_NAMES, LINUX_DBMS_KEYS_LIST File "/usr/lib/python2.6/site-packages/ambari_server/dbConfiguration.py", line 28, in from ambari_server.serverConfiguration import decrypt_password_for_alias, get_ambari_properties, get_is_secure, \ File "/usr/lib/python2.6/site-packages/ambari_server/serverConfiguration.py", line 36, in from ambari_commons.os_utils import run_os_command, search_file, set_file_permissions, parse_log4j_file ImportError: cannot import name parse_log4j_fileCause: This occurs because os_utils.py and other python files inside of /usr/lib/ambari-agent/lib/ambari_commons are upgraded by the agent's RPM and are used by the server's scripts to find which d