Resources

I will add the NiFi flow information here tomorrow.For a rough draft, I wanted to show what it could do as it's pretty cool.cat all.txt| jq --raw-output '.["text"]' | syntaxnet/demo.shFrom NiFi I collect a stream of Twitter data and send that to a file as JSON (all.txt). There are many ways to parse that, but I am a fan of the simple command line tool JQ which is an awesome tool to parse JSON from the command line and is available for MacOSX and Linux. So from the Twitter feed I just grab the tweet text to parse with Parsey.Initially I was going to install TensorFlow Syntaxnet (Parsey McParseface) on the HDP 2.4 Sandbox, but Centos 6 and TensorFlow do not play well. So for now the easiest route is to install HDF on a Mac and build Syntaxnet on your Mac.The install instructions are very detailed, but the build is very particular and very machine intensive. It's best to let the build run and go off and do something else with everything else shutdown (no Chrome, VM, editors, ...).Aft

Quite often, hbase users face the following exception when submitting job(s) to secure hbase cluster:2016-05-27 21:16:06,806 WARN [hconnection-0x6be9ad6c-metaLookup-shared--pool2-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] 2016-05-27 21:16:06,806 FATAL [hconnection-0x6be9ad6c-metaLookup-shared--pool2-t1] org.apache.hadoop.hbase.ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'. javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect

In order to make it faster and easier to exchange data from/to Hadoop, a new data integration tool, Skool, has been developed by BT and made available as an open-source product to the wider community. The code is available on the following URL:https://github.com/BT-Plc/skool This tool covers the following aspects:Seamless data transfer from Hadoop into a relational databaseSeamless data transfer from a relational database into HadoopFile transfer and Hive table creation for file based transfers into HadoopAutomatic generation and deployment of file creation scripts and jobs from Hadoop or HivetablesKey FeaturesThe tool generates code which can be automatically executed (or scheduled) for delta and milestone replication with defined frequency of data refresh.The tool is configurable to select tables/columns/files which are to be transferred in or out of Hadoop.Inbuilt optimization of storage to deliver performant code – the tool takes into consideration table size, database pa

This part 1 article is an introduction to Burrow’s design, an overview of the features meant to address some of the known Kafka’s monitoring challenges, a step-by-step tutorial to setup Burrow on sandbox. The part 2 article will explore Consumer lag evaluation rules, HTTP endpoint APIs, email and HTTP notifiers. Stay tuned for part 2 article to be released on May.The source reference is Burrow’s documentation wiki: https://github.com/linkedin/Burrow/wikiWhy Monitoring Kafka? Collecting metrics can help troubleshooting and set alerts on those that require an action.What to Monitor? Process state, memory usage, swap usage, network bandwidth, disk usage, disk IO, under-replicated partitions, offline partitions, active controller brokers, incoming messages per second, incoming/outgoing bytes per second, requests per second, total and split time to process a request, disputed leader elections rate, asynchronous disk log flush and time, unclean leader election rate, number of partitions, I

I call twitter with a filter on some terms, grab the key twitter attributes then call a filter to remove profanities.http://www.purgomalum.com/service/plain?text= or http://www.purgomalum.com/service/json=text work and are free REST API services.For fun, I send the tweet as a search keyword to Guardian using their API (you need to register for a key).http://content.guardianapis.com/search?order-by=newest&q=${tweet}&api-key=StuffNumbers

A remote Linux system can use NFS (Network File System) to mount an HDFS file system and interact with the file system. Before proceeding, it's important to understand that your linux instance is directly accessing your HDFS system through the network, therefore you will incur network latency. Depending on your dataset size, you have to remember you could be potentially processing gigabytes or more of data on a single machine therefore this is not the best approach for large datasets. These steps will show you how to mount and interact with a remote HDFS node within your Linux system:1) The linux system must have NFS installed (CentOS for demo)yum install nfs-utils nfs-utils-lib2) Your HDP cluster must have an NFS Gateway installed (Ambari allows this option with one click)* Keep track of either the FQDN or IP address of the NFSGateway3) In Ambari, under HDFS > Advanced > General set Access time precision = 36000003) Mount the NFS Gateway on your linux system (must be root)moun

EMC Blog Post containing instructions when upgrading HDP 2.2 to HDP 2.3 with IsilonWe have been working with the Isilon engineering team to document the upgrade process from HDP 2.2. to HDP 2.3 when the underlying storage layer is OneFS instead of HDFS. The instructions in the blog post are to be used in conjunction with Ambari instructions on upgrading HDP.Important to note that Isilon is working on making enhancements to OneFS to support rolling upgrade. This will be made available in an upcoming release of OneFS.

Predict Stock Portfolio Gains Using Monte CarloWhy? Why create yet another VaR example? To demonstrate VaR running on a modern architecture that has no vertical limit. This is a functional, immutable, scaleable interpretation of a basic technique commonly used in finance. Code Available here and on github. https://github.com/kirkhas/zeppelin-notebooks/ link to Vlad's article for history of Monte Carlo and VaR - https://community.hortonworks.com/articles/36321/predicting-stock-portfolio-losses-using-monte-carl.htmlSome modifications from original posting include: scala calling Yahoo API directly, alleviating the need for shell scripting and adding interopability between variables. All data loaded dynamically in memory, removing the need to store files (which inherently adds manual customizations to a generic process). Code all in Zeppelin for readability. Visualizations in Zeppelin. Inputs built in using Zep forms so the user can interact with the model. Percentiles not only o

This beautiful demo is courtesy of Vadim Vaks. It utilizes a Lambda architecture built using the Hortonworks Data Platform and Hortonworks Data Flow. The demo shows how a Telecom can manage customer device outages using predictive maintenance and a connected workforce.Overview:Customer devices that are simulated in this specific telecom usecase are SetTopBoxes(STB) in individual homes that might need assistance from a technician, when something goes wrong. Attributes associated with STB are:1. SignalStrength2. Internal Temperature3. StatusLocation of technician is tracked and plotted on MapUI using the latitude and longitudes.Two cycles of operation of STBs are:1. Normal Cycle: When status is normal and internal temperature of STB fluctuates up and down2. Failure Cycle: When status is not normal and internal temperature of STBox incrementally goes high until 109 degrees. Step 1: Prerequisites for the Demo Set Up: For instructions to install this demo on an HDP 2.4 Sandbox,a good p

Platform: HDP 2.4 (Sandbox) Hadoop version: 2.7 OS: CentOS 6.8 For every data node that Spark will run on, Python 2.7 and all dependent libraries that your code uses must be installed on each data node - e.g.Pandas, Matplotlib, Scipy Below are the steps I performed to get iPython up and running. My notebook jobs are executed via YARN. I did this install as ‘root’ user. STEP 1 Install/configure the following 5 iPython dependencies by typing in the following commands: yum install nano centos-release-SCL zlib-devel yum install bzip2-devel openssl-devel ncurses-devel yum install sqlite-devel readline-devel tk-devel yum install gdbm-devel db4-devel libpcap-devel xz-devel yum install libpng-devel libjpg-devel atlas-devel STEP 2 iPython has a requirement for Python 2.7 or higher. Check which version of Python you are using:In my case, I had an older version of Python so I then had to install a new version. Install the “Development tools” dependency for Python 2.7Command to run: yum gro

Most enterprises require data transport between applications to be via RESTful interfaces in order to facilitate cross application and organization communication and to reduce the degree of data integration coupling between applications. The summary table below identifies out the major HDP services ability to support RESTful communication over HTTP. HTTP Operations Supported HDP Service Read from HTTP Write to HTTP Retrieve Meta information Execute Queries Flume X X Sqoop X X X X Hive X X X X HBase X X X X Storm Java programming Java programming Java programming X Spark Spark Programming Spark Programming Spark Programming Spark Programming Kafka X (requires plugin) X (requires plugin) X (requires plugin) X (requires plugin)

LDAP Usage Hadoop may be configured to use LDAP as the source for resolving an authenticated user's list of group memberships. A common example where Hadoop needs to resolve group memberships is the permission checks performed by HDFS at the NameNode. The Apache documentation's HDFS Permissions Guide contains further discussion of how the group mapping works: the NameNode calls a configurable plugin to get the user's group memberships before checking permissions.Despite that document's focus on group resolution at the NameNode, many other Hadoop processes also call the group mapping. The information in this document applies to the entire ecosystem of Hadoop-related components.As described in that document, the exact implementation of the group mapping is configurable. Here is the documentation of the configuration property from core-default.xml and its default value. <property> <name>hadoop.security.group.mapping</name> <value>org.apache.hadoop.security

https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/host_scripts/alert_disk_space.pyvi /var/lib/ambari-server/resources/host_scripts/alert_disk_space.py# defaults in case no script parameters are passedMIN_FREE_SPACE_DEFAULT = 5000000000L #5GBPERCENT_USED_WARNING_DEFAULT = 50PERCENT_USED_CRITICAL_DEFAULT = 80You can change the above parameter to avoid alerting at 80% threshold reached in case it needs to be aletered to 85 or 90

SYMPTOM: Specifying a valid group in Ranger Knox policy results in a 403 authorization error.ERROR: The result is a 403 forbidden error. ROOT CAUSE: Most likely the cause of this issue is that the topology is not setup in Knox for ldap groups to be passed to Ranger from the Knox plugin. RESOLUTION: Make sure the following values are present and correct in the topology:<!-- changes needed for group sync--><param> <name>main.ldapRealm.authorizationEnabled</name> <value>true</value></param><param> <name>main.ldapRealm.groupSearchBase</name> <value>OU=MyUsers,DC=AD-HDP,DC=COM</value></param><param> <name>main.ldapRealm.groupObjectClass</name> <value>group</value></param><param> <name>main.ldapRealm.groupIdAttribute</name> <value>cn</value></param> ALTERNATIVE SOLUTION:Instead of getting the above LDAP group settings w

I've received a couple of "how to" questions after folks successfully deploy the Hortonworks Data Platform Standard on Microsoft's Azure. I've collected my responses here as a reference to others:What is Hortonworks Data Platform (HDP) Standard?It is a muti-node HDP 2.4/Ambari 2.2.1 cluster on Microsoft’s Azure Cloud launched in a few mouse clicks. Hortonworks aims to match this service with the latest version of HDP. You provide: your nameemailpasswords or ssh key the number of nodes the VM types for your masters and workers should your cluster be HA or notas shown in this screenshot:Where's Ambari?Once the cluster is successfully deployed, the Azure Dashboard will go from something like:tothe Ambari service is located on the first master server. In the Azure portal goto Resource group, <your resource group name which was selected at first "Basics" step>, master1, settings and look for Public IP address. Use a web browser to access Ambari with:<master1 Public IP address>:

When adding a net new data source to Metron, the first step is to decide how to push the events from the new telemetry data source into Metron. You can use a number of data collection tools and that decision is decoupled from Metron. However, we recommend evaluating Apache Nifi as it is an excellent tool to do just that (this article uses Nifi to push data into Metron). The second step is to configure Metron to parse the telemetry data source so that downstream processing can be done on it. In this article we will walk you through how to perform both of these steps. In the previous article of this blog series, we described the following set of requirements for Customer Foo who wanted to add the Squid telemetry data source Into Metron. The proxy events from Squid logs need to be ingested in real-time. The proxy logs must be parsed into a standardized JSON structure that Metron can understand. In real-time, the squid proxy event must be enriched so that the domain names are enriched wit

The documentation is straight forward BUT based on my experience, we always end up some kind of LDAP error messages.This article is walking through the Hive and LDAP integration.HDP 2.3.4Ambari 2.2Security workshop was used to setup the openLDPA Most important: 2016-02-10 08:02:50,705 ERROR [HiveServer2-Handler-Pool: Thread-39]: transport.TSaslTransport (TSaslTransport.java:open(315)) - SASL negotiation failurejavax.security.sasl.SaslException: Error validating the login [Caused by javax.security.sasl.AuthenticationException: Error validating LDAP user [Caused by javax.naming.AuthenticationException: [LDAP: error code 32 - No Such Object]]]at org.apache.hive.service.auth.PlainSaslServer.evaluateResponse(PlainSaslServer.java:109)I kept hitting the above error because of wrong baseDNCorrect setting:Test:

SYMPTOM: KMS gets 500 error when decrypting files when being accessed from another one way trust realm. ERRORS: Command line error: [root@support ~]$ hdfs dfs -cat /zone_encr3/abc1.txt cat: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 500, message: Internal Server Error KMS Stack Trace error: 2016-05-04 15:44:21,677 ERROR [webservices-driver] - Servlet.service() for servlet [webservices-driver] in context with path [/kms] threw exceptionorg.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to user06@HDP.COM at org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:389) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:377) at org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:347) at java.secu

1. Make sure webhbase is running:[root@ambari-server]# curl http://localhost:60080/versionrest 0.0.3 [JVM: Oracle Corporation 1.8.0_60-25.60-b23] [OS: Linux 2.6.32-504.el6.x86_64 amd64] [Server: jetty/6.1.26.hwx] [Jersey: 1.9] 2. If the webhdfs REST daemon is not running then you will need to start it. [root@ambari-server]# /usr/hdp/current/hbase-client/bin/hbase-daemon.sh start rest -p 60080starting rest, logging to /var/log/hbase/hbase-root-rest-ambari-server.support.com.out [root@ambari-server security]# ps -ef | grep 60080root 19147 1 0 20:17 pts/1 00:00:00 bash /usr/hdp/current/hbase-client/bin/hbase-daemon.sh --config /usr/hdp/current/hbase-client/bin/../conf foreground_start rest -p 60080root 19161 19147 28 20:17 pts/1 00:00:03 /usr/jdk64/jdk1.8.0_60/bin/java -Dproc_rest -XX:OnOutOfMemoryError=kill -9 %p -Dhdp.version=2.3.4.0-3485 -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -Djava.security.auth.login.config=/usr/hdp/current/hbase-re

1. Username and password input fields in ranger should be the ldap user and password authenticating through knox. 2. Configure the admin topology to meet your configuration needs by default it points to the demo ldap server on knox server. 3. To test manually, login to the URL directly using ldap credentials from the ranger server. https://<knoxhost>:8443/gateway/admin/api/v1/topologies 4. Make sure the ranger server trusts the knox certificate being presented; if it doesn't; you will need to add the knox gateway trust certificate to the ranger/java truststore. 5. Currently knox service lookups don't work in ranger and a bug has been filed to fix this. Currently, as of HDP 2.4, only topology lookup in ranger works.

Predicting stock portfolio losses using Monte Carlo simulation in Spark Summary Have you ever asked yourself: what is the most money my stock holdings could lose in a single day? If you own stock through a 401k, a personal trading account, or employer provided stock options then you should absolutely ask yourself this question. Now think about how to answer it. Your first guess maybe to pick a random number, say 20%, and assume that is the worst case scenario. While simple, this is likely to be wildly inaccurate and certainly doesn’t take into account the positive impacts of a diversified portfolio. Surprisingly, a good estimate is hard to calculate. Luckily, financial institutions have to do this for their stock portfolios (called Value at Risk (VaR)), and we can apply their methods to individual portfolios. In this article we will run a Monte Carlo simulation using real trading data to try to quantify what can happen to your portfolio. You should now go to your broker website

NULL

Apache Storm is a distributed real-time computation system for processing large volumes of high-velocity data. Debugging a distributed system is inherently hard since there are many moving parts spread across a large number of hosts in a cluster. Tracing failures to a particular component or a host in the system is hard and requires one to collect and analyze a lot of logs and debug/trace processes running in the distributed cluster.In the past the support for debugging topologies in an Apache storm cluster was very minimal. Users would have to first turn on debug logs, restart the workers, redeploy the topologies and then collect logs from different worker nodes and analyze them. If they wanted to take a jstack dump or profile the workers, they would have to login to the worker hosts and run the commands manually which was painful. If one wanted to inspect the tuples flowing through the topology pipeline, they would have to add special “debug” bolts to log the tuples to some log and t

Modern CPU and NUMAProcessor clock speed has increased dramatically in the recent years, however, the CPU needs to be supplied with a large amount of memory bandwidth to use its processing power effectively. Even a single CPU running a memory-intensive workload, such as a scientific computing application, can be constrained by memory bandwidth. This problem is amplified on symmetric multiprocessing (SMP) systems, where many processors must compete for bandwidth on the same system bus.Non-uniform memory access (NUMA) is a computer memory design used in multiprocessing, where the memory access time depends on the memory location relative to the processor. NUMA systems have more than one system bus and they can harness large numbers of processors in a single system, linking several small, cost-effective nodes using a high-performance connection.. Each node contains processors and memory, much like a small SMP system. However, an advanced memory controller allows a node to use memory on

When using smartsense 1.2 or below in conjunction with OpenJDK, you get the following error upon startup. It's a none issue which will be resolved in the next smart sense version. Traceback (most recent call last): File "/usr/sbin/hst-agent.py", line 420, in <module> main(sys.argv) File "/usr/sbin/hst-agent.py", line 397, in main setup(options) File "/usr/sbin/hst-agent.py", line 323, in setup server_hostname = get_server_hostname(server, tries, try_sleep, options.quiet) File "/usr/sbin/hst-agent.py", line 107, in get_server_hostname hostname = validate_server_hostname(default_hostname, tries, try_sleep) File "/usr/sbin/hst-agent.py", line 125, in validate_server_hostname elif not register_agent(server_hostname): File "/usr/sbin/hst-agent.py", line 143, in register_agent if not server_api.register_agent(agent_version): File "/usr/hdp/share/hst/hst-agent/lib/hst_agent/ServerAPI.py", line 104, in register_agent content = self.call(request) File "/usr/hdp/share/hst/hst-agent