Resources

There's a number of tools from the command line, that I like to use from NiFi as part of a Big Data Flow.The first tool I wanted to use was Ghostscript.On Centos/RHEL, you can install it via:yum install ghostscriptI use GhostScript to extract content from PDFs (this can be passed in from an existing flow using ExecuteStreamCommand). It then outputs to the Standard Output the text from those files. run.sh gs -dBATCH -dNOPAUSE -sDEVICE=txtwrite -dFirstPage=1 -dLastPage=500 -sOutputFile=- $@Output from the Hadoop documentation:GPL Ghostscript 9.07 (2013-02-14)Copyright (C) 2012 Artifex Software, Inc. All rights reserved.This software comes with NO WARRANTY: see the file PUBLIC for details.Processing pages 1 through 8.Page 1Can't find (or can't open) font file NimbusSanL-ReguItal.Querying operating system for font files...Loading NimbusSanL-ReguItal font from /usr/share/fonts/default/Type1/n019023l.pfb... 3984660 2473586 2498328 1197484 3 done.Loading NimbusSanL-Bold

I was working on one of the case and came across an issue where balancer utility was not working in namenode HA environment.HDP Version - 2.3.2.0Ambari Version - 2.1.2Issue - balancer was failing with below error:[hdfs@slave1 ~]$ hdfs balancer 15/11/20 11:46:10 INFO balancer.Balancer: namenodes = [hdfs://mycluster, hdfs://mycluster:8020] 15/11/20 11:46:10 INFO balancer.Balancer: parameters = Balancer.Parameters [BalancingPolicy.Node, threshold = 10.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, run during upgrade = false] [..] 15/11/20 11:46:11 INFO block.BlockTokenSecretManager: Setting block keys 15/11/20 11:46:11 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec java.io.IOException: Another Balancer is running.. Exiting ... Nov 20, 2015 11:46:11 AM Balancing took 1.883 seconds Resolution(workaround) -1. Copy /etc/hadoop/conf/* /home/hdfs/copied_conf/ 2. Edit /home/hdfs/copied_conf/hdfs-site.xml and delete below property &lt

When executing Step 3 of the Ambari installation wizard "Confirm Hosts", Ambari will (by default) SSH to each node and start an instance of the Ambari Agent process. In some cases, it is possible that the local RPM database is corrupted and this registration process will fail.The error message in Ambari would look something like:INFO:root:Executing parallel bootstrapERROR:root:ERROR: Bootstrap of host myhost.mydomain fails because previous action finished with non-zero exit code (1)ERROR MESSAGE: tcgetattr: Invalid argumentConnection to myhost.mydomain closed.STDOUT: Error: database disk image is malformedError: database disk image is malformedDesired version (2.5.0.0) of ambari-agent package is not available.tcgetattr: Invalid argumentConnection to myhost.mydomain closed.In this case, the local RPM database is malformed and all actions to alter the installed packages on the system will fail until the database is rebuilt.This can be done by the following commands as root on the host re

After creating a cluster on the HDCLoud for AWS, you may notice that certain ports are not opened by default, so you may need to manually open these ports by editing the inbound access on the security group.In this tutorial, I will show you how to open YARN Resource Manager UI (8088) and Hive UI (10502) ports by manually editing the inbound access on the master node security group.Let’s get started!1. On AWS, from the Services menu, select EC2 to navigate to the EC2 console:2. In the left pane, in the INSTANCES section, click on Instances.Note: If you can’t see your instances, check the top right corner to make sure that you are in the correct region.3. Identify the instance corresponding to your master node and. The name of the instance should be <your-cluster-name>-1-master. Next, select that instance. This will allow you to see the Description tab, which includes the link to the security group configuration:4. Click on the security group URL to open the Security Group section.

Use Case: Store log data in Hadoop Data Lake and Send Curated, Reduced Set to Sumologic via REST APIIntegrationThe integration point for sending log data to Sumologic is their HTTP Source. To send data you must setup an HTTP Source in Sumologic from your web console as shown below. Take the HTTP string they give to you and put into an InvokeHTTP processor with POST, it will look something like this: https://endpoint1.collection.us2.sumologic.com/receiver/v1/http/ZaLongCodeLongI noticed IP Address in the data, so I decided to parse it out:${regex.6:substringAfterLast('source ip: '):replaceAll('\)','')}Then send it to MaxMind for processing. The MaxMind GeoIP free database is easy to download and use with NiFi. Just add the GeoIP processor and connect the field and the file location.Finally displaying and charting data is up next, easy as pie in Zeppelin. Just query my Phoenix data. The flow is a bit long as I am using RegEx to convert the logs from NiFi Log4J format to indiv

If you are on a Mac and run into the following error when running start-all.sh:% sh start-all.shstarting org.apache.spark.deploy.master.Master, logging to ...localhost: ssh: connect to host localhost port 22: Connection refusedYou need to enable "Remote Login" for your machine. From System Preferences, select Sharing, and then turn on Remote Login.

SAS users are very excited about leveraging Hadoop with their existing deployments. This article will cover the basic concepts of SAS Access to Hadoop, and add on product from SAS which can be deployed into existing SAS enviroments. This product give a SAS user the ability to leverage HDFS storage, access HiveServer2 and execute inline within a SAS program, HDFS, Pig, Hive and MapReduce programs. Lets start by discussing how to leverage HDFS storage.1) Leveraging HDFS for flat files using the SAS Filename StatementThe SAS Filename statement allows a SAS programmer to setup a pointer to an inbound or outbound filesystem directory. With Hadoop, the SAS Filename statement can reference an HDFS directory. Once the file reference is established, this file reference can be used within a SAS Data Step on an Infile or File statement. This enables SAS programmers to read and write flat files to and from HDFS inline within their programs.2) Leveraging HDFS for SAS Libraries using the Libna

Issue:When you are trying to run queries from Tableau, SQuirrel etc.. connecting to Hiveserver2 via JDBC/ODBC driver in a Knox environment, you might intermittently come across the above error. This error pops-up especially when the query has more columns in the select statement. Interestingly, the same query works fine from hive cli and beeline cli.Resolution:This is a bug in Knox - https://issues.apache.org/jira/browse/KNOX-492Log into Ambari > Knox > Config > Add the below under "Advanced Topology" file section<service> <role>HIVE</role> <url>http://localhost:10001/cliservice</url> <param> <name>replayBufferSize</name> <value>16</value> </param></service>

SYMPTOM While adding new service to the HDP cluster, it fails with below error:Traceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/hook.py", line 37, in <module> AfterInstallHook().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/hook.py", line 31, in hook setup_hdp_symlinks() File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/hooks/after-INSTALL/scripts/shared_initialization.py", line 49, in setup_hdp_symlinks hdp_select.select_all(version) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/hdp_select.py", line 122, in select_all Execute(command, only_if = only_if_command) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/s

Problem Statement: Whenever HBase is running, the "hdfs fsck /" reports four hbase-related files in the path "hbase/data/WALs/" as CORRUPT. Even after letting the cluster sit idle for a couple hours, it is still in the corrupt state. If HBase is shut down, the problem goes away. If HBase is then restarted, the problem recurs.ERROR:hdfs@test1:>$ hdfs fsck /Connecting to namenode via http://test1.example.com:50070/fsck?ugi=hdfs&path=%2FFSCK started by hdfs (auth:SIMPLE) from /39.0.8.2 for path / at Wed Jun 24 20:40:17 GMT 2015.../apps/hbase/data/WALs/test2.example.com,16020,1435168292684/test2.example.com%2C16020%2C1435168292684.default.1435175500556: MISSING 1 blocks of total size 83 B./apps/hbase/data/WALs/test3.example.com,16020,1435168290466/test3.example.com%2C16020%2C1435168290466..meta.1435175562144.meta: MISSING 1 blocks of total size 83 B./apps/hbase/data/WALs/test3.example.com,16020,1435168290466/test3.example.com%2C16020%2C1435168290466.default.1435175498500: MISSING 1

[users] # To use a different strategy (LDAP / Database / ...) check the shiro doc at http://shiro.apache.org/configuration.html#Configuration-INISections admin = admin [main] ldapRealm = org.apache.zeppelin.server.LdapGroupRealm ldapRealm.contextFactory.environment[ldap.searchBase] = dc=mgmt,dc=example,dc=net ldapRealm.contextFactory.url = ldaps://me.abc.example.net:636 ldapRealm.userDnTemplate = uid={0},cn=group,dc=mgmt,dc=example,dc=netldapRealm.contextFactory.authenticationMechanism = SIMPLEldapRealm.contextFactory.systemUsername=uid=example,ou=example,dc=mgmt,dc=example,dc=netldapRealm.contextFactory.systemPassword=examplesessionManager=org.apache.shiro.web.session.mgt.DefaultWebSessionManagersecurityManager.sessionManager = $sessionManager securityManager.sessionManager.globalSessionTimeout = 86400000 shiro.loginUrl = /api/login[urls] /** = authc

hcc-58548.zipUntil recently, Hive could only read and write UTF-8 text files, and no other character sets were supported forcing people to convert their possibly huge and/or multiple input files to UTF-8 using "iconv" or other such utility which can be cumbersome (for example, iconv supports only files smaller than 16G), and time-consuming. However, since Hive-0.14, by way of LazySimpleSerDe, conversion from and to other character sets is supported. The SerDe can be specified by, for example hive> CREATE TABLE mytable(...) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES("serialization.encoding"='SJIS'); or by declaring the table in a usual way, omitting the SerDe, specifying the field delimiter, and then doing: hive> ALTER TABLE mytable SET SERDEPROPERTIES ('serialization.encoding'='SJIS'); where, instead of "SJIS" one can insert the encoding of the input file (note that char-set self-detection is not supported), like for example o

My first caveat would be that in my tests, the pre-trained models is missing a lot of names. If this is for a production work load, I would recommend training your own models using your own data. Maybe use all of your corporate directory, client list, Salesforce data, LinkedIn and social media. I would recommend full name, first names and any nicknames that are commonly used.The current version is 1.7.0 and there are pre-trained 1.5.0 models that work. They have a number of pre-trained models in a few human languages. I chose English (http://opennlp.sourceforge.net/models-1.5/en-ner-person.bin).Walk Through:Create TokenNameFinderModel from pre-built person model.Tokenize the input sentence.Find the identified people.Convert to JSON array.You can easily plug this into a custom NiFi processor, microservice, command line tool or routine in a larger Apache Storm or Apache Spark pipeline.Code (JavaBean)public class PersonName { private String name = ""; public String getName() { re

Was an interesting issue faced last week. Putting here for bigger audience, might be helpful to others too.PROBLEM On one of the node, datanode and nodemanager were not coming up. Below is the error after starting from ambari. resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'ulimit -c unlimited ; /usr/hdp/current/hadoop-client/sbin/hadoop-daemon.sh --config /usr/hdp/current/hadoop-client/conf start datanode'' returned 1. starting datanode, logging to /var/log/hadoop/hdfs/hadoop-hdfs-datanode-ny-node3.hwxblr.com.outError: Could not find or load main class org.apache.hadoop.hdfs.server.datanode.DataNodeAs datanode process itself wasn't loaded, so nothing was printed in datanode logs. Only thing we see in .out file is Error: Could not find or load main class org.apache.hadoop.hdfs.server.datanode.DataNodeVerified in the jar, DataNode class was present./usr/jdk64/jdk1.8.0_77/bin/jar -tvf /usr/hdp/2.5.0.0-1245/hadoop-hdfs/hadoop-hdfs-2.7.3

Using SaltStack to run commands on HDCloud and CloudbreakHDCloud and Cloudbreak make it easy to provision, configure and elastically grow HDP clusters on cloud infrastructure. During provisioning time Cloudbreak can execute recipes on the nodes participating in the cluster. Recipes are simple script written in bash or python or any other scripting language available on the nodes.It is common that users would like to execute ad-hoc commands (e.g. collecting logs, installing extra packages, executing scripts) on nodes not only during provisioning time, but when the cluster is used.Infrastructure management of CloudbreakUnder the hood Cloudbreak uses SaltStack to manage nodes of the cluster, install packages, change configuration files and execute recipes. After provisioning phase, users can take advantage of this infrastructure management tool and execute their own scripts and create their own Salt States. Connectivity checkBy default Salt master is installed on one of the master nodes,

Below article will guide you on how to access Ranger KMS policies via rest api -1. I have Cluster with Ranger and Ranger KMS installed.2. From the documentation it is clearly given on "how to access Ranger policies using rest api". Please check the link below -https://cwiki.apache.org/confluence/display/RANGER/REST+APIs+for+Policy+Management3. Below is the example for Range rest api GET method -a.Sample Ranger Policy for HDFS repository in UI is as below -Below is the rest api GET method call which we can use to get the policy as displayed above - curl -iv -u <username>:<password> -H "Content-type:application/json" -X GET http://localhost:6080/service/public/api/repository/{id}Eg. In above screenshot my policy id is "2"curl -iv -u admin:admin -H "Content-type:application/json" -X GET http://localhost:6080/service/public/api/repository/24. Below is my Ranger KMS UI with policy -But if you try the same steps for Ranger KMS it will fail -curl -iv -u <username>:<passwo

User tries to decommission/recommission nodes from Ambari UI, nothing happens on the UI and it seems like the operation did not go through.Ambari-server Logs:WARN [C3P0PooledConnectionPoolManager[identityToken->2s8bny9j1mxgjkn9oj5d8|79679221]-HelperThread-#0] StatementUtils:223 - Statement close FAILED.com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=DEFAULT' at line 1 at sun.reflect.GeneratedConstructorAccessor198.newInstance(Unknown Source)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.Util.getInstance(Util.java:386) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1052)ROOT CAUSE The default JDBC driver

USE CASEACCESS MODEL [Let's say in this case TERADATA] User U1 can only read all tables in Database D1 & D2User U2 can Read Database D1 and INSERT , UPDATE , DELETE , SELECT all tables in Database D2. User U2 Cannot DROP or Create Table in Database D2.User U3 can SELECT,INSERT,UPDATE,DELETE,DROP,CREATE (ALL ACCESS) on Database D1 and D2OBJECTIVE :- Want to have same model on Hadoop with one improvement.We will have Storage Groups and ACLs – grouping the tables of same subject area. One Database may have more than one storage Group. Say SG11 , SG12 and SG21 and SG22 (SG11 and SG12 are associated with database D1 and SG21 & SG22 with D2)User U1 should read all Tables in D1 and D2 .User U2 should only INSERT,UPDATE,DELETE and SELECT Tables covered by SG11 ( in Database D1) - U2 will not be able to Update tables in SG12 ( in Database D1) but can readUser U3 can do all operations on SG11,SG12 ( D1) and SG21,SG22 (D2) and is OWNER of all the objects in D1 and D2 .OU

Additional Use of Color Color has been added throughout the UI for important messaging and to highlight changes in state/status, such as whether processors are running, stopped, or invalid and whether controller services are disabled or enabled. Here are some sample screenshots: Status Bar Processors Process Groups Remote Process Groups Summary Settings Customization of Processor Color To further highlight when a processor’s color has been customized, the processor icon now appears on a square background of the selected color. Additionally, the processor itself is outlined by the custom color. If users wish to return the processor to the default color, they can select the top most left corner of the color palette: Back Pressure Indicators Processor connections now visually indicate when back pressure is enabled. Progress bars appear on the connection label for “Back Pressure Object Threshold" and "Back Pressure Data Size Threshold.”The progress bars change color

Property: tez.am.view-aclsLocation: tez-site.xml Navigation Option: Ambari > Tez > Advanced tez-site > tez-site.xmlDefault Value: Empty stringAM view will only be displayed to users who submitted the job.Description: The Property "tez.am.view-acls" allows you to control users or groups that can view Tez AM jobs. "No default value" or Empty Value indicate that only the users who submitted the job can view their job status.AM view ACLs. This setting enables the specified users or groups to view the status of the AM and all DAGs that run within the AM. Format: a comma-separated list of users, a white space, and then a comma-separated list of groups. For example, "lhale,msmith administrators,users"If you want to allow all users have visibility of all Tez jobs, set tez.am.view-acls=*Services to be restarted to apply configuration: Tez, Hive, Oozie.This Article assumes the following:1. Environment is Ambari Managed Cluster2. Tez View is enabled.

When enabling Kerberos, where Ambari is managing the Kerberos identities (principals and keytab files), the KDC administrator's credentials are needed in order to create the needed accounts in the KDC. During the process of enabling Kerberos via Ambari's web-based UI, the user is prompted for this information and has the option to store the credentials in either the temporary or persisted credential store. The temporary credential store is a keystore in memory where each entry is removed after 90 minutes (from initial creation), when Ambari is restarted, or by user request. The persisted credential store is a keystore stored on disk where each entry is removed only by user request. The option to store a credential in the persisted store is only available if Ambari's credential store has been setup. Creating Ambari's Credential StoreTo set up Ambari's credential store, the following commands must be invoked from the Ambari server host's command line:ambari-server setup-securityOpt

Apache Zeppelin is a flexible, multi-purpose notebook used for code editing and data visualization. It supports a variety of languages (using interpreters) and enables users to quickly prototype, write code, share code, and visualize the results using standard charts.But as you move towards data apps and front-end development, you may need to create custom visualizations. Fortunately, Zeppelin comes with built-in support for Angular through both a front-end API and back-end API. I recently worked on a project to identify networking issues. These issues could range from bad/weak network signals, wiring issues, down nodes, etc. To replicate this, I created an example project where I mapped out the path that a packet takes from my home wifi connection to the google.com servers (using traceroute). The complete project is here, but for this post I want to focus on the steps I took to display the Spark output within Google Maps, all within Zeppelin. Here's a couple screenshots showing the e

PROBLEM Zookeeper transaction logs and snapshot files are created very frequently (multiple files in every minute) and that fills up the FileSystem in a very short time. ROOT CAUSE One or more application are creating or modifying the znodes too frequently, causing too many transactions in a short duration. This leads to the creation of too many transactional log files and snapshot files since they get rolled over after 100,000 entries by default (as defined by zookeeper property 'snapCount') RESOLUTION The resolution for such cases involves reviewing the zookeeper transaction logs to find the znodes that are updated/created most frequently using the following command on one of the zookeeper servers: # cd /usr/hdp/current/zookeeper-server# java -cp zookeeper.jar:lib/* org.apache.zookeeper.server.LogFormatter /hadoop/zookeeper/version-2/logxxx(where 'dataDir' is set to '/hadoop/zookeeper' within zookeeper configuration)Once the frequently updating znodes are identified using the abov

SYMPTOM Namenode crash might be observed when jni based unix group mappings is enabled. The crash will usually generate the "hs_err" log file which will have the stack as below: ## A fatal error has been detected by the Java Runtime Environment:## SIGSEGV (0xb) at pc=0x00007fbc814dd2a0, pid=380582, tid=140448021370624## JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)# Problematic frame:# C [libnss_uxauth.so.2+0x4e2a0] sqlite3ExprCodeTarget+0xcc3#......Stack: [0x00007fbc9a5c5000,0x00007fbc9a6c6000], sp=0x00007fbc9a6c2860, free space=1014kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)C [libnss_uxauth.so.2+0x4e2a0] sqlite3ExprCodeTarget+0xcc3C [libnss_uxauth.so.2+0x4e8db] evalConstExpr+0xf7C [libnss_uxauth.so.2+0x47ae2] sqlite3WalkExpr+0x34C [libnss_uxauth.so.2+0x47bdd] sqlite3WalkExprList+0x42C [libnss_uxauth.so.2+0x4

SYMPTOM: While adding a new host via the Ambari , results in an exception as below:#####11 Apr 2016 10:52:24,629 ERROR [qtp-client-81] AbstractResourceProvider:279 - Caught AmbariException when creating a res ource org.apache.ambari.server.HostNotFoundException: Host not found, hostname=hostname_123.abc.xyz.com at org.apache.ambari.server.state.cluster.ClustersImpl.getHost(ClustersImpl.java:343) at org.apache.ambari.server.state.ConfigHelper.getEffectiveDesiredTags(ConfigHelper.java:108) at org.apache.ambari.server.controller.AmbariManagementControllerImpl.findConfigurationTagsWithOverrides(AmbariM anagementControllerImpl.java:1820) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.google.inject.internal.DelegatingInvocationHandler.invoke(Delegatin