Resources

I am sharing this again with the configurations and prerequisites which were missing from above example.This was done using HDP version 2.3.4 on both source and target cluster.Pre-requisites :Source and target clusters should be up and running with required services. HDP stack services required on the clusters are HDFS, Yarn, Falcon, Hive, Oozie, Pig, Tez and Zookeeper as seen below:Staging and working directories should be present on hdfs with falcon users:To do that run below commands on both source and target clusters as falcon user.[falcon@src1 ~]$ hdfs dfs -mkdir /apps/falcon/staging /apps/falcon/working[falcon@src1 ~]$ hdfs dfs -chmod 777 /apps/falcon/stagingCreate source and target cluster entity using falcon CLI as below : For source cluster:falcon entity -type cluster -submit -file source-cluster.xmlsource-cluster.xml :<?xml version="1.0" encoding="UTF-8" standalone="yes"?><cluster name="source" description="primary" colo="primary" xmlns="uri:falcon:cluster:0.1">&

If I approached you on the street and challenged you to name a state capital building from a flashcard image how many capital buildings out of the 50 do you think you could recognize? That would be pretty difficult right? We humans tend to have a narrow scope of knowledge focused around our intimate interactions with day to day life. I would have no problem identifying Atlanta Georgia’s capital building for example because I live in Atlanta and am exposed to it almost daily. If presented with a image of Kansas's capital in Topeka however I would be stumped since I have never been there or even seen a picture of that capital building. The point is experience drives our ability to recognize images. Most millennials tend to also share these “experiences” to social media, blogs, text, etc. Google is dominate at crunching these digital “experiences” from its users and artfully marrying those impressions against validated datasets. Google Vision is a rest API hosted and managed by Google tha

Problem description :When upgrading to HDP 2.5.3 , some queries to Hive were failing Especially when executing Insert Overwrite queries. with the following error :ERROR : Failed with exception java.util.ConcurrentModificationException org.apache.hadoop.hive.ql.metadata.HiveException: java.util.ConcurrentModificationException at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2883) at ..... etcCaused by: java.util.ConcurrentModificationException Workaround:"Add SORT BY 0" to your query.INSERT OVERWRITE TABLE AAA SELECT A , B , C FROM BBBB SORT BY 0 ;More investigations to follow up ...

Problem summary: Users can end up in a situation where the SQL Engine for all Ambari tables is set to MyISAM instead of InnoDB which is required for Ambari. There are few ways to land up in this situation. Prior to MySQL version 5.5.5 the default engine was MyISAM. It could also be a configured default as a global default in my.cnf which affected Ambari versions before 2.5. AMBARI-18951 addressed this issue by making InnoDB default for new databases (2.5+ Ambari deployments) as well as throwing an explicit error preventing Ambari upgrade if the wrong engine is set in order to avoid landing up in an intermediate non-upgraded state.Following is an sample error that would be encountered on an upgrade path from 2.2.2 to 2.4.2ERROR [main] DBAccessorImpl:830 - Error executing query: ALTER TABLE hostcomponentdesiredstate ADD CONSTRAINT hstcmpnntdesiredstatecmpnntnme FOREIGN KEY (component_name, service_name, cluster_id) REFERENCES servicecomponentdesiredstate (component_name, service_name, cl

Setting up GPU-enabled Tensorflow to work with ZeppelinSometimes we want to do some quick Deep Learning prototyping using TensorFlow. We also want to take advantage of Spark for data pre-processing, scaling, feature extraction while keeping it all in the same place for demo. This step-by-step guide will go through process of setting up those tools to work with each other.My setup: AWS GPU-enabled instance (any g2) Ubuntu 14.04HDP 2.5This tutorial has been partly based on this script: https://gist.github.com/erikbern/78ba519b97b440e10640, but has been heavily updated and modified.First, let’s install and set up some pre-reqs:sudo apt-get update sudo apt-get upgrade -y # choose maintainers versionsudo apt-get install -y build-essential python-pip python-dev git python-numpy swig python-dev default-jdk zip zlib1g-devThen, disable nouveau and update initramfsecho -e "blacklist nouveau\nblacklist lbm-nouveau\noptions nouveau modeset=0\nalias nouveau off\nalias lbm-nouveau off\n" | sudo tee

Hive is having an improvised explain plan with the hive.explain.user set to true. Now the indentation and arrows clearly depict the execution flow. Retrieving the join order needed a lot of back and forth references in the old style which was made as easy as following the arrows now, even with the details of mappers and reducers. Let us take a look at one sample explain plan and retrieve the join order Plan optimized by CBO.Vertex dependency in root stageMap 2 <- Map 1 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE), Map 5 (BROADCAST_EDGE), Map 6 (BROADCAST_EDGE), Map 7 (BROADCAST_EDGE), Map 8 (BROADCAST_EDGE), Map 9 (BROADCAST_EDGE)Reducer 3 <- Map 2 (SIMPLE_EDGE)Stage-0 Fetch Operator limit:-1 Stage-1 Reducer 3 vectorized, llap File Output Operator [FS_101] Group By Operator [GBY_100] (rows=1 width=736) Output:["_col0","_col1","_col2","_col3"],aggregations:["min(VALUE._col0)","min(VALUE._col1)","min(VALUE._col2)","min(VALUE._col3)"] <-Map 2

People were asking me when you could throw some more acronyms into an article. So I did that.iBeacons are tiny devices that are broadcasting over Bluetooth Low Energy (BLE) that are very good for advertising and for broadcasting proximity. I have been thinking of the retail space so I purchased a beacon, got a beacon simulator on my phone and found a really cool gateway device. It reads the BLE transmissions from these devices, packages them as JSON MQTT messages and sends them over WIFI to a MQTT Broker of your choice. I am using a cloud hosted MQTT broker to hold these and then Apache NiFi subscribes to these messages. This can happen amazingly fast as MQTT is very light weight, NIFI is fast and BLE is fast. These tiny messages are constantly advertising their identity and other information. With multiple beacons in a building, store, room, facility; you can see who is close to where.As you might expect there's multiple standards for this, iBeacon is Apple's and Google Ed

Interest in Time Series data bases and historian augmentation/replacement is currently at an all-time high with the explosion of IoT use cases. So many devices sending out unbounded streams of data provides data scientists opportunities for predictive failure analysis, there needs to be a robust scalable architecture to land that and at the same time provide a responsive environment that allows a high numbers of concurrent users access to this data.This article discusses how the union of features that exist in Apache Phoenix and Hive to can be used construct a robust time series oriented architecture the enables a scalable streaming ingest layer while simultaneously providing a low latency data exploration layer for data scientists and BI reports.The examples referenced in this article are a subset of a larger CDR processing project I’ve been working with for a while now. This will focus on the downstream data collection dealing specifically with Phoenix and Hive. Data is streamed into

After a few hours of debugging SSSD and mapping users/groups I wanted to make a post here to try and save someone the pain.I had SSD configured correctly using the following document:https://github.com/HortonworksUniversity/Security_Labs#lab-1What I found by adding debug_level=7 to the sssd.conf file was this cryptic message:Trying to resolve service 'AD_GC' I realized at some point I was firewall'd off to the Active Directory Global Catalog port 3286, once I opened this I can now get the correct groups mapped to my SSSD users.Hope this saves someone some time in the future!

FlowFile Continuation Sometimes you need to backup your current running flow, let that flow run at a later date, or make a backup of what is in process now. You want this in a permanent storage and want to reconstitute it later like Orange Juice. And add it back into the flow or restart it.This could be do to failures, for integration testing, for testing new versions of components, as a checkpoint or for many other purposes. You don't always want to reprocess the original source or files (they may be gone).Option 1:You can save that raw data that came in originally in local files or HDFS. Then read it out of there later.Option 2: Preferred: MergeContent to FlowFileV3 then Reload with Get* to IdentifyMimeType to UnpackContentUsing MergeContent with FlowFileV3 option. After that step you can PutFile, PutS3Object, PutHDFS or other file saving options. Perhaps send it to an FTP or sFTP server for storage elsewhere.Now you have a pkg file.cat /opt/demo/flow/904381478117605.pk

Affected versions:1.2.xSymptoms:Ambari was running as root during installation, then after configured for non-root, the issue occurs.When we create a bundle, we are not able to encrypt it because ambari is not able to read the public key under /var/lib/smartsense/hst-common/encryption/keys.File permissions[root@DPDev hdfs]# ls -lsR /var/lib/smartsense/hst-common/var/lib/smartsense/hst-common:total 84 drwxr-xr-x 3 ambari ambari 4096 Apr 20 16:59 anonymization4 drwxr-xr-x 3 ambari ambari 4096 Apr 20 16:59 encryption/var/lib/smartsense/hst-common/anonymization:total 44 drwxr-xr-x 2 ambari ambari 4096 Apr 20 18:21 keys/var/lib/smartsense/hst-common/anonymization/keys:total 84 -rw-r--r-- 1 root root 44 Apr 20 16:59 private_anonymization.key4 -rw-r--r-- 1 root root 44 Apr 20 18:01 shared_anonymization.key/var/lib/smartsense/hst-common/encryption:total 44 drw-r----- 2 ambari ambari 4096 Apr 20 16:59 keys/var/lib/smartsense/hst-common/encryption/keys:total 84 -rw-r----- 1 ambari ambari 216 Mar

Affected versions:1.4.xSymptoms: New recommendations are not displayed in the recommendations tab in the SmartSense view. There is a failure in hst-server.log :2017-04-20 17:20:11,904 INFO[RecommendationAutoDownloadTask] GatewayClientHelper:74 - HTTP/1.1 200 OK 2017-04-20 17:20:11,904 INFO[RecommendationAutoDownloadTask] GatewayClientHelper:75 - OK 2017-04-20 17:20:11,907 ERROR[RecommendationAutoDownloadTask] ArchiveFileCrypto:240 - Invalid key. Illegalkey size or default parameters 2017-04-20 17:20:11,907 ERROR [RecommendationAutoDownloadTask] RecommendationAutoDownloadTask:41 - Failed to download recommendations. Illegal key size or default parameters com.hortonworks.smartsense.anonymization.crypto.CryptoException: Illegal key size or default parameters at com.hortonworks.smartsense.anonymization.crypto.impl.ArchiveFileCrypto.decrypt(ArchiveFileCrypto.java:241)at com.hortonworks.smartsense.anonymization.crypto.FileDecryptor.decrypt(FileDecryptor.java:69)at com.hortonworks.support.too

Affected versions: 1.2.0, 1.2.1Symptoms: Bundle upload fails with following error in hst-server.log:10 Aug 2016 16:56:17,131 INFO [pool-1-thread-1] GatewayHttpsClient:219- Executing Request:POST https://gateway0host.gateway.com:9451/api/v1/upload/bundleHTTP/1.110 Aug 2016 16:56:17,142 ERROR[pool-1-thread-1] GatewayHttpsClient:241 - SSL error occurred. Cleaning upcertificates/keys at client side is required.10 Aug 2016 16:56:17,142 WARN [pool-1-thread-1] UploadRunnable:106 -SSL error occurred while connecting to gateway. Reseting local keys andcertificates.Reason: Support for using md5 algorithm to setup 2-way SSL was stopped in JDK 8. We had md5 as our default algorithm in SmartSense 1.2.1 and lower. This causes conflict in SSL setup hst-server jdk version is lower than JDK 8.Solution:This happens when HST is java version 1.6 and Gateway is on version 1.8. This is fixed in 1.2.2. Upgrade HST to version higher than 1.2.1. Upgrading Gateway mostly will not cause any impact because it p

Affected versions: 1.2.x, 1.3.0Symptoms: Standalone Gateway is up and running. But still there is a failure pop up when trying to upload a bundle. Error details are provided in logs below:##hst-server.logCaused by:com.hortonworks.smartsense.gateway.client.GatewayClientException:java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1column 7at com.hortonworks.smartsense.gateway.client.GatewayHttpsClient.uploadBundle(GatewayHttpsClient.java:249)at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:100)... 12 moreCaused by: com.google.gson.JsonSyntaxException:java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1column 7at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176)at com.google.gson.Gson.fromJson(Gson.java:795)at com.google.gson.Gson.fromJson(Gson.java:761)at com.google.gson.Gson.fromJson(Gson.java:710)at com.google.gson.Gson.fromJson(Gso

Affected versions: SmartSense 1.3.xSymptoms: Gateway start fails with following exception in hst-gateway.log : INFO 2016-10-21 12:32:42,881hst-gateway.py:520 - SmartSense Gateway is not running... INFO 2016-10-21 12:32:42,882hst-gateway.py:539 - Running server: ['/bin/sh', '-c', 'ulimit -n 10000;/usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC-XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -Dlog.file.name=hst-gateway.log-Xms512m -Xmx1024m -cp /etc/hst/conf:/usr/hdp/share/hst/hst-common/lib/*com.hortonworks.smartsense.gateway.server.GatewayServer>/var/log/hst/hst-gateway.out 2>&1 &'] INFO 2016-10-21 12:32:42,884 hst-gateway.py:542 - Executing Gateway Server start command 21 Oct 2016 12:32:43,505 DEBUG main GatewayConfigValidator:40 - Found supported transfer type:HTTPS 21 Oct 2016 12:32:43,507 INFO main GatewayConfigValidator:67 - Found configurations to be used for connectivity. 21 Oct 2016 12:32:43,509 INFO main GatewayValidator:149

Affected versions: SmartSense 1.2.x, 1.3.x with Ambari lower than 2.4.1 Symptoms: Users / Groups are pre-created before setting up the cluster. As such, ignore_groupsusers_create needs to be set to true so groups are not modified when services are added. This works fine for all components except smartsense which fails with error performing usermod : resource_management.core.exceptions.Fail:Execution of 'usermod -G -adp-hdfs -g adp-hadoop adp-xxxx' returned 6. usermod: group 'adp-hdfs' does not exist This group does NOT exist. Note, within the stdout you do actually see the following message: Skipping creation of User andGroup as host is sys prepped or ignore_groupusers_create flag is on Reason: The “ignore_groupsusers_create” configurations was not considered while adding / installing SmartSense service. Solution: New fix is available in Ambari 2.4.2.6-5 or higher. Upgrade is the cleanest solution in this case.

Affected versions: All SmartSense versionsSymptoms:YARN and MR dashboards work fine but there is no data for HDFS dashboard. Activity Analyzer logs are available at /var/log/smartsense-activity/activity-analyzer.log. These logs will shows an error stack trace similar to something as provided below: ERROR [pool-14-thread-1] ActivityManager:89 - Failed to processactivity id /hadoop/hdfs/namenode/current/fsimage_0009281 of type HDFScom.hortonworks.smartsense.activity.ActivityException: Failed to process activity. Activity Type: HDFS; ActivityId:/hadoop/hdfs/namenode/current/fsimage_0009281; ActivityData: null at com.hortonworks.smartsense.activity.hdfs.HDFSImageProcessor.processActivity(HDFSImageProcessor.java:104) at com.hortonworks.smartsense.activity.ActivityManager$1.call(ActivityManager.java:80) at com.hortonworks.smartsense.activity.ActivityManager$1.call(ActivityManager.java:73) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecu

The HDFS Balancer program can be invoked to rebalance HDFS blocks when data nodes are added to or removed from the cluster. For more information about the HDFS balancer, see this HCC article.Since Kerberos tickets are designed to expire, a common question that arises in secure clusters is whether one needs to account for ticket expiration (namely, TGT) when invoking long-running Balancer jobs. To cut to the chase: the answer depends on just how long the job takes to run. Let's discuss some background context (I am referencing Chris Nauroth's excellent answer on Stack Overflow as well as HDFS-9698 below).The primary use case for Kerberos authentication in the Hadoop ecosystem is Hadoop's RPC framework, which uses SASL for authentication. Many daemon processes, i.e., non-interactive processes, call UserGroupInformation#loginUserFromKeytab at process startup, using a keytab to authenticate to the KDC.Moreover, Hadoop implements an automatic re-login mechanism directly inside the RPC clien

if you want to verify the Certificate contents of KNOX Server execute below commandopenssl s_client -showcerts -connect 127.0.0.1:8443if developers want to connect to KNOX with SSL enabled copy cert contents from above command to knox.crt file and import to a Keystore by executing below commandkeytool -import -keystore myLocalTrustStore.jks -file knox.crtNow developers use as below beeline> !connect "jdbc:hive2://hadoop-knox.dev.XXXX.com:8443/default;transportMode=http;httpPath=gateway/default/hive;ssl=true;sslTrustStore=/tmp/knoxhacerts/new/myLocalTrustStore.jks;trustStorePassword=knoxdev"Hive JDBCjdbc:hive2://{gateway-host}:{gateway-port}/; ssl=true; sslTrustStore={gateway-trust-store-path}; trustStorePassword={gateway-trust-store-password}; transportMode=http; httpPath={gateway-path}/{cluster-name}/hiveIf you want to list the imported certs in a JKS file execute below command.keytool -v -list -keystore gateway.jkscommand to create new truststore myNewTrustSTore.jkskeytool -

This article assumes you have already identified the GUID associated with your Atlas entity and that the tag in question you wish to associate to this entity already exists.For more information, on how to identify the GUID for your entity, please see this HCC article by Mark Johnson. For example, if we wanted to add a new tag to all hive tables containing the word "claim", we could use the Full Text Search capability to identify all entities (replace admin:admin with the username:password values for an Atlas administrator):curl -u admin:admin http://$ATLAS_SERVER:21000/api/atlas/discovery/search/fulltext?query=claimNow we are ready to assign a new tag, called "PII", to all such entities.We are using the v1 API for Atlas in HDP 2.5. In HDP 2.6, the Atlas API has been revamped and simplified. Please see this HCC article for more details.Let's construct an example using the first GUID, f7a24ec6-5b0c-42d8-ba8a-1ac654d24f45. We will use the traits resource associated with this entity, and P

Q1) What is the difference between using the programs kadmin.local and kadmin, respectively?A1) The difference between kadmin and kadmin.local is that kadmin.local directly accesses the KDC database (a file called principal in /var/kerberos/krb5kdc) and does not use Kerberos for authentication. Since kadmin.local directly accesses the KDC database, it must be run directly on the master KDC as a user with sufficient permissions to read the KDC database.When using kadmin to administer the KDC database, the user is communicating with the kadmind daemon over the network and will authenticate using Kerberos to the KDC master. Hence, the first principal must already exist before connecting over the network—this is the motivation for the existence of kadmin.local. This also means the KDC administrator will need to kinit as the administrator principal (by default, admin/admin@REALM) to run kadmin from another box.Q2) How can we restrict which users can administer the MIT-KDC service?A2) The

hdfs dfsadmin -report outputs a brief report on the overall HDFS filesystem. It’s a useful command to quickly view how much disk is available, how many DataNodes are running, corrupted blocks etc. Note: This article explains the disk space calculations as seen by the HDFS. Command: Run the command with sudo -u hdfs prefixed to ensure you don't get a permission denied error. sudo -u hdfs hdfs dfsadmin -reportYou will see an output similar to:Configured Capacity: 270082531328 (251.53 GB)Present Capacity: 190246318080 (177.18 GB)DFS Remaining: 143504465920 (133.65 GB)DFS Used: 46741852160 (43.53 GB)DFS Used%: 24.57%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0Missing blocks (with replication factor 1): 0 -------------------------------------------------Live datanodes (4): Name: 123.45.678.910:50010 (kharearpit4.local)Hostname: kharearpit4.localRack: /rack4Decommission Status : NormalConfigured Capacity: 20063055872 (18.69 GB)DFS Used: 40960 (40 KB)Non DFS Use

HDP2.3 Installation on Single Node CentOSIn this step we will start creating the HDP Compute Cluster. We will create VM with CentOS 6.7. Deploy HDP 2.3 with Ambari 2.1 on a single node. In the second part you will create a Docker instance and make it a Datanode to the existing instance in the same VM.The tutorial is to show how easy it is use docker and create a multi node instance.Details Create HDP on a Single Node VM Create a docker node and add it as an data node to the above VMHave Fun!

QuestionHow does a user generate an AWS keypair which is used during Environment Quickstart?  AnswerAt the last step of Environment Quickstart, user will be asked to enter AWS keypair. Following are the steps to create the keypair:  Sign into the https://console.aws.amazon.comBeneath IAM -> Users ->  Select your user ->  Security Credentials tab,  click the "Create access key" button to create a new AWS access key for your accountCopy and paste Access Key ID and the access key value to a file for later usage.Creation the access keyClick the Show to be able to copy the key value

Tibco/Spotfire JDBC Connection to Phoenix on Kerberized ClusterVersions : Tibco Spotfire : 7.5Phoenix: 4.4HDP : 2.4.xIssues: Phoenix Client Jars from HDP cluster has dependency conflicts with Spotfire Jars. Step 1: Use the Phoenix core jar with below dependencies to avoid the conflicts1) hbase-client 2) phoenix core 3) betty-all 4) zookeeper 5) hbase-protocol 6) commons-lang 7) commons-configuration 😎 protobuf-java 9) hbase-server 10) guava 11) hadoop-auth 12) hadoop-common 13) commons-collections 14) htrace-core 15) phoenix-core 16) hbase-commonStep 2: Add the krb5.conf file from the cluster to JDK_HOME/jre/lib/security/ that Spotfire is using. The JDK could be embedded in tomcat of spotfire or external. Step 3: Add hbase-site.xml, core-site.xml & hdfs-site.xml to a folder on the spotfire server. And add the folder to the class