Community Articles

People were asking me when you could throw some more acronyms into an article. So I did that.iBeacons are tiny devices that are broadcasting over Bluetooth Low Energy (BLE) that are very good for advertising and for broadcasting proximity. I have been thinking of the retail space so I purchased a beacon, got a beacon simulator on my phone and found a really cool gateway device. It reads the BLE transmissions from these devices, packages them as JSON MQTT messages and sends them over WIFI to a MQTT Broker of your choice. I am using a cloud hosted MQTT broker to hold these and then Apache NiFi subscribes to these messages. This can happen amazingly fast as MQTT is very light weight, NIFI is fast and BLE is fast. These tiny messages are constantly advertising their identity and other information. With multiple beacons in a building, store, room, facility; you can see who is close to where.As you might expect there's multiple standards for this, iBeacon is Apple's and Google Ed

Interest in Time Series data bases and historian augmentation/replacement is currently at an all-time high with the explosion of IoT use cases. So many devices sending out unbounded streams of data provides data scientists opportunities for predictive failure analysis, there needs to be a robust scalable architecture to land that and at the same time provide a responsive environment that allows a high numbers of concurrent users access to this data.This article discusses how the union of features that exist in Apache Phoenix and Hive to can be used construct a robust time series oriented architecture the enables a scalable streaming ingest layer while simultaneously providing a low latency data exploration layer for data scientists and BI reports.The examples referenced in this article are a subset of a larger CDR processing project I’ve been working with for a while now. This will focus on the downstream data collection dealing specifically with Phoenix and Hive. Data is streamed into

After a few hours of debugging SSSD and mapping users/groups I wanted to make a post here to try and save someone the pain.I had SSD configured correctly using the following document:https://github.com/HortonworksUniversity/Security_Labs#lab-1What I found by adding debug_level=7 to the sssd.conf file was this cryptic message:Trying to resolve service 'AD_GC' I realized at some point I was firewall'd off to the Active Directory Global Catalog port 3286, once I opened this I can now get the correct groups mapped to my SSSD users.Hope this saves someone some time in the future!

FlowFile Continuation Sometimes you need to backup your current running flow, let that flow run at a later date, or make a backup of what is in process now. You want this in a permanent storage and want to reconstitute it later like Orange Juice. And add it back into the flow or restart it.This could be do to failures, for integration testing, for testing new versions of components, as a checkpoint or for many other purposes. You don't always want to reprocess the original source or files (they may be gone).Option 1:You can save that raw data that came in originally in local files or HDFS. Then read it out of there later.Option 2: Preferred: MergeContent to FlowFileV3 then Reload with Get* to IdentifyMimeType to UnpackContentUsing MergeContent with FlowFileV3 option. After that step you can PutFile, PutS3Object, PutHDFS or other file saving options. Perhaps send it to an FTP or sFTP server for storage elsewhere.Now you have a pkg file.cat /opt/demo/flow/904381478117605.pk

Affected versions:1.2.xSymptoms:Ambari was running as root during installation, then after configured for non-root, the issue occurs.When we create a bundle, we are not able to encrypt it because ambari is not able to read the public key under /var/lib/smartsense/hst-common/encryption/keys.File permissions[root@DPDev hdfs]# ls -lsR /var/lib/smartsense/hst-common/var/lib/smartsense/hst-common:total 84 drwxr-xr-x 3 ambari ambari 4096 Apr 20 16:59 anonymization4 drwxr-xr-x 3 ambari ambari 4096 Apr 20 16:59 encryption/var/lib/smartsense/hst-common/anonymization:total 44 drwxr-xr-x 2 ambari ambari 4096 Apr 20 18:21 keys/var/lib/smartsense/hst-common/anonymization/keys:total 84 -rw-r--r-- 1 root root 44 Apr 20 16:59 private_anonymization.key4 -rw-r--r-- 1 root root 44 Apr 20 18:01 shared_anonymization.key/var/lib/smartsense/hst-common/encryption:total 44 drw-r----- 2 ambari ambari 4096 Apr 20 16:59 keys/var/lib/smartsense/hst-common/encryption/keys:total 84 -rw-r----- 1 ambari ambari 216 Mar

Affected versions:1.4.xSymptoms: New recommendations are not displayed in the recommendations tab in the SmartSense view. There is a failure in hst-server.log :2017-04-20 17:20:11,904 INFO[RecommendationAutoDownloadTask] GatewayClientHelper:74 - HTTP/1.1 200 OK 2017-04-20 17:20:11,904 INFO[RecommendationAutoDownloadTask] GatewayClientHelper:75 - OK 2017-04-20 17:20:11,907 ERROR[RecommendationAutoDownloadTask] ArchiveFileCrypto:240 - Invalid key. Illegalkey size or default parameters 2017-04-20 17:20:11,907 ERROR [RecommendationAutoDownloadTask] RecommendationAutoDownloadTask:41 - Failed to download recommendations. Illegal key size or default parameters com.hortonworks.smartsense.anonymization.crypto.CryptoException: Illegal key size or default parameters at com.hortonworks.smartsense.anonymization.crypto.impl.ArchiveFileCrypto.decrypt(ArchiveFileCrypto.java:241)at com.hortonworks.smartsense.anonymization.crypto.FileDecryptor.decrypt(FileDecryptor.java:69)at com.hortonworks.support.too

Affected versions: 1.2.0, 1.2.1Symptoms: Bundle upload fails with following error in hst-server.log:10 Aug 2016 16:56:17,131 INFO [pool-1-thread-1] GatewayHttpsClient:219- Executing Request:POST https://gateway0host.gateway.com:9451/api/v1/upload/bundleHTTP/1.110 Aug 2016 16:56:17,142 ERROR[pool-1-thread-1] GatewayHttpsClient:241 - SSL error occurred. Cleaning upcertificates/keys at client side is required.10 Aug 2016 16:56:17,142 WARN [pool-1-thread-1] UploadRunnable:106 -SSL error occurred while connecting to gateway. Reseting local keys andcertificates.Reason: Support for using md5 algorithm to setup 2-way SSL was stopped in JDK 8. We had md5 as our default algorithm in SmartSense 1.2.1 and lower. This causes conflict in SSL setup hst-server jdk version is lower than JDK 8.Solution:This happens when HST is java version 1.6 and Gateway is on version 1.8. This is fixed in 1.2.2. Upgrade HST to version higher than 1.2.1. Upgrading Gateway mostly will not cause any impact because it p

Affected versions: 1.2.x, 1.3.0Symptoms: Standalone Gateway is up and running. But still there is a failure pop up when trying to upload a bundle. Error details are provided in logs below:##hst-server.logCaused by:com.hortonworks.smartsense.gateway.client.GatewayClientException:java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1column 7at com.hortonworks.smartsense.gateway.client.GatewayHttpsClient.uploadBundle(GatewayHttpsClient.java:249)at com.hortonworks.support.tools.gateway.UploadRunnable.uploadBundle(UploadRunnable.java:100)... 12 moreCaused by: com.google.gson.JsonSyntaxException:java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1column 7at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176)at com.google.gson.Gson.fromJson(Gson.java:795)at com.google.gson.Gson.fromJson(Gson.java:761)at com.google.gson.Gson.fromJson(Gson.java:710)at com.google.gson.Gson.fromJson(Gso

Affected versions: SmartSense 1.3.xSymptoms: Gateway start fails with following exception in hst-gateway.log : INFO 2016-10-21 12:32:42,881hst-gateway.py:520 - SmartSense Gateway is not running... INFO 2016-10-21 12:32:42,882hst-gateway.py:539 - Running server: ['/bin/sh', '-c', 'ulimit -n 10000;/usr/java/default/bin/java -server -XX:NewRatio=3 -XX:+UseConcMarkSweepGC-XX:-UseGCOverheadLimit -XX:CMSInitiatingOccupancyFraction=60 -Dlog.file.name=hst-gateway.log-Xms512m -Xmx1024m -cp /etc/hst/conf:/usr/hdp/share/hst/hst-common/lib/*com.hortonworks.smartsense.gateway.server.GatewayServer>/var/log/hst/hst-gateway.out 2>&1 &'] INFO 2016-10-21 12:32:42,884 hst-gateway.py:542 - Executing Gateway Server start command 21 Oct 2016 12:32:43,505 DEBUG main GatewayConfigValidator:40 - Found supported transfer type:HTTPS 21 Oct 2016 12:32:43,507 INFO main GatewayConfigValidator:67 - Found configurations to be used for connectivity. 21 Oct 2016 12:32:43,509 INFO main GatewayValidator:149

Affected versions: SmartSense 1.2.x, 1.3.x with Ambari lower than 2.4.1 Symptoms: Users / Groups are pre-created before setting up the cluster. As such, ignore_groupsusers_create needs to be set to true so groups are not modified when services are added. This works fine for all components except smartsense which fails with error performing usermod : resource_management.core.exceptions.Fail:Execution of 'usermod -G -adp-hdfs -g adp-hadoop adp-xxxx' returned 6. usermod: group 'adp-hdfs' does not exist This group does NOT exist. Note, within the stdout you do actually see the following message: Skipping creation of User andGroup as host is sys prepped or ignore_groupusers_create flag is on Reason: The “ignore_groupsusers_create” configurations was not considered while adding / installing SmartSense service. Solution: New fix is available in Ambari 2.4.2.6-5 or higher. Upgrade is the cleanest solution in this case.

Affected versions: All SmartSense versionsSymptoms:YARN and MR dashboards work fine but there is no data for HDFS dashboard. Activity Analyzer logs are available at /var/log/smartsense-activity/activity-analyzer.log. These logs will shows an error stack trace similar to something as provided below: ERROR [pool-14-thread-1] ActivityManager:89 - Failed to processactivity id /hadoop/hdfs/namenode/current/fsimage_0009281 of type HDFScom.hortonworks.smartsense.activity.ActivityException: Failed to process activity. Activity Type: HDFS; ActivityId:/hadoop/hdfs/namenode/current/fsimage_0009281; ActivityData: null at com.hortonworks.smartsense.activity.hdfs.HDFSImageProcessor.processActivity(HDFSImageProcessor.java:104) at com.hortonworks.smartsense.activity.ActivityManager$1.call(ActivityManager.java:80) at com.hortonworks.smartsense.activity.ActivityManager$1.call(ActivityManager.java:73) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecu

The HDFS Balancer program can be invoked to rebalance HDFS blocks when data nodes are added to or removed from the cluster. For more information about the HDFS balancer, see this HCC article.Since Kerberos tickets are designed to expire, a common question that arises in secure clusters is whether one needs to account for ticket expiration (namely, TGT) when invoking long-running Balancer jobs. To cut to the chase: the answer depends on just how long the job takes to run. Let's discuss some background context (I am referencing Chris Nauroth's excellent answer on Stack Overflow as well as HDFS-9698 below).The primary use case for Kerberos authentication in the Hadoop ecosystem is Hadoop's RPC framework, which uses SASL for authentication. Many daemon processes, i.e., non-interactive processes, call UserGroupInformation#loginUserFromKeytab at process startup, using a keytab to authenticate to the KDC.Moreover, Hadoop implements an automatic re-login mechanism directly inside the RPC clien

if you want to verify the Certificate contents of KNOX Server execute below commandopenssl s_client -showcerts -connect 127.0.0.1:8443if developers want to connect to KNOX with SSL enabled copy cert contents from above command to knox.crt file and import to a Keystore by executing below commandkeytool -import -keystore myLocalTrustStore.jks -file knox.crtNow developers use as below beeline> !connect "jdbc:hive2://hadoop-knox.dev.XXXX.com:8443/default;transportMode=http;httpPath=gateway/default/hive;ssl=true;sslTrustStore=/tmp/knoxhacerts/new/myLocalTrustStore.jks;trustStorePassword=knoxdev"Hive JDBCjdbc:hive2://{gateway-host}:{gateway-port}/; ssl=true; sslTrustStore={gateway-trust-store-path}; trustStorePassword={gateway-trust-store-password}; transportMode=http; httpPath={gateway-path}/{cluster-name}/hiveIf you want to list the imported certs in a JKS file execute below command.keytool -v -list -keystore gateway.jkscommand to create new truststore myNewTrustSTore.jkskeytool -

This article assumes you have already identified the GUID associated with your Atlas entity and that the tag in question you wish to associate to this entity already exists.For more information, on how to identify the GUID for your entity, please see this HCC article by Mark Johnson. For example, if we wanted to add a new tag to all hive tables containing the word "claim", we could use the Full Text Search capability to identify all entities (replace admin:admin with the username:password values for an Atlas administrator):curl -u admin:admin http://$ATLAS_SERVER:21000/api/atlas/discovery/search/fulltext?query=claimNow we are ready to assign a new tag, called "PII", to all such entities.We are using the v1 API for Atlas in HDP 2.5. In HDP 2.6, the Atlas API has been revamped and simplified. Please see this HCC article for more details.Let's construct an example using the first GUID, f7a24ec6-5b0c-42d8-ba8a-1ac654d24f45. We will use the traits resource associated with this entity, and P

Q1) What is the difference between using the programs kadmin.local and kadmin, respectively?A1) The difference between kadmin and kadmin.local is that kadmin.local directly accesses the KDC database (a file called principal in /var/kerberos/krb5kdc) and does not use Kerberos for authentication. Since kadmin.local directly accesses the KDC database, it must be run directly on the master KDC as a user with sufficient permissions to read the KDC database.When using kadmin to administer the KDC database, the user is communicating with the kadmind daemon over the network and will authenticate using Kerberos to the KDC master. Hence, the first principal must already exist before connecting over the network—this is the motivation for the existence of kadmin.local. This also means the KDC administrator will need to kinit as the administrator principal (by default, admin/admin@REALM) to run kadmin from another box.Q2) How can we restrict which users can administer the MIT-KDC service?A2) The

hdfs dfsadmin -report outputs a brief report on the overall HDFS filesystem. It’s a useful command to quickly view how much disk is available, how many DataNodes are running, corrupted blocks etc. Note: This article explains the disk space calculations as seen by the HDFS. Command: Run the command with sudo -u hdfs prefixed to ensure you don't get a permission denied error. sudo -u hdfs hdfs dfsadmin -reportYou will see an output similar to:Configured Capacity: 270082531328 (251.53 GB)Present Capacity: 190246318080 (177.18 GB)DFS Remaining: 143504465920 (133.65 GB)DFS Used: 46741852160 (43.53 GB)DFS Used%: 24.57%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0Missing blocks (with replication factor 1): 0 -------------------------------------------------Live datanodes (4): Name: 123.45.678.910:50010 (kharearpit4.local)Hostname: kharearpit4.localRack: /rack4Decommission Status : NormalConfigured Capacity: 20063055872 (18.69 GB)DFS Used: 40960 (40 KB)Non DFS Use

HDP2.3 Installation on Single Node CentOSIn this step we will start creating the HDP Compute Cluster. We will create VM with CentOS 6.7. Deploy HDP 2.3 with Ambari 2.1 on a single node. In the second part you will create a Docker instance and make it a Datanode to the existing instance in the same VM.The tutorial is to show how easy it is use docker and create a multi node instance.Details Create HDP on a Single Node VM Create a docker node and add it as an data node to the above VMHave Fun!

QuestionHow does a user generate an AWS keypair which is used during Environment Quickstart?  AnswerAt the last step of Environment Quickstart, user will be asked to enter AWS keypair. Following are the steps to create the keypair:  Sign into the https://console.aws.amazon.comBeneath IAM -> Users ->  Select your user ->  Security Credentials tab,  click the "Create access key" button to create a new AWS access key for your accountCopy and paste Access Key ID and the access key value to a file for later usage.Creation the access keyClick the Show to be able to copy the key value

Tibco/Spotfire JDBC Connection to Phoenix on Kerberized ClusterVersions : Tibco Spotfire : 7.5Phoenix: 4.4HDP : 2.4.xIssues: Phoenix Client Jars from HDP cluster has dependency conflicts with Spotfire Jars. Step 1: Use the Phoenix core jar with below dependencies to avoid the conflicts1) hbase-client 2) phoenix core 3) betty-all 4) zookeeper 5) hbase-protocol 6) commons-lang 7) commons-configuration 😎 protobuf-java 9) hbase-server 10) guava 11) hadoop-auth 12) hadoop-common 13) commons-collections 14) htrace-core 15) phoenix-core 16) hbase-commonStep 2: Add the krb5.conf file from the cluster to JDK_HOME/jre/lib/security/ that Spotfire is using. The JDK could be embedded in tomcat of spotfire or external. Step 3: Add hbase-site.xml, core-site.xml & hdfs-site.xml to a folder on the spotfire server. And add the folder to the class

Accessing data from industrial control devices can be made easier and safer by utilizing an OPC Server. OPC Servers provide protocol proxy services by translating requests for data from a standardized protocol like OPC to the native protocol of the industrial control device. Since the typical OPC Server can proxy many requests into many different protocols each for a different family of data sources they also make ideal aggregation points for control system data. Centralized data access also has advantages in security that should not be overlooked. The OPC Sever and Nifi can work together to provide certificate based authentication and role based data access to devices in process control networks. Nifi's unique pedigree is capable of complying with the stringent network ingress and regress rules.Get started fast with Kepware and Apache Nifi Out of the box tools available in KepServer Ex can make data available via HTTP and MQTT. ----|Kepware|---HTTP REST API-----|NIFI|or ----|K

PROBLEM: Running the job in notebook through Zeppelin sometimes I am getting 500 error. The following can be seen in livy-livy-server.log file...17/04/25 14:04:30 ERROR SessionServlet$: internal errorcom.fasterxml.jackson.core.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 13)): has to be escaped using backslash to be included in string value at [Source: HttpInputOverHTTP@62c493f2; line: 1, column: 76] at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1419) at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:508) at com.fasterxml.jackson.core.base.ParserMinimalBase._throwUnquotedSpace(ParserMinimalBase.java:472) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2235) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:2165) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser

PROBLEM: I have just started with livy through Zeppelin and have made some livy interpreter config changes i.e. spark memory. When running %livy.sparksc.version I am getting Cannot start sparkSOLUTION:When checking the application log from RM UI I noticed the following...End of LogType:launch_container.shLogType:stderrLog Upload Time:Tue Apr 18 15:13:02 +0200 2017LogLength:142Log Contents:Invalid maximum heap size: -Xmx0mError: Could not create the Java Virtual Machine.Error: A fatal exception has occurred. Program will exit....As you can see, the Xmx value is set to 0m. Doing the further research I noticed the following in livy interpreter:livy.spark.executor.memory 512livy.spark.driver.memory 512Seting the values tolivy.spark.executor.memory 512m livy.spark.driver.memory 512m saving the changes and restarting livy interpreter fixed the problem

This article is to announce the availability of HDP 2.5.5. This release builds on top of the last releases, HDP 2.5.3 for on prem, and HDP 2.5.4 on Azure HDInsight.The key point and changes in this release are fixing big issues for our customers. This release has over 300 product changes that increase stability, fix errors, and improve usability of the product. There are no large features introduced.The release details can be found in the HDP 2.5.5 release notes.Supported OS platforms are CentOS 6/7, SLES 11/12, Debian 6/7, and Ubuntu 12/14This is also going to Azure HDInsight, and should be available there in several days.Thanks everybody! This was a big release, and thanks for all the employees who helped. And of course, the Apache committers who got the fixes into the community.AndrewEng Prog Mgr

Basics of WebSocket protocol WebSocket is a fully duplex and bi-directional protocol. The most interesting characteristics (at least for me) is that there is no difference or roll such as server or client for each peer after those established a WebSocket connection. Each peer can send messages at will. However, a connection is always initiated by a WebSocket client. For example: A client sends a HTTP request to a server (URL e.g. ws://example.com/web-socket). The server accepts the request and upgrade HTTP protocol to WebSocket protocol. (at this point, each peer can send or receive messages asynchronously) The client can send a message to the server The server receives it The server can send a message to the client The client receives it I wondered, should NiFi be a client? or a server? Then decided to support both. NiFi can be not only a WebSocket client, but also a WebSocket server! How it works? Modules Since the protocol itself is more complex tha

SYMPTOM: Recently, we upgraded from Ambari 2.1.2.1 to Ambari 2.2.2.0. Now, we also would like to utilize „Smart Sense“. Per Ambari status it is running. But when opening the „SmartSense“ View we are receiving following error: ERROR: SmartSense Service Unavailable The SmartSense service is currently unavailable. Please make sure the SmartSense Server is up and running. ROOT CAUSE: Customer was having problem with DNS registry, hence it was giving issue.RESOLUTION: Using ipaddress instead of hostname resolved the issue.