Resources

Accessing data from industrial control devices can be made easier and safer by utilizing an OPC Server. OPC Servers provide protocol proxy services by translating requests for data from a standardized protocol like OPC to the native protocol of the industrial control device. Since the typical OPC Server can proxy many requests into many different protocols each for a different family of data sources they also make ideal aggregation points for control system data. Centralized data access also has advantages in security that should not be overlooked. The OPC Sever and Nifi can work together to provide certificate based authentication and role based data access to devices in process control networks. Nifi's unique pedigree is capable of complying with the stringent network ingress and regress rules.Get started fast with Kepware and Apache Nifi Out of the box tools available in KepServer Ex can make data available via HTTP and MQTT. ----|Kepware|---HTTP REST API-----|NIFI|or ----|K

PROBLEM: Running the job in notebook through Zeppelin sometimes I am getting 500 error. The following can be seen in livy-livy-server.log file...17/04/25 14:04:30 ERROR SessionServlet$: internal errorcom.fasterxml.jackson.core.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 13)): has to be escaped using backslash to be included in string value at [Source: HttpInputOverHTTP@62c493f2; line: 1, column: 76] at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1419) at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:508) at com.fasterxml.jackson.core.base.ParserMinimalBase._throwUnquotedSpace(ParserMinimalBase.java:472) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2235) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:2165) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser

PROBLEM: I have just started with livy through Zeppelin and have made some livy interpreter config changes i.e. spark memory. When running %livy.sparksc.version I am getting Cannot start sparkSOLUTION:When checking the application log from RM UI I noticed the following...End of LogType:launch_container.shLogType:stderrLog Upload Time:Tue Apr 18 15:13:02 +0200 2017LogLength:142Log Contents:Invalid maximum heap size: -Xmx0mError: Could not create the Java Virtual Machine.Error: A fatal exception has occurred. Program will exit....As you can see, the Xmx value is set to 0m. Doing the further research I noticed the following in livy interpreter:livy.spark.executor.memory 512livy.spark.driver.memory 512Seting the values tolivy.spark.executor.memory 512m livy.spark.driver.memory 512m saving the changes and restarting livy interpreter fixed the problem

This article is to announce the availability of HDP 2.5.5. This release builds on top of the last releases, HDP 2.5.3 for on prem, and HDP 2.5.4 on Azure HDInsight.The key point and changes in this release are fixing big issues for our customers. This release has over 300 product changes that increase stability, fix errors, and improve usability of the product. There are no large features introduced.The release details can be found in the HDP 2.5.5 release notes.Supported OS platforms are CentOS 6/7, SLES 11/12, Debian 6/7, and Ubuntu 12/14This is also going to Azure HDInsight, and should be available there in several days.Thanks everybody! This was a big release, and thanks for all the employees who helped. And of course, the Apache committers who got the fixes into the community.AndrewEng Prog Mgr

Basics of WebSocket protocol WebSocket is a fully duplex and bi-directional protocol. The most interesting characteristics (at least for me) is that there is no difference or roll such as server or client for each peer after those established a WebSocket connection. Each peer can send messages at will. However, a connection is always initiated by a WebSocket client. For example: A client sends a HTTP request to a server (URL e.g. ws://example.com/web-socket). The server accepts the request and upgrade HTTP protocol to WebSocket protocol. (at this point, each peer can send or receive messages asynchronously) The client can send a message to the server The server receives it The server can send a message to the client The client receives it I wondered, should NiFi be a client? or a server? Then decided to support both. NiFi can be not only a WebSocket client, but also a WebSocket server! How it works? Modules Since the protocol itself is more complex tha

SYMPTOM: Recently, we upgraded from Ambari 2.1.2.1 to Ambari 2.2.2.0. Now, we also would like to utilize „Smart Sense“. Per Ambari status it is running. But when opening the „SmartSense“ View we are receiving following error: ERROR: SmartSense Service Unavailable The SmartSense service is currently unavailable. Please make sure the SmartSense Server is up and running. ROOT CAUSE: Customer was having problem with DNS registry, hence it was giving issue.RESOLUTION: Using ipaddress instead of hostname resolved the issue.

ProblemThis problem happened on HDP 2.5.3 when running Spark On HBase. Here is the error seen in the application log:... 17/04/11 10:12:04 WARN RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid17/04/11 10:12:05 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)17/04/11 10:12:05 WARN ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnectjava.net.ConnectException: Connection refusedat sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125) ...SolutionTo fix that problem ensur

Problem Statement: User cannot modify or start restricted components such as getHDFS in NIFI 1.1 with Ranger enabled.Solution: In Ranger you must grant /restricted_components permissions to the user in order for them to have access to the restricted components in NIFI.

To remove a already installed Grafana, following needs to be done:1. Stop the AMS service from Ambari UI.2. Execute following curl API commands to delete Grafana:# curl -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop HDFS via REST"}, "Body": {"ServiceInfo": {"state": "INSTALLED"}}}' http://<strong><ambari-server-hostname></strong>:8080/api/v1/clusters/<strong><CLUSTERNAME></strong>/services/AMBARI_METRICS/components/METRICS_GRAFANA# curl -u admin:admin -i -H 'X-Requested-By: ambari' -X PUT -d '{"RequestInfo": {"context" :"Stop HDFS via REST"}, "Body": {"ServiceInfo": {"state": "INSTALLED"}}}' http://<strong><ambari-server-hostname></strong>:8080/api/v1/clusters/<strong><CLUSTERNAME></strong>/services/AMBARI_METRICS/# curl -u admin:admin -H "X-Requested-By: ambari" -X DELETE http://<strong><ambari-server-hostname></strong>:8080/api/v1/clusters/<strong&

How to increase Read time out value from Ambari stand alone view server to Ambari main serverStand alone Ambari view server makes below call to fetch configurations from Remote server.http://REMOTEAMBARI:8080/api/v1/clusters/CLUSTER_NAME?fields=services/ServiceInfo,hosts,ClustersIf this takes more than 10 seconds you will be exception in the logs. because of that non of the views would work.Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) at sun.net.www.http.HttpClie

some times if you run some curl calls expecting to fix some other issues - it may turn out to that Ambari UI does not even load the UI.Running the below call would fix the issue and load the UI.curl -u admin:admin -i -H 'X-Requested-By: ambari' -X POST -d '{"wizard-data":"{\"userName\":\"admin\",\"controllerName\":\"\"}"}'http://AMBARIHOST:8080/api/v1/persistNote: Please note that these call should run with caution. contact Hortonworks support team.This is tried and tested with Ambari-2.4.x

QuestionWhat steps should a Cloudera Altus administrator take if a Cloudera Altus end-user leaves their company, or should no longer have access to Cloudera Altus? AnswerIf a user leaves their company, a Cloudera Altus administrator will need to delete the Cloudera Altus CLI Keys and remove user access from the Web UI to prevent unauthorized access.

QuestionWhere are Cloudera Altus Clusters created and how are Data Engineering jobs run? AnswerClusters are spun up and reside within customer's public cloud account. For example, in case of AWS, EC2 instances are spun-up and managed by Cloudera Altus in the subnet and VPC specified by customer created environment; These clusters handle specific Data Engineering workloads (Hive-on-Spark, Spark-on-YARN, MR2 on YARN, etc). 

QuestionWhat are the various Cloudera Altus interfaces available and how can I access them? AnswerCloudera Altus is accessible via command line interface (CLI), console / graphical user interface (GUI) and via a software development kit (SDK) for use with Java. The Cloudera Altus Web console is available via: https://console.altus.cloudera.com/ Instructions for installing the Cloudera Altus CLI is available here (link to Cloudera Altus CLI Documentation) Additional details for the Cloudera Altus SDK and coding examples are available here (link to Cloudera Altus SDK) Additional details on interfaces are available in the Cloudera Altus Interfaces section of the Cloudera Altus Documentation. 

QuestionAre there any current limitations with Cloudera Altus? AnswerDefault limit for commercial have the following default soft-limits: Up to 10 active clusters at any given timeLimit of 100 usersLimit of 10 defined Cloudera Altus environmentsIf there are needs required that will utilize beyond the current soft-limits, please reach out to Cloudera Support to request an increase. Users that are given access as part of a free trial will have no ability to create additional users and may have smaller limits. 

QuestionCan I change an environment once created in Cloudera Altus? AnswerBy design, Cloudera Altus environments are not modifiable once created. To make a change to the environment, we recommend to clone the environment into a new one with the appropriate changes.

QuestionCan I use a bootstrap script to further configure the cluster?  AnswerInstance bootstrap scripts are executed on all the cluster instances immediately after startup before any services are configured and started.  You can use it to install additional OS packages or add application dependencies (e.g. copy a jar to the cluster, etc). This is not intended to be a mechanism for cluster configuration.

QuestionIs logging enabled by default for jobs run in Cloudera Altus Data Engineering? AnswerWhile we recommend enabling logging for troubleshooting Data Engineering jobs, It is not enabled by default.   Details for enabling logging within the Cloudera Altus Environment Setup is available in the online doc Cloudera Altus Environment Setup:https://console.altus.cloudera.com/support/documentation.html?type=topics&page=ag_dataengr_get_start_env_setupNOTE: If logging is not enabled, cluster bundles that are automatically generated during shutdown of a Cloudera Altus-managed cluster will be lost.

QuestionWhat are the minimum and maximum cluster sizes? AnswerThe minimum cluster size is 3 worker nodes, 1 Master node, and 1 Cloudera Manager node, for a total of five nodes.  The maximum cluster size varies based on cluster type and public cloud. Further information about this could be found in this  Cloudera Altus Documentation section NOTE: Each cluster spun up by Cloudera Altus requires 2 additional (master) nodes, one for CM and the other for Master services used for each cluster.  These 2 nodes are a hard-requirement per cluster as they're required for a functional cluster and will need to be accounted for when calculating the public cloud instance limits. 

Use Cases: Seperate out hiveservers for batch jobs which can ensure user jobs or too many connections from users dont effect batch workloads or one groups workloads dont effect other group workloads.Secure hive connections through Knox which only works with http mode for hiveservers2.Custom configurations for different workloads such as different engine's MR or tez or anything else. Method:Use ambari config groups to customize the default hive.server2.zookeeper.namespace to something else other than default value “hiveserver2”Reference: http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_hadoop-high-availability/content/ha-hs2-service-discovery.htmlhttp://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-operations/content/using_host_config_groups.htmlCaveat: The beeline connection string will be different from the default one. The new one needs different value for the zooKeeperNamespace which can cause confusion. Example: I will be using Ambari 2.4.2 with HDP 2.3.2 b

These checklists help ensure that you avoid unexpected problems when adding new hosts to an HDP cluster. Preparation checklist (what to do before adding the new hosts via the Ambari Add Host wizard): Have you followed the guidance in the Ambari installation documentation (e.g. example install doc is here)? It is recommended to follow the instructions for the HDP version that is installed on the cluster (go to http://docs.hortonworks.com). Does your file system partitioning scheme, storage options, memory, network, and power configurations match what is used for other nodes on the cluster? It is good to either follow the pattern of what is already on the cluster or to follow recommendations from docs.hortonworks.com (see example doc here). Compare and verify that settings and configurations match, including: THP, selinux, ssh keys, user integration, LDAP/AD setup, SSSD integration, drive layouts, etc. Have you verified that JAVA_HOME is set properly with same Java version across

QuestionWhat type of access Cloudera Altus Cluster provides for Cloudera Manager user? AnswerRead-only access. User (guest by the default) can't modify or change settings or configurations in Cloudera Manager. 

QuestionCan I choose the Linux distribution that a Cloudera Altus Cluster will be running on?  AnswerCurrently, the selected Linux distribution for deployment in Cloudera Altus Clusters is based on CentOS 7.x 64-bit.  At this time, no other Linux distrubtions are available for use with Cloudera Altus.  

QuestionDoes Cloudera save AWS user credentials (AWS Access Key ID or AWS Access Key)? AnswerNo, Cloudera doesn't save AWS user credentials. If Cloudera Altus environment is set-up using the Cloudera Altus Quickstart, the user is prompted for these credentials, but they are never sent to Cloudera Altus. They are used by the web application residing within the user's brower that sends commands directly to AWS to create necessary resources using a CloudFormation script.In case of the Environment being created via Cloudera Altus Wizard, credentials will never need to appear inside the Cloudera Altus console. User needs to create necessary resources in AWS console and then grant access to them using AWS cross account access mechanism, which does not involve explicit key management.

QuestionDoes Cloudera have access to the customer data in Cloudera Altus Clusters? Answer Spoiler (Highlight to read)  No.  Cloudera Altus clusters are setup with the expectation that data will reside on the cloud provider's object storage (AWS S3, Azure ADLS).  When data resides in this object storage (which exists inside a customer account within AWS or Azure), the Cloudera Altus service does not have access to this data.