Resources

ENVIRONMENT and SETUPI did test the below solution withHDP-2.6.0.3-8Ambari 2.5.0.3To configure this environment, follow up the steps defined for HDP 2.5 (Zeppelin 0.6) - https://community.hortonworks.com/articles/81069/how-to-enable-user-impersonation-for-sh-interprete.htmlAs the Zeppelin UI -> Interpreter window has changed, follow this up.I. Default config looks like below: II. Do the following change:- edit the interpreter - change The interpreter will be instantiated from Globally to Per User and shared to isolated - then you will be able to see and select User Impersonate check-box - also removezeppelin.shell.auth.typezeppelin.shell.keytab.locationzeppelin.shell.principal III. Changed config looks like below:After Saving the changes and running below as admin%shwhoamiWe get

The key thing to remember is that in Spark RDD/DF are immutable. So once created you can not change them.However there are many situation where you want the column type to be different.E.g By default Spark comes with cars.csv where year column is a String. If you want to use a datetime function you need the column as a Datetime. You can change the column type from string to date in a new dataframe.Here is an example to change the column type.val df2 = sqlContext.load("com.databricks.spark.csv", Map("path" -> "file:///Users/vshukla/projects/spark/sql/core/src/test/resources/cars.csv", "header" -> "true"))df2.printSchema()val df4 = df2.withColumn("year2", 'year.cast("Int")).select('year2 as 'year, 'make, 'model, 'comment)df4.printSchema

PROBLEM: Oozie jobs gets stuck in PREP modeROOT CAUSE : Below are the possible reasons:1. Wrong Namenode host/port in job.properties2. Wrong Resource manager host/port in the configurations.If there are lot of jobs stuck in the RESOLUTION : 1. Stop Oozie server from Ambari.2. Backup Oozie DB is cluster is production.3. Remove entries for stuck jobs from below tablesWF_JOBSCOORD_JOBSWF_ACTIONS4. Start Oozie server

delete-scripts.zipAttached are scripts you can use for ORACLE, PSQL, and MYSQL for deleting users. Directions to how these scripts work are included in the script.Good practice is to make sure to backup your db before attempting to use and test these scripts.

I tried logging out and then logging back in several times but could not find the button to ask a question. Does this happen to anyone else?

httpfs is needed to support a centralized WebHDFS interface to an HA enable NN Cluster. This can be used by Hue or any other WebHDFS enabled client that needs to use a cluster configured with a High-Availability Namenode.The installation is a piece of cake:yum install hadoop-httpfsBut that's were the fun ends!!! Configuring is a whole other thing. It's not hard, if you know the right buttons to push. Unfortunately, the buttons and directions for doing this can be quite aloof.The httpfs service is a tomcat application that relies on having the Hadoop libraries and configuration available, so it can resolve your HDP installation.When you do the installation (above), a few items are installed./usr/hdp/2.2.x.x-x/hadoop-httpfs/etc/hadoop-httpfs/conf/etc/hadoop-httpfs/tomcat-deploymentConfiguring - The Short VersionSet the version for current with hdp-select set hadoop-httpfs 2.2.x.x-xFrom this point on, many of our changes are designed to "fix" the "hardcoded" implementations in the dep

This article is related to creating SNMP alerts through the custom script and how to troubleshoot. Install SNMP on sandbox or local environment.yum install net-snmp net-snmp-utils net-snmp-libs –y Change the script /etc/snmp/snmptrapd.conf file and include "disableAuthorization yes".# Example configuration file for snmptrapd## No traps are handled by default, you must edit this file!## authCommunity log,execute,net public# traphandle SNMPv2-MIB::coldStart /usr/bin/bin/my_great_script colddisableAuthorization yes To understand why this change required, Refer to the Access Control section of this link. Copy APACHE-AMBARI-MIB.txt file to /usr/share/snmp/mibs folder. cp /var/lib/ambari-server/resources/APACHE-AMBARI-MIB.txt /usr/share/snmp/mibs Note: Ensure it has proper permission Startup a simple SNMP trap daemon to log traps to the /tmp/traps.log file for testing purposes.nohup snmptrapd -m ALL -A -n -Lf /tmp/traps.log &​ Invoke a test trap to ensure that the snmptrapd

1 OverviewIn this article I am going to talk about three main integration pieces between SAP and Hadoop. First I will cover the SAP integration products and how they can be used to ingest data into hadoop systems. Second, I am going to talk about SAP hana and how that integrated with hadoop. I will provide a overview of various methods that can be used to push/pull data from SAP Hana as well as aspects around data federation and data offload architecture. Third I am going to touch on topic around how SAP BI tools integrate with Hadoop and talk briefly about SAP Hana Vora.2 Hadoop using SAP Data Integration componentsSAP has three primary products around data integration. Based upon their integration pattern, here is a brief overview of these productsBatch processing SAP Data services aka BODS (Business Objects Data Services) is the batch ETL tool from SAP. SAP acquired this product as part of its Business Objects acquisition.Real time ingestion SLT (SAP Landscape Transformation) Repl

Short Description:This article will help to renew kerberos ticket with Falcon. ArticleIt has been seen that Falcon server fails to perform operations after kerberos credentials expire and following exception occur.  Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)   To solve this issue, we must set following parameter in Falcon startup.properties through ambari-ui to revalidate kerberos credentials. Value for this properties is in seconds.Stop Falcon serverSet parameter : *.falcon.service.authentication.token.validity=<value in seconds>Start Falcon serverNote: This article is for version greater than HDP-2.5.* .

OBJECTIVE:Add more useful service-level metrics for NiFi in Cloudera Flow Management (CFM)/Cloudera Data Flow. OVERVIEW:The default install of early releases of Cloudera Flow Management (CFM) includes only a few basic metrics for NiFi in the Cloudera Manager dashboards. This article explains how to add several more service and host level metrics that improve the performance monitoring in Cloudera Manager 5.x or 6.x. PREREQUISITES:CFM (CDF) 1.0.1 or later and CM 5.16 or CM 6.2 or later ADDING NiFi METRICS TO CM Basic Metrics1. From the NiFi status page in Cloudera Manager, select “Chart Builder” from the “Charts” menu.2. Insert one of the following SELECT statements in the query text box: NiFi Memory SELECT total_mem_rss_across_nifi_nodes, total_mem_virtual_across_nifi_nodes WHERE category = SERVICE and entityName = $SERVICENAME  CPU Rates SELECT cpu_user_rate_across_nifi_nodes, cpu_system_rate_across_nifi_nodes WHERE entityName = $SERV

When developing NiFi flows with python, it's sometimes a headache to build customized configurations for your processors. Every processor is different and has many different parameters to customize depending on your flows needs and requirements. To accelerate development, I use the development tools in my chrome browser to help accelerate development by extracting the config files from the rest calls the NiFi application makes.  Open your browser and point it to your NiFi instances. Once in the UI right click on the gray bar and select "inspect".This will open up a development view of the UI. Next, click on the network tab to view and debug network activity.  In this view, all network related traffic will be displayed here.   Notice you can see all the rest calls that the NiFi is making behind the scene. This can be quite helpful when customizing processors. When you drag a processor on the canvas and parameterize it, a rest call is made that creates

Building Basic Flows with Nipyapi If building flows in NiFi wasn't easy enough, developers can use the Nipyapi library to build flows with python. With this model, one can programmatically build flows without having access to the canvas. This comes in handy when deploying highly configurable and repeatable flows.  In this simple example below, we'll walk through using the python libraries to build a very simple flow whereby we use the GenerateFlowFile processor to write data to the local file system. Very contrived but one can get the picture of the flexibility afforded by the nipyapi libraries.  For more information on the python libraries discussed here, please go to:https://nipyapi.readthedocs.io/en/latest/index.html First things first, you'll need to import the required library and classes for the example. from nipyapi import canvas,configfrom random import randrangeNext set the configuration details of your NiFi environment. Here, I'm setting the

Hi, all I got a misspelled word in my nickname when I joined cloudera.how can I change my nickname in profile?( because for now I cant change it )

Why Share a Hive Metastore? Many organizations are moving workloads to the cloud to take advantage of the flexibility it offers. A key to this flexibility is a shared, persistent repository of metadata and services. Ephemeral compute clusters scale up and down and connect to this shared service layer. All clusters and metadata services share a unified storage layer. These capabilities are at the core of Cloudera’s next-generation product, the Cloudera Data Platform. Can we deploy this architecture today, with Hortonworks Data Platform 3?  A key piece to this architecture is sharing a single Hive Metastore between all clusters. Hive is HDP’s SQL engine. The Hive metastore contains the metadata which allows services on each cluster to know where and how Hive tables are stored, and access those tables.  Let’s look at our options. Standalone Hive Metastore Service A standalone Hive Metastore Service could be installed on a node outside of t

To encrypt LDAP bind password in zeppelin Shiro configuration follow below steps:1. Removed system password line from the config (this line contained the password for the system user that is doing the LDAP lookups) 2. Added the following line in shiro.ini:  ldapRealm.hadoopSecurityCredentialPath=jceks://file/home/zeppelin/conf/zeppelin.jceks  3. Run the following command as Zeppelin user on Zeppelin host:  # hadoop credential create ldapRealm.systemPassword -provider jceks://file/home/zeppelin/conf/zeppelin.jceks #Entered password <test>  Where <test> is my password 4. Confirmed that file has 700 attributes 5. Restart Zeppelin 

Jdbc interpreter is will take default queue to run a job.    To run the job in specific queue add the queue name in hive.url property in JDBC interpreter settings  as shown below       hive.url=jdbc:hive2://<zookeeper quorum>/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2?tez.queue.name=<Queue Name>     EXAMPLE:      hive.url= jdbc:hive2://c186-node2.-labs.com:2181,c186-node3.-labs.com:2181,c186-node4.-labs.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2?tez.queue.name=sai     Where sai is my queue name.   To check and confirm execute below query in zeppelin using JDBC interpreter and check-in Yarn Resource manager UI     %jdbc(hive)show databases;use <database name>;show tables;select count (*) from <table name>;    

  Apache NiFi is an extremely versatile tool that can be used in just about ANY streaming/real-time architecture to enable rapid development and efficient monitoring and maintenance. Often times it is easiest to learn by doing something hands on and tangible.   With that in mind let's look at something most of us are familiar with. Our irrigation control system. Almost every new home in Texas has one of these systems included, and all commercial land establishments such as Golf Courses, Farms, Parks and Businesses with outdoor landscaping. This can also easily be applied to Fire Safety systems where a hydrant is triggered remotely to minimize damages caused by a fire.    To get started we will need to set up our irrigation control system to accept commands remotely. You can skin this cat several different ways including buying a wifi enabled irrigation system, but where is the fun in that. Let's get our hands dirty and kick off with some embedded programming.&nbsp

HDFS Snapshots In Part 1 we looked at the basics of HDFS snapshots. In next section we'll look at what happens with managing multiple snapshots in a given directory and what to look out for... primarily phantom data that exists when you delete a dataset that has a snapshot or multiple snapshots linked to it.   Working With Multiple Snapshots First lets take a new snapshot on the directory we were previously working on:   hdfs dfs -createSnapshot /tmp/snapshot_dir/dir1 20190819v1   Next lets add another file in the same base directory that we have file1.txt in:   hdfs dfs -put file2.txt /tmp/snapshot_dir/dir1   Let's pretend that 8 hours later, you take another snapshot of the same directory. Here we named the snapshot v2:   hdfs dfs -createSnapshot /tmp/snapshot_dir/dir1 20190819v2   Let's pause and take a look at the size of the directory:   hdfs dfs -

 Part 2 of Autoscaling MiNiFi on K8S is focused on deploying the artifacts on AKS - Amazon Kubernetes Service.  My knew jerk reaction was all Kubernetes as a Service would play well but that is definitely not the case.  Hence why GCP Anthos product direction for this space is a key. The net-net of my observation is k8s app deployment on any single cloud vendor would cause deployment complexities any other k8s deployment, cloud or OnPrem..  Vendor lock in theory is ALIVE and WELL.  In Azure I leveraged ACS for EFM and NiFi Registry; however, the natural evolution was to deploy EFM and NiFi Registry (NR) on K8S.  EFM, NR, and MiNiFi are integrated components (refer to part 1 on architecture).  I will leverage several key out of the box k8s components to make this all work together.  The good news is, the deployment is super simple! PrerequisitesSome knowledge of Kubernetes and AKSAKS Clusterkubectl clieksctl cliVPC2 public subnets wi

SYMPTOM: We have some alerts about high heap size in datanode in Ambari for production cluster. The maximum of heap size of the datanode is set to 16GERROR: Below is the snapshot ROOT CAUSE: DN operations are IO expensive do not require 16GB of the heap. RESOLUTION: Tuning GC parameters resolved the issue -4GB Heap recommendation : -Xms4096m -Xmx4096m -XX:NewSize=800m -XX:MaxNewSize=800m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=70 -XX:ParallelGCThreads=8

In HDP-2.6/Ambari-2.6, it was not mandatory enable HS2 metrics explicitly. Thus, all metrics would be emitted without defining any configs explicitly. In HDP-3/Ambari-2.7, we will see similar erros in AMS Collector Log:Error : 2019-06-10 02:42:59,215 INFO timeline timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20Debug Error shows this :2019-06-14 20:35:29,538 DEBUG main timeline.HadoopTimelineMetricsSink: Trying to find live collector host from : exp5.lab.com,exp4.lab.com 2019-06-14 20:35:29,538 DEBUG main timeline.HadoopTimelineMetricsSink: Requesting live collector nodes : http://exp5.lab.com,exp4.lab.com:6188/ws/v1/timeline/metrics/livenodes 2019-06-14 20:35:29,557 DEBUG main timeline.HadoopTimelineMetricsSink: Unable to connect to collector, http://exp5.lab.com,exp4.lab.com:6188/ws/v1/timeline/metrics/livenodes 2019-06-14 20:35:29,557 DEBUG main timeline.HadoopTimelineMet

Hello, I had trouble to post questions on the community. It keeps marking my question as spam and I dispute to moderator get no answers. Does anyone know how to post questions without being treated as spam thanks,Harry

This article describes the configuration of the authorizers.xml for using the Composite User Group Provider for both LDAP and File based authentication in Cloudera Manager for CDF 1.0 This article assumes TLS has already been configured for NiFi using either the NiFi CA or your own certs. Here is the Active Directory Structure that I will use for my sync:Here are the groups in the Groups OU:Here are the users in the Users OU:Here are the service users in the ServiceUsers OU: First, I will import my Active Directory root certificate into the NiFi and Java Truststores:  keytool -import -file adcert.pem -alias ad -keystore /var/lib/nifi/cert/truststore.jkskeytool -import -file adcert.pem -alias ad -keystore /etc/pki/java/cacerts  Now login to Cloudera Manager and proceed to the NiFi Configuration. Update the configuration to use the "composite-user-group-provider" as follows:  xml.authorizers.accessPolicyProvider.file-access-policy-provide

HDFS Snapshots HDFS Snapshots are a great way to backup important data on HDFS. It's extremely easy to implement and it helps safeguard your data from instances where a user or admin accidentally deletes data. In the article below, we'll walkthrough some simple examples of using snapshots and some of the gotchas to look out for when implementing them.   Part 1: Understanding Snapshots First lets create some files and directories for testing:  echo "Hello World" > file1.txtecho "How are you" > file2.txtecho "hdfs snapshots are great" > file3.txthdfs dfs -mkdir /tmp/snapshot_dirhdfs dfs -mkdir /tmp/snapshot_dir/dir1   Next lets put file1.txt in the directory:  hdfs dfs -put file1.txt /tmp/snapshot_dir/dir1   Creating a snapshot is a really simple process. You will need a user with superuser permissions to create a snapshottable directory. With this process, we enable the directory to have snapshots but we

hi all   after hortonworks migration I see that nickname displayed instead my name? ( nickname is my secret name and should not be displayed !! how can I change my nickname in profile? ( because for now I cant change it )