Community Articles

Issue:Whilst running a 'select count(*)' or 'analyze table compute statistics' on a regex table & if you see the below error, it indicates you have created the table with old serde. Prior to Hive 0.10, 'RegexSerDe' is part of 'hive-contrib' library. From Hive 0.10 onwards the serde is part of 'hive-serde-<version>.jar'ERROR : Status: FailedERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1496529203645_0883_2_00, diagnostics=[Task failed, taskId=task_1496529203645_0883_2_00_000010, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtim

I recently worked with a customer to demo HDF/NiFi DataFlows for their uses cases. One use case involved converting delimited row-formatted text files into Parquet and ORC columnar formats. This can quickly be done with HDF/NiFi. Here is an easy to follow DataFlow that will convert row-formatted text files to Parquet and ORC. It may look straightforward; however, it requires some basic knowledge of Avro file formats and use of the Kite API. This article will explain the DataFlow in detail. STEP 1: Create an Avro schema for the source data My source data is tab delimited and consists of 8 fields. Notice in the DataFlow that before converting the data to Parquet and ORC the data is first converted to Avro. This is done so we have schema information for the data. Prior to converting the data to Avro we need to create an Avro schema definition file for the source data. Here is what my schema definition file looks like for the 8 fields. I stored this in a file named sample.avsc. More info

Background: when it comes to rely completely on ranger, and if you are specific about configuring authorization for a resource to an end user then you have to create one policy for each resource. There should be a way to configure a policy that provide access to specific resources based on the User who is making call. Soultion:{USER} Support: {USER} support solves this problem , It allows us to create a policy where we can configure resource as {USER} Eg. /user/{USER} and select user also as {USER}. that means all users will get access of their corresponding home directory. Eg. Hdfs: resource: /user/{USER} user1 will have access to /user/user1 user2 will have access to /user/user2 Hive: resource: database:database_{USER} user1 will have access to database database_user1 user2 will have access to database database_user2 resource may contrain {USER} partially or fully. delimiter can be customised also , Steps to configure {USER}:1) go to ranger admin, and create policy page, there on re

PROBLEM: While implementing AutoHDFS for storm-hdfs integration, observed following error:2017-05-19 11:21:44.865 o.a.s.h.c.s.AutoHDFS [ERROR] Could not populate HDFS credentials. java.lang.RuntimeException: Failed to get delegation tokens. at org.apache.storm.hdfs.common.security.AutoHDFS.getHadoopCredentials(AutoHDFS.java:242) at org.apache.storm.hdfs.common.security.AutoHDFS.populateCredentials(AutoHDFS.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) at org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__11226.submitTopologyWithOpts(nimbus.clj:1544) at org.apache.storm.generated.Nimbus$Processor$submitTopolo

The NiFi S2S protocol is used by NiFi's Remote Process Group (RPG) components to distribute FlowFiles from one NiFi instance to another. When the target NiFi is a NiFi cluster, load-balancing of the FlowFie delivery is done across all nodes in the target NiFi cluster. The default way this works (and the only way it works in versions of NiFi previous to Apache NiFi 1.2.0 or HDF 3.0) is as follows: The RPG regularly communicates with the target NiFi cluster to get load status information about each node in the cluster. This information includes the number of currently connected nodes in the target cluster, each node's hostname, port information, and the number of total queued FlowFiles on each target NiFi node. The Source NiFi uses this information to determine a data distribution strategy for its source FlowFiles it has queued. - - - Let's assume a 4 node target NiFi cluster all reporting a zero queue count. - Each node will then be scheduled to receive 25% of the data. This means a di

Upon setting up a Socks Proxy Connection to access Cloudera Manager on an Altus cluster per:   https://www.cloudera.com/documentation/altus/topics/ug_dataengr_trblshoot_socks_proxy.html#socks_proxy   You may see:   ssh: connect to host X.X.X.X port 22: Connection timed out 2017-06-26 09:16:05,759 - MainThread - altuscli.extensions.dataeng.socksproxy - ERROR - IP address of Cloudera Manager (X.X.X.X) is unreachable. Please check security permissions on instances of the cluster.   The same steps should be followed, listed here:   https://www.cloudera.com/documentation/altus/topics/ug_dataengr_trblshoot_ssh_conn.html   However, you should slightly modify these steps to allow access Cloudera Manager (port 7180 In the AWS Console, go to VPC > Security Groups and find the security group you created for the Altus environment. Verify that you are in the correct region. On the Inbound Rules tab, edit the security group and add another rule of type Custom TCP Ru

SYMPTOM :=> This problem occurs in case of a partitioned table without any null partitions and contains approximately more than 600 columns in the table=> Following stacktrace is observed in hive metastore logsNested Throwables StackTrace:org.datanucleus.store.rdbms.exceptions.MappedDatastoreException: INSERT INTO "PARTITION_PARAMS" ("PARAM_VALUE","PART_ID","PARAM_KEY") VALUES (?,?,?) at org.datanucleus.store.rdbms.scostore.JoinMapStore.internalPut(JoinMapStore.java:1056) at org.datanucleus.store.rdbms.scostore.JoinMapStore.put(JoinMapStore.java:307) at org.datanucleus.store.types.wrappers.backed.Map.put(Map.java:653) at org.apache.hadoop.hive.common.StatsSetupConst.setColumnStatsState(StatsSetupConst.java:285) at org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:6237) at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Dele

SYMPTOM:Incorrect status shown for the DAGs in Tez UI ROOT CAUSE This is a known issue (https://issues.apache.org/jira/browse/TEZ-3656). It will only happen for the killed applications or if there was a failure to write into Application Timeline Server. It should not cause any issues, except for the wrong status for the DAG in the TezUI. RESOLUTION:This is fixed in HDP 2.6.1 release

PROBLEM DEFINITION: CREATE TABLE DT(Dérivation string, Pièce_Générique string);Throws ParserException ErrorROOT CAUSE/ WORKAROUND: Hive database name, table name and/or column names cannot contain Unicode string. However, Hive supports UTF-8 and Unicode string for only the table data/comments. LINKS:https://cwiki.apache.org/confluence/display/Hive/User+FAQ

The original configurations and the original CLI options are respectively discussed in Section 1 & 2. The new configurations and the new CLI options are respectively discussed in Section 3 & 4.1. Original ConfigurationsConcurrent Movesdfs.datanode.balance.max.concurrent.moves, default is 5This configuration is to limit the maximum number of concurrent block moves that a Datanode is allowed for balancing the cluster. If this configuration is set in a Datanode, the Datanode will throw an exception if the limit is exceeded. If it is set in the Balancer, the Balancer will schedule concurrent block movements within the limit. Note that the Datanode setting and the Balancer setting could be different. Since both settings impose a restriction, the effective setting is the minimum of them.Before HDFS-9214, changing this configuration in a Datanode requires restarting the Datanode. Therefore, it is recommended to set it to the highest possible value in Datanodes and adjust the runt

Steps to configure 2way SSL between Ambari-server and Ambari-agent by using custom certs. Here I have used CA Signed certs at server side and Agent certs are generated dynamically. if you are planning to use CA Signed cert at Agents side then for every Agent install you may have to copy the certs and do manual work.1. Make sure to have fresh keys folder. (if you do not have one, you can copy the folder from one of the fresh install machine or do following). - Delete all the crt and csr files that starts with hostname at /var/lib/ambari-server/keys. - Empty /var/lib/ambari-server/keys/db/index.txt file - Delete any certs under /var/lib/ambari-server/keys/db/newcerts/ 2. - Copy your own Signed Certificate, key files /var/lib/ambari-server/keys/ Ex: certificate name is - ca-cust.crt, ca-cust.key 3. Create PKCS keystore file from your cert and key files. Ex:openssl pkcs12 -export -inkey /tmpr/keys/ca.key -in ca-cust.crt -out /tmp/keys/keystore.p12 -password pass:bigdat

We had kerberised the HDP cluster and storm UI was not opening and we need to troubleshoot StormUI issue.To put DEBUG on for STormUI follow this steps:--------------------------------------------------------------In Ambari - Storm - Config - STORM UI Server section add below :ui.childopts(original)= -Xmx768m_JAAS_PLACEHOLDERui.childopts = -Xmx768m_JAAS_PLACEHOLDER -Dsun.security.krb5.debug=truewe wanted to troubleshoot zookeeper as well To put DEBUG on for Zookeeper follow this steps:------------------------------------------------------------------In Amabri - Zookeeeper - Advanced Zookeeper-log4jadd below:#log4j.rootLogger=INFO,CONSOLE ***Comment this linelog4j.rootLogger=DEBUG,CONSOLE,ROLLINGFILE ***uncomment this lineThe log file will be generated by default to /home/zookeeper/zookeeper.log if below property is set as default.log4j.appender.ROLLINGFILE.File=zookeeper.log

Overview This article reviews the steps necessary to update Hive entities within Atlas the Description and Comment fields. The 0.70 Atlas release will display and allow text searches on the ‘‘description’ field, but the Atlas UI does not at this time support the ability to manually enter those properties into a given data Asset. Examined in this article includes: Searching for a Hive_Table entity Update a single property for the Hive_Table entity definition (“description”)The Problem: In release 0.70, Atlas has the ability to monitor additions as well as changes to Hive table and Hive columns. When Atlas identifies a new entry or change the appropriate Metadata property is updated for that entity. One very cool aspect to Atlas is the ability to conduct either DSL or free text searches on any properties set for the entity. Anyone trying to identify datasets to support a specific analytic activity will definitely appreciate the ability search through all of the entities and qui

STEPS TO REPRODUCE:1. Create a Phoenix connection 2. CREATE SCHEMA test 3. DROP SCHEMA test4. Now create the test schema again. This will throw below exceptions :0: jdbc:phoenix:> create schema "test"; Error: ERROR 721 (42M04): Schema with given name already exists schemaName=test (state=42M04,code=721) org.apache.phoenix.schema.SchemaAlreadyExistsException: ERROR 721 (42M04): Schema with given name already exists schemaName=test5. Now attempt to drop the test schema again:0: jdbc:phoenix:> drop schema "test";Error: ERROR 722(43M05):Schema does not exists schemaName=test (state=43M05,code=722) org.apache.phoenix.schema.SchemaNotFoundException: ERROR 722(43M05):Schema does not exists schemaName=testIf you look in Hbase and list_namespace, you will see the test schema still exists.ROOT CAUSE : This is a known issue discussed in Apache JIRA PHOENIX-3694SOLUTION : Issue is fixed in Phoenix 4.10. For a backport hotfix , please log a case with HWX.

The HDFS NameNode ensures that each block is sufficiently replicated. When it detects the loss of a DataNode, it instructs remaining nodes to maintain adequate replication by creating additional block replicas. For each lost replica, the NameNode picks a (source, destination) pair where the source is an available DataNode with another replica of the block and the destination is the target for the new replica. The re-replication work can be massively parallelized in large clusters since the replica distribution is randomized. In this article, we estimate a lower bound for the recovery time.Simplifying Assumptions The maximum IO bandwidth of each disk is 100MB/s (reads + writes). This is true for the vast majority of clusters that use spinning disks.The aggregate IO capacity of the cluster is limited by disk and not the network. This is not always true but helps us establish lower bounds without discussing network topologies. Block replicas are uniformly distributed across the clus

QuestionIs the Cloudera Altus CLI supported for Windows systems? AnswerStarting with the Cloudera Altus CLI Version 1.1.3, installation is available through pip, as long as the following minimum requirements are met: Minimum version of Python 3.6+ 1) Download and Install Python for Windows according to the website's instructions:     https://www.python.org/downloads/windows/ 2) Open a Command Prompt (Windows Key + R), then type cmd and hit enter. 3) In the Command prompt window, run the command to update the Windows environment to add the newly installed Python folder to the system path:  setx "%PATH%";C:\<Python_Install_destination>\Scripts;    For example, if Python 2.7 was installed in it's default location, the following command would be run:    setx "%PATH%";C:\Python36\Scripts; 4) In the same Window, run the pip installer to install the Cloudera Altus

Device Behavior Analytics allows administrators to detect straggler nodes/disks which reduce the cluster performance. Finding and fixing these nodes can improve the overall cluster throughput. This feature will be available in HDP-2.6.1 and later. There are two parts to Device Behavior Analytics – Slow Datanode Detection and Slow Disk Detection.SLOW DATANODE (PEER) DETECTION Datanode will collect latency statistics about their peers during the normal operation of the datanode write pipeline. These latency stats are used to detect outliers among the peers. Slow peer detection is not performed unless the datanode has statistics of at least 10 peers. Namenode maintains the list of slow peers and administrators can read it via JMX. Datanodes also expose the average write latency of their peers through Datanode JMX.SLOW DISK DETECTION Each datanode will collect I/O statistics from all its disks. We can configure the percentage of file I/O events to be sampled to limit the performance i

Current on version HDP-2.6.x - Ambari-2.5.x if the zookeeper principal name is changed or customized manual changes are required for HDFS, Yarn and Ambari-Infra.In Ambari - config - for yarn and HDFS: yarn-env.sh YARN_OPTS="-Dzookeeper.sasl.client=true -Dzookeeper.sasl.client.username=sand-box-zookeeper -Djava.security.auth.login.config=/etc/hadoop/2.6.0.3-8/0/yarn_jaas.conf -Dzookeeper.sasl.clientconfig=Client $YARN_OPTS" hadoop/conf/hadoop-env.sh export HADOOP_ZKFC_OPTS="-Dzookeeper.sasl.client=true -Dzookeeper.sasl.client.username=sandbox-zookeeper -Djava.security.auth.login.config=/usr/hdp/current/hadoop-client/conf/secure/hdfs_jaas.conf -Dzookeeper.sasl.clientconfig=Client $HADOOP_ZKFC_OPTS" For Ambari-Infra/Solr edit /usr/lib/ambari-infra-solr-client/solrCloudCli.sh PATH=$JAVA_HOME/bin:$PATH $JVM -classpath "$sdir:$sdir/libs/*" -Dzookeeper.sasl.client.username=sandbox-zookeeper org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI ${1+"$@"} edit /usr/lib/ambari-infra-solr/

QuestionWhere does Workload Analytics ingest user workloads into and analyze it? AnswerWorkload Analytics runs as part of Cloudera Altus functionality on an environment operated by Cloudera. Telemetry Publisher, part of Cloudera Manager installation, sends user's workload to the environment as soon as a job ends and analyze it. Thereon the result shows up in the Cloudera Altus UI. https://www.cloudera.com/documentation/altus/topics/wa_overview.html

QuestionIs it possible to tune the thresholds for Health Check which Workload Analytics does based on user's requirements? AnswerNo, it's not possible to tune the thresholds. Health Check uses predefined thresholds which are described in the document: https://www.cloudera.com/documentation/altus/topics/wa_analyze_jobs.html

SYMPTOMS: Cannot access Tez View. On accessing Tez View, we get the below error: error code: 500, message: Internal Server Error{"message":"Failed to fetch results by the proxy from url: http://<hostname>:8188/ws/v1/timeline/TEZ_DAG_ID?limit=11&_=1470838899327&...Error.","status":500,"trace":"Connection refused"} ROOT CAUSE: The issue is due to "YARN ResourceManager URL" pointing out to the standby RM node when you have a RM HA configured. This is usually a configuration change under "Cluster Configuration" while setting up a TEZ view where the user updates opts for the "Custom" option and includes only a single Resource Manager address in "YARN ResourceManager URL" configuration. SOLUTION: In case of RM HA configuration , the "YARN ResourceManager URL" should point out to a both the Resource Manager addresses which should be in a semi-colon separated format (e.g. sumeshnode1.openstacklocal:8088;sumeshnode3.openstacklocal:8088)

This article will give a quick demonstration to add Leaflet maps to an Angular paragraph in Zeppelin. Background: Leaflet Maps Leaflet is an open-source JavaScript library for interactive maps. Leaflet is designed with simplicity, performance and usability in mind.Step 1: Adding a map to a Zeppelin notebook.Code Source: %angular <link rel="stylesheet" href="https://unpkg.com/leaflet@1.0.3/dist/leaflet.css" /><div id="map" style="height: 800px; width: 100%"> </div><script type="text/javascript">function initMap() { var map = L.map('map').setView([33.415, -111.831], 6); L.tileLayer('http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png', { attribution: 'Map data © <a href="http://openstreetmap.org">OpenStreetMap</a> contributors', maxZoom: 15, minZoom: 3 }).addTo(map); var geoMarkers = L.layerGroup().addTo(map); }if (window.L) { initMap();} else { console.log('Loading Leaflet library'); var sc = document

QuestionWhy do I get below message "Request Time is Invalid" after logging into Cloudera Altus?  AnswerThe clock on the machine from where you are accessing the Cloudera Altus Web UI is not accurate. Please make sure the clock is set up correctly.

Release - HDP2.5 Background: Before this feature was released, Ranger allowed User to create only access policy for hive, that can provide security till column level, but folks might be interested in enabling cell level security then this is the solution they should opt. Introduction: This is a newly introduced feature,allows to create new type of ranger-hive policies that help administrator to restrict User from accessing some specific rows in table based on the filter condition in policy or mask the data that is sensitive , this masking can be full or partial. Lets go in details for both policy type: Row Filter policy: Row filter policy allows ranger to specify filter expression in the hive policy, so that users sees only some specifc rows in the table that belong to him. eg: if user belong to US, and query the employee table and we want to restrict him to see only those employee who belong to US, then filter expression will be location = 'US'. there is a new tab we can see HDP2.5

Change Log directory for Ranger Admin logsOn Ranger Admin nodes: ADMIN LOGS (Change the symbolic link for in ews directory):mkdir /opt/ranger/adminchmod -R 775/opt/ranger/adminchown -R ranger:ranger /opt/ranger/admincd /usr/hdp/current/ranger-admin/ewsunlink logln -s /opt/ranger/admin logsChange log directory for USERSYNC LOGSOn Ranger Usersync node:mkdir /opt/ranger/usersyncchmod -R 775/opt/ranger/usersyncchown -R ranger:ranger /opt/ranger/usersynccd /usr/hdp/current/ranger-usersync/unlink logsln -s /opt/ranger/usersync logscd /usr/hdp/current/ranger-usersync/cp ranger-usersync-services.sh{,.backup01122015}vim ranger-usersync-services.sh #logdir=/var/log/ranger/usersync (If it is present) logdir=/opt/ranger/usersyncChange log directory for TAGSYNC LOGS:On Ranger Tagsync nodes:mkdir /opt/ranger/tagsyncchmod -R 775/opt/ranger/tagsyncchown -R ranger:ranger /opt/ranger/tagsynccd /usr/hdp/current/ranger-tagsync/unlink logsln -s /opt/ranger/tagsync logscp ranger-tagsync-services.sh{,.backup