Resources

select * from t1 where col1 = 'aaa' limit 2;If you see such scenario, make sure to verify the no. of files in each partitions of the table. It is possible that if there are too many small files in the order of KBs under each partition then most of the time is spent in just opening the file in ORC. Check if it is expected to have so many files in each partition or if it is possible to merge them before loading data.These are the parameters to consider hive.merge.tezfiles=true hive.merge.mapredfiles=truehive.merge.mapfiles=true hive.merge.orcfile.stripe.level=true In such situation, checking the no of files in the partition should be the first protocol & verifying anything else should be only secondary.

IBM Spectrum Scale 4.2.3 has been certified with the Hortonworks Data Platform (HDP) 2.6 / Ambari 2.5 on IBM Power Systems. IBM and Hortonworks collaborated on an optimized and integrated solution that was validated against a comprehensive suite of integration test cases across the full stack of HDP components and Ambari. Testing covered secure and non-secure scenarios with Accumulo, Atlas, Falcon, Flume, Hbase, HDFS, Hive, HiveServer2, Kafka, Knox, Mahout, Map Reduce, Oozie, Phoenix, Pig, Spark, Sqoop, Storm, Tez, Yarn, Zeppelin, and Zookeeper. This certification is for Spectrum Scale software and hence applies to all deployment models of Spectrum Scale, including Elastic Storage Server (ESS). Further, this certification includes a paper certification for Hortonworks Data Flow (HDF) V3.0 use with IBM Spectrum Scale. IBM’s Power platform is already certified to run HDP and offers 3x price performance compared with x86.IBM ESS (pre-integrated system powered by IBM Spectrum Scale) includ

Problem Description:On deleting few kafka topics and creating same topics again there Not Leader for this partition exception while Producing/Consuming and was reading old meta data. Getting Below error:17/03/10 10:45:35 ERROR ApplicationMaster: User class threw exception: org.apache.spark.SparkException: org.apache.spark.SparkException: Couldn't find leaders for Set([akrnohij-wng-fp9,47], [akrnohij-wng-fp4,224], [akrnohij-wng-fp11,172], [akrnohij-wng-fp1,84], [akrnohij-wng-fp10,117], [akrnohij-wng-fp10,168], [akrnohij-wng-fp4,176], [akrnohij-wng-fp1,136], [akrnohij-wng-fp9,167], [akrnohij-wng-fp2,174], [akrnohij-wng-fp11,74], [akrnohij-wng-fp3,189], [akrnohij-wng-fp11,200], [akrnohij-wng-fp11,168], [akrnohij-wng-fp4,149], [akrnohij-wng-fp7,127], [akrnohij-wng-fp6,39], [akrnohij-wng-fp10,133], [akrnohij-wng-fp9,171], [akrnohij-wng-fp5,175], [akrnohij-wng-fp7,181] Cause:The zookeeper ACL for state znode was not correct : getAcl /brokers/topics/akrnohij-wng-fp4/partitions/135/state ‘aut

SYMPTOM: While performing HDP upgrade [HDP-2.3.0 upgrade to HDP-2.5.3] there was error for which we had to start ambari server. But ambari server startup failed with below error - ERROR: Caused by: Exception [EclipseLink-116] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.DescriptorException Exception Description: No conversion value provided for the value [UPGRADED] in field [host_version.state]. Mapping: org.eclipse.persistence.mappings.DirectToFieldMapping[state-->host_version.state] Descriptor: RelationalDescriptor(org.apache.ambari.server.orm.entities.HostVersionEntity --> [DatabaseTable(host_version)]) at org.eclipse.persistence.exceptions.DescriptorException.noFieldValueConversionToAttributeValueProvided(DescriptorException.java:1066) at org.eclipse.persistence.mappings.converters.ObjectTypeConverter.convertDataValueToObjectValue(ObjectTypeConverter.java:226) at org.eclipse.persistence.mappings.converters.EnumTypeConverter.conv

SYMPTOM: During upgrade from HDP 2.4.3 to HDP2.6.0 the express upgrade failed on the step "Yarn NodeMangers / Restarting NodeManger."ERROR: 2017-06-2215:02:22,791 INFO recovery.ZKRMStateStore(ZKRMStateStore.java:runWithRetries(1227)) - Exception while executing a ZKoperation. org.apache.zookeeper.KeeperException$AuthFailedException:KeeperErrorCode = AuthFailed for /rmstore atorg.apache.zookeeper.KeeperException.create(KeeperException.java:123) atorg.apache.zookeeper.KeeperException.create(KeeperException.java:51) atorg.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) ROOT CAUSE: The issue was with property Ambari-Yarn-Configs-"yarn-resourcemanager-zk-address" was set to localhost:2181 RESOLUTION: Modified address to "{zk-server-hostname}:2181" resolved the issue. After which node managers were able to start successfully.

Issue:Whilst running a 'select count(*)' or 'analyze table compute statistics' on a regex table & if you see the below error, it indicates you have created the table with old serde. Prior to Hive 0.10, 'RegexSerDe' is part of 'hive-contrib' library. From Hive 0.10 onwards the serde is part of 'hive-serde-<version>.jar'ERROR : Status: FailedERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1496529203645_0883_2_00, diagnostics=[Task failed, taskId=task_1496529203645_0883_2_00_000010, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtim

I recently worked with a customer to demo HDF/NiFi DataFlows for their uses cases. One use case involved converting delimited row-formatted text files into Parquet and ORC columnar formats. This can quickly be done with HDF/NiFi. Here is an easy to follow DataFlow that will convert row-formatted text files to Parquet and ORC. It may look straightforward; however, it requires some basic knowledge of Avro file formats and use of the Kite API. This article will explain the DataFlow in detail. STEP 1: Create an Avro schema for the source data My source data is tab delimited and consists of 8 fields. Notice in the DataFlow that before converting the data to Parquet and ORC the data is first converted to Avro. This is done so we have schema information for the data. Prior to converting the data to Avro we need to create an Avro schema definition file for the source data. Here is what my schema definition file looks like for the 8 fields. I stored this in a file named sample.avsc. More info

Background: when it comes to rely completely on ranger, and if you are specific about configuring authorization for a resource to an end user then you have to create one policy for each resource. There should be a way to configure a policy that provide access to specific resources based on the User who is making call. Soultion:{USER} Support: {USER} support solves this problem , It allows us to create a policy where we can configure resource as {USER} Eg. /user/{USER} and select user also as {USER}. that means all users will get access of their corresponding home directory. Eg. Hdfs: resource: /user/{USER} user1 will have access to /user/user1 user2 will have access to /user/user2 Hive: resource: database:database_{USER} user1 will have access to database database_user1 user2 will have access to database database_user2 resource may contrain {USER} partially or fully. delimiter can be customised also , Steps to configure {USER}:1) go to ranger admin, and create policy page, there on re

PROBLEM: While implementing AutoHDFS for storm-hdfs integration, observed following error:2017-05-19 11:21:44.865 o.a.s.h.c.s.AutoHDFS [ERROR] Could not populate HDFS credentials. java.lang.RuntimeException: Failed to get delegation tokens. at org.apache.storm.hdfs.common.security.AutoHDFS.getHadoopCredentials(AutoHDFS.java:242) at org.apache.storm.hdfs.common.security.AutoHDFS.populateCredentials(AutoHDFS.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) at org.apache.storm.daemon.nimbus$mk_reified_nimbus$reify__11226.submitTopologyWithOpts(nimbus.clj:1544) at org.apache.storm.generated.Nimbus$Processor$submitTopolo

The NiFi S2S protocol is used by NiFi's Remote Process Group (RPG) components to distribute FlowFiles from one NiFi instance to another. When the target NiFi is a NiFi cluster, load-balancing of the FlowFie delivery is done across all nodes in the target NiFi cluster. The default way this works (and the only way it works in versions of NiFi previous to Apache NiFi 1.2.0 or HDF 3.0) is as follows: The RPG regularly communicates with the target NiFi cluster to get load status information about each node in the cluster. This information includes the number of currently connected nodes in the target cluster, each node's hostname, port information, and the number of total queued FlowFiles on each target NiFi node. The Source NiFi uses this information to determine a data distribution strategy for its source FlowFiles it has queued. - - - Let's assume a 4 node target NiFi cluster all reporting a zero queue count. - Each node will then be scheduled to receive 25% of the data. This means a di

Upon setting up a Socks Proxy Connection to access Cloudera Manager on an Altus cluster per:   https://www.cloudera.com/documentation/altus/topics/ug_dataengr_trblshoot_socks_proxy.html#socks_proxy   You may see:   ssh: connect to host X.X.X.X port 22: Connection timed out 2017-06-26 09:16:05,759 - MainThread - altuscli.extensions.dataeng.socksproxy - ERROR - IP address of Cloudera Manager (X.X.X.X) is unreachable. Please check security permissions on instances of the cluster.   The same steps should be followed, listed here:   https://www.cloudera.com/documentation/altus/topics/ug_dataengr_trblshoot_ssh_conn.html   However, you should slightly modify these steps to allow access Cloudera Manager (port 7180 In the AWS Console, go to VPC > Security Groups and find the security group you created for the Altus environment. Verify that you are in the correct region. On the Inbound Rules tab, edit the security group and add another rule of type Custom TCP Ru

SYMPTOM :=> This problem occurs in case of a partitioned table without any null partitions and contains approximately more than 600 columns in the table=> Following stacktrace is observed in hive metastore logsNested Throwables StackTrace:org.datanucleus.store.rdbms.exceptions.MappedDatastoreException: INSERT INTO "PARTITION_PARAMS" ("PARAM_VALUE","PART_ID","PARAM_KEY") VALUES (?,?,?) at org.datanucleus.store.rdbms.scostore.JoinMapStore.internalPut(JoinMapStore.java:1056) at org.datanucleus.store.rdbms.scostore.JoinMapStore.put(JoinMapStore.java:307) at org.datanucleus.store.types.wrappers.backed.Map.put(Map.java:653) at org.apache.hadoop.hive.common.StatsSetupConst.setColumnStatsState(StatsSetupConst.java:285) at org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:6237) at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Dele

SYMPTOM:Incorrect status shown for the DAGs in Tez UI ROOT CAUSE This is a known issue (https://issues.apache.org/jira/browse/TEZ-3656). It will only happen for the killed applications or if there was a failure to write into Application Timeline Server. It should not cause any issues, except for the wrong status for the DAG in the TezUI. RESOLUTION:This is fixed in HDP 2.6.1 release

PROBLEM DEFINITION: CREATE TABLE DT(Dérivation string, Pièce_Générique string);Throws ParserException ErrorROOT CAUSE/ WORKAROUND: Hive database name, table name and/or column names cannot contain Unicode string. However, Hive supports UTF-8 and Unicode string for only the table data/comments. LINKS:https://cwiki.apache.org/confluence/display/Hive/User+FAQ

The original configurations and the original CLI options are respectively discussed in Section 1 & 2. The new configurations and the new CLI options are respectively discussed in Section 3 & 4.1. Original ConfigurationsConcurrent Movesdfs.datanode.balance.max.concurrent.moves, default is 5This configuration is to limit the maximum number of concurrent block moves that a Datanode is allowed for balancing the cluster. If this configuration is set in a Datanode, the Datanode will throw an exception if the limit is exceeded. If it is set in the Balancer, the Balancer will schedule concurrent block movements within the limit. Note that the Datanode setting and the Balancer setting could be different. Since both settings impose a restriction, the effective setting is the minimum of them.Before HDFS-9214, changing this configuration in a Datanode requires restarting the Datanode. Therefore, it is recommended to set it to the highest possible value in Datanodes and adjust the runt

Steps to configure 2way SSL between Ambari-server and Ambari-agent by using custom certs. Here I have used CA Signed certs at server side and Agent certs are generated dynamically. if you are planning to use CA Signed cert at Agents side then for every Agent install you may have to copy the certs and do manual work.1. Make sure to have fresh keys folder. (if you do not have one, you can copy the folder from one of the fresh install machine or do following). - Delete all the crt and csr files that starts with hostname at /var/lib/ambari-server/keys. - Empty /var/lib/ambari-server/keys/db/index.txt file - Delete any certs under /var/lib/ambari-server/keys/db/newcerts/ 2. - Copy your own Signed Certificate, key files /var/lib/ambari-server/keys/ Ex: certificate name is - ca-cust.crt, ca-cust.key 3. Create PKCS keystore file from your cert and key files. Ex:openssl pkcs12 -export -inkey /tmpr/keys/ca.key -in ca-cust.crt -out /tmp/keys/keystore.p12 -password pass:bigdat

We had kerberised the HDP cluster and storm UI was not opening and we need to troubleshoot StormUI issue.To put DEBUG on for STormUI follow this steps:--------------------------------------------------------------In Ambari - Storm - Config - STORM UI Server section add below :ui.childopts(original)= -Xmx768m_JAAS_PLACEHOLDERui.childopts = -Xmx768m_JAAS_PLACEHOLDER -Dsun.security.krb5.debug=truewe wanted to troubleshoot zookeeper as well To put DEBUG on for Zookeeper follow this steps:------------------------------------------------------------------In Amabri - Zookeeeper - Advanced Zookeeper-log4jadd below:#log4j.rootLogger=INFO,CONSOLE ***Comment this linelog4j.rootLogger=DEBUG,CONSOLE,ROLLINGFILE ***uncomment this lineThe log file will be generated by default to /home/zookeeper/zookeeper.log if below property is set as default.log4j.appender.ROLLINGFILE.File=zookeeper.log

Overview This article reviews the steps necessary to update Hive entities within Atlas the Description and Comment fields. The 0.70 Atlas release will display and allow text searches on the ‘‘description’ field, but the Atlas UI does not at this time support the ability to manually enter those properties into a given data Asset. Examined in this article includes: Searching for a Hive_Table entity Update a single property for the Hive_Table entity definition (“description”)The Problem: In release 0.70, Atlas has the ability to monitor additions as well as changes to Hive table and Hive columns. When Atlas identifies a new entry or change the appropriate Metadata property is updated for that entity. One very cool aspect to Atlas is the ability to conduct either DSL or free text searches on any properties set for the entity. Anyone trying to identify datasets to support a specific analytic activity will definitely appreciate the ability search through all of the entities and qui

STEPS TO REPRODUCE:1. Create a Phoenix connection 2. CREATE SCHEMA test 3. DROP SCHEMA test4. Now create the test schema again. This will throw below exceptions :0: jdbc:phoenix:> create schema "test"; Error: ERROR 721 (42M04): Schema with given name already exists schemaName=test (state=42M04,code=721) org.apache.phoenix.schema.SchemaAlreadyExistsException: ERROR 721 (42M04): Schema with given name already exists schemaName=test5. Now attempt to drop the test schema again:0: jdbc:phoenix:> drop schema "test";Error: ERROR 722(43M05):Schema does not exists schemaName=test (state=43M05,code=722) org.apache.phoenix.schema.SchemaNotFoundException: ERROR 722(43M05):Schema does not exists schemaName=testIf you look in Hbase and list_namespace, you will see the test schema still exists.ROOT CAUSE : This is a known issue discussed in Apache JIRA PHOENIX-3694SOLUTION : Issue is fixed in Phoenix 4.10. For a backport hotfix , please log a case with HWX.

The HDFS NameNode ensures that each block is sufficiently replicated. When it detects the loss of a DataNode, it instructs remaining nodes to maintain adequate replication by creating additional block replicas. For each lost replica, the NameNode picks a (source, destination) pair where the source is an available DataNode with another replica of the block and the destination is the target for the new replica. The re-replication work can be massively parallelized in large clusters since the replica distribution is randomized. In this article, we estimate a lower bound for the recovery time.Simplifying Assumptions The maximum IO bandwidth of each disk is 100MB/s (reads + writes). This is true for the vast majority of clusters that use spinning disks.The aggregate IO capacity of the cluster is limited by disk and not the network. This is not always true but helps us establish lower bounds without discussing network topologies. Block replicas are uniformly distributed across the clus

QuestionIs the Cloudera Altus CLI supported for Windows systems? AnswerStarting with the Cloudera Altus CLI Version 1.1.3, installation is available through pip, as long as the following minimum requirements are met: Minimum version of Python 3.6+ 1) Download and Install Python for Windows according to the website's instructions:     https://www.python.org/downloads/windows/ 2) Open a Command Prompt (Windows Key + R), then type cmd and hit enter. 3) In the Command prompt window, run the command to update the Windows environment to add the newly installed Python folder to the system path:  setx "%PATH%";C:\<Python_Install_destination>\Scripts;    For example, if Python 2.7 was installed in it's default location, the following command would be run:    setx "%PATH%";C:\Python36\Scripts; 4) In the same Window, run the pip installer to install the Cloudera Altus

Device Behavior Analytics allows administrators to detect straggler nodes/disks which reduce the cluster performance. Finding and fixing these nodes can improve the overall cluster throughput. This feature will be available in HDP-2.6.1 and later. There are two parts to Device Behavior Analytics – Slow Datanode Detection and Slow Disk Detection.SLOW DATANODE (PEER) DETECTION Datanode will collect latency statistics about their peers during the normal operation of the datanode write pipeline. These latency stats are used to detect outliers among the peers. Slow peer detection is not performed unless the datanode has statistics of at least 10 peers. Namenode maintains the list of slow peers and administrators can read it via JMX. Datanodes also expose the average write latency of their peers through Datanode JMX.SLOW DISK DETECTION Each datanode will collect I/O statistics from all its disks. We can configure the percentage of file I/O events to be sampled to limit the performance i

Current on version HDP-2.6.x - Ambari-2.5.x if the zookeeper principal name is changed or customized manual changes are required for HDFS, Yarn and Ambari-Infra.In Ambari - config - for yarn and HDFS: yarn-env.sh YARN_OPTS="-Dzookeeper.sasl.client=true -Dzookeeper.sasl.client.username=sand-box-zookeeper -Djava.security.auth.login.config=/etc/hadoop/2.6.0.3-8/0/yarn_jaas.conf -Dzookeeper.sasl.clientconfig=Client $YARN_OPTS" hadoop/conf/hadoop-env.sh export HADOOP_ZKFC_OPTS="-Dzookeeper.sasl.client=true -Dzookeeper.sasl.client.username=sandbox-zookeeper -Djava.security.auth.login.config=/usr/hdp/current/hadoop-client/conf/secure/hdfs_jaas.conf -Dzookeeper.sasl.clientconfig=Client $HADOOP_ZKFC_OPTS" For Ambari-Infra/Solr edit /usr/lib/ambari-infra-solr-client/solrCloudCli.sh PATH=$JAVA_HOME/bin:$PATH $JVM -classpath "$sdir:$sdir/libs/*" -Dzookeeper.sasl.client.username=sandbox-zookeeper org.apache.ambari.logsearch.solr.AmbariSolrCloudCLI ${1+"$@"} edit /usr/lib/ambari-infra-solr/

QuestionWhere does Workload Analytics ingest user workloads into and analyze it? AnswerWorkload Analytics runs as part of Cloudera Altus functionality on an environment operated by Cloudera. Telemetry Publisher, part of Cloudera Manager installation, sends user's workload to the environment as soon as a job ends and analyze it. Thereon the result shows up in the Cloudera Altus UI. https://www.cloudera.com/documentation/altus/topics/wa_overview.html

QuestionIs it possible to tune the thresholds for Health Check which Workload Analytics does based on user's requirements? AnswerNo, it's not possible to tune the thresholds. Health Check uses predefined thresholds which are described in the document: https://www.cloudera.com/documentation/altus/topics/wa_analyze_jobs.html