Hello, we have a phoenix table with about 44M records. We're creating a secondary index on it, but, after about 3 days, the index table on hbase is populated with ~ 340k records. At this rate, we expect the index to be active not before one month or more. Is there a way to speed up the index creation? We're using HDP 2.6.4 with Hbase 1.1.2 and Phoenix 4.7 Thank you ... View more

Hi all, I installed and configured Cloudbreak 1.5.0 (TP) on an azure virtual machine, and I configured it to authenticate users against Active Directory, mapping AD groups with UAA roles. Now my question is if it is possible to configure role based authorization inside Cloudbreak UI, in particular I need to configure some user to be able to see (at least in read only mode) resources like clusters, bluprints, etc... created by other users. Is it possible? Thank you, Davide ... View more

Hi all, I have an HDP ambari managed cluster with HDF-3.0.1 management pack installed. Ambari release is 2.6.1. I have successfully upgraded the HDF management pack to version HDF-3.1.0, but when I click on the "Install on... <cluster name>" button from the Admin view I been redirected to the "manage version" tab in Ambari, in which I'm only able to see the existing HDP version (2.6.4). How can I perform the upgrade from HDF-3.0.1 to HDF-3.1.0? thank you, ... View more

Hi all, I'm using HDP 2.4.2 and I'm getting some issues running simple test like TestDFSIO and Teragen. If I execute the test with an high number of containers, some of them will be killed after 300 seconds, according to mapreduce.task.timeout property. For example, running this command: hadoop jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar teragen -Dmapreduce.job.queuename=ETL_APP -Dmapred.map.tasks=500 10000000000 /benchmarks/teragen_1T results in 64 map tasks killed by timeout. On yarn I have 6.68 TB of RAM and 1330 vCores, and only few others applications are running, so no container of the teragen job are in waiting state. (the minimum size for a container is 2048MB). The strange thing is that, if I look at the Application Master UI, I can see that those container are in state "RUNNING", but the status column is "NEW", until they will be killed by the Node Manager. In the attached file therenm-container-id-container-e244-1515410092608-62086.txt is the log from the NM for one of those container. You can see that the container request arrived at 2018-01-11 11:24:41,445 but the container was killed at 11:30:13,532, without logging anything else. The next attempt was running fine in few minutes. I then did another test, setting the property mapreduce.task.timeout to 600000 milliseconds and all containers started before the timeout, so the problem is not the container itself, but how long it tooks to start. Did anyone knows why some containers took long time to start? Thank you very much, Davide ... View more

Hi all, I see in the HDP 2.5.3 documentation that mysql 5.6 is no longer supported as hive nor oozie metastore backend. (hdp docs), but it was in the previous release(hdp docs). Is it an error in the documentation? What if I need to upgrade HDP from 2.4.x to HDP 2.5.x? Also, is mysql 5.7 supported? Thank you very much ... View more

Hi all, does anyone know when RHEL/CentOS 7 will be supported by Apache Hawq (HDB)? Thank you! ... View more

Hi all, I'm experiencing an high cpu usage from a mysql instance used as metastore for hive. I'm using HDP 2.5, with ACID enabled (I need ACID 'cause I'm using storm and hive streaming). I configured the Hive metastore daemon on 2 server and the compactor to run on one of them using config groups. The issue seems related to the fact that the hive transaction reaper is always running, as I can see from the logs. I see thousand of this entries: 016-10-05 13:47:17,592 INFO [org.apache.hadoop.hive.ql.txn.AcidHouseKeeperService-0]: txn.TxnHandler (TxnHandler.java:performTimeOuts(2949)) - Aborted the following transactions due to timeout: [] 2016-10-05 13:47:17,592 INFO [org.apache.hadoop.hive.ql.txn.AcidHouseKeeperService-0]: txn.TxnHandler (TxnHandler.java:performTimeOuts(2960)) - Aborted 0 transactions due to timeout 2016-10-05 13:47:17,592 DEBUG [org.apache.hadoop.hive.ql.txn.AcidHouseKeeperService-0]: txn.TxnHandler (TxnHandler.java:performTimeOuts(2925)) - Going to execute query <select txn_id from TXNS where txn_state = 'o' and txn_last_heartbeat < 1475674924000 limit 250000> 2016-10-05 13:47:17,666 INFO [org.apache.hadoop.hive.ql.txn.AcidHouseKeeperService-0]: txn.TxnHandler (TxnHandler.java:performTimeOuts(2949)) - Aborted the following transactions due to timeout: [] 2016-10-05 13:47:17,666 INFO [org.apache.hadoop.hive.ql.txn.AcidHouseKeeperService-0]: txn.TxnHandler (TxnHandler.java:performTimeOuts(2960)) - Aborted 0 transactions due to timeout 2016-10-05 13:47:17,666 DEBUG [org.apache.hadoop.hive.ql.txn.AcidHouseKeeperService-0]: txn.TxnHandler (TxnHandler.java:performTimeOuts(2925)) - Going to execute query <select txn_id from TXNS where txn_state = 'o' and txn_last_heartbeat < 1475674924000 limit 250000> 2016-10-05 13:47:17,735 INFO [org.apache.hadoop.hive.ql.txn.AcidHouseKeeperService-0]: txn.TxnHandler (TxnHandler.java:performTimeOuts(2949)) - Aborted the following transactions due to timeout: [] 2016-10-05 13:47:17,735 INFO [org.apache.hadoop.hive.ql.txn.AcidHouseKeeperService-0]: txn.TxnHandler (TxnHandler.java:performTimeOuts(2960)) - Aborted 0 transactions due to timeout 2016-10-05 13:47:17,735 DEBUG [org.apache.hadoop.hive.ql.txn.AcidHouseKeeperService-0]: txn.TxnHandler (TxnHandler.java:performTimeOuts(2925)) - Going to execute query <select txn_id from TXNS where txn_state = 'o' and txn_last_heartbeat < 1475674924000 limit 250000> 2016-10-05 13:47:17,805 INFO [org.apache.hadoop.hive.ql.txn.AcidHouseKeeperService-0]: txn.TxnHandler (TxnHandler.java:performTimeOuts(2949)) - Aborted the following transactions due to timeout: [] 2016-10-05 13:47:17,805 INFO [org.apache.hadoop.hive.ql.txn.AcidHouseKeeperService-0]: txn.TxnHandler (TxnHandler.java:performTimeOuts(2960)) - Aborted 0 transactions due to timeout 2016-10-05 13:47:17,805 DEBUG [org.apache.hadoop.hive.ql.txn.AcidHouseKeeperService-0]: txn.TxnHandler (TxnHandler.java:performTimeOuts(2925)) - Going to execute query <select txn_id from TXNS where txn_state = 'o' and txn_last_heartbeat < 1475674924000 limit 250000> Why the metastore tries to abort the transactions every few milliseconds? Is there a way to manage this? I tried to set the property hive.timedout.txn.reaper.interval=360sbut nothing changed. Thank you very much! ... View more

Hi all, I'm trying to execute an oozie script that read data from an hbase table mapped in hive. If I execute the script via beeline it runs well, but using oozie I get the following error: 13795 [main] ERROR hive.log - error in initSerDe: java.lang.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.hbase.HBaseSerDe not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:395) at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:276) at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:258) at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:605) In the oozie shared library I added the hive-hbase-handler jar, under this path: /user/oozie/share/lib/lib_20151020075400/hive/hive-hbase-handler-1.2.1.2.3.2.0-2950.jar I also added hive-site.xml and hbase-site.xml files in the oozie shared library This is my job.properties file: nameNode=hdfs://masterHA jobTracker=master03:8032 queueName=HOYA oozie.use.system.libpath=true #oozie.libpath=/user/oozie/share/lib/lib_20151020075400 oozie.wf.application.path=${nameNode}/user/root/workflow.xml By the way, in the container path on the slave, I'm unable to find the jar Current (local) dir = /grid/5/hadoop/yarn/local/usercache/root/appcache/application_1464329584635_67853/container_e70_1464329584635_67853_01_000002 ------------------------ tmp container_tokens .container_tokens.crc launch_container.sh .launch_container.sh.crc default_container_executor_session.sh .default_container_executor_session.sh.crc default_container_executor.sh .default_container_executor.sh.crc json-simple-1.1.jar curator-client-2.6.0.jar servlet-api-2.5.jar slf4j-api-1.6.6.jar curator-recipes-2.5.0.jar joda-time-2.1.jar jackson-databind-2.2.3.jar stax-api-1.0.1.jar jersey-client-1.9.jar parquet-hadoop-bundle-1.6.0.jar jdo-api-3.0.1.jar libthrift-0.9.2.jar commons-compress-1.4.1.jar httpclient-4.2.5.jar hadoop-annotations-2.7.1.2.3.2.0-2950.jar xz-1.0.jar job.xml commons-codec-1.4.jar jetty-all-7.6.0.v20120127.jar hadoop-yarn-common-2.7.1.2.3.2.0-2950.jar bonecp-0.8.0.RELEASE.jar antlr-runtime-3.4.jar guice-3.0.jar hive-shims-1.2.1.2.3.2.0-2950.jar stringtemplate-3.2.1.jar commons-httpclient-3.1.jar libfb303-0.9.2.jar commons-io-2.4.jar hive-contrib-1.2.1.2.3.2.0-2950.jar oozie-hadoop-utils-hadoop-2-4.2.0.2.3.2.0-2950.jar calcite-avatica-1.2.0.2.3.2.0-2950.jar jline-2.12.jar geronimo-jta_1.1_spec-1.1.1.jar objenesis-2.1.jar commons-collections4-4.0.jar hive-shims-scheduler-1.2.1.2.3.2.0-2950.jar aopalliance-1.0.jar commons-collections-3.2.1.jar activation-1.1.jar jersey-json-1.9.jar opencsv-2.3.jar jersey-core-1.9.jar oozie-sharelib-hive-4.2.0.2.3.2.0-2950.jar hive-shims-0.20S-1.2.1.2.3.2.0-2950.jar avro-1.7.5.jar aws-java-sdk-1.7.4.jar netty-3.6.2.Final.jar commons-pool-1.5.4.jar tez-runtime-internals-0.7.0.2.3.2.0-2950.jar commons-cli-1.2.jar curator-framework-2.6.0.jar ant-1.9.1.jar hadoop-azure-2.7.1.2.3.2.0-2950.jar guava-11.0.2.jar jackson-core-2.2.3.jar hive-shims-common-1.2.1.2.3.2.0-2950.jar ST4-4.0.4.jar datanucleus-rdbms-3.2.9.jar httpcore-4.2.4.jar asm-tree-3.1.jar javassist-3.18.1-GA.jar slf4j-log4j12-1.6.6.jar tez-api-0.7.0.2.3.2.0-2950.jar hive-service-1.2.1.2.3.2.0-2950.jar hive-cli-1.2.1.2.3.2.0-2950.jar ivy-2.4.0.jar jaxb-impl-2.2.3-1.jar pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar commons-lang3-3.3.2.jar mr-framework jpam-1.1.jar asm-commons-3.1.jar jetty-6.1.14.jar datanucleus-api-jdo-3.2.6.jar asm-3.1.jar hive-exec-1.2.1.2.3.2.0-2950.jar calcite-linq4j-1.2.0.2.3.2.0-2950.jar hive-serde-1.2.1.2.3.2.0-2950.jar log4j-1.2.16.jar datanucleus-core-3.2.10.jar hadoop-yarn-server-web-proxy-2.7.1.2.3.2.0-2950.jar mail-1.4.jar refreshCurrencyList.sql jackson-xc-1.9.13.jar jetty-util-6.1.26.hwx.jar commons-logging-1.1.jar guice-servlet-3.0.jar snappy-java-1.0.5.jar protobuf-java-2.5.0.jar tez-common-0.7.0.2.3.2.0-2950.jar janino-2.7.6.jar tez-runtime-library-0.7.0.2.3.2.0-2950.jar tez-mapreduce-0.7.0.2.3.2.0-2950.jar hadoop-yarn-api-2.7.1.2.3.2.0-2950.jar oro-2.0.8.jar fst-2.24.jar tez-dag-0.7.0.2.3.2.0-2950.jar jsr305-2.0.3.jar commons-lang-2.4.jar groovy-all-2.1.6.jar json-20090211.jar hive-shims-0.23-1.2.1.2.3.2.0-2950.jar hive-common-1.2.1.2.3.2.0-2950.jar hadoop-yarn-server-resourcemanager-2.7.1.2.3.2.0-2950.jar calcite-core-1.2.0.2.3.2.0-2950.jar jackson-jaxrs-1.9.13.jar hadoop-yarn-registry-2.7.1.2.3.2.0-2950.jar geronimo-annotation_1.0_spec-1.1.1.jar hadoop-aws-2.7.1.2.3.2.0-2950.jar apache-curator-2.6.0.pom oozie-sharelib-oozie-4.2.0.2.3.2.0-2950.jar hadoop-yarn-server-applicationhistoryservice-2.7.1.2.3.2.0-2950.jar eigenbase-properties-1.1.5.jar hadoop-yarn-server-common-2.7.1.2.3.2.0-2950.jar zookeeper-3.4.6.2.3.2.0-2950-tests.jar geronimo-jaspic_1.0_spec-1.0.jar commons-compiler-2.7.6.jar hive-ant-1.2.1.2.3.2.0-2950.jar zookeeper-3.4.6.2.3.2.0-2950.jar tez-yarn-timeline-history-0.7.0.2.3.2.0-2950.jar jettison-1.3.4.jar hive-metastore-1.2.1.2.3.2.0-2950.jar ant-launcher-1.9.1.jar servlet-api-2.5-6.1.14.jar jta-1.1.jar jackson-annotations-2.2.3.jar commons-dbcp-1.4.jar derby-10.10.1.1.jar velocity-1.5.jar antlr-2.7.7.jar jaxb-api-2.2.2.jar paranamer-2.3.jar jersey-guice-1.9.jar azure-storage-2.2.0.jar javax.inject-1.jar stax-api-1.0-2.jar tez-yarn-timeline-history-with-acls-0.7.0.2.3.2.0-2950.jar apache-log4j-extras-1.1.jar leveldbjni-all-1.8.jar .job.xml.crc action.xml .action.xml.crc propagation-conf.xml hive-site.xml hive-log4j.properties hive-exec-log4j.properties I'm running hbase on slider. Do you have any tips? Thank you! ... View more

Repo Description Ambari service to run and manage Apache Drill. For more information about Apache Drill visit https://drill.apache.org/ Requirements: - RHEL/CENTOS 7.1 - Ambari 2.X - HDP 2.4 - You need HDFS and Zookeeper up & running on your cluster Features: - Allows to install Apache Drill on an Ambari-managed cluster - Edit drill-overrides.conf and drill-env.sh via ambari - Integration with zookeeper Repo Info Github Repo URL https://github.com/dvergari/ambari-drill-service Github account name dvergari Repo name ambari-drill-service ... View more