Hi - Looking for some better methods of reducing existing table region count. We have a few tables that were originally pre split with far too many regions. Time goes on and Issues like performance and compaction storms are evident. With smaller tables I've been using Export/Import or CopyTable to move the data to new tables with less regions, but larger tables (TBs) are very challenging to run to completion. Are there any better strategies for accomplishing the above? In some cases the region counts are so high that manually merging them is not feasible so I find myself back at Export/Import. ... View more

I added a few nodes to install Storm, but Ambari (2.2.2.0) is apparently trying to to use the wrong version. On the destination node, the path /var/lib/ambari-agent/cache/common-services/STORM contains 2 versions, 0.9.1 and 0.9.1.2.1. The directory 0.9.1 is empty and is the one Ambari is trying to execute script in. The other directory contains the items needed for installation. So the install fails with errors like: Caught an exception while executing custom service command: <class 'ambari_agent.AgentException.AgentException'>: 'Script /var/lib/ambari-agent/cache/common-services/STORM/0.9.1/package/scripts/drpc_server.py does not exist'; 'Script /var/lib/ambari-agent/cache/common-services/STORM/0.9.1/package/scripts/drpc_server.py does not exist' I suppose the question is how do I get Ambari to into shape here? Any help/thoughts are greatly appreciated. edit: I found that my repo_version table in the Ambari database has 2 different entries for the stack_id I'm using: 53 | 2.4.0.0-169 | HDP-2.4.0.0 | [{"repositories":[{"Repositories/repo_id":"HDP-2.4","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.0.0","Repositories/repo_name ":"HDP"},{"Repositories/repo_id":"HDP-UTILS-1.1.0.20","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6","Repositories/repo_name":"HDP-UTILS"}],"OperatingSystems/os_type":"r edhat6"}] | 51 103 | 2.4.2.0-258 | HDP-2.4.2.0 | [{"repositories":[{"Repositories/repo_id":"HDP-2.4","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.2.0","Repositories/repo_name ":"HDP"},{"Repositories/repo_id":"HDP-UTILS-1.1.0.20","Repositories/base_url":"http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6","Repositories/repo_name":"HDP-UTILS"}],"OperatingSystems/os_type":"r edhat6"}] | 51 ... View more

Hello- I have a Kafka cluster that has a dedicated (not managed by Ambari) Zookeeper ensemble for which I need to change the Hostnames. No Ambari, no HDP, no anything except Kafka and ZK Beyond updating Kafka's configuration files, is there more to consider (e.g. consumer retaining consumer offsets)? The ZK ensemble will remain untouched aside from the hostname change. Thanks. ... View more

Hello - I'm having trouble using the Sqoop CLI with HCatalog. The error is always: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.mapreduce.HCatOutputFormat not found at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:519) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:499) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1598) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:499) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:285) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1556) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1553) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1486) Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.mapreduce.HCatOutputFormat not found at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) at org.apache.hadoop.mapreduce.task.JobContextImpl.getOutputFormatClass(JobContextImpl.java:222) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:515) ... 11 more Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.mapreduce.HCatOutputFormat not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) ... 13 more I added the following to the sqoop-env template within Amabari (advice I found in another post) with no luck. This job works fine when in an Oozie Sqoop action, but failing with the above when executed via CLI: $ sqoop import --skip-dist-cache --username xx --password-file /dir/xx.dat --connect jdbc:postgresql://server.x.lan/xx --split-by download_id --hcatalog-table project_xx_0 --hcatalog-database default --query "select [lots of stuff with several joins] AND \$CONDITIONS" So this is a sqoop free-form query import from a Postgresql database/table to an existing hive-hcatalog table. Any help, tips are greatly appreciated, Thanks. ... View more

Upgrading to Ambari 2.2.2 from 2.2 and when executing $ambari-server upgrade am receiving the following warnings: "Updating properties in ambari.properties ... WARNING: Can not find ambari-env.sh.rpmsave file from previous version, skipping restore of environment settings Fixing database objects owner" I am using an external PostreSQL database. I do have backups, but what is this going to wipeout? ... View more

During an HDP deployment (automated with Ambari) we've found some issues on our end to resolve. Am we able resume installation from the point I left off in the Ambari installer at a later time, or will we need to start from scratch? In this case the stopping point is after cluster configuration, but before deployment. Ambari 2.2.1, HDP 2.4 Thanks ... View more

Hello- I the process of upgrading Ambari to 2.2.1.1 I hit the following error after $ ambari-server upgrade: 06 Apr 2016 13:22:52,129 ERROR [main] SchemaUpgradeHelper:315 - Exception occurred during upgrade, failed org.apache.ambari.server.AmbariException: Errors found while populating the upgrade table with values for columns upgrade_type and upgrade_package. at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.executePreDMLUpdates(SchemaUpgradeHelper.java:215) at org.apache.ambari.server.upgrade.SchemaUpgradeHelper.main(SchemaUpgradeHelper.java:302) Caused by: org.apache.ambari.server.AmbariException: Errors found while populating the upgrade table with values for columns upgrade_type and upgrade_package. Took a look at my (external) Postgres database and found the ambari database does not have an "upgrade" table. I checked another cluster with a similar external Postgres instance and it does have that table. Should I create the table manually? If so, anyone have the appropriate create statement for the table? If there's a more ambari-friendly way get the tables straight, that would be preferable. Thanks. ... View more

I know the mapred config "yarn.app.mapreduce.am.job.client.port-range" can be configured, but I've read conflicting info on whether this setting actually sticks. Is this known to be working on HDP 2.3? ... View more