I am currently doing an upgrade from HDP-2.6.0.3 to HDP-3.1.0.0 on a HDP Cluster Previously I installed HDF management pack for ambari (HDF version 3.1.0.0-564) and Nifi through the mpack MongoDB management pack from https://github.com/nikunjness/mongo-ambari Unfortunately did not read upgrade Guide for HDF before staring the upgrade. For the upgrade process I upgraded ambari to version 2.7.3.0 (without touching the mpacks) Before starting the HDP upgrade there was the warning that MongoDB and Nifi would be uninstalled because they don't belong to the HDP stack, which was fine for me. The HDP Upgrade, which I started after the ambari upgrade at first halted / failed in the step Remove unsupported services and components due to MongoDB and HDF. I paused the upgrade and uninstalled the MongoDB and Nifi Services through ambari as well as uninstalling the mongodb mpack and upgrading the hdp mpack to version 3.4.0.0-155. Then the HDP upgrade proceeded but failed at Update Target Repositories with a stdout of Updating the desired repository version to 3.1.0.0-78 for all cluster services. and a stderr of org.apache.ambari.server.ServiceNotFoundException: Service not found, clusterName=itpp, serviceName=MONGODB     at org.apache.ambari.server.state.cluster.ClusterImpl.getService(ClusterImpl.java:888)     at org.apache.ambari.server.state.UpgradeHelper.setDesiredRepositories(UpgradeHelper.java:922)     at org.apache.ambari.server.state.UpgradeHelper.updateDesiredRepositoriesAndConfigs(UpgradeHelper.java:894)     at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:118)     at org.apache.ambari.server.serveraction.upgrades.UpdateDesiredRepositoryAction.updateDesiredRepositoryVersion(UpdateDesiredRepositoryAction.java:160)     at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:128)     at org.apache.ambari.server.serveraction.upgrades.UpdateDesiredRepositoryAction.execute(UpdateDesiredRepositoryAction.java:96)     at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:550)     at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:466) at java.lang.Thread.run(Thread.java:745) I checked https://community.hortonworks.com/questions/202455/upgrade-from-hdp-264-to-300-fails-at-update-target.html but unfortunately the answers provided there where either not helpful for me or missing some details. I did check for MONGODB and NIFI in the tables servicecomponentdesiredstate hostcomponentstate hostcomponentdesiredstate servicedesiredstate clusterservices and did not find any related entries. It seems that there are still some references to Nifi and/or Mongodb somewhere the in ambari database. In the ambari-server.log there is also the error: Failed to obtain service info for stack HDP-2.6, service name NIFI org.apache.ambari.server.StackAccessException: Stack data, stackName=HDP, stackVersion=2.6, serviceName=NIFI Any advice how to proceed or how to restart the upgrade (if advisable)? ... View more

Hi, We have setup a Phoenix table which we access through an external Hive table using the org.apache.phoenix.hive.PhoenixStorageHandler. We are using HBase version 1.1.2.2.6 and Hive version 1.2.1.2.6 (HDP 2.6.0.3). When we are accessing the data in data_table (through Hive or Phoenix (sqlline)) using a sql statement with the first element of the Phoenix composite row-key (mk0) in the where clause (mk0 = 12345) the performance is fine. However we got an additional requirement for queries based on a different column (mk1), which is not part of the composite phoenix row key. Therefore we created a lookup table (lookup_table) in Phoenix (also exposed though an external Hive table) with only a composite row key consisting of mk1 and mk0, in that order. Using that lookup table the query in the phoenix shell select <md1> from <data_table> where <mk0> in (select <mk0> from <lookup_table> where <mk1> = 12345); has an acceptable performance which is not really surprising since the queries should access the the phoenix / hbase tables using a known row-key prefix. The Phoenix query plan: CLIENT 22955-CHUNK 12756848943 ROWS 7200887136211 BYTES PARALLEL 1-WAY ROUND ROBIN FULL SCAN OVER <data_table> SKIP-SCAN-JOIN TABLE 0 CLIENT 1-CHUNK 0 ROWS 0 BYTES PARALLEL 1-WAY RANGE SCAN OVER <lookup_table> [12345] SERVER FILTER BY FIRST KEY ONLY SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [<mk0>] CLIENT MERGE SORT DYNAMIC SERVER FILTER BY <data_table>.<mk0> IN ($1.$3) However when we use the same table setup through hive and hive.PhoenixStorageHandler the query gets executed as map-reduce job which takes way too long (using DISTINCT and ORDER BY does not change query performance in a noticeable fashion). Hive query: select b.<md1> from <data_table> as b where <mk0> in (select distinct t.<mk0> from <lookup_table> as t where t.<mk1> = 12345 order by mk0); The Hive query plan: Plan not optimized by CBO. Vertex dependency in root stage Map 1 <- Reducer 4 (BROADCAST_EDGE) Reducer 3 <- Map 2 (SIMPLE_EDGE) Reducer 4 <- Reducer 3 (SIMPLE_EDGE) Stage-0 Fetch Operator limit:-1 Stage-1 Map 1 File Output Operator [FS_19] compressed:false Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"} Select Operator [SEL_18] outputColumnNames:["_col0"] Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE Map Join Operator [MAPJOIN_24] | condition map:[{"":"Left Semi Join 0 to 1"}] | HybridGraceHashJoin:true | keys:{"Map 1":"<mk0> (type: int)","Reducer 4":"_col0 (type: int)"} | outputColumnNames:["_col11"] | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE |<-Reducer 4 [BROADCAST_EDGE] | Reduce Output Operator [RS_15] | key expressions:_col0 (type: int) | Map-reduce partition columns:_col0 (type: int) | sort order:+ | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE | Group By Operator [GBY_11] | keys:_col0 (type: int) | outputColumnNames:["_col0"] | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE | Select Operator [SEL_9] | | outputColumnNames:["_col0"] | | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE | |<-Reducer 3 [SIMPLE_EDGE] vectorized | Reduce Output Operator [RS_27] | key expressions:_col0 (type: int) | sort order:+ | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE | Group By Operator [OP_26] | | keys:KEY._col0 (type: int) | | outputColumnNames:["_col0"] | | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE | |<-Map 2 [SIMPLE_EDGE] | Reduce Output Operator [RS_5] | key expressions:_col0 (type: int) | Map-reduce partition columns:_col0 (type: int) | sort order:+ | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE | Group By Operator [GBY_4] | keys:<mk0> (type: int) | outputColumnNames:["_col0"] | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE | Filter Operator [FIL_23] | predicate:<mk0> is not null (type: boolean) | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE | TableScan [TS_1] | alias:t | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE |<-Filter Operator [FIL_22] predicate:<mk0> is not null (type: boolean) Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE TableScan [TS_0] alias:b Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE Since the hive table sits on top of the phoenix table I had expected hive to directly hand over the query to phoenix for execution and not running a map-reduce job. Also according to the hive query plan, the equals condition in th e where clause is not used as filter for HBase. We have set TEZ as execution engine and enabled CBO. Is there a way to speed up the Hive query using basically data_table(lookup_table(rowKey)) instead of runnning a map-reduce job? Maybe indexing is another option, but would queries from hive actually use a Phoenix index? Maybe an index in Hive would be another option. ... View more