@Jasper Thanks for your comments. Could you please also let me know if this is the usual way Kafka consumers are run in hadoop? if not, could you let me know how the consumers/producers are usually scheduled in hadoop cluster? ... View more

Could not paste both the explain plans in previous comment. Here is the explain plan by disabling hive.explain.user=false. hive> set hive.explain.user=false; hive> explain select a.* from big_part a, small_np b where a.jdate = b.jdate; OK STAGE DEPENDENCIES: Stage-1 is a root stage Stage-0 depends on stages: Stage-1 STAGE PLANS: Stage: Stage-1 Tez DagId: A515595_20170117140547_4494cba3-581e-441c-8fb6-8175b74d89c2:3 Edges: Map 1 <- Map 2 (BROADCAST_EDGE) DagName: Vertices: Map 1 Map Operator Tree: TableScan alias: a filterExpr: jdate is not null (type: boolean) Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: PARTIAL Filter Operator predicate: jdate is not null (type: boolean) Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: PARTIAL Map Join Operator condition map: Inner Join 0 to 1 keys: 0 jdate (type: date) 1 jdate (type: date) outputColumnNames: _col0, _col1, _col6 input vertices: 1 Map 2 Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE HybridGraceHashJoin: true Filter Operator predicate: (_col1 = _col6) (type: boolean) Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE Select Operator expressions: _col0 (type: int), _col1 (type: date) outputColumnNames: _col0, _col1 Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE table: input format: org.apache.hadoop.mapred.TextInputFormat output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Execution mode: vectorized Map 2 Map Operator Tree: TableScan alias: b filterExpr: jdate is not null (type: boolean) Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE Filter Operator predicate: jdate is not null (type: boolean) Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE Reduce Output Operator key expressions: jdate (type: date) sort order: + Map-reduce partition columns: jdate (type: date) Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE Stage: Stage-0 Fetch Operator limit: -1 Processor Tree: ListSink Time taken: 0.428 seconds, Fetched: 68 row(s) ... View more

Thanks for your comments! Here is the explain plan and create table statements. Hive version is 0.14. And also for the 3rd answer, in case both are partitioned tables, is there anyway to ensure that bigger of the two tables undergo partition pruning instead of the small one? (or is this the default behavior?) What does hive.explain.user = false do? I have attached explain plan by both enabling and disabling this. > show create table big_part; OK CREATE TABLE `big_part`( `id` int) PARTITIONED BY ( `jdate` date) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://littleredns/apps/hive/warehouse/big_part' TBLPROPERTIES ( 'transient_lastDdlTime'='1484615054') Time taken: 1.749 seconds, Fetched: 14 row(s) hive> show create table small_np; OK CREATE TABLE `small_np`( `id2` int, `jdate` date) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://littleredns/apps/hive/warehouse/small_np' TBLPROPERTIES ( 'transient_lastDdlTime'='1484615162') Time taken: 0.16 seconds, Fetched: 13 row(s) hive> set hive.optimize.ppd=true; hive> set hive.tez.dynamic.partition.pruning=true; hive> explain select a.* from big_part a, small_np b where a.jdate = b.jdate; OK Plan not optimized by CBO. Vertex dependency in root stage Map 1 <- Map 2 (BROADCAST_EDGE) Stage-0 Fetch Operator limit:-1 Stage-1 Map 1 vectorized File Output Operator [FS_21] compressed:false Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"} Select Operator [OP_20] outputColumnNames:["_col0","_col1"] Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE Filter Operator [FIL_19] predicate:(_col1 = _col6) (type: boolean) Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE Map Join Operator [MAPJOIN_18] | condition map:[{"":"Inner Join 0 to 1"}] | HybridGraceHashJoin:true | keys:{"Map 2":"jdate (type: date)","Map 1":"jdate (type: date)"} | outputColumnNames:["_col0","_col1","_col6"] | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE |<-Map 2 [BROADCAST_EDGE] | Reduce Output Operator [RS_4] | key expressions:jdate (type: date) | Map-reduce partition columns:jdate (type: date) | sort order:+ | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE | Filter Operator [FIL_14] | predicate:jdate is not null (type: boolean) | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE | TableScan [TS_1] | alias:b | Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE |<-Filter Operator [FIL_17] predicate:jdate is not null (type: boolean) Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: PARTIAL TableScan [TS_0] alias:a Statistics:Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: PARTIAL Time taken: 1.459 seconds, Fetched: 45 row(s) ... View more

Thanks for the suggestion. I have not tried these parameters.. What are these parameters for? Are these the ones that help set the mapper memory size in pig? ... View more

Hi, I checked the logs. No information found as to why the script aborted. This is all is shown in the log. INFO hive.HiveImport: Loading uploaded data into Hive WARN conf.HiveConf: HiveConf of name hive.metastore.pre-event.listeners does not exist WARN conf.HiveConf: HiveConf of name hive.semantic.analyzer.factory.impl does not exist Logging initialized using configuration in jar:file:/grid/8/hadoop/yarn/local/filecache/5470/hive-common-1.2.1.2.3.4.0-3485.jar!/hive-log4j.properties Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] ... View more

Thanks, I was able to set up the SSH and it is working. But I have a question. What is this oozie ID? I am logging in to Linux using my ID. I submit the workflow either using my ID or I do a kinit to other ID and submit the workflow using that ID. When I see in the UI logs, oozie workflow is shown to be submitted using either my ID or other ID for which ticket was obtained using Kinit. What I dont understand is where does this "oozie" user id fit in. Even I had to go to the home directory of this user "oozie" and get its keys and added it to my destination server authprized_keys file. Can you please explain the purpose of this id? And also how to find this "oozie user id" Is this user id available in oozie-site.xml? Based on this article I also searched for oozie id, but what if it were different? how to find this oozie id? Thanks! ... View more

And one more question, I am actually loading the Hbase table through pig, where I am generating the sequence number before loading this table using Rank. Even though I pre split the table, when I looked at the UI, it showed that only three regions of the Hbase table was getting loaded initially for first few minutes. Why would this happen? When we pre-split the table, shouldn't all the regions of the Hbase table get its data from previous operator simultaneously and get loaded in parallel? Why is it that only three of the 10 regions were loading? Could it be because it was streaming the data as and when the previous operator generator row number and passed it down to Storer? Any suggestions would be helpful. Thanks! ... View more

Actually the main purpose of the table is to be a lookup table. Major problem is that the lookup with this table is based on the Sequence number so I chose sequence number as the row key and that is the reason I think this caused hot spots. May I know how the phoenix ensures a good distribution? Because I thought phoenix was just a SQL layer on top of Hbase to enable query on Hbase tables. ... View more

Thanks for the info. ... View more

Hi Ben, Thanks for taking time to explain each of these questions. For qn. 4, I actually meant to type Vertex, but instead I mentioned it as node. What I meant to ask was, by setting the number of reducers, we affect all the vertices that run only the reducers. Based on your explanation. I think it affects all the vertices running reducers. And it would not affect any vertex running mappers or combination of mappers and combiners. Right? By mapper and reducer, my understanding was that any class extending Mapper is mapper and any class extending Reducer is reducer. Just out of curiosity when I look into the pig source code, there are many operators like PORank, POFRJoin etc.. and these are the ones that are showed in explain plan also as tasks of each vertex. So essentially in Tez DAG pig latin gets converted to these operators right? Are these operators run as part of Mapper and reducers? So irrespective of the underlying task being a true mapper or reducer class or one of the tez pig operators, is it correct to assume that that the parallelism of root vertices which read the data from file or table to be controlled based on file split or table parttions and the leaf vertices and other vertices in between are all like reducers and its parallelism is controlled by reducers properties? like number of reducers or bytes per reducers? And if I write a UDF, is it possible to identify if it is run inside mapper class or reducer class? ... View more

Hi Artem, Thanks for your reply. I will try hashing my row-key. But even then I think I might need to pre-split the Hbase table so that there are many regions even in the beginning of the load. (I was just reading about creating the pre splits). But even then Hbase might end up creating all the regions in same server right? Is there any option to ensure that all the regions of same table are distributed across multiple servers? ... View more

Hi Josh, I saw the table split as you suggested. I see that the table has 18 regions. But the problem is all 18 regions of the table is in same node. (or region server). How do I split the regions across multiple region servers? And also is there a Command to check the table splits and Hbase regions via CLI? Any parameter I can use to improve the load performance using pig? currently I am using only hbase.scanner.caching to reduce roundtrips. Thanks for your help! ... View more

Hi, Thanks for your suggestion. I have just started using this so can you please help me in understanding few more things. I found the webhcat server using ambari and the templeton.libjars value in webhcat.-site.xml are as follows <name>templeton.libjars</name> <value>/usr/hdp/${hdp.version}/zookeeper,/usr/hdp/${hdp.version}/hive/lib/hive-common.jar/zookeeper.jar</value> I think this has wrong values or typo in it as you suggested. I dont have access to edit this file. 1. Is there any other way to use webhcat without editing the webhcat-site.xml file? Like passing as a post parameter in Curl or something? 2. My cluster has edge node in it. Why is it that the webhcat-site.xml file is present only in webhcat server. Why is it not present in edge node. Edge node only has webhcat-default.xml? Shouldn't all the *-site.xml be present in edge node as well? 3. How to access hiveserver2 via knox? Is it possible to use hiveserver2 to insert values into hive table from outside the cluster using knox? ... View more

I just logged into Ambari and found the hiveserver2 under summary tab. There are five hiveserver2 processes listed there. And each of them are running in different hosts. Why would there be so many hiveserver2 components? Isnt hiveserver2 a server process for processing client requests from cli? so shouldnt there be only one hiveserver2? Thanks! ... View more

Sorry, I just started learning hadoop. Not sure where you want me look for this summary. I checked from hue. I could not see this detail. Can you please elaborate a little? ... View more

I found the port information under the property hive.server2.thrift.port. But the only properties where server names are listed are under hive.zookeeper.quorum hive.metastore.uris. Can you please let me know which property I should be looking at to find the host that runs hiveserver2 service? ... View more