All, I am very new to micro-services and Containers world and concept, and trying to related how it can be possibly be accommodated with my Spark and Hive applications, Can someone please point me in the right direction to figure this out. Are there examples or blog which will show some sample code and demos.. I googled but not much luck ... View more

All, I want to learn how to use various settings ( set hive.tez... etc ) in hive for tuning and whats meaning of each and impacts of each ... where can i study each of them and see the effect of each setting during execution. Also is there extensive list of each of them ? We been strugling big time for identifying right settings to use, and simulate exact issues in lower environment like prod ? Thanks Freakabhi ... View more

All, I am working on writing to RDBMS ( Sql Server ) using Spark from hive, process works with great speed. But there is big issue, that each tasks until completes does not commits - which utilizes transaction log of the database and can cause impacts to other running jobs. Need to have some way to commits at regular interval ( 10000 K or so) . Can someone please suggest how this can be done?? Spark version : 2.2 SQL Server 2016 Thanks freakabhi ... View more

I wanted to understand how to approach to the solution of hive query failures; in our environment, we get numerous failure for "Vertex issue" / "Out of memory" etc ? is it common? how to check Hive Server2 logs/ yarn-logs and resource manager logs ( what to look for ? ) how to know if there is SKEW or another issue - Are there article covering these topics What hive setting/parameters means what and how it affects --> where to look for this details? what tools are good or being used to analyze issues in hive queries or map/reduce level? ... View more

I am planning to enable full-fledged development environment for Spark Practice, with Intellij / eclipse installed I am trying to enable Desktop mode for HDP 2.6 VM and it keeps on failing while trying to Add VNC Server; is it possible to achieve this? Error I am getting is shown in the screen-shot attached. ... View more

I wanted to get the suggestion on the incremental strategy for tables be implemented : We have set of source table which are getting refreshed on the daily basis in the source ( DB2 ) and we need to refresh then in hive db as well, which approach will you suggest. Please note : Source table have new inserts as well as updates to existing records; 1) approach 1: USe Hbase to store the data since updates are allowed and build hive external table referring to the same I doubt if this will affect queries using the joins for hive-hbase table with large ORC hive tables? 2) approach 2 : USe 4 step incremental table approach suggested by HDP ? https://hortonworks.com/blog/four-step-strategy-incremental-updates-hive/ thanks Abhijeet ... View more

All, I would like to get the suggestions and correct way to convert very large queries like ( 1000 lines ) joining 10+ tables and complicated transforms to Py-Spark program   Also if there are relevent examples for large sqls.   and also I have question regarding writing SparkSQL program, is there difference of performance between writing   1) SQLContext.sql("select count(*) from (select distinct col1,col2 from table))") 2) using pyspark Api : df.select("col1,col2").distinct().count().   I am from SQL background and we are working on converting existing logic to hadoop, hence SQL is handy. ... View more

All,  I am working on importing 1.2 billion rows from one of the db2 table, There is composite primary index in the table with at least 5 columns, hence I need to manually specify the --split-by column ( since SQOOP does not support multi-part split-by column), I tried to run the Sqoop import with one of the columns from the index which is numeric , and import ran for almost 8 hrs. I am being suggested to try with different choice for split by key, but now I have question how to choose column for split by  1) is it supposed to be numeric?  or varchar is fine too? 2) can it have nulls 3)  I suppose distribution of values for the column should be as even as possible, but will it matter how many maps I am choosing (--num-map ## )? or any other criteria to pay attention to?   thanks  Abhijeet  ... View more

All , I am working on creating and loading very high volume (450 million ) ORC table and process keeps on failing for vortex error . Platform: hdp 2.4 Engine : tez Why does this happens and what's solution? ... View more

I have question regarding, if we are creating common reservoir for the files in hadoop which can later be used for any purpose, may it be spark processing or pig or hive ...etc. Now I have question on while sqooping data into the hadoop, which file format to choose, any industry wide standards ? 1)Text delimited ( compressed or un-compressed ?) 2) AVRO 3)Parquet ... View more