All, I am very new to micro-services and Containers world and concept, and trying to related how it can be possibly be accommodated with my Spark and Hive applications, Can someone please point me in the right direction to figure this out. Are there examples or blog which will show some sample code and demos.. I googled but not much luck ... View more

All, We are working for setting up Kafka cluster for internal message and event communications - I am very novice with Kafka, and have some architectural questions - 1) When you create Topic and publish information - Can we/Should we keep all the information ( 100+ fields ) as message 2) Message length and retention days, are there any standards we can follow? 3) Recommandation on Format of the message ( Text / JSON / Avro ?? ) - We might have hortonworks Kafka Setup, so not sure Schema registry is supposed to be there 4) I know Partitions should give good throughput - but at what message volume and sizes we should consider partitioning -= 5) How can Kafka be possible using for Batched Datasets? - e.g. nightly data snapshot? I stumbled on web for above, but no direct answer to most of them - Please advise. ... View more

All, I want to learn how to use various settings ( set hive.tez... etc ) in hive for tuning and whats meaning of each and impacts of each ... where can i study each of them and see the effect of each setting during execution. Also is there extensive list of each of them ? We been strugling big time for identifying right settings to use, and simulate exact issues in lower environment like prod ? Thanks Freakabhi ... View more

All, I am working on writing to RDBMS ( Sql Server ) using Spark from hive, process works with great speed. But there is big issue, that each tasks until completes does not commits - which utilizes transaction log of the database and can cause impacts to other running jobs. Need to have some way to commits at regular interval ( 10000 K or so) . Can someone please suggest how this can be done?? Spark version : 2.2 SQL Server 2016 Thanks freakabhi ... View more

All, I am trying to read hive table and load into SQL Server using Spark API. Process works fine for small tables, but recently I tried to apply the same for one of the large hive source table and started writing to Target database using partitions. Process was running fine, except when I looked at the transactions logs it was getting filled up. My job had 20 spark tasks or executors - I realized, Spark job was pumping data into Database in 20 ways and was commiting at the end of each Task only, which was causing transaction log to fill. Is there way to avoid or have intermediate commits while writing to database using spark? thanks, Freakabhi ... View more

while SQOOP import,Is there any way to use SQL functions in split by Key like below? since below is only legit split-key which yields even sized mappers. to_char(srvc_date, 'YYYYMM') or any other solution ? ... View more

I am working in cluster environment for HDP 2.6; I am trying to create oozie workflow using hive2 action to call simple hive script. oozie workflow keeps on failing with "unable to retrive delegation token for user <abcd> ( id job submitted with )" I have seen several links and none of them helps; is there any solution exists ? I see couple of solution which might work on VM ( where you can su with oozie/hive users and kinit ), but this is not possible in realtime working environment. job.properties. ---------------- nameNode=hdfs://<hostname>:8020 job-tracker=<hostname>:8050 oozie.use.system.libpath=true oozie.wf.application.path=${nameNode}/user/${user.name}/hive_action dry_run=false security_enabled=false workflow.xml ---------------- <workflow-app xmlns='uri:oozie:workflow:0.1' name='hive-wf'> ... <credentials> <credential name='howlauth' type='hive2'> <property> <name>hive2.jdbc.url</name> <value>jdbc:hive2//hostname:100000/default</value> </property> <property> <name>hive2.server.principal</name> <value>hive/hostname@abc.com</value> </property> </credential> </credentials> ... <action name='hive212' cred='howlauth'> <hive> <job-tracker>JT</job-tracker> <name-node>NN</name-node> <configuration> <property> <name>oozie.action.sharelib.for.hive</name> <value>hive2</value> </property> <property> <name>oozie.action.launcher.main.class</name> <value>org.apache.oozie.action.hadoop.hive2Main</value> </property> </configuration> <jdbc-url></jdbc-url> <password>####</password> <script>###script_location###</script> </hive> </action> ... </workflow-app> ... View more

All, I have big query which is data for data-science model pulling data from 10+ sources, Query is quite extensive and I am trying to convert into Hive ELT or Spark-SQL solution, Can you please suggest whats recomanded as per the experiece. where I try below query with hive , it fails with vertex error and when I try with pyspark - or spark-SQL , it fails with OOM and GC overhead .. Do we have any suggestions or references which talks about solutionizing big Queries with pyspark or HQL.... Original Query is some what like : select driving_src left outer join big_table t1 where t1.col = 'xyz' left outer join big_table t2 where t2.col = 'fsdds' left outer join big_table t3 where t3.col = 'vvvv ... View more

All, I have following questions on OOZIE 1) when I create OOZIE job - how do I maintain and pass environment level parameters ; e.g. output path for Unit/ Test/ stage and where to maintain those , should I have 3 job properties file one for each environment ? 2) how to migrate OOZIE workflows from one environment to other, ( unit to Test/stage and prod ) ? since oozie is kept in HDFS directories and job properties 3) My workflows are failing for Kerberoze auth / how to avoid and solutionize this, if I do kinit, it will be local to host and not to cluster.. if I understand it right , Please let us know. ... View more

I wanted to understand how to approach to the solution of hive query failures; in our environment, we get numerous failure for "Vertex issue" / "Out of memory" etc ? is it common? how to check Hive Server2 logs/ yarn-logs and resource manager logs ( what to look for ? ) how to know if there is SKEW or another issue - Are there article covering these topics What hive setting/parameters means what and how it affects --> where to look for this details? what tools are good or being used to analyze issues in hive queries or map/reduce level? ... View more

I am planning to enable full-fledged development environment for Spark Practice, with Intellij / eclipse installed I am trying to enable Desktop mode for HDP 2.6 VM and it keeps on failing while trying to Add VNC Server; is it possible to achieve this? Error I am getting is shown in the screen-shot attached. ... View more

All, I am trying to install hue 3.10 on the HDP 2.6 sandbox and following the below link and followed all the steps; I can see hue 3.10 is getting installed into /usr/local/hue directory, but when I try to access the localhost:8000 is still pointing to hue 2.6 ; I believe some setting is still pointing to /usr/lib/hue ( where hue 2.6 is installed ) I am trying to run /usr/local/hue/...supervisor -d .. still no lunk am i missing something, how can i have hue to point to correct directories and run .. Are there any other link for sandbox hue upgrade? link followed: http://gethue.com/hadoop-hue-3-on-hdp-installation-tutorial/ https://mapredit.blogspot.com/2016/12/hue-311-with-hdp-25.html ... View more

I wanted to get the suggestion on the incremental strategy for tables be implemented : We have set of source table which are getting refreshed on the daily basis in the source ( DB2 ) and we need to refresh then in hive db as well, which approach will you suggest. Please note : Source table have new inserts as well as updates to existing records; 1) approach 1: USe Hbase to store the data since updates are allowed and build hive external table referring to the same I doubt if this will affect queries using the joins for hive-hbase table with large ORC hive tables? 2) approach 2 : USe 4 step incremental table approach suggested by HDP ? https://hortonworks.com/blog/four-step-strategy-incremental-updates-hive/ thanks Abhijeet ... View more

All, I would like to get the suggestions and correct way to convert very large queries like ( 1000 lines ) joining 10+ tables and complicated transforms to Py-Spark program   Also if there are relevent examples for large sqls.   and also I have question regarding writing SparkSQL program, is there difference of performance between writing   1) SQLContext.sql("select count(*) from (select distinct col1,col2 from table))") 2) using pyspark Api : df.select("col1,col2").distinct().count().   I am from SQL background and we are working on converting existing logic to hadoop, hence SQL is handy. ... View more

All,  I am working on importing 1.2 billion rows from one of the db2 table, There is composite primary index in the table with at least 5 columns, hence I need to manually specify the --split-by column ( since SQOOP does not support multi-part split-by column), I tried to run the Sqoop import with one of the columns from the index which is numeric , and import ran for almost 8 hrs. I am being suggested to try with different choice for split by key, but now I have question how to choose column for split by  1) is it supposed to be numeric?  or varchar is fine too? 2) can it have nulls 3)  I suppose distribution of values for the column should be as even as possible, but will it matter how many maps I am choosing (--num-map ## )? or any other criteria to pay attention to?   thanks  Abhijeet  ... View more

All , I am working on creating and loading very high volume (450 million ) ORC table and process keeps on failing for vortex error . Platform: hdp 2.4 Engine : tez Why does this happens and what's solution? ... View more

I have question regarding, if we are creating common reservoir for the files in hadoop which can later be used for any purpose, may it be spark processing or pig or hive ...etc. Now I have question on while sqooping data into the hadoop, which file format to choose, any industry wide standards ? 1)Text delimited ( compressed or un-compressed ?) 2) AVRO 3)Parquet ... View more

All, Is there any way to generate .XDF files ( extensible file format ) for consumption in R from hive datasets or any files in HDFS? if anyone has done this in the past, Please suggest some possible solutions Problem statement : we have source data in hive and want to Land .xdf dataset for few tables. ... View more