About muthaiyaprabhu

muthaiyaprabhu · ‎10-02-2017

@mqureshi Thank you for the prompt response. I am new to this space. Could you please elaborate little bit on setting up spark- env.sh. I understand this is to hand shake with hive, trying to get exact values. My current set up is: HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/usr/hdp/2.6.2.0-205/hadoop/conf} How to add hdfs,hive,core site xmls. In the java code you put above, i don't see the hive connection parameters. Do i need to replace the values on this? .config("spark.some.config.option","some-value") Please advise.

muthaiyaprabhu · ‎09-30-2017

Hi All, I have table 1 in hive say emp1, which has columns empid int, name string, dept string, salary double. In spark, using data frame i would like to read the data from hive emp 1 table, and i need to load them into another table called emp2(assume emp2 is empty and has same DDL as that of emp1). It would be great if i get java reference code. No scala or python code needed. Thanks in advance!

muthaiyaprabhu · ‎08-24-2017

Hive: I have a column name(varchar), table has now both alphabets and pure numeric values. I want to count the column name that has only numeric values

muthaiyaprabhu · ‎04-07-2017

Thank you Bala!

muthaiyaprabhu · ‎04-06-2017

Thanks Namit Maheshwari, data i am bringing into test is hive data, i need to filter using some criteria, like where condition in hive query. distcp -filters to exclude some files right, not on the data level. I want to filter the hive data using some criteria in production, and then want to bring the filtered data into test region.

muthaiyaprabhu · ‎04-06-2017

I need to bring prod hive table data into test hive table. Since it's a hadoop to hadoop, i can't use sqoop, hence i can use discp to transfer data across the clusters. But i have one more scenario to be handled while bringing data, that is filtering. Say i have 10 million records in prod hive table, i want to filter using some criteria and bring it to test table. is there a way to give filter parameters in distcp command on the fly? Or any other suggestions? Thanks in advance.

Online	Offline
Last Visited	‎03-08-2018 07:15 PM

Member Since	‎04-01-2016 06:57 PM
Last Visited	‎03-08-2018 07:15 PM
Posts	8

Cloudera Community

Re: How to read hive table1 from spark, using data...

How to read hive table1 from spark, using datafram...

numeric check in hive

Re: Need to bring prod hive table data into test e...

Re: Need to bring prod hive table data into test e...

Need to bring prod hive table data into test envir...