About subhrajit

subhrajit · ‎05-01-2018

I have one requirement where I just want to use spark JDBC driver to read oracle data which is encrypted and after processing, I want to load into Oracle and while loading data should be encrypted.

subhrajit · ‎03-18-2018

for Cloudera manager, it needs minimum 8GB RAM. Is there any way I can replace current spark 1.6 version to 2 without using parcel so that I can run the VM with less than 8 GB?

subhrajit · ‎02-15-2018

Hi, I have follwed all the steps to install Spark2 in cloudera vm but getting following error when I am trying to Host spark2 parcel. CDH (5.8 and higher) parcel required for SPARK2 (2.2.0.cloudera2-1.cdh5.12.0.p0.232957) is not available. I am using CDH version : 5.12 and using http://archive.cloudera.com/spark2/parcels/2.2.0.cloudera2/ link for parcel. Please help me with the configurtion.

subhrajit · ‎01-17-2017

dt1 = {'one':[0.3, 1.2, 1.3, 1.5, 1.4, 1],'two':[0.6, 1.2, 1.7, 1.5,1.4, 2]} dt = sc.parallelize([ (k,) + tuple(v[0:]) for k,v in dt1.items()]).toDF() dt.show() I want 'one' and 'two' to be column header and all list values should be column values

subhrajit · ‎12-15-2016

Hi Tim, thanks for replying. my start_ipython_notebook.sh contains #!/bin/bashsource/opt/rh/python27/enable IPYTHON_OPTS="notebook --port 8889 \ --notebook-dir='/usr/hdp/current/spark-client/' \ --ip='*' --no-browser" pyspark I am using linux. I tried with both pc password and admin as password but both are not working. I am sending you the screenshot please check once.jupyter.png

subhrajit · ‎12-14-2016

Hi paul after installation jupyter asks for password what will be the password as I didn't set any password couldn't start jupyter notebook. Can you please help me?

subhrajit · ‎12-14-2016

subhrajit · ‎11-08-2016

error.png I am getting steaming json data but got issue while analysing further.

subhrajit · ‎11-05-2016

I am able to get now. One thing I had to import all twitter streaming related jars during spark submit. Do we have any other option for this?

subhrajit · ‎11-05-2016

Hey Artem, I have done this in sbt project and I have uploaded my build.sbt file as well can you please check once.

subhrajit · ‎11-05-2016

can you please suggst me how to add twitter dependency as I am using hdp 2.4 and created jar and trying to submit. I guess while using spark-submit I need to add some packages.

subhrajit · ‎11-05-2016

error.pngtwitterdata.txt buildsbt.txt I have uploaded all the files along with code.

subhrajit · ‎09-22-2016

I have scenario like this Array((0,(27,20,9)), //(p1,(a1,b1,c1)) (1,(16,15,270)), //(p2,(a2,b2,c2)) (2,(7,15,650))) //(p3,(a3,b3,c3)) I want to calculate some scenarios based upon the element for first column d1=c1,e1=d1-1,f1=e1/c1 from second column onwards it depending from previous column d2=f1*c1,e2=f1*c2-1,f2=e2/c2 if you could help it will be great as I am new to spark

subhrajit · ‎09-21-2016

subhrajit · ‎08-30-2016

empinfo.txtbuilderror.png I have used SBT for build and submit the code but getting error for windows function I have uploaded the code and the error please check once.

subhrajit · ‎08-30-2016

Thanks Arun. I am able to create jars 🙂

subhrajit · ‎08-29-2016

subhrajit · ‎08-20-2016

thanks Arun for your response. I have tried in scala IDE by creating maven project and when ever I am adding dependency of spark getting compile time error for which I couldn't proceed. I am using HDP 2.4 which has spark 1.6.0 and scala 2.10.5. I am not sure which scala IDE and what versions I have to use. please suggest me. I just want to build a jar and submit the job.

subhrajit · ‎08-19-2016

I have done package and made a jar but while running I am facing this issue with that jar. I have tried but getting same class not found exception.

subhrajit · ‎08-17-2016

I am getting class not found exception. I have uploaded my scala as well as build.sbt file please have a look. files.zip

subhrajit · ‎08-16-2016

Thanks for the information. How I can configure to lower version. I have scala 2.10.5 as I am using hdp 2.4. I have tried to do in quick fix but not able to configure.

subhrajit · ‎08-15-2016

subhrajit · ‎07-31-2016

Thanks for the explanation.

subhrajit · ‎05-09-2016

thank you 🙂

subhrajit · ‎05-05-2016

Thanks for replying. if not hive can impala read all versions in HBase?

subhrajit · ‎05-04-2016

subhrajit · ‎02-24-2016

@Saurabh Kumar thanx saurabh for the explanation now I got it what was the issue.. 🙂

subhrajit · ‎02-23-2016

can you please explain me what does it mean as I am not familiar with linux command as well

subhrajit · ‎02-19-2016

you can send me to my mail id(subhr.mohanty@gmail.com).

subhrajit · ‎02-19-2016

hiveissue.png

Online	Offline
Last Visited	‎07-09-2020 11:50 AM

Date Registered	‎06-29-2016 05:57 AM
Last Visited	‎07-09-2020 11:50 AM
Total Messages Posted	30
Total Kudos Received	5

Can spark read encrypted data from Oracle as well ...

Re: Can I install Spark 2 on the QuickStart VM?

Re: Installing Spark2

How to transpose a pyspark dataframe?

Re: After installing jupyter it is asking for pass...

Re: Hi all, Ifollowed the tutorial to install jupy...

After installing jupyter it is asking for password...

how to resolve corrupt json streaming records. I w...

Re: Issue while trying to implement spark streamin...

Re: Issue while trying to implement spark streamin...

Re: Issue while trying to implement spark streamin...

Issue while trying to implement spark streaming to...

Re: I have one RDD where the first row is fixed a...

I have one RDD where the first row is fixed and t...

Error when using window function: "Note that, usin...

Re: I am getting Exception in thread "main" java.l...

In spark dataframe how to get non null last value ...

Re: I am getting Exception in thread "main" java.l...

Re: I am getting Exception in thread "main" java.l...

Re: I am getting Exception in thread "main" java.l...

Re: I am getting Exception in thread "main" java.l...

I am getting Exception in thread "main" java.lang....

Re: save spark-csv output to hive in HDP 2.3.2

Re: how hive can read Hbase current and previous v...

Re: how hive can read Hbase current and previous v...

how hive can read Hbase current and previous versi...

Re: I am new to hadoop and I am trying for hive sh...

Re: I am new to hadoop and I am trying for hive sh...

Re: I am new to hadoop and I am trying for hive sh...

I am new to hadoop and I am trying for hive shell ...