Member since
06-29-2016
30
Posts
5
Kudos Received
0
Solutions
05-01-2018
06:54 AM
I have one requirement where I just want to use spark JDBC driver to read oracle data which is encrypted and after processing, I want to load into Oracle and while loading data should be encrypted.
... View more
Labels:
03-18-2018
09:12 AM
for Cloudera manager, it needs minimum 8GB RAM. Is there any way I can replace current spark 1.6 version to 2 without using parcel so that I can run the VM with less than 8 GB?
... View more
02-15-2018
08:56 AM
Hi, I have follwed all the steps to install Spark2 in cloudera vm but getting following error when I am trying to Host spark2 parcel. CDH (5.8 and higher) parcel required for SPARK2 (2.2.0.cloudera2-1.cdh5.12.0.p0.232957) is not available. I am using CDH version : 5.12 and using http://archive.cloudera.com/spark2/parcels/2.2.0.cloudera2/ link for parcel. Please help me with the configurtion.
... View more
01-17-2017
05:56 AM
dt1 = {'one':[0.3, 1.2, 1.3, 1.5, 1.4, 1],'two':[0.6, 1.2, 1.7, 1.5,1.4, 2]} dt = sc.parallelize([ (k,) + tuple(v[0:]) for k,v in dt1.items()]).toDF() dt.show() I want 'one' and 'two' to be column header and all list values should be column values
... View more
Labels:
12-15-2016
02:19 AM
Hi Tim, thanks for replying. my start_ipython_notebook.sh contains #!/bin/bashsource/opt/rh/python27/enable
IPYTHON_OPTS="notebook --port 8889 \
--notebook-dir='/usr/hdp/current/spark-client/' \
--ip='*' --no-browser" pyspark
I am using linux. I tried with both pc password and admin as password but both are not working.
I am sending you the screenshot please check once.jupyter.png
... View more
12-14-2016
05:41 PM
Hi paul after installation jupyter asks for password what will be the password as I didn't set any password couldn't start jupyter notebook. Can you please help me?
... View more
11-08-2016
04:38 PM
error.png
I am getting steaming json data but got issue while analysing further.
... View more
Labels:
11-05-2016
07:01 PM
I am able to get now. One thing I had to import all twitter streaming related jars during spark submit. Do we have any other option for this?
... View more
11-05-2016
06:14 PM
Hey Artem, I have done this in sbt project and I have uploaded my build.sbt file as well can you please check once.
... View more
11-05-2016
03:25 PM
can you please suggst me how to add twitter dependency as I am using hdp 2.4 and created jar and trying to submit. I guess while using spark-submit I need to add some packages.
... View more
11-05-2016
12:41 PM
error.pngtwitterdata.txt
buildsbt.txt
I have uploaded all the files along with code.
... View more
Labels:
09-22-2016
06:37 AM
I have scenario like this Array((0,(27,20,9)), //(p1,(a1,b1,c1))
(1,(16,15,270)), //(p2,(a2,b2,c2))
(2,(7,15,650))) //(p3,(a3,b3,c3))
I want to calculate some scenarios based upon the element
for first column
d1=c1,e1=d1-1,f1=e1/c1
from second column onwards it depending from previous column
d2=f1*c1,e2=f1*c2-1,f2=e2/c2 if you could help it will be great as I am new to spark
... View more
08-30-2016
02:44 AM
empinfo.txtbuilderror.png I have used SBT for build and submit the code but getting error for windows function I have uploaded the code and the error please check once.
... View more
Labels:
08-30-2016
02:42 AM
Thanks Arun. I am able to create jars 🙂
... View more
08-20-2016
03:59 AM
thanks Arun for your response. I have tried in scala IDE by creating maven project and when ever I am adding dependency of spark getting compile time error for which I couldn't proceed. I am using HDP 2.4 which has spark 1.6.0 and scala 2.10.5. I am not sure which scala IDE and what versions I have to use. please suggest me. I just want to build a jar and submit the job.
... View more
08-19-2016
03:59 PM
I have done package and made a jar but while running I am facing this issue with that jar. I have tried but getting same class not found exception.
... View more
08-17-2016
12:22 AM
I am getting class not found exception. I have uploaded my scala as well as build.sbt file please have a look.
files.zip
... View more
08-16-2016
05:48 AM
Thanks for the information. How I can configure to lower version. I have scala 2.10.5 as I am using hdp 2.4. I have tried to do in quick fix but not able to configure.
... View more
07-31-2016
01:44 AM
Thanks for the explanation.
... View more
05-09-2016
06:19 AM
thank you 🙂
... View more
05-05-2016
05:58 AM
Thanks for replying. if not hive can impala read all versions in HBase?
... View more
02-24-2016
06:01 AM
@Saurabh Kumar thanx saurabh for the explanation now I got it what was the issue.. 🙂
... View more
02-23-2016
03:10 PM
can you please explain me what does it mean as I am not familiar with linux command as well
... View more
02-19-2016
03:15 AM
you can send me to my mail id(subhr.mohanty@gmail.com).
... View more