Member since
08-02-2017
11
Posts
0
Kudos Received
0
Solutions
12-05-2017
04:24 AM
I have few questions... 1) Is it necessary to write the producer application in structured spark when we are writing the consumer application in structured spark? I have written producer application in Kafka and consumer application in Structured spark . But i am not able to fetch the data from the kafka topic. Can anyone help me with this as soon as possible. 2) How to use the custom Deserializer on Values in Structured spark?
... View more
Labels:
12-04-2017
10:40 AM
I have wrote a Producer application which writes data to kafka topic (converting from strings to bytes). Now i have written structured spark based consumer application which i am trying to read from that topic. I have setup the schema also. But my i get all fields as NULL values as shown below.. +-------------+--------------------+--------+---------+---------+--------------+----+---------+----------------+ | |SENT_TIME|LAST_OCCURANCE|NODE|NODE_TYPE|X733SPECIFICPROB| +-------------+--------------------+--------+---------+---------+--------------+----+---------+----------------+ | null | null | null | null |null | null | null | | null | null| null| null|null| null| null| |null| null| null| null|null| null| null| | null| null| null| null|null| null| null| | null| null| null| null|null| null| null| | null| null| null| null|null| null| null| |null| null| null| null|null| null| null| | null| null| null| null|null| null| null| +-------------+--------------------+-------- May i know where i might have done the mistake?
... View more
Labels:
11-10-2017
05:12 AM
1) where i can find the code on windowing function that is used in spark streaming. (written in scala) 2) By default on which timestamp the windowing works? Is it received time or sent time ?
... View more
Labels:
10-16-2017
08:51 AM
17/10/16 14:17:00 ERROR Utils: Exception encountered java.lang.ClassNotFoundException: scala.Some at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:67) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027)
... View more
10-16-2017
05:55 AM
I am getting the following error: 17/10/16 11:01:07 INFO JobGenerator: Started JobGenerator at 1508131870000 ms 17/10/16 11:01:07 INFO JobScheduler: Started JobScheduler 17/10/16 11:01:07 INFO StreamingContext: StreamingContext started 17/10/16 11:01:10 INFO JobScheduler: Added jobs for time 1508131870000 ms 17/10/16 11:01:10 INFO JobScheduler: Starting job streaming job 1508131870000 ms.0 from job set of time 1508131870000 ms ------------------------------------------- Time: 1508131870000 ms ------------------------------------------- 17/10/16 11:01:10 INFO JobScheduler: Finished job streaming job 1508131870000 ms.0 from job set of time 1508131870000 ms
... View more
08-17-2017
02:51 PM
Steps to be performed before submiting the spark jobs from a remote machine. I have tried adding the core.xml and yarn.xml file form master to the remote machine in the spark conf foldern. Still its not working. Can anyone suggest how to perform this step.
... View more
Labels:
08-14-2017
03:22 AM
I have tried doing it by installing mysql and created user and database for hive and granted all its privileges. But when i do the ambari-start, the ambari server wont get strated. I get a message as saying "couldnt get the PID of ambari-server". If anyone knows how to setup ambari with mysql database please reply as soon as possible.
... View more
Labels:
08-03-2017
10:57 AM
error.txt Can anyone suggest me what to be done to resolve this.
... View more
Labels:
08-03-2017
05:43 AM
I tried doing that and the result what i found was as connection refused. @Sindhu #OUTPUT Trying 192.168.56.1... telnet: Unable to connect to remote host: Connection refused So next what is should do to solve this issue.
... View more
Labels:
08-03-2017
02:16 AM
Thank you for the reply@Sindhu I tried doing that and the result what i found was as connection refused. #OUTPUT Trying 192.168.56.1... telnet: Unable to connect to remote host: Connection refused So next what is should do to solve this issue.
... View more
08-02-2017
07:47 AM
Metastore connection URL: jdbc:mysql://erblrpc02/hive?createDatabaseIfNotExist=true
Metastore Connection Driver : com.mysql.jdbc.Driver
Metastore connection User: hive
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.
Underlying cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException : Communications link failure
... View more
Labels: