I have few questions... 1) Is it necessary to write the producer application in structured spark when we are writing the consumer application in structured spark? I have written producer application in Kafka and consumer application in Structured spark . But i am not able to fetch the data from the kafka topic. Can anyone help me with this as soon as possible. 2) How to use the custom Deserializer on Values in Structured spark? ... View more

I have wrote a Producer application which writes data to kafka topic (converting from strings to bytes). Now i have written structured spark based consumer application which i am trying to read from that topic. I have setup the schema also. But my i get all fields as NULL values as shown below.. +-------------+--------------------+--------+---------+---------+--------------+----+---------+----------------+ | |SENT_TIME|LAST_OCCURANCE|NODE|NODE_TYPE|X733SPECIFICPROB| +-------------+--------------------+--------+---------+---------+--------------+----+---------+----------------+ | null | null | null | null |null | null | null | | null | null| null| null|null| null| null| |null| null| null| null|null| null| null| | null| null| null| null|null| null| null| | null| null| null| null|null| null| null| | null| null| null| null|null| null| null| |null| null| null| null|null| null| null| | null| null| null| null|null| null| null| +-------------+--------------------+-------- May i know where i might have done the mistake? ... View more

1) where i can find the code on windowing function that is used in spark streaming. (written in scala) 2) By default on which timestamp the windowing works? Is it received time or sent time ? ... View more

17/10/16 14:17:00 ERROR Utils: Exception encountered java.lang.ClassNotFoundException: scala.Some at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:67) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) ... View more

I am getting the following error: 17/10/16 11:01:07 INFO JobGenerator: Started JobGenerator at 1508131870000 ms 17/10/16 11:01:07 INFO JobScheduler: Started JobScheduler 17/10/16 11:01:07 INFO StreamingContext: StreamingContext started 17/10/16 11:01:10 INFO JobScheduler: Added jobs for time 1508131870000 ms 17/10/16 11:01:10 INFO JobScheduler: Starting job streaming job 1508131870000 ms.0 from job set of time 1508131870000 ms ------------------------------------------- Time: 1508131870000 ms ------------------------------------------- 17/10/16 11:01:10 INFO JobScheduler: Finished job streaming job 1508131870000 ms.0 from job set of time 1508131870000 ms ... View more

Steps to be performed before submiting the spark jobs from a remote machine. I have tried adding the core.xml and yarn.xml file form master to the remote machine in the spark conf foldern. Still its not working. Can anyone suggest how to perform this step. ... View more

I have tried doing it by installing mysql and created user and database for hive and granted all its privileges. But when i do the ambari-start, the ambari server wont get strated. I get a message as saying "couldnt get the PID of ambari-server". If anyone knows how to setup ambari with mysql database please reply as soon as possible. ... View more

error.txt Can anyone suggest me what to be done to resolve this. ... View more

I tried doing that and the result what i found was as connection refused. @Sindhu #OUTPUT Trying 192.168.56.1... telnet: Unable to connect to remote host: Connection refused So next what is should do to solve this issue. ... View more

Metastore connection URL: jdbc:mysql://erblrpc02/hive?createDatabaseIfNotExist=true Metastore Connection Driver : com.mysql.jdbc.Driver Metastore connection User: hive org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version. Underlying cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException : Communications link failure ... View more