About bigspark

bigspark · ‎07-23-2019

Do we have Analytical Window function in SAM by any chance? I tried to use Analytical Window functions in structured streaming but no luck. SO wondering if i solve this using SAM? I looked into it we can do stream-stream join which is what iam looking for. However not sure if we have window RANK/LAG functionality in SAM? Thank you

bigspark · ‎07-22-2019

I would like to use an Analytical function like RANK/LAG in Spark Structured streaming. Do we have Analytical Window function in Spark Structured Streaming by any chance?I am aware org.apache.spark.sql.expressions.Window is available, but this probably works with Data Frames only, not with Spark Structured streaming. Also, org.apache.spark.sql.window method is purely used for sliding data in window in Structured streaming. Also, Any suggestions on how to pick Max Date record in a group, in Spark structured streaming if we don’t have rank option. val groupedOutput = joinedDf.groupBy( window(col("timestampIpfr"), "2 minutes"), col("userid"), col("timestampIpfr") ).agg(max("timestampMME").alias("maxTimestampMME"), max("locationMME").alias("locationMME") //Max on locationMME doesn't work here, looking for help on the same to find locationMME of maxTimestampMME record ) Expecting Grouped output value with max timestampMME record and corresponding locationMME

bigspark · ‎06-27-2019

We are using kafka 1.0 in hdp/hdf 2.6. can we write data to hdfs from kafka topic using kafka connect. To be clear we are referring to Apache kafka not confluent kafka. Any suggestions please.

bigspark · ‎05-14-2019

Can you please try removing the master("local[*]") from the spark code and pass it as a parameter in spark submit -- master yarn --deploy-mode cluster.. It should work..

bigspark · ‎10-05-2018

we are using spark2.3 spark sql, java8 running on Yarn. In spark ui we can observe task straggling, data skew and lot of GC. We are currently using default java serialisation. hopefully there will be some performance improvement when we move to kryo. in terms of gc plz suggest best practices to tune gc. we are planning for g1gc. data skew and task straggler we are planning to use salting any salting examples will be helpful and can we use bucketing but we are not using Hive here just file based and spark sql typed dataset will spark bucketing work for file based? any bucketing example would be great plz. tried parallelising with partitions and spark sql shuffle partitions no luck. only 160mb of data but its growing as a massive data when loaded and performing aggregations in spark not sure why its exponentially growing? thank u

bigspark · ‎10-05-2018

Hi all, Using java 8 & spark 2.3 typed dataset api for our single parent nested json object of around 150 elements and soem nested elements as well in the same json. My question is we created a single java pojo object of the parent json and we perform lot of transformations and actions on this single dataset<jsonobject>. its been passed around atleast 15 java classes filtered and aggregated. Just wondering about the memory issues in spark when we pass the beafy java object.. plz suggest

bigspark · ‎07-16-2018

Original source is Mainframe systems or RDBMS and ingested into hive tables in hadoop. Now Application reads from hive tables orc format as source data. We have a sub-set of columns that we need for our v1 release. And the schema will be fixed for v1. In future for v2 and so on we will have another set of columns coming but as this is the batch application business will notify for v2 the new columns or existing columns being renamed or deleted etc. I have worked in streaming applications where we receive files from different organisations where the schema is not fixed. In this application do we need schema evolution to be considered even for future proposition. Or do i have to consider Avro instead of ORC. The application has a requirement to read subset of columns and aggregate on few columns I have mentioned data storage as hive (1.2) ORC format with Snappy compression to be processed in Spark 1.6 in HDP 2.5. And adding new columns in ORC is supported. https://community.hortonworks.com/questions/26718/orc-schema-evolution-in-hive.html Please confim.

bigspark · ‎04-06-2018

@bkosaraju thanks for your reply. So you are suggesting for Hive with LLAP or postgres DB(any RDBMS) in cluster? If you suggest Hive with LLAP: We are using HDP-2.5 how can i configure Hive with LLAP? Ya in the batch processing i ll be required to insert and update records. we are using hive1.2 and hope it supports ACID and updates? Do you think it will help to create indexes on these hive tables? https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Indexing

bigspark · ‎04-05-2018

Hi All, Currently we are doing batch processing using spark (1.6 )and hive (1.2) and HDP 2.5. While processing batches we need to store information about the batches saying the batchid, start time of batch , end time of batch etc i.e. a control table. Will it work if i store this data in a hive table for spark to read it before every batch process or do i use Hbase for it as its quick to lookup records? plz suggest the best practice.

bigspark · ‎04-03-2018

Currently running spark on Intellij in my local machine. I also have hdp-2.5 sandbox on local machine on top of virtual box. In my sandbox hive is running and i have created a few tables. Iam planning to read data from those hive tables using Spark on Intellij running on my local laptop. Any instructions to do that plz

bigspark · ‎02-27-2018

we have hbase in our datalake with hdp-2.5. We bulk load onto it from spark sql jobs and we need end users to query the processed data sitting on hbase via a REST-API. 1.is it best practice to provide end user access via api to the data in hbase? 2. Or we have cassandra or DB2 outside of hadoop cluster purely to provide rest api on top of it for user to access it? 3. Is cassandra faster in terms of random read than Hbase? thank you

bigspark · ‎02-21-2018

https://github.com/hortonworks-spark/shc Does the above SparkOnHbase Hortonworks connector support bulk loading i.e. petabytes of data onto hbase?

bigspark · ‎02-21-2018

@Josh Elser @Harald Berghoff If the jira patch is pushed then i believe we can use this cloudera connector and donot need to rewrite hbase salting and spark partitioner from the below post is it? We can just use the connector api instead isnt it? http://www.opencore.com/blog/2016/10/efficient-bulk-load-of-hbase-using-spark/ Also we have another connector from hortonworks not sure if this supports bulk loading the previous one is from cloudera and rdd version. however the below one supports dataframe version of spark https://github.com/hortonworks-spark/shc It would be good if this connector supports dataframe bulk loading? It supports the standard write not sure abt bulk load though https://issues.apache.org/jira/browse/HBASE-15336

bigspark · ‎02-21-2018

We have spark-cassandra connector from datastax. When we are bulk loading petabytes of data to shall we use the datastax spark-cassandra connector or is there any other alternative way to bulk load into cassandra? Also when would we use spark-cassandra connector i mean threshold volume it supports loading? thank you https://github.com/datastax/spark-cassandra-connector

bigspark · ‎02-21-2018

@Pavan Kumar KondaIt depends on lot of constraints like compression, serialization and whether the storage format is splittable etc. I think ORC is just for hive. Avro Avro is language neutral data serialization Writables has the drawback that they do not provide language portability. Avro formatted data can be described through language independent schema. Hence Avro formatted data can be shared across applications using different languages. Avro stores the schema in header of file so data is self-describing. Avro formatted files are splittable and compressible and hence it’s a good candidate for data storage in Hadoop ecosystem. Schema Evolution – Schema used to read a Avro file need not be same as schema which was used to write the files. This makes it possible to add new fields. Parquet Parquet is a columnar format. Columnar formats works well where only few columns are required in query/ analysis. Only required columns would be fetched / read, it reduces the disk I/O. Parquet is well suited for data warehouse kind of solutions where aggregations are required on certain column over a huge set of data. Parquet provides very good compression upto 75% when used with compression formats like snappy. Parquet can be read and write using Avro API and Avro Schema. It also provides predicate pushdown, thus reducing further disk I/O cost.

bigspark · ‎02-21-2018

@Harald Berghoff & @Josh Elser thanks for your reply. OK will try to use HFiles from according to the link which i knew exists. However in that link it says about Spark-hbase connector to support bulk loading in future. Now the jira says its resolved then does it mean now we can use the spark-hbase connector? https://issues.apache.org/jira/browse/HBASE-14150

bigspark · ‎02-19-2018

Iam looking to read really large petabytes scale hive tables into spark, which is a better option in terms of performance to read directly from hive table using select stmt or giving the hdfs path for the ORC files of hive tables to read? Looking for best practice and better performance as well. Also looking to save petabytes scale data to hbase tables. HW has a spark-hbase connector not sure how its to do with petabytes volume of data and performance as well. Shall i consider using HFILES or spark-hbase connector? Any suggestions will be really appreciated... thank you

bigspark · ‎02-15-2018

On a highlevel i know the difference between kafka streams and structured streaming. However which is currently better to use in production?

bigspark · ‎03-22-2017

Hi All, Just looking for various open source options available to monitor the input data rate to show some metrics on the input data for last 24 hours etc. Also will be using this monitoring tools for other options as well to purely monitor everything related t spark streaming? Any ideas in terms of the tools i know we currently have Spark Streaming UI, which will show the number of rdds processed but currently i just wanted to know the input data rate before these are processed? Any ideas in terms of the open source tools to use will be appreciated? 1. Ganglia? 2. Push pull receivers? 3. Spark logs i.e. as soon as the messages are received shall we log them and connect a UI tool to it. Please let me know if there are any more efficient options. Thank you

bigspark · ‎02-23-2017

@mqureshi yes i tried making it transient as well no luck and inside .map map(path => sc.textFile(path)) its not reading the contents of the file i thnk its just returning the string which is strange if i do sc.textfile outside map function it returns the data. Above is the whole code nothing else to it. Looks simple.. probably rather than calling sc.textfile inside map may be i need to fetch data using s3 api as shown in this link. http://michaelryanbell.com/processing-whole-files-spark-s3.html but not sure i might still get task serailization error???? any ideas to better implement this requirement

bigspark · ‎02-22-2017

Iam trying to read the gzip files in a dir parallely. I followed the steps advised by Matei in the following link http://apache-spark-user-list.1001560.n3.nabble.com/Strategies-for-reading-large-numbers-of-files-td15644.html but i get the below exception. looks like other people as well got the same exception. Just wanted to know if its possible to acheive this in spark 2.1.0. iam running on local VM at the moment using a simple two line code. splitFilesPathList = List("s3n://pathtos3file1","file2,file3)etc... val lineRDD = sc.parallelize(splitFilesPathList, 4).map(path => sc.textFile(path)).take(10).toList.foreach(println) The above 2 lines code doesnt work. Any help is really appreciated please. I have checked for closures and moved the code into a new scala class which extends serializable but i still get the task not serializable exception I tried almost all possible ways. I checked this as well https://forums.databricks.com/questions/369/how-do-i-handle-a-task-not-serializable-exception.html

bigspark · ‎02-09-2017

I have a dataset of structure i.e. .gz file which iam reading in spark abcdefghij; abc=1234 xyz=987 abn=567 ubg=345 after pivot abcdefghij abc ubg abn xyz abcdefghij 1234 345 567 987 and so on. All the above columns are string columns and abn values are duplicated. So as they are strings and iam just looking to split the data thats why i dont need aggregation, i just need pivot. Any ideas to acheive this in spark scala? thank u

bigspark · ‎02-09-2017

Hi Tibor thanks for your reply. I have looked at the above link. I have a dataset of structure i.e. .gz file which iam reading in spark abcdefghij; abc=1234 xyz=987 abn=567 ubg=345 after pivot abcdefghij abn ubg abc xyz abcdefghij 567 987 1234 987 and so on. All the above columns are string columns and abn values are duplicated. So as they are strings and iam just looking to split the data thats why i dont need aggregation, i just need pivot. Any ideas to acheive this in spark scala? thank u

bigspark · ‎02-09-2017

Iam looking to perform spark pivot without aggregation, is it really possible to use the spark 2.1.0 api and generate a pivot without aggregation. I tried but as the pivot returns groupeddataset without aggregation the api is not working for me. Any ideas how to convert to DF or dataset without performing aggregation and show the DF please. Thank you

bigspark · ‎02-02-2017

Just wondering if spark supports Reading *.gz files from an s3 bucket or dir as a Dataframe or Dataset.. I think we can read as RDD but its still not working for me. Any help would be appreciated. Thank you. iam using s3n://.. but spark says invalid input path exception. val df = spark.sparkContext.textFile("s3n://..../*.gz) doesnt work for me 😞 I prefer to the s3 dir of .gz files as a DF or Dataset if possible else atleast RDD please. thank you

bigspark · ‎12-21-2016

@Mridul M Thanks for the reply. Actually i was using the old version of spark testing base. Didn't know there was a 2.0.2 version. Holden karau pointed me to maven central where i found the latest version of spark testing base. In github readme of spark testing base it was mentioned 1.6 so i assumed the latest version was 1.6. But now its sorted and spark 2.0.2 Dataframe testing works for me. In case people might need this it also needs Hive dependency. Currently its new and couldnt find forums in spark 2.0 testing so iam posting this might help save time for other developers 🙂

bigspark · ‎12-20-2016

A needed class was not found. This could be due to an error in your runpath. Missing class: org/apache/spark/Logging java.lang.NoClassDefFoundError: org/apache/spark/Logging https://community.hortonworks.com/questions/58286/noclassdeffounderror-orgapachesparklogging-using-s.html Looked at the above question i think its valid for spark prior to 2.0 but iam looking for 2.0.2 🙂 I tried adding dependencies in sbt for Spark streaming, twitter and many more. Still i get this error for spark 2.0.2. Any ideas how to get this resolved? Thank you

bigspark · ‎12-08-2016

@Bernhard Walter thanks for the reply.

bigspark · ‎12-06-2016

Just wondering if we write a inner join query in plain sql or if we use dataframe api to perform join, do we get the same performance? I can see that the auery and dataframe are pushed to catalyst optimiser from the diagram but just wanted to confirm if i write plain sql queries can i use it for bigdata production usecases or shall i use dataframe for better performance? thank u

bigspark · ‎11-02-2016

@Matt Burgess currently we are using hive 0.13 version with cdh 5.3.1 and also we have nifi 0.7. Its running a hiveserver2 service. Currently we cannot change hive version on the cdh server. Is there any other quicker alternative to extract data from hive? Thank you..

Online	Offline
Last Visited	‎10-19-2020 08:16 AM

Date Registered	‎09-25-2015 09:05 AM
Last Visited	‎10-19-2020 08:16 AM
Total Messages Posted	112
Total Kudos Received	37

Re: Spark Testing base 1.6.1_0.3.3 for spark versi...

Does SAM provide Window ANalytical function like R...

Do we have Analytical function like RANK/LAG in Sp...

Is there a HDFS kafka connector for Kakfa-connect ...

Re: “User did not initialize spark context” Error ...

Spark2 performance tuning running on YARN

Spark SQL Dataset 2.3 single nested json parent ob...

We have a batch application using spark 1.6 & hive...

Re: Using hive tables as control table or lookup t...

Using hive tables as control table or lookup table...

Spark on Intellij running locally on mac or window...

Which one to pick Hbase vs cassandra vs db2 for cr...

SparkOnHbase Hortonworks connector does it support...

Re: Spark DF Reading Petabytes of data from Hive a...

Spark DF persisting Petabytes of data to CASSANDRA...

Re: Between Avro, Parquet, and RC/ORC which is use...

Re: Spark DF Reading Petabytes of data from Hive a...

Spark DF Reading Petabytes of data from Hive and p...

Kafka Streams vs Structured Streaming which is bet...

Spark Streaming Monitoring options to monitor inpu...

Re: Parallelly load all the gzip files from a s3 d...

Parallelly load all the gzip files from a s3 dir t...

spark scala pivot without aggregation or alternati...

Re: Performing Spark pivot without aggregation?

Performing Spark pivot without aggregation?

spark 2.1.0 Reading *.gz files from an s3 bucket o...

Re: Spark Testing base 1.6.1_0.3.3 for spark versi...

Spark Testing base 1.6.1_0.3.3 for spark version 2...

Re: Spark SQL 2.0 - performance of Plain SQL query...

Spark SQL 2.0 - performance of Plain SQL query in ...

Re: SelectHiveQL processor hive connection pool co...