About mak88

mak88 · ‎10-01-2019

Tried to delete this as noone seems to answer here. But if someone has a similiar issue, just UpdateAttribute on attributes you want to create a new flowfile and then pass them to AttributesToJSON processor.

mak88 · ‎09-25-2019

Can anyone at Cloudera/Horton aswer this??? Having same issues.

mak88 · ‎11-01-2016

At the moment I made it past this...unfortunately I added the extra options to the end of the command line and if you notice, the options "--master...memory 5g" are actually being fed into my jar. So I just moved the "Main.jar..." to the end and it works now. Corrrected cmd line: spark-submit --packages com.databricks:spark-avro_2.10:2.0.1 --class Main --master yarn --executor-memory 5g --driver-memory 5g Main.jar --avro-file file_1.avro

mak88 · ‎10-19-2016

We passed this problem by eventually just matching up one of the versions of the lucene library.

mak88 · ‎09-20-2016

Almost forgot about this... I access my avro files like so: First as Tim said, include proper avro lib, in my case DataBricks. spark-submit --packages com.databricks:spark-avro_2.10:2.0.1 --class MyMain MyMain.jar val df = sqlContext.read.format("com.databricks.spark.avro"). option("header", "true").load("/user/user1/writer_test.avro") df.select("time").show() ... Thanks all

Online	Offline
Last Visited	‎12-16-2019 12:38 PM

Member Since	‎09-05-2016 02:43 AM
Last Visited	‎12-16-2019 12:38 PM
Posts	24
Kudos received	2

Cloudera Community

Re: Attributes to flow file

Re: Spark Heap error with large avro

Re: Flume with Elasticsearch noSuchFieldError

Re: Attributes to flow file

Re: [NIFI] Cannot create PoolableConnectionFactory

Re: Spark Heap error with large avro

Re: Flume with Elasticsearch noSuchFieldError

Re: How to access file in HDFS from Spark-shell or...