About chris_herssens

chris_herssens · ‎11-23-2018

I had limited resource on the yarn queue that I used for my spark job.

chris_herssens · ‎11-22-2018

Hello All, I try to run a spark 2 job with oozie on HDP 2.6.5. Oozie starts successful, but my spark 2 job keeps in running state. on Resource manager UI, I got Application Type: SPARK Application Tags: oozie-7bf4f1cfb7c848de2db37230975b4450 Application Priority: 0 (Higher Integer value indicates higher priority) YarnApplicationState: ACCEPTED: waiting for AM container to be allocated, launched and register with RM. Kindly suggest.

chris_herssens · ‎03-29-2018

A there ways to solve this problem. Is it possible to compile the PutHDFS code with a new version of Hadoop code ?

chris_herssens · ‎10-12-2017

I have done this, but this gives performance issues. I found how to do this with the JoltTransform processor.

chris_herssens · ‎10-12-2017

Hello All, I want to flatten json records using nifi. The records are store in kafka as AVRO. I use the ConsumeKafkaRecord processor to convert avro to json. For flattening I would like to use the JoltTransformJSON processor, but this doesn't work since it expect as input a single json and not an array of json records. Any ideas, how I can do this ? Ideally is that I don't have to convert the records to json, but it seems that currently there is no processor to flatten avro. Regards, Chris

chris_herssens · ‎10-12-2017

Yes, you can use one schema for read and a different one for write

chris_herssens · ‎08-03-2017

Hello All, Does Stream Analytics manager support spark streaming ?

chris_herssens · ‎05-10-2017

can you change outputStream.wrtie(text) with outputStream.write(text.encode('latin-1'))

chris_herssens · ‎05-09-2017

Hello, In the "process" method you can read for instance data from the inputstream, change some things and output it see https://community.hortonworks.com/content/kbentry/75032/executescript-cookbook-part-1.html for more information on executescript text = IOUtils.toString(inputStream, IOUtils.toString(inputStream,StandardCharsets.ISO_8859_1) ... outputStream.write(text)

chris_herssens · ‎02-03-2017

Hi, I try to use the executeScript Processor with python for converting strings with special characters like é . For encoding I use latin-1. The script is : text = IOUtils.toString(inputStream, StandardCharsets.ISO_8859_1) .... outputStream.write(bytearray(out.encode('latin-1'))) using this I get the follow error: org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: TypeError: write(): 1st arg can't be coerced to int, byte[] in <script> at line number 33 if I loop over the bytearray : text = IOUtils.toString(inputStream, StandardCharsets.ISO_8859_1) .... out = bytearray(out.encode('latin-1')) for o in out : outputStream.write(o) I don't get this error. Thanks for your help Chris

Online	Offline
Last Visited	‎08-19-2019 04:04 AM

Member Since	‎10-14-2016 06:28 AM
Last Visited	‎08-19-2019 04:04 AM
Posts	32
Kudos received	1

Cloudera Community

Cloudera Community

Re: spark 2 jobs with OOZIE on HDP 2.6.5 hangs

spark 2 jobs with OOZIE on HDP 2.6.5 hangs

Re: Connecting to Azure Data Lake from a NiFi data...

Re: flatten json using nifi

flatten json using nifi

Re: NiFi 1.3+ Multiple Schemas

Spark streaming support for stream analytics manag...

Re: NiFi ExecuteScript Processor: error using stri...

Re: NiFi ExecuteScript Processor: error using stri...

NiFi ExecuteScript Processor: error using string i...