About alvinuw

alvinuw · ‎07-31-2017

Hello, Let's say I have HOSTNAME as an env variable. May I set below property in nifi.properties file? nifi.web.http.host=$HOSTNAME Thanks.

alvinuw · ‎07-25-2017

Hi Guys, I noticed there is official docker image for NiFi. Is there Kubernetes yaml for building a NiFi cluster, or have this in plan? Thanks.

alvinuw · ‎07-10-2017

Hello, I work with NiFi v1.3 and publish avro messages by using PublishKafkaRecord_0_10 to Kafka. However, when I use kafka-avro-console-consumer to consume avro from Kafka topic, I saw below errors: ERROR Unknown error when running consumer: (kafka.tools.ConsoleConsumer$:105) org.apache.kafka.common.errors.SerializationException: Error deserializing Avro message for id -1 Caused by: org.apache.kafka.common.errors.SerializationException: Unknown magic byte! When I use kafka-console-consumer to consume, everything goes well. Any ideas to make the avro messages published from NiFi deserializable for Confluent Avro deserializer? Thanks.

alvinuw · ‎06-30-2017

In my case, the csv columns are not all strings, there are long types. Yes, I can provide schema text without using Schema Registry. For your first solution, I think the index starts from 1. ${fragment.index:gt(1)} Thanks.

alvinuw · ‎06-30-2017

Hi @Matt Burgess Thank you for your response. The first solution works for me. For the second solution, may I ask which processors should I use, since CSVReader is a service, which also requires schema and schema registry. Thanks.

alvinuw · ‎06-28-2017

Hello, I have a csv file with the first line as header. When I use SplitText processor, the split tiny files contain that header as in first line. Is there an easy way to generate the split file without header? Thanks.

alvinuw · ‎06-27-2017

Hi @Greg Keys, Thank you for this document. May I ask if I would like to search for datetime type(YYY-MM-DD HH:MM:SS) in a row and convert it to unix timestamp type. Any idea on it? Thanks.

alvinuw · ‎06-27-2017

What you described is exactly what I need. PublishKafka processor doesn't support Schema Registry. That's why I don't use it. Otherwise, it looks good to me. Thanks.

alvinuw · ‎06-27-2017

Hi @Bryan Bende, Yes, you are right. I agree that in most cases, the message key should be chosen from the record columns. In my case, I want all messages(rows) from the same sources(could be file, sftp, etc) go to the same partition. I won't enable the compaction log in the Kafka broker. I am a little bit confused about the term "message key field" used in the NiFi Kafka processor. Is it the same as "record key" concept in Kafka used for partition? Thank you for creating this ticket NIFI-4133

alvinuw · ‎06-27-2017

@Sonu Sahi Thank you for your response. ${now():toNumber():format('yyyy-MM-dd')} It seems above expression works for attributes. How to use it to convert a column in csv file? Thanks.

Online	Offline
Last Visited	‎01-18-2018 07:36 PM

Member Since	‎07-19-2016 02:09 AM
Last Visited	‎01-18-2018 07:36 PM
Posts	91
Kudos received	10

Cloudera Community

Re: Two NiFi Instances/Clusters Communication Time...

Does nifi.properties file accept Env variables?

Is There Kubernetes Config to build a NiFi cluster

Can't consume avro messages, whcih are published b...

Re: How to remove the header when using NiFi Split...

Re: How to remove the header when using NiFi Split...

How to remove the header when using NiFi SplitText...

Re: NiFi ETL: Removing columns, filtering rows, ch...

Re: How does PublishKafkaRecord_0_10 use filename ...

Re: How does PublishKafkaRecord_0_10 use filename ...

Re: NiFi processor: Convert string(datetime format...