Member since
08-21-2017
13
Posts
0
Kudos Received
0
Solutions
09-19-2017
01:28 AM
Hi @Matt Clarke, Your explanation was useful for me to build my Nifi flow. But I am experiencing a data loss of 7 records.I have posted about the same in forum. Below is the link: https://community.hortonworks.com/questions/138873/data-loss-found-with-tcp-and-mergecontent-processo.html Can you help me in figuring out the mistake I am doing in configuration of the processors? Currently, I am using PutFile instead of PutHDFS for the sake of easy checks with lines count of merged content. Sravanthi
... View more
09-14-2017
12:21 PM
09-10-2017
07:48 PM
Hi, I am getting below exception, while I try to restart kafka service (which is running over HDF environment- Kafka 0.10.2). I was getting this exception, as I had the disk space consuming 100% of the log directory and Kafka service went down because of that. As, I did cleared the space for the logs and restarted the kafka service, I was getting the below exception: 2017-09-09 16:56:36,751] FATAL (kafka.Kafka$)
java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.sink.kafka.KafkaTimelineMetricsReporter
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at kafka.utils.CoreUtils$.createObject(CoreUtils.scala:174)
at kafka.metrics.KafkaMetricsReporter$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:65)
at kafka.metrics.KafkaMetricsReporter$anonfun$startReporters$1.apply(KafkaMetricsReporter.scala:64)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
at kafka.metrics.KafkaMetricsReporter$.startReporters(KafkaMetricsReporter.scala:64)
at kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:27)
at kafka.Kafka$.main(Kafka.scala: 6] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.NumberFormatException: For input string: "logs.bk"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229) Any ideas here, like how can I overcome this issue? Thanks in advance, Sravanthi
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Kafka
09-08-2017
02:31 PM
09-05-2017
06:37 PM
Hi Everyone, I have a issue with Kafka here. On a cluster I have installed Zookeeper-client and Zookeeper server and are up and running. Along the same I have Kafka-server up and running and on Ambari it shows the health as good and fine. But, while I do execute the producer command for kafka as below, I am getting the below error: Kafka Producer Command: bin/kafka-console-producer.sh --broker-list kafka.broker:6667 --topic testTopic Error Message: [2017-09-05 12:22:02,638] ERROR Error when sending message to topic testTopic with key: null, value: 8 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Expiring 2 record(s) for testTopic-0: 1535 ms has passed since batch creation plus linger time e 345[2017-09-05 12:22:05,439] ERROR Error when sending message to topic testTopic with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for testTopic-0: 1536 ms has passed since batch creation plus linger time Not getting the actual issue here. Any help will be appritiated. Thanks, Sravanthi
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Kafka
09-02-2017
04:50 AM
Hi, I am running a spark job which I have ran the command in yarn cluster mode. But, while running the job I got the below logs and it was in hung state printing the same set of messages: 17/09/01 09:58:17 INFO Client: Application report for application_1503983924258_0018 (state: ACCEPTED) 17/09/01 09:58:18 INFO Client: Application report for application_1503983924258_0018 (state: ACCEPTED) running-spark-job-issue.png Below is the spark command that I am running: spark-submit --class com.spark.sparkStreaming --master yarn --deploy-mode cluster SparkStreaming-0.0.1-jar-with-dependencies.jar --executor-memory 16g --driver-memory 4g --executor-cores 4 --num-executors 8 Below is the spark server configuration: Memory ranges from 64GB - 256GB and CPU is 16 cores. Thanks, Sravanthi
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
08-30-2017
09:51 AM
Hi, We have installed Hive in Hortonworks environment (within cluster). The Hive service is shown as started up and running in Ambari UI, but could not able to access Hive UI. Below is the error it shows for Hive view: hive-view-error-in-ambari-ui.png Tried opening Hive from command line even, but the same error: hive-command-line-error-with-tez-engine.png We tried changing the Tez engine with mapreduce engine and restarting ambari-server, but seems the issue still persist. Also, we do get HDFS Block health alert in Ambari UI. Below is the error: ambari-error-for-hdfs-health.png Not sure, of how to resolve this issue. Hence, requesting for any open suggestion. Thanks, Sravanthi
... View more
Labels:
- Labels:
-
Apache Hive
08-29-2017
05:23 AM
Thanks a lot Bryan. I did added the new-line using shift+enter on the value field and I am succeeded to get the desired result of batch of records. Earlier I did tried using '\n' and '\\n' for this attribute. Out of curiosity, '\\n' should be equivalent to 'shift+enter'. Please correct me here. Thanks, Sravanthi
... View more
08-27-2017
04:22 AM
Hi, We are using below NiFi processors to generate a sample pipline data flow using kafka: GetFile --> RouteOnAttribute --> publishKafka_0_10 Workflow steps: ----------------------- - We used a csv file (50MB) to be read using 'GetFile' processor. Here, we have provided the folder location of the file. Consider each record is of 1KB size and as such there are 50K records on the file - Which is working. - Then we have connected to 'RouteOnAttribute' processor. To pick only this file from the GetFile processor - Which is working. - Then we did connected to 'publishKafka_0_10' processor. Here, we did provided a topic and started the configured server with properties - Which is working too. Following is the issue we are facing when we are trying publish data into Kafka topic using PublishKafka_0_10 1.2.0 processor of Nifi 1.2.0 in HDF 3.0.1.0-43 : - I did use my Spark-Kafka consumer (which is my custom spark job, I am running in cluster). Here, I did provided the maximum batch fetch size: 30MB and buffer size as 15MB. - But, while running the spark job, I am getting only 10 records as my batch size (as my spark consumer is consuming 10 records as a batch). NOTE: I tried using custom kafka producer (sample kafka producer code) which can produce set of messages (1KB/record) by iterating from 1 - 50K count. Here, using the same spark-consumer, I did received maximum set of records as a batch (15K records/batch). Seems the issue with publishKafka processor, which is able to send only few records to the topic. Is there any way I can tune the parameters to achieve maximum throughput writes to a topic using this processor ?? Please find attached were configuration I did used for publishKafka and spark-kafka consumer (custom spark code): Thanks in advance, Sravanthi
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
08-21-2017
10:05 AM
Hi, Following is the issue we are facing when we are trying publish data into Kafka topic using PublishKafka0.10.1.2 processor of Nifi 1.2.0 in HDF 3.0.1.0-43 : Error: Failed to process session due to org.apache.nifi.processor.exception.FlowFileHandlingException. Transfer relationship not specified. Attached is the Image for a more clarified detail about the error. I have got this error in the following scenarios:
1)When I check in the checkbox for both success and failure in Automatically Terminate Relationships.
2)When I check in the checkbox for success and route the data to a local file for failure in Automatically Terminate Relationships. processorconfiguration.png Are we missing out any relationships apart from failure and success?How to fix this?Any Suggestions for would be highly appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi