Member since
01-18-2017
58
Posts
5
Kudos Received
0
Solutions
08-14-2018
03:35 PM
Thank you very much @Bryan Bende I need to insert each message into sql server. The database table has a clientno and jdonmessage field. I think I has to use Splitrecod to get the clientno from Json and insert the whole json record into the jsonmessage field.
... View more
08-14-2018
02:29 PM
Thanks for the quick reply, @ Bryan Bende I tried to use the Jason record writer, but I don't have parquet schema information. How could I configure the Jason record writer, so It will get each message as a flow file? Schema Write Strategy
Set 'schema.name' Attribute
Schema Access Strategy
Use 'Schema Name' Property
Schema RegistryIncompatible Controller Service ConfiguredSchema Name
${schema.name}
Schema Text
${avro.schema}
Date FormatNo value setTime FormatNo value setTimestamp FormatNo value setPretty Print JSON
false
Schema Write Strategy
Set 'schema.name' Attribute
Schema Access Strategy
Use 'Schema Name' Property
Schema RegistryIncompatible Controller Service ConfiguredSchema Name
${schema.name}
Schema Text
${avro.schema}
Date FormatNo value setTime FormatNo value setTimestamp FormatNo value setPretty Print JSON
false
... View more
08-14-2018
01:03 PM
I want to read the parquet file and convert the each record into a json flow file. However FetchParquet will get the .parquet file and put its content in a single flowFile, but it doesn't read each record invidually from the parquet file into a flow files record by record. Anyway that I can read the parquet file and convert the each record into a json flow file using NiFi?
... View more
07-17-2018
10:26 PM
Hi, I am going to convert the result of Nifi ExecuteSQL procecessor and save the result as putparquet into hdfs. Any suggestion on how to convert avro format (result of executesql) to parquet using putparquet ? I will use avroreader in putparquet processor, but it requires Schema Registry. could I auto generate the Schema Registry of the result from executesql.? Thanks Andy
... View more
Labels:
- Labels:
-
Apache NiFi
07-17-2018
09:01 PM
Thank you for the quick reply, Matt
... View more
07-17-2018
04:02 PM
Could NIfi Nifi - executeSQL process run sql to join two Databases?
... View more
Labels:
- Labels:
-
Apache NiFi
03-03-2018
08:42 PM
1 Kudo
@Constantin Stanca @Jeff Storck I had created a 2 nodes clusters with 4 zookeepers from scratch on 2 VM (1 nifi and 2 zookeepers on each VM), but I still get the same errors. I also setup DistributedMapCacheClientService & DistributedMapCacheServe. I attached the logs (nifi and zookeeprs). Any suggestion will be appreciated. Thanks a lot. Andy Here is the setttings 1) server 1
zookeeper-3.4.11/zookeeper-1/bin/ (Follower)
zookeeper-3.4.11/zookeeper-2/bin/ (Follower) nifi-1.4.0/bin (primary, Coordinator) 2) server 2
zookeeper-3.4.11/zookeeper-3/bin/ (Leader) zookeeper-3.4.11/zookeeper-4/bin/ (Follower) nifi-1.4.0/bin 3) zookeeper logs
lsof -i:2181 |grep LISTEN
lsof -p 1649 | grep log
lsof -p 12093 | grep out
$ lsof -i:2181 |grep LISTEN
java 12041 pdn0app1 25u IPv4 356369590 0t0 TCP *:eforward (LISTEN)
$ lsof -p 12041 | grep log
java 12041 pdn0app1 mem REG 253,5 481535 54658050 /app/DN00/zookeeper-3.4.11/zookeeper-3/lib/log4j-1.2.16.jar
java 12041 pdn0app1 mem REG 253,5 9753 54658055 /app/DN00/zookeeper-3.4.11/zookeeper-3/lib/slf4j-log4j12-1.6.1.jar
java 12041 pdn0app1 5r REG 253,5 9753 54658055 /app/DN00/zookeeper-3.4.11/zookeeper-3/lib/slf4j-log4j12-1.6.1.jar
java 12041 pdn0app1 8r REG 253,5 481535 54658050 /app/DN00/zookeeper-3.4.11/zookeeper-3/lib/log4j-1.2.16.jar
java 12041 pdn0app1 34w REG 253,5 67108880 54791040 /app/DN00/zookeeper-3.4.11/zookeeper-3/data/version-2/log.200000001
$ lsof -p 12041 | grep out
java 12041 pdn0app1 1w REG 253,5 14225 54659929 /app/DN00/zookeeper-3.4.11/zookeeper-3/bin/zookeeper.out
java 12041 pdn0app1 2w REG 253,5 14225 54659929 /app/DN00/zookeeper-3.4.11/zookeeper-3/bin/zookeeper.out
... View more
03-02-2018
09:41 PM
2 Kudos
I got the same issue, and in my nifi-app.log, I have the following message: ( I have 3 nodes and set the Server Hostname for DistributedMapCacheClientService, I used myserver.dev.test.com. I tried localhost, and got the same error too.) Any suggestion is appreciated. Thanks a lot. 2018-03-02 16:04:52,177 ERROR [StandardProcessScheduler Thread-8] o.a.n.p.standard.MonitorActivity MonitorActivity[id=ab5f82a3-ba56-3764-abf5-5d665d0049fb] Failed to clear cluster state due to java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID ab5f82a3-ba56-3764-abf5-5d665d0049fb: java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID ab5f82a3-ba56-3764-abf5-5d665d0049fb
java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID ab5f82a3-ba56-3764-abf5-5d665d0049fb
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:347)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.clear(ZooKeeperStateProvider.java:453)
at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.clear(StandardStateManagerProvider.java:292)
at org.apache.nifi.controller.state.StandardStateManager.clear(StandardStateManager.java:85)
at org.apache.nifi.processors.standard.MonitorActivity.onStopped(MonitorActivity.java:392)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)
at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotations(ReflectionUtils.java:233)
at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotation(ReflectionUtils.java:85)
at org.apache.nifi.controller.StandardProcessorNode$2.run(StandardProcessorNode.java:1404)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by: java.net.UnknownHostException: myserver.dev.test.com
at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getZooKeeper(ZooKeeperStateProvider.java:170)
at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:316)
... 22 common frames omitted
... View more
03-01-2018
08:17 PM
Hi, Anybody can help? Thanks. I also got the same error for ListHDFS processor and in environment where I have a cluster with 3 nifi node 1) DistributedMapCacheServer is setup and enabled.
2) PutDistributedMapCache controller is added and the invalid message is go through this controller before saving to hdfs.
3) DistributedMapCacheClientService is created and enabledfor PutDistributedMapCache controller 4) DistributedMapCacheClientService is created and enabled for listHdfs controller
However, I am still getting the following error. Did you see this before? Any suggestions?
ListHDFS[id=e286d390-0161-1000-0000-000066a787b2] Failed to retrieve timestamp of last listing from Distributed Cache Service. Will not perform listing until this is accomplished.
... View more
09-18-2017
10:26 PM
I am new to spark and Kafka. I am following the tutorial at http://henning.kropponline.de/2016/12/25/simple-spark-streaming-kafka-example-in-a-zeppelin-notebook/ In the artilces, kafka and spark packages are org.apache.spark:spark-streaming-kafka_2.10:1.6.2 org.apache.kafka:kafka_2.10:0.8.2.2 org.apache.kafka:kafka-clients:0.8.2.2 I installed hdp 2.6. What are the version of those packages should I use? how could I find it out?
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Spark
09-11-2017
03:57 PM
My servers was auto reboot yesterday, and I cannot start Nifi. Any suggestion to solve this issue? Thanks Andy I got the following error: Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: No enum constant org.wali.UpdateType.
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) ~[na:na]
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) ~[na:na]
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) ~[na:na]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254) ~[na:na]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[na:na]
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) ~[na:na]
at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) ~[na:na]
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[na:na]
... 33 common frames omitted
Caused by: java.lang.IllegalArgumentException: No enum constant org.wali.UpdateType.
at java.lang.Enum.valueOf(Enum.java:238) ~[na:1.8.0_101]
at org.wali.UpdateType.valueOf(UpdateType.java:24) ~[na:na]
at org.apache.nifi.controller.state.StateMapSerDe.deserializeRecord(StateMapSerDe.java:76) ~[na:na]
at org.apache.nifi.controller.state.StateMapSerDe.deserializeEdit(StateMapSerDe.java:69) ~[na:na]
at org.apache.nifi.controller.state.StateMapSerDe.deserializeEdit(StateMapSerDe.java:30) ~[na:na]
at org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1072) ~[na:na]
at org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:459) ~[na:na]
at org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301) ~[na:na]
at org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider.init(WriteAheadLocalStateProvider.java:99) ~[na:na]
at org.apache.nifi.controller.state.providers.AbstractStateProvider.initialize(AbstractStateProvider.java:34) ~[na:na]
at org.apache.nifi.controller.state.manager.StandardStateManagerProvider.createStateProvider(StandardStateManagerProvider.java:189) ~[na:na]
at org.apache.nifi.controller.state.manager.StandardStateManagerProvider.createLocalStateProvider(StandardStateManagerProvider.java:81) ~[na:na]
at org.apache.nifi.controller.state.manager.StandardStateManagerProvider.create(StandardStateManagerProvider.java:67) ~[na:na]
at org.apache.nifi.controller.FlowController.<init>(FlowController.java:469) ~[na:na]
at org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:375) ~[na:na]
at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:74) ~[na:na]
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[na:na]
... 40 common frames omitted
2017-09-11 10:21:10,334 INFO [Thread-2] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise).
... View more
Labels:
- Labels:
-
Apache NiFi
05-15-2017
09:53 PM
Nifi Cannot connect to kafka with Kerberos while I try to configure it using the following instruction https://community.hortonworks.com/articles/26551/accessing-kerberos-enabled-kafka-topics-using-getk.html I got the error: unable to find LoginModule class: com.sun.security.auth.module.Krb5LoginModule Any advice is appreciated. 2017-05-15 17:00:00,703 ERROR [Timer-Driven Process Thread-7] o.a.n.p.kafka.pubsub.ConsumeKafka ConsumeKafka[id=11281745-115c-12b5-0451-b59b388b0d9b] ConsumeKafka[id=11281745-115c-12b5-0451-b59b388b0d9b] failed to process due to org.apache.kafka.common.KafkaException: Failed to construct kafka consumer; rolling back session: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer 2017-05-15 17:00:00,707 ERROR [Timer-Driven Process Thread-7] o.a.n.p.kafka.pubsub.ConsumeKafka
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:648) ~[kafka-clients-0.9.0.1.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:512) ~[kafka-clients-0.9.0.1.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:494) ~[kafka-clients-0.9.0.1.jar:na]
at org.apache.nifi.processors.kafka.pubsub.ConsumerPool.createKafkaConsumer(ConsumerPool.java:137) ~[nifi-kafka-0-9-processors-1.1.1.jar:1.1.1]
at org.apache.nifi.processors.kafka.pubsub.ConsumerPool.obtainConsumer(ConsumerPool.java:106) ~[nifi-kafka-0-9-processors-1.1.1.jar:1.1.1]
at org.apache.nifi.processors.kafka.pubsub.ConsumeKafka.onTrigger(ConsumeKafka.java:286) ~[nifi-kafka-0-9-processors-1.1.1.jar:1.1.1]
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.1.1.jar:1.1.1]
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.1.jar:1.1.1]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.1.jar:1.1.1]
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.1.jar:1.1.1]
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.1.jar:1.1.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522) [na:1.8.0-internal]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:319) [na:1.8.0-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:191) [na:1.8.0-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [na:1.8.0-internal]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) [na:1.8.0-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [na:1.8.0-internal]
at java.lang.Thread.run(Thread.java:785) [na:1.8.0-internal] Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: unable to find LoginModule class: com.sun.security.auth.module.Krb5LoginModule at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:74) ~[kafka-clients-0.9.0.1.jar:na]
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:60) ~[kafka-clients-0.9.0.1.jar:na]
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:79) ~[kafka-clients-0.9.0.1.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:577) ~[kafka-clients-0.9.0.1.jar:na]
... 17 common frames omitted Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: com.sun.security.auth.module.Krb5LoginModule at javax.security.auth.login.LoginContext.invoke(LoginContext.java:828) ~[na:1.8.0-internal]
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:196) ~[na:1.8.0-internal]
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:698) ~[na:1.8.0-internal]
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:696) ~[na:1.8.0-internal]
at java.security.AccessController.doPrivileged(AccessController.java:650) ~[na:1.8.0-internal]
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:696) ~[na:1.8.0-internal]
at javax.security.auth.login.LoginContext.login(LoginContext.java:597) ~[na:1.8.0-internal]
at org.apache.kafka.common.security.kerberos.Login.login(Login.java:298) ~[kafka-clients-0.9.0.1.jar:na]
at org.apache.kafka.common.security.kerberos.Login.<init>(Login.java:104) ~[kafka-clients-0.9.0.1.jar:na]
at org.apache.kafka.common.security.kerberos.LoginManager.<init>(LoginManager.java:44) ~[kafka-clients-0.9.0.1.jar:na]
at org.apache.kafka.common.security.kerberos.LoginManager.acquireLoginManager(LoginManager.java:85) ~[kafka-clients-0.9.0.1.jar:na]
ble to find LoginModule class: com.sun.security.auth.module.Krb5LoginModuleat org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:55) ~[kafka-clients-0.9.0.1.jar:na]
... 20 common frames omitted
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
05-15-2017
05:48 PM
@Wynner Thanks for your help, I had updated it to java.arg.2=-Xms512m java.arg.3=-Xmx8g
... View more
05-15-2017
03:21 PM
Hi, My Nifi UI is down On the nifi-bootstrap log, I found the OutOfMemoryError. Any advice on this issue is appraciated. Thanks, Andy here is the nifi-bootstrap log: 2017-05-13 02:55:41,053 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/05/13 02:55:41 - please wait. 2017-05-13 02:55:41,054 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Heap dump using '/app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0001.phd' in response to an event 2017-05-13 02:55:48,525 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Heap dump written to /app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0001.phd 2017-05-13 02:55:48,529 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested System dump using '/app/Project/nifi-1.1.1/core.20170513.025541.31792.0002.dmp' in response to an event 2017-05-13 02:55:56,811 ERROR [NiFi logging handler] org.apache.nifi.StdErr Warning: unable to move dump to "/app/Project/nifi-1.1.1/core.20170513.025541.31792.0002.dmp" across file systems (check kernel core_pattern). Using alternate file location "/var/core/core.20170513.025541.31792.0002.dmp" 2017-05-13 02:55:56,818 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I System dump written to /var/core/core.20170513.025541.31792.0002.dmp 2017-05-13 02:55:56,818 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Java dump using
'/app/Project/nifi-1.1.1/javacore.20170513.025541.31792.0003.txt' in response to an event 2017-05-13 02:55:56,971 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Java dump written to /app/Project/nifi-1.1.1/javacore.20170513.025541.31792.0003.txt 2017-05-13 02:55:56,972 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Snap dump using '/app/Project/nifi-1.1.1/Snap.20170513.025541.31792.0005.trc' in response to an event 2017-05-13 02:55:56,972 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Snap dump written to /app/Project/nifi-1.1.1/Snap.20170513.025541.31792.0005.trc 2017-05-13 02:55:56,973 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError". 2017-05-13 02:55:56,974 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Heap dump using '/app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0004.phd' in response to an event
... View more
Labels:
- Labels:
-
Apache NiFi
05-15-2017
02:49 PM
My Nifi UI is down daily in the past few days. I wounder if it is relative to OutOfMemoryError. Any suggestion on this issue? 1) On the vi nifi-bootstrap_2017-05-13.log file 2017-05-13 02:55:41,053 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/05/13 02:55:41 - please wait. 2017-05-13 02:55:41,05ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Heap dump using '/app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0001.phd' in response to an event 2017-05-13 02:55:48,525 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Heap dump written to /app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0001.phd 2017-05-13 02:55:48,529 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested System dump using '/app/Project/nifi-1.1.1/core.20170513.025541.31792.0002.dmp' in response to an event
2017-05-13 02:55:56,811 ERROR [NiFi logging handler] org.apache.nifi.StdErr Warning: unable to move dump to "/app/Project/nifi-1.1.1/core.20170513.025541.31792.0002.dmp" across file systems (check kernel core_pattern). Using alternate file location "/var/core/core.20170513.025541.31792.0002.dmp" 2017-05-13 02:55:56,818 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I System dump written to /var/core/core.20170513.025541.31792.0002.dmp 2017-05-13 02:55:56,818 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Java dump using '/app/Project/nifi-1.1.1/javacore.20170513.025541.31792.0003.txt' in response to an event 2017-05-13 02:55:56,971 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Java dump written to /app/Project/nifi-1.1.1/javacore.20170513.025541.31792.0003.txt 2017-05-13 02:55:56,972 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Snap dump using '/app/Project/nifi-1.1.1/Snap.20170513.025541.31792.0005.trc' in response to an event 2017-05-13 02:55:56,972 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Snap dump written to /app/Project/nifi-1.1.1/Snap.20170513.025541.31792.0005.trc 2017-05-13 02:55:56,973 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError". 2017-05-13 02:55:56,974 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Heap dump using '/app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0004.phd' in response to an event 2017-05-13 02:56:05,809 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Heap dump written to /app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0004.phd
2017-05-13 02:56:05,810 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Java dump using '/app/Project/nifi-1.1.1/javacore.20170513.025541.31792.0006.txt' in response to an event
2017-05-13 02:56:05,970 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Java dump written to /app/Project/nifi-1.1.1/javacore.20170513.025541.31792.0006.txt
2017-05-13 02:56:05,970 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/05/13 02:56:05 - please wait. 2017-05-13 02:56:05,970 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Snap dump using '/app/Project/nifi-1.1.1/Snap.20170513.025541.31792.0007.trc' in response to an event 2017-05-13 02:56:05,970 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Snap dump written to /app/Project/nifi-1.1.1/Snap.20170513.025541.31792.0007.trc 2017-05-13 02:56:05,970 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError".
2017-05-13 02:56:05,971 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Heap dump using '/app/Project/nifi-1.1.1/heapdump.20170513.025605.31792.0008.phd' in response to an event
2017-05-13 02:56:14,533 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Heap dump written to /app/Project/nifi-1.1.1/heapdump.20170513.025605.31792.0008.phd 2017-05-13 02:56:22,867 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Java dump using '/app/Project/nifi-1.1.1/javacore.20170513.025605.31792.0009.txt' in response to an event 2017-05-13 02:56:23,055 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Java dump written to /app/Project/nifi-1.1.1/javacore.20170513.025605.31792.0009.txt 2017-05-13 02:56:23,064 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Snap dump using '/app/Project/nifi-1.1.1/Snap. 20170513.025605.31792.0010.trc' in response to an event
2017-05-13 02:56:23,064 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Snap dump written to /app/Project/nifi-1.1.1/Snap. 20170513.025605.31792.0010.trc
2017-05-13 02:56:23,065 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError". 2017-05-13 02:56:23,073 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/05/13 02:56:23 - please wait.
... View more
- Tags:
- Hadoop Core
- NiFi
Labels:
- Labels:
-
Apache NiFi
05-15-2017
02:20 PM
@Matt Clarke Hi Matt, Thank you very much for helping. 1) here is the NiFi Bulletin Board error 10:19:17 EDT
ERROR
12d4119d-1128-1761-becf-ab1016926114
GetHDFS[id=12d4119d-1128-1761-becf-ab1016926114] GetHDFS[id=12d4119d-1128-1761-becf-ab1016926114] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.; processor will not be scheduled to run for 30 seconds: java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. 2) Here is the error log 2017-05-15 10:20:17,535 ERROR [StandardProcessScheduler Thread-8] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1449) ~[na:na]
at org.apache.nifi.controller.StandardProcessorNode.access$100(StandardProcessorNode.java:100) ~[na:na]
at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1275) ~[na:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522) [na:1.8.0-internal]
at java.util.concurrent.FutureTask.run(FutureTask.java:277) [na:1.8.0-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:191) [na:1.8.0-internal]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [na:1.8.0-internal]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) [na:1.8.0-internal]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [na:1.8.0-internal]
at java.lang.Thread.run(Thread.java:785) [na:1.8.0-internal]
Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
at java.util.concurrent.FutureTask.report(FutureTask.java:133) [na:1.8.0-internal]
at java.util.concurrent.FutureTask.get(FutureTask.java:217) [na:1.8.0-internal]
at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1432) ~[na:na]
... 9 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.GeneratedMethodAccessor406.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[na:1.8.0-internal]
at java.lang.reflect.Method.invoke(Method.java:507) ~[na:2.6 (01-06-2016)]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) ~[na:na]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) ~[na:na]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) ~[na:na]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) ~[na:na]
at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1279) ~[na:na]
at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1275) ~[na:na]
... 6 common frames omitted
... View more
05-11-2017
10:50 AM
Hi, When I use GetHDFS, I got the following error and I cannot find a solution for it. Any suggestion for this issue will be appreciated. Thanks Andy GetHDFS[id=115c12d7-1761-1128-7f1a-c31b56a24aaf] GetHDFS[id=115c12d7-1761-1128-7f1a-c31b56a24aaf] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.; processor will not be scheduled to run for 30 seconds: java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
... View more
Labels:
- Labels:
-
Apache NiFi
03-08-2017
02:35 PM
Hi @Andrew Grande and @Neeraj Sabharwal 1) I was able to use the TransFormXml processor to convert the xml in the following format <?xml version="1.0" encoding="UTF-8"?> -<ROWSET> <ROW kind="element" name="Product" pid="" id="d1e1"> </ROW> <ROW kind="attribute" name="Type" pid="d1e1" id="d1e1a1373">Laptop</ROW> <ROW kind="element" name="Notebook" pid="d1e1" id="d1e3"> </ROW> <ROW kind="attribute" name="Brand" pid="d1e3" id="d1e3a1403">HP</ROW> <ROW kind="attribute" name="Model" pid="d1e3" id="d1e3a1938">Pavilion dv6-3132TX Notebook</ROW> <ROW kind="element" name="Harddisk" pid="d1e3" id="d1e5">640 GB</ROW> <ROW kind="element" name="Processor" pid="d1e3" id="d1e8">Intel Core i7</ROW> <ROW kind="element" name="RAM" pid="d1e3" id="d1e11">4 GB</ROW> <ROW kind="element" name="Notebook" pid="d1e1" id="d1e15"> </ROW> <ROW kind="attribute" name="Brand" pid="d1e15" id="d1e15a1403">HP</ROW> <ROW kind="attribute" name="Model" pid="d1e15" id="d1e15a1938">HP Pavilion dv6-3032TX Notebook</ROW> <ROW kind="element" name="Harddisk" pid="d1e15" id="d1e17">640 GB</ROW> <ROW kind="element" name="Processor" pid="d1e15" id="d1e20">Intel Core i7</ROW> <ROW kind="element" name="RAM" pid="d1e15" id="d1e23">6 GB</ROW> <ROW kind="element" name="Notebook" pid="d1e1" id="d1e27"> </ROW> <ROW kind="attribute" name="Brand" pid="d1e27" id="d1e27a1403">Toshiba</ROW> <ROW kind="attribute" name="Model" pid="d1e27" id="d1e27a1938">Satellite A660/07R 3D Notebook</ROW> <ROW kind="element" name="Harddisk" pid="d1e27" id="d1e29">640 GB</ROW> <ROW kind="element" name="Processor" pid="d1e27" id="d1e32">Intel Core i7</ROW> <ROW kind="element" name="RAM" pid="d1e27" id="d1e35">4 GB</ROW> <ROW kind="element" name="Notebook" pid="d1e1" id="d1e39"> </ROW> <ROW kind="attribute" name="Brand" pid="d1e39" id="d1e39a1403">Toshiba</ROW> <ROW kind="attribute" name="Model" pid="d1e39" id="d1e39a1938">Satellite A660/15J Notebook</ROW> <ROW kind="element" name="Harddisk" pid="d1e39" id="d1e41">640 GB</ROW> <ROW kind="element" name="Processor" pid="d1e39" id="d1e44">Intel Core i5</ROW> <ROW kind="element" name="RAM" pid="d1e39" id="d1e47">6 GB</ROW> </ROWSET> 2) my question is how to convert the following xquery in the sample for Nifi's EvaluateXQuery : (In the example, oracle is using.) SELECT x.* FROM xml_test t
, XMLTable('/ROWSET/ROW' passing xmltransform(t.object_value, xmltype(:xsldoc)) columns node_id varchar2(100) path '@id'
, node_name varchar2(30) path '@name'
, node_value varchar2(2000) path 'text()'
, parent_node_id varchar2(100) path '@pid'
, node_kind varchar2(30) path '@kind' ) x
;
... View more
03-06-2017
06:51 PM
@Bryan Bende Do I have to use StandardSSLContextService to connect to IBM mq if my nifi.properties is configured with SSL Authentication for web browser? If I have to, could I use the same certificate that I created for web browser for IBM MQ (I use nifi.pfx for the web browser)?
... View more
03-06-2017
05:02 PM
1) I had Configured Apache NiFi SSL Authentication using the following instruction https://www.batchiq.com/nifi-configuring-ssl-auth.html I attached the Certificate to a web Browser and the browser is able to connect to Nifi. 2) I have Nifi connected to IBM MQ, and I know I need to configured StandardSSLContextService of JMSConnectionFactoryProvider. 3) My question is that do I need to attach the certificate to MQ? Any suggestion on how to do it? Thanks a lot Andy
... View more
Labels:
- Labels:
-
Apache NiFi
02-24-2017
10:32 PM
update: I finally figured out how to setup IBM MQ for Nifi Here is my steps to setup IBM MQ for the Nifi connection 1) Create queue manager myMQManager using MQ Explorer 2) create Channel and queue using the following commands runmqsc myMQManager def chl(myMQChannel) chltype(svrconn) replace def chl(myMQChannel) chltype(clntconn) conname('localhost(1420)') QMNAME('myMQManager') replace def ql(myMQqueue) 2) Grant necessary authorization for a user using MQ Explorer 2.1 Queue Manager Authorities Setting right click your queue manager-->Object Authorities-->Add Role Based Authorities -->give the user access permission 2.2 Queue Authorities Setting right click your queue-->Object Authorities-->Manage Authority Records-->Specific Profiles--> your queue--> Accumulated authorities-->setup your user Access Permission for the queue 3) After the configuration and setup your Nifi ConsumeJMS processor, you should be able connect to IBM MQ
... View more
02-24-2017
10:29 PM
@ozhurakousky Here is the steps to setup IBM MQ for the Nifi connection 1) Create queue manager myMQManager using MQ Explorer 2) create Channel and queue using the following commands
runmqsc myMQManager def chl(myMQChannel) chltype(svrconn) replace def chl(myMQChannel) chltype(clntconn) conname('localhost(1420)') QMNAME('myMQManager') replace def ql(myMQqueue) 2) Grant necessary authorization for a user using MQ Explorer 2.1 Queue Manager Authorities Setting right click your queue manager-->Object Authorities-->Add Role Based Authorities -->give the user access permission 2.2 Queue Authorities Setting right click your queue-->Object Authorities-->Manage Authority Records-->Specific Profiles--> your queue--> Accumulated authorities-->setup your user Access Permission for the queue 3) After the configuration and setup your Nifi ConsumeJMS processor, you should be able connect to IBM MQ
... View more
02-23-2017
06:49 PM
@ozhurakousky of course, I will post what I did on the IBM MQ here later today.
... View more
02-23-2017
06:40 PM
@ozhurakousky Hi Oleg, I finally got the connection between Nifi and IBM MQ working now. It's some configuration issue on MQ. Thank you very much for your help and time. I am very appreciated. Andy
... View more
02-22-2017
09:14 PM
@ozhurakousky Hi Oleg, I had re-installed IBM MQ and setting up the queue manager, Channel, listener, and queue. I will do more testing on Nifi and post the result. Thanks a lot Andy
... View more
02-21-2017
05:11 PM
@ozhurakousky Thanks a lot for taking time to help me. Yeah, I had tired your suggestion. and I just wrote a very detail message on what I did on the previous post.
... View more
02-21-2017
05:00 PM
@ozhurakouskyThanks for the advice and your time. I had setup the processor as the following: Controller Service and Processor
For the IBM MQ side 1) I found the channel SYSTEM.ADMIN.SVRCONN , do I need to create a sender channel or it's already built in? 2) I already have a listener for Nifi (localhost:1419) 3) I create a local queue Q1 (do I have to set it's Usage to transmission?) 4) I got the following error: ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] failed to process due to org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'NifiMQ' with connection mode 'Client' and host name 'localhost(1419)'.
Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2540' ('MQRC_UNKNOWN_CHANNEL_NAME').; rolling back session: org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'NifiMQ' with connection mode 'Client' and host name 'localhost(1419)'.
Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2540' ('MQRC_UNKNOWN_CHANNEL_NAME'). Is there Any other configuration that I missed? Thanks a lot. @ozhurakousky I am really appreciated for your help and your time.
... View more
02-21-2017
03:58 PM
@Matt Clarke I had read the posts at https://community.hortonworks.com/questions/35871/consumejms-failed-to-load-the-websphere-mq-native.html and https://community.hortonworks.com/questions/35871/consumejms-failed-to-load-the-websphere-mq-native.html I followed the instructions to configure the onsumeJMS and PublishJMS processors, but I cannot get it working.
I wonder it's the configuration issue on the IBM MQ. For example, how could I set up the SYSTEM.ADMIN.SVRCONN for channel or is it build-in already? Do I need to create a transmission queue for the channel?
... View more
02-21-2017
03:35 PM
I need help to Setup MQ for Nifi. I have a very tight deadline for a project to retrieve data from IBM MQ to Nifi. I installed IBM MQ on my local computer. I was able to configure the queues to read one from another. but I have issue on setup configurations for Nifi. Any advice on the MQ configuration for Nifi will be very appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi