toandyliang
Explorer

Re: How to export parquet file to csv (without Hiv...

by Explorer in Support Questions
‎08-14-2018 03:35 PM
‎08-14-2018 03:35 PM
Thank you very much @Bryan Bende I need to insert each message into sql server. The database table has a clientno and jdonmessage field. I think I has to use Splitrecod to get the clientno from Json and insert the whole json record into the jsonmessage field. ... View more

Re: How to export parquet file to csv (without Hiv...

by Explorer in Support Questions
‎08-14-2018 02:29 PM
‎08-14-2018 02:29 PM
Thanks for the quick reply, @ Bryan Bende I tried to use the Jason record writer, but I don't have parquet schema information. How could I configure the Jason record writer, so It will get each message as a flow file? Schema Write Strategy Set 'schema.name' Attribute Schema Access Strategy Use 'Schema Name' Property Schema RegistryIncompatible Controller Service ConfiguredSchema Name ${schema.name} Schema Text ${avro.schema} Date FormatNo value setTime FormatNo value setTimestamp FormatNo value setPretty Print JSON false Schema Write Strategy Set 'schema.name' Attribute Schema Access Strategy Use 'Schema Name' Property Schema RegistryIncompatible Controller Service ConfiguredSchema Name ${schema.name} Schema Text ${avro.schema} Date FormatNo value setTime FormatNo value setTimestamp FormatNo value setPretty Print JSON false ... View more

Re: How to export parquet file to csv (without Hiv...

by Explorer in Support Questions
‎08-14-2018 01:03 PM
‎08-14-2018 01:03 PM
I want to read the parquet file and convert the each record into a json flow file. However FetchParquet will get the .parquet file and put its content in a single flowFile, but it doesn't read each record invidually from the parquet file into a flow files record by record. Anyway that I can read the parquet file and convert the each record into a json flow file using NiFi? ... View more

how could I convert avro format (result of execute...

by Explorer in Support Questions
‎07-17-2018 10:26 PM
‎07-17-2018 10:26 PM
Hi, I am going to convert the result of Nifi ExecuteSQL procecessor and save the result as putparquet into hdfs. Any suggestion on how to convert avro format (result of executesql) to parquet using putparquet ? I will use avroreader in putparquet processor, but it requires Schema Registry. could I auto generate the Schema Registry of the result from executesql.? Thanks Andy ... View more
Labels:

Re: Could NIfi - executeSQL process run sql to jo...

by Explorer in Support Questions
‎07-17-2018 09:01 PM
‎07-17-2018 09:01 PM
Thank you for the quick reply, Matt ... View more

Could NIfi - executeSQL process run sql to join t...

by Explorer in Support Questions
‎07-17-2018 04:02 PM
‎07-17-2018 04:02 PM
Could NIfi Nifi - executeSQL process run sql to join two Databases? ... View more
Labels:

Re: 'ListHDFS' processor is not working.

by Explorer in Support Questions
‎03-03-2018 08:42 PM
1 Kudo
‎03-03-2018 08:42 PM
1 Kudo
@Constantin Stanca @Jeff Storck I had created a 2 nodes clusters with 4 zookeepers from scratch on 2 VM (1 nifi and 2 zookeepers on each VM), but I still get the same errors. I also setup DistributedMapCacheClientService & DistributedMapCacheServe. I attached the logs (nifi and zookeeprs). Any suggestion will be appreciated. Thanks a lot. Andy Here is the setttings 1) server 1 zookeeper-3.4.11/zookeeper-1/bin/ (Follower) zookeeper-3.4.11/zookeeper-2/bin/ (Follower) nifi-1.4.0/bin (primary, Coordinator) 2) server 2 zookeeper-3.4.11/zookeeper-3/bin/ (Leader) zookeeper-3.4.11/zookeeper-4/bin/ (Follower) nifi-1.4.0/bin 3) zookeeper logs lsof -i:2181 |grep LISTEN lsof -p 1649 | grep log lsof -p 12093 | grep out $ lsof -i:2181 |grep LISTEN java 12041 pdn0app1 25u IPv4 356369590 0t0 TCP *:eforward (LISTEN) $ lsof -p 12041 | grep log java 12041 pdn0app1 mem REG 253,5 481535 54658050 /app/DN00/zookeeper-3.4.11/zookeeper-3/lib/log4j-1.2.16.jar java 12041 pdn0app1 mem REG 253,5 9753 54658055 /app/DN00/zookeeper-3.4.11/zookeeper-3/lib/slf4j-log4j12-1.6.1.jar java 12041 pdn0app1 5r REG 253,5 9753 54658055 /app/DN00/zookeeper-3.4.11/zookeeper-3/lib/slf4j-log4j12-1.6.1.jar java 12041 pdn0app1 8r REG 253,5 481535 54658050 /app/DN00/zookeeper-3.4.11/zookeeper-3/lib/log4j-1.2.16.jar java 12041 pdn0app1 34w REG 253,5 67108880 54791040 /app/DN00/zookeeper-3.4.11/zookeeper-3/data/version-2/log.200000001 $ lsof -p 12041 | grep out java 12041 pdn0app1 1w REG 253,5 14225 54659929 /app/DN00/zookeeper-3.4.11/zookeeper-3/bin/zookeeper.out java 12041 pdn0app1 2w REG 253,5 14225 54659929 /app/DN00/zookeeper-3.4.11/zookeeper-3/bin/zookeeper.out ... View more

Re: 'ListHDFS' processor is not working.

by Explorer in Support Questions
‎03-02-2018 09:41 PM
2 Kudos
‎03-02-2018 09:41 PM
2 Kudos
I got the same issue, and in my nifi-app.log, I have the following message: ( I have 3 nodes and set the Server Hostname for DistributedMapCacheClientService, I used myserver.dev.test.com. I tried localhost, and got the same error too.) Any suggestion is appreciated. Thanks a lot. 2018-03-02 16:04:52,177 ERROR [StandardProcessScheduler Thread-8] o.a.n.p.standard.MonitorActivity MonitorActivity[id=ab5f82a3-ba56-3764-abf5-5d665d0049fb] Failed to clear cluster state due to java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID ab5f82a3-ba56-3764-abf5-5d665d0049fb: java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID ab5f82a3-ba56-3764-abf5-5d665d0049fb java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID ab5f82a3-ba56-3764-abf5-5d665d0049fb at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:347) at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283) at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228) at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.clear(ZooKeeperStateProvider.java:453) at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.clear(StandardStateManagerProvider.java:292) at org.apache.nifi.controller.state.StandardStateManager.clear(StandardStateManager.java:85) at org.apache.nifi.processors.standard.MonitorActivity.onStopped(MonitorActivity.java:392) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotations(ReflectionUtils.java:233) at org.apache.nifi.util.ReflectionUtils.quietlyInvokeMethodsWithAnnotation(ReflectionUtils.java:85) at org.apache.nifi.controller.StandardProcessorNode$2.run(StandardProcessorNode.java:1404) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.UnknownHostException: myserver.dev.test.com at java.net.InetAddress.getAllByName0(InetAddress.java:1280) at java.net.InetAddress.getAllByName(InetAddress.java:1192) at java.net.InetAddress.getAllByName(InetAddress.java:1126) at org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61) at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445) at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380) at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.getZooKeeper(ZooKeeperStateProvider.java:170) at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:316) ... 22 common frames omitted ... View more

Re: Failed to retrieve timestamp of last listing f...

by Explorer in Support Questions
‎03-01-2018 08:17 PM
‎03-01-2018 08:17 PM
Hi, Anybody can help? Thanks. I also got the same error for ListHDFS processor and in environment where I have a cluster with 3 nifi node 1) DistributedMapCacheServer is setup and enabled. 2) PutDistributedMapCache controller is added and the invalid message is go through this controller before saving to hdfs. 3) DistributedMapCacheClientService is created and enabledfor PutDistributedMapCache controller 4) DistributedMapCacheClientService is created and enabled for listHdfs controller However, I am still getting the following error. Did you see this before? Any suggestions? ListHDFS[id=e286d390-0161-1000-0000-000066a787b2] Failed to retrieve timestamp of last listing from Distributed Cache Service. Will not perform listing until this is accomplished. ... View more

how to find kafka and spark packages versions?

by Explorer in Support Questions
‎09-18-2017 10:26 PM
‎09-18-2017 10:26 PM
I am new to spark and Kafka. I am following the tutorial at http://henning.kropponline.de/2016/12/25/simple-spark-streaming-kafka-example-in-a-zeppelin-notebook/ In the artilces, kafka and spark packages are org.apache.spark:spark-streaming-kafka_2.10:1.6.2 org.apache.kafka:kafka_2.10:0.8.2.2 org.apache.kafka:kafka-clients:0.8.2.2 I installed hdp 2.6. What are the version of those packages should I use? how could I find it out? ... View more
Labels:

Unable to start NIFI ( Error creating bean with na...

by Explorer in Support Questions
‎09-11-2017 03:57 PM
‎09-11-2017 03:57 PM
My servers was auto reboot yesterday, and I cannot start Nifi. Any suggestion to solve this issue? Thanks Andy I got the following error: Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalArgumentException: No enum constant org.wali.UpdateType. at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) ~[na:na] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:254) ~[na:na] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[na:na] at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) ~[na:na] at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) ~[na:na] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[na:na] ... 33 common frames omitted Caused by: java.lang.IllegalArgumentException: No enum constant org.wali.UpdateType. at java.lang.Enum.valueOf(Enum.java:238) ~[na:1.8.0_101] at org.wali.UpdateType.valueOf(UpdateType.java:24) ~[na:na] at org.apache.nifi.controller.state.StateMapSerDe.deserializeRecord(StateMapSerDe.java:76) ~[na:na] at org.apache.nifi.controller.state.StateMapSerDe.deserializeEdit(StateMapSerDe.java:69) ~[na:na] at org.apache.nifi.controller.state.StateMapSerDe.deserializeEdit(StateMapSerDe.java:30) ~[na:na] at org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1072) ~[na:na] at org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:459) ~[na:na] at org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301) ~[na:na] at org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider.init(WriteAheadLocalStateProvider.java:99) ~[na:na] at org.apache.nifi.controller.state.providers.AbstractStateProvider.initialize(AbstractStateProvider.java:34) ~[na:na] at org.apache.nifi.controller.state.manager.StandardStateManagerProvider.createStateProvider(StandardStateManagerProvider.java:189) ~[na:na] at org.apache.nifi.controller.state.manager.StandardStateManagerProvider.createLocalStateProvider(StandardStateManagerProvider.java:81) ~[na:na] at org.apache.nifi.controller.state.manager.StandardStateManagerProvider.create(StandardStateManagerProvider.java:67) ~[na:na] at org.apache.nifi.controller.FlowController.<init>(FlowController.java:469) ~[na:na] at org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:375) ~[na:na] at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:74) ~[na:na] at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[na:na] ... 40 common frames omitted 2017-09-11 10:21:10,334 INFO [Thread-2] org.apache.nifi.NiFi Jetty web server shutdown completed (nicely or otherwise). ... View more
Labels:

Nifi Cannot connect to kafka with Kerberos: unab...

by Explorer in Support Questions
‎05-15-2017 09:53 PM
‎05-15-2017 09:53 PM
Nifi Cannot connect to kafka with Kerberos while I try to configure it using the following instruction https://community.hortonworks.com/articles/26551/accessing-kerberos-enabled-kafka-topics-using-getk.html I got the error: unable to find LoginModule class: com.sun.security.auth.module.Krb5LoginModule Any advice is appreciated. 2017-05-15 17:00:00,703 ERROR [Timer-Driven Process Thread-7] o.a.n.p.kafka.pubsub.ConsumeKafka ConsumeKafka[id=11281745-115c-12b5-0451-b59b388b0d9b] ConsumeKafka[id=11281745-115c-12b5-0451-b59b388b0d9b] failed to process due to org.apache.kafka.common.KafkaException: Failed to construct kafka consumer; rolling back session: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer 2017-05-15 17:00:00,707 ERROR [Timer-Driven Process Thread-7] o.a.n.p.kafka.pubsub.ConsumeKafka org.apache.kafka.common.KafkaException: Failed to construct kafka consumer at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:648) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:512) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:494) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.nifi.processors.kafka.pubsub.ConsumerPool.createKafkaConsumer(ConsumerPool.java:137) ~[nifi-kafka-0-9-processors-1.1.1.jar:1.1.1] at org.apache.nifi.processors.kafka.pubsub.ConsumerPool.obtainConsumer(ConsumerPool.java:106) ~[nifi-kafka-0-9-processors-1.1.1.jar:1.1.1] at org.apache.nifi.processors.kafka.pubsub.ConsumeKafka.onTrigger(ConsumeKafka.java:286) ~[nifi-kafka-0-9-processors-1.1.1.jar:1.1.1] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.1.1.jar:1.1.1] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) [nifi-framework-core-1.1.1.jar:1.1.1] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.1.jar:1.1.1] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.1.jar:1.1.1] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.1.jar:1.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522) [na:1.8.0-internal] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:319) [na:1.8.0-internal] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:191) [na:1.8.0-internal] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [na:1.8.0-internal] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) [na:1.8.0-internal] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [na:1.8.0-internal] at java.lang.Thread.run(Thread.java:785) [na:1.8.0-internal] Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: unable to find LoginModule class: com.sun.security.auth.module.Krb5LoginModule at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:74) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:60) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:79) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:577) ~[kafka-clients-0.9.0.1.jar:na] ... 17 common frames omitted Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: com.sun.security.auth.module.Krb5LoginModule at javax.security.auth.login.LoginContext.invoke(LoginContext.java:828) ~[na:1.8.0-internal] at javax.security.auth.login.LoginContext.access$000(LoginContext.java:196) ~[na:1.8.0-internal] at javax.security.auth.login.LoginContext$4.run(LoginContext.java:698) ~[na:1.8.0-internal] at javax.security.auth.login.LoginContext$4.run(LoginContext.java:696) ~[na:1.8.0-internal] at java.security.AccessController.doPrivileged(AccessController.java:650) ~[na:1.8.0-internal] at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:696) ~[na:1.8.0-internal] at javax.security.auth.login.LoginContext.login(LoginContext.java:597) ~[na:1.8.0-internal] at org.apache.kafka.common.security.kerberos.Login.login(Login.java:298) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.common.security.kerberos.Login.<init>(Login.java:104) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.common.security.kerberos.LoginManager.<init>(LoginManager.java:44) ~[kafka-clients-0.9.0.1.jar:na] at org.apache.kafka.common.security.kerberos.LoginManager.acquireLoginManager(LoginManager.java:85) ~[kafka-clients-0.9.0.1.jar:na] ble to find LoginModule class: com.sun.security.auth.module.Krb5LoginModuleat org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:55) ~[kafka-clients-0.9.0.1.jar:na] ... 20 common frames omitted ... View more
Labels:

Re: Nifi GetHDFS OnScheduled error

by Explorer in Support Questions
‎05-15-2017 06:01 PM
‎05-15-2017 06:01 PM
We are using nifi-1.1.1 and java version "1.8.0_71" ... View more

Re: Nifi UI is down and Nifi OutOfMemoryError

by Explorer in Support Questions
‎05-15-2017 05:48 PM
‎05-15-2017 05:48 PM
@Wynner Thanks for your help, I had updated it to java.arg.2=-Xms512m java.arg.3=-Xmx8g ... View more

Nifi UI is down and Nifi OutOfMemoryError

by Explorer in Support Questions
‎05-15-2017 03:21 PM
‎05-15-2017 03:21 PM
Hi, My Nifi UI is down On the nifi-bootstrap log, I found the OutOfMemoryError. Any advice on this issue is appraciated. Thanks, Andy here is the nifi-bootstrap log: 2017-05-13 02:55:41,053 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/05/13 02:55:41 - please wait. 2017-05-13 02:55:41,054 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Heap dump using '/app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0001.phd' in response to an event 2017-05-13 02:55:48,525 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Heap dump written to /app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0001.phd 2017-05-13 02:55:48,529 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested System dump using '/app/Project/nifi-1.1.1/core.20170513.025541.31792.0002.dmp' in response to an event 2017-05-13 02:55:56,811 ERROR [NiFi logging handler] org.apache.nifi.StdErr Warning: unable to move dump to "/app/Project/nifi-1.1.1/core.20170513.025541.31792.0002.dmp" across file systems (check kernel core_pattern). Using alternate file location "/var/core/core.20170513.025541.31792.0002.dmp" 2017-05-13 02:55:56,818 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I System dump written to /var/core/core.20170513.025541.31792.0002.dmp 2017-05-13 02:55:56,818 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Java dump using '/app/Project/nifi-1.1.1/javacore.20170513.025541.31792.0003.txt' in response to an event 2017-05-13 02:55:56,971 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Java dump written to /app/Project/nifi-1.1.1/javacore.20170513.025541.31792.0003.txt 2017-05-13 02:55:56,972 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Snap dump using '/app/Project/nifi-1.1.1/Snap.20170513.025541.31792.0005.trc' in response to an event 2017-05-13 02:55:56,972 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Snap dump written to /app/Project/nifi-1.1.1/Snap.20170513.025541.31792.0005.trc 2017-05-13 02:55:56,973 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError". 2017-05-13 02:55:56,974 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Heap dump using '/app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0004.phd' in response to an event ... View more
Labels:

Nifi UI is down and Nifi OutOfMemoryError

by Explorer in Support Questions
‎05-15-2017 02:49 PM
‎05-15-2017 02:49 PM
My Nifi UI is down daily in the past few days. I wounder if it is relative to OutOfMemoryError. Any suggestion on this issue? 1) On the vi nifi-bootstrap_2017-05-13.log file 2017-05-13 02:55:41,053 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/05/13 02:55:41 - please wait. 2017-05-13 02:55:41,05ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Heap dump using '/app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0001.phd' in response to an event 2017-05-13 02:55:48,525 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Heap dump written to /app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0001.phd 2017-05-13 02:55:48,529 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested System dump using '/app/Project/nifi-1.1.1/core.20170513.025541.31792.0002.dmp' in response to an event 2017-05-13 02:55:56,811 ERROR [NiFi logging handler] org.apache.nifi.StdErr Warning: unable to move dump to "/app/Project/nifi-1.1.1/core.20170513.025541.31792.0002.dmp" across file systems (check kernel core_pattern). Using alternate file location "/var/core/core.20170513.025541.31792.0002.dmp" 2017-05-13 02:55:56,818 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I System dump written to /var/core/core.20170513.025541.31792.0002.dmp 2017-05-13 02:55:56,818 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Java dump using '/app/Project/nifi-1.1.1/javacore.20170513.025541.31792.0003.txt' in response to an event 2017-05-13 02:55:56,971 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Java dump written to /app/Project/nifi-1.1.1/javacore.20170513.025541.31792.0003.txt 2017-05-13 02:55:56,972 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Snap dump using '/app/Project/nifi-1.1.1/Snap.20170513.025541.31792.0005.trc' in response to an event 2017-05-13 02:55:56,972 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Snap dump written to /app/Project/nifi-1.1.1/Snap.20170513.025541.31792.0005.trc 2017-05-13 02:55:56,973 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError". 2017-05-13 02:55:56,974 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Heap dump using '/app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0004.phd' in response to an event 2017-05-13 02:56:05,809 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Heap dump written to /app/Project/nifi-1.1.1/heapdump.20170513.025541.31792.0004.phd 2017-05-13 02:56:05,810 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Java dump using '/app/Project/nifi-1.1.1/javacore.20170513.025541.31792.0006.txt' in response to an event 2017-05-13 02:56:05,970 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Java dump written to /app/Project/nifi-1.1.1/javacore.20170513.025541.31792.0006.txt 2017-05-13 02:56:05,970 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/05/13 02:56:05 - please wait. 2017-05-13 02:56:05,970 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Snap dump using '/app/Project/nifi-1.1.1/Snap.20170513.025541.31792.0007.trc' in response to an event 2017-05-13 02:56:05,970 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Snap dump written to /app/Project/nifi-1.1.1/Snap.20170513.025541.31792.0007.trc 2017-05-13 02:56:05,970 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError". 2017-05-13 02:56:05,971 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Heap dump using '/app/Project/nifi-1.1.1/heapdump.20170513.025605.31792.0008.phd' in response to an event 2017-05-13 02:56:14,533 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Heap dump written to /app/Project/nifi-1.1.1/heapdump.20170513.025605.31792.0008.phd 2017-05-13 02:56:22,867 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Java dump using '/app/Project/nifi-1.1.1/javacore.20170513.025605.31792.0009.txt' in response to an event 2017-05-13 02:56:23,055 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Java dump written to /app/Project/nifi-1.1.1/javacore.20170513.025605.31792.0009.txt 2017-05-13 02:56:23,064 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP032I JVM requested Snap dump using '/app/Project/nifi-1.1.1/Snap. 20170513.025605.31792.0010.trc' in response to an event 2017-05-13 02:56:23,064 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP010I Snap dump written to /app/Project/nifi-1.1.1/Snap. 20170513.025605.31792.0010.trc 2017-05-13 02:56:23,065 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP013I Processed dump event "systhrow", detail "java/lang/OutOfMemoryError". 2017-05-13 02:56:23,073 ERROR [NiFi logging handler] org.apache.nifi.StdErr JVMDUMP039I Processing dump event "systhrow", detail "java/lang/OutOfMemoryError" at 2017/05/13 02:56:23 - please wait. ... View more
Labels:

Re: Nifi GetHDFS OnScheduled error

by Explorer in Support Questions
‎05-15-2017 02:20 PM
‎05-15-2017 02:20 PM
@Matt Clarke Hi Matt, Thank you very much for helping. 1) here is the NiFi Bulletin Board error 10:19:17 EDT ERROR 12d4119d-1128-1761-becf-ab1016926114 GetHDFS[id=12d4119d-1128-1761-becf-ab1016926114] GetHDFS[id=12d4119d-1128-1761-becf-ab1016926114] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.; processor will not be scheduled to run for 30 seconds: java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. 2) Here is the error log 2017-05-15 10:20:17,535 ERROR [StandardProcessScheduler Thread-8] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1449) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode.access$100(StandardProcessorNode.java:100) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1275) ~[na:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522) [na:1.8.0-internal] at java.util.concurrent.FutureTask.run(FutureTask.java:277) [na:1.8.0-internal] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:191) [na:1.8.0-internal] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [na:1.8.0-internal] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153) [na:1.8.0-internal] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [na:1.8.0-internal] at java.lang.Thread.run(Thread.java:785) [na:1.8.0-internal] Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException at java.util.concurrent.FutureTask.report(FutureTask.java:133) [na:1.8.0-internal] at java.util.concurrent.FutureTask.get(FutureTask.java:217) [na:1.8.0-internal] at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1432) ~[na:na] ... 9 common frames omitted Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor406.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55) ~[na:1.8.0-internal] at java.lang.reflect.Method.invoke(Method.java:507) ~[na:2.6 (01-06-2016)] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) ~[na:na] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) ~[na:na] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) ~[na:na] at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1279) ~[na:na] at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1275) ~[na:na] ... 6 common frames omitted ... View more

Nifi GetHDFS OnScheduled error

by Explorer in Support Questions
‎05-11-2017 10:50 AM
‎05-11-2017 10:50 AM
Hi, When I use GetHDFS, I got the following error and I cannot find a solution for it. Any suggestion for this issue will be appreciated. Thanks Andy GetHDFS[id=115c12d7-1761-1128-7f1a-c31b56a24aaf] GetHDFS[id=115c12d7-1761-1128-7f1a-c31b56a24aaf] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.; processor will not be scheduled to run for 30 seconds: java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. ... View more
Labels:

Re: HDF - Flatten a xml file

by Explorer in Support Questions
‎03-08-2017 02:35 PM
‎03-08-2017 02:35 PM
Hi @Andrew Grande and @Neeraj Sabharwal 1) I was able to use the TransFormXml processor to convert the xml in the following format <?xml version="1.0" encoding="UTF-8"?> -<ROWSET> <ROW kind="element" name="Product" pid="" id="d1e1"> </ROW> <ROW kind="attribute" name="Type" pid="d1e1" id="d1e1a1373">Laptop</ROW> <ROW kind="element" name="Notebook" pid="d1e1" id="d1e3"> </ROW> <ROW kind="attribute" name="Brand" pid="d1e3" id="d1e3a1403">HP</ROW> <ROW kind="attribute" name="Model" pid="d1e3" id="d1e3a1938">Pavilion dv6-3132TX Notebook</ROW> <ROW kind="element" name="Harddisk" pid="d1e3" id="d1e5">640 GB</ROW> <ROW kind="element" name="Processor" pid="d1e3" id="d1e8">Intel Core i7</ROW> <ROW kind="element" name="RAM" pid="d1e3" id="d1e11">4 GB</ROW> <ROW kind="element" name="Notebook" pid="d1e1" id="d1e15"> </ROW> <ROW kind="attribute" name="Brand" pid="d1e15" id="d1e15a1403">HP</ROW> <ROW kind="attribute" name="Model" pid="d1e15" id="d1e15a1938">HP Pavilion dv6-3032TX Notebook</ROW> <ROW kind="element" name="Harddisk" pid="d1e15" id="d1e17">640 GB</ROW> <ROW kind="element" name="Processor" pid="d1e15" id="d1e20">Intel Core i7</ROW> <ROW kind="element" name="RAM" pid="d1e15" id="d1e23">6 GB</ROW> <ROW kind="element" name="Notebook" pid="d1e1" id="d1e27"> </ROW> <ROW kind="attribute" name="Brand" pid="d1e27" id="d1e27a1403">Toshiba</ROW> <ROW kind="attribute" name="Model" pid="d1e27" id="d1e27a1938">Satellite A660/07R 3D Notebook</ROW> <ROW kind="element" name="Harddisk" pid="d1e27" id="d1e29">640 GB</ROW> <ROW kind="element" name="Processor" pid="d1e27" id="d1e32">Intel Core i7</ROW> <ROW kind="element" name="RAM" pid="d1e27" id="d1e35">4 GB</ROW> <ROW kind="element" name="Notebook" pid="d1e1" id="d1e39"> </ROW> <ROW kind="attribute" name="Brand" pid="d1e39" id="d1e39a1403">Toshiba</ROW> <ROW kind="attribute" name="Model" pid="d1e39" id="d1e39a1938">Satellite A660/15J Notebook</ROW> <ROW kind="element" name="Harddisk" pid="d1e39" id="d1e41">640 GB</ROW> <ROW kind="element" name="Processor" pid="d1e39" id="d1e44">Intel Core i5</ROW> <ROW kind="element" name="RAM" pid="d1e39" id="d1e47">6 GB</ROW> </ROWSET> 2) my question is how to convert the following xquery in the sample for Nifi's EvaluateXQuery : (In the example, oracle is using.) SELECT x.* FROM xml_test t , XMLTable('/ROWSET/ROW' passing xmltransform(t.object_value, xmltype(:xsldoc)) columns node_id varchar2(100) path '@id' , node_name varchar2(30) path '@name' , node_value varchar2(2000) path 'text()' , parent_node_id varchar2(100) path '@pid' , node_kind varchar2(30) path '@kind' ) x ; ... View more

Re: how to setup StandardSSLContextService:

by Explorer in Support Questions
‎03-06-2017 06:51 PM
‎03-06-2017 06:51 PM
@Bryan Bende Do I have to use StandardSSLContextService to connect to IBM mq if my nifi.properties is configured with SSL Authentication for web browser? If I have to, could I use the same certificate that I created for web browser for IBM MQ (I use nifi.pfx for the web browser)? ... View more

how to setup StandardSSLContextService:

by Explorer in Support Questions
‎03-06-2017 05:02 PM
‎03-06-2017 05:02 PM
1) I had Configured Apache NiFi SSL Authentication using the following instruction https://www.batchiq.com/nifi-configuring-ssl-auth.html I attached the Certificate to a web Browser and the browser is able to connect to Nifi. 2) I have Nifi connected to IBM MQ, and I know I need to configured StandardSSLContextService of JMSConnectionFactoryProvider. 3) My question is that do I need to attach the certificate to MQ? Any suggestion on how to do it? Thanks a lot Andy ... View more
Labels:

Re: how to Setup IBM MQ Configuration for Nifi

by Explorer in Support Questions
‎02-24-2017 10:32 PM
‎02-24-2017 10:32 PM
update: I finally figured out how to setup IBM MQ for Nifi Here is my steps to setup IBM MQ for the Nifi connection 1) Create queue manager myMQManager using MQ Explorer 2) create Channel and queue using the following commands runmqsc myMQManager def chl(myMQChannel) chltype(svrconn) replace def chl(myMQChannel) chltype(clntconn) conname('localhost(1420)') QMNAME('myMQManager') replace def ql(myMQqueue) 2) Grant necessary authorization for a user using MQ Explorer 2.1 Queue Manager Authorities Setting right click your queue manager-->Object Authorities-->Add Role Based Authorities -->give the user access permission 2.2 Queue Authorities Setting right click your queue-->Object Authorities-->Manage Authority Records-->Specific Profiles--> your queue--> Accumulated authorities-->setup your user Access Permission for the queue 3) After the configuration and setup your Nifi ConsumeJMS processor, you should be able connect to IBM MQ ... View more

Re: nifi disabled controller-services and ConsumeJ...

by Explorer in Support Questions
‎02-24-2017 10:29 PM
‎02-24-2017 10:29 PM
@ozhurakousky Here is the steps to setup IBM MQ for the Nifi connection 1) Create queue manager myMQManager using MQ Explorer 2) create Channel and queue using the following commands runmqsc myMQManager def chl(myMQChannel) chltype(svrconn) replace def chl(myMQChannel) chltype(clntconn) conname('localhost(1420)') QMNAME('myMQManager') replace def ql(myMQqueue) 2) Grant necessary authorization for a user using MQ Explorer 2.1 Queue Manager Authorities Setting right click your queue manager-->Object Authorities-->Add Role Based Authorities -->give the user access permission 2.2 Queue Authorities Setting right click your queue-->Object Authorities-->Manage Authority Records-->Specific Profiles--> your queue--> Accumulated authorities-->setup your user Access Permission for the queue 3) After the configuration and setup your Nifi ConsumeJMS processor, you should be able connect to IBM MQ ... View more

Re: nifi disabled controller-services and ConsumeJ...

by Explorer in Support Questions
‎02-23-2017 06:49 PM
‎02-23-2017 06:49 PM
@ozhurakousky of course, I will post what I did on the IBM MQ here later today. ... View more

Re: nifi disabled controller-services and ConsumeJ...

by Explorer in Support Questions
‎02-23-2017 06:40 PM
‎02-23-2017 06:40 PM
@ozhurakousky Hi Oleg, I finally got the connection between Nifi and IBM MQ working now. It's some configuration issue on MQ. Thank you very much for your help and time. I am very appreciated. Andy ... View more

Re: nifi disabled controller-services and ConsumeJ...

by Explorer in Support Questions
‎02-22-2017 09:14 PM
‎02-22-2017 09:14 PM
@ozhurakousky Hi Oleg, I had re-installed IBM MQ and setting up the queue manager, Channel, listener, and queue. I will do more testing on Nifi and post the result. Thanks a lot Andy ... View more

Re: how to Setup IBM MQ Configuration for Nifi

by Explorer in Support Questions
‎02-21-2017 05:11 PM
‎02-21-2017 05:11 PM
@ozhurakousky Thanks a lot for taking time to help me. Yeah, I had tired your suggestion. and I just wrote a very detail message on what I did on the previous post. ... View more

Re: nifi disabled controller-services and ConsumeJ...

by Explorer in Support Questions
‎02-21-2017 05:00 PM
‎02-21-2017 05:00 PM
@ozhurakouskyThanks for the advice and your time. I had setup the processor as the following: Controller Service and Processor For the IBM MQ side 1) I found the channel SYSTEM.ADMIN.SVRCONN , do I need to create a sender channel or it's already built in? 2) I already have a listener for Nifi (localhost:1419) 3) I create a local queue Q1 (do I have to set it's Usage to transmission?) 4) I got the following error: ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] failed to process due to org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'NifiMQ' with connection mode 'Client' and host name 'localhost(1419)'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2540' ('MQRC_UNKNOWN_CHANNEL_NAME').; rolling back session: org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'NifiMQ' with connection mode 'Client' and host name 'localhost(1419)'. Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2540' ('MQRC_UNKNOWN_CHANNEL_NAME'). Is there Any other configuration that I missed? Thanks a lot. @ozhurakousky I am really appreciated for your help and your time. ... View more

Re: how to Setup IBM MQ Configuration for Nifi

by Explorer in Support Questions
‎02-21-2017 03:58 PM
‎02-21-2017 03:58 PM
@Matt Clarke I had read the posts at https://community.hortonworks.com/questions/35871/consumejms-failed-to-load-the-websphere-mq-native.html and https://community.hortonworks.com/questions/35871/consumejms-failed-to-load-the-websphere-mq-native.html I followed the instructions to configure the onsumeJMS and PublishJMS processors, but I cannot get it working. I wonder it's the configuration issue on the IBM MQ. For example, how could I set up the SYSTEM.ADMIN.SVRCONN for channel or is it build-in already? Do I need to create a transmission queue for the channel? ... View more

how to Setup IBM MQ Configuration for Nifi

by Explorer in Support Questions
‎02-21-2017 03:35 PM
‎02-21-2017 03:35 PM
I need help to Setup MQ for Nifi. I have a very tight deadline for a project to retrieve data from IBM MQ to Nifi. I installed IBM MQ on my local computer. I was able to configure the queues to read one from another. but I have issue on setup configurations for Nifi. Any advice on the MQ configuration for Nifi will be very appreciated. ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎04-04-2019 03:36 PM
Public Statistics
Date Registered ‎01-18-2017 02:43 PM
Last Visited ‎04-04-2019 03:36 PM
Total Messages Posted 58
Total Kudos Received 5