1973
Posts
1225
Kudos Received
124
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1999 | 04-03-2024 06:39 AM | |
| 3171 | 01-12-2024 08:19 AM | |
| 1725 | 12-07-2023 01:49 PM | |
| 2505 | 08-02-2023 07:30 AM | |
| 3517 | 03-29-2023 01:22 PM |
10-27-2016
01:07 PM
1 Kudo
https://github.com/aperepel/nifi-api-deploy Copy all your configuration, all your conf/archive/*.gz and your conf/flow.xml.gz file. Also from the UI, save and export all your templates separately to be sure you have them.
... View more
10-26-2016
01:44 AM
AttributestoJSON as FlowFileContent
... View more
10-25-2016
05:31 PM
Have you tried DeepLearning4J with Spark? or H2O with Spark. Both of those are very sophisticated NLP. https://dzone.com/articles/in-progress-natural-language-processing
... View more
10-25-2016
01:46 PM
Look at the Ambari Metrics and Grafana reports http://hortonworks.com/blog/3-things-to-know-hdf-2-0/
... View more
10-24-2016
10:13 PM
1 Kudo
SysDig SysDig (Github) is an open source tool that allows for the exploration, analysis and trouble shooting of Linux systems and containers. It is well documented and very easy to install and use. It can be used for container and Linux system diagnostics, security analysis, monitoring and basic system information capture. Remember that sysdig can produces thousands of lines of messages and can continue doing so forever depending on the options selected. Check out the examples and read through all the options, you can monitor a ton of data really fast and also check for security anomalies. NiFi Ingesting SysDig Sysdig can produce amazing amounts of logs. I chose to ingest 1 second chunks as ASCII JSON. I selected those options and listed them below. The results are arrays of JSON. I decided it's best to save them as a large JSON files for now and convert them to ORC later for Hive analysis in Zeppelin. You could also split them into individual JSON rows and process those. I also save them to Apache Phoenix for fast queries. ExecuteProcess Command sysdig -A -j -M 1 --unbuffered I just wrap that in a shell script for neatness. HDF 2.0 / NIFI 1.0.0 Flow Event JSON from SysDig {"evt.cpu":6,"evt.dir":">","evt.info":"fd=7(<f>/usr/lib64/python2.7/lib-dynload/_elementtree.so) ","evt.num":111138,"evt.outputtime":1477313882635597873,"evt.type":"fstat",
"proc.name":"python","thread.tid":14602}
Apache Phoenix Table CREATE TABLE sysdigevents
(
evtcpu varchar,
evtdir varchar,
evtinfo varchar,
evtoutputtime varchar,
evttype varchar,
procname varchar,
threadtid varchar,
evtnum varchar not null primary key
);
Links Sysdig Examples Sysdig Cheatsheet mapping to legacy tools Monitor Linux Server with sysdig NiFi Flow sysdig.xml
... View more
Labels:
10-24-2016
09:04 PM
1 Kudo
https://community.hortonworks.com/questions/3037/what-size-of-tables-make-the-best-out-of-orc-forma.html Large ORC files with large stripes should be best performance. Look at this Yahoo article on Hive and ORC at scale http://www.slideshare.net/Hadoop_Summit/hive-at-yahoo-letters-from-the-trenches
... View more
10-24-2016
09:03 PM
Did you open that port on the sandbox through VirtualBox? That needs to be available from your Mac/PC to the Sandbox. It is probably not open. https://community.hortonworks.com/questions/17168/lab-0-setup-start-the-sandbox-vm-and-open-ambari.html
... View more
10-24-2016
06:36 PM
1 Kudo
I am getting strange HBase connection errors in my logs must they are listed as INFO. 2016-10-24 18:34:27,065 INFO [StandardProcessScheduler Thread-7] o.a.nifi.hbase.HBase_1_1_2_ClientService HBase_1_1_2_ClientService[id=01571015-91d2-1dc1-f96e-a783702c1ea2] Simple Authentication
2016-10-24 18:34:27,066 ERROR [StandardProcessScheduler Thread-7] o.a.n.c.s.StandardControllerServiceNode HBase_1_1_2_ClientService[id=01571015-91d2-1dc1-f96e-a783702c1ea2] Failed to invoke @OnEnabled method due to java.io.IOException: java.lang.reflect.InvocationTargetException
2016-10-24 18:34:27,070 ERROR [StandardProcessScheduler Thread-7] o.a.n.c.s.StandardControllerServiceNode
java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240) ~[hbase-client-1.1.2.jar:1.1.2]
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218) ~[hbase-client-1.1.2.jar:1.1.2]
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119) ~[hbase-client-1.1.2.jar:1.1.2]
at org.apache.nifi.hbase.HBase_1_1_2_ClientService.createConnection(HBase_1_1_2_ClientService.java:238) ~[nifi-hbase_1_1_2-client-service-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
at org.apache.nifi.hbase.HBase_1_1_2_ClientService.onEnabled(HBase_1_1_2_ClientService.java:178) ~[nifi-hbase_1_1_2-client-service-1.0.0.2.0.0.0-579.jar:1.0.0.2.0.0.0-579]
at sun.reflect.GeneratedMethodAccessor808.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_101]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_101]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137) ~[na:na]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125) ~[na:na]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70) ~[na:na]
at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47) ~[na:na]
at org.apache.nifi.controller.service.StandardControllerServiceNode$2.run(StandardControllerServiceNode.java:348) ~[na:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_101]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.GeneratedConstructorAccessor501.newInstance(Unknown Source) ~[na:na]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_101]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_101]
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238) ~[hbase-client-1.1.2.jar:1.1.2]
... 19 common frames omitted
Caused by: java.lang.UnsupportedOperationException: Unable to find org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory
at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:36) ~[hbase-common-1.1.2.jar:1.1.2]
at org.apache.hadoop.hbase.ipc.RpcControllerFactory.instantiate(RpcControllerFactory.java:58) ~[hbase-client-1.1.2.jar:1.1.2]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.createAsyncProcess(ConnectionManager.java:2242) ~[hbase-client-1.1.2.jar:1.1.2]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:690) ~[hbase-client-1.1.2.jar:1.1.2]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:630) ~[hbase-client-1.1.2.jar:1.1.2]
... 23 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_101]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_101]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_101]
at java.lang.Class.forName0(Native Method) ~[na:1.8.0_101]
at java.lang.Class.forName(Class.java:264) ~[na:1.8.0_101]
at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:32) ~[hbase-common-1.1.2.jar:1.1.2]
... 27 common frames omitted
2016-10-24 18:34:27,070 ERROR [StandardProcessScheduler Thread-7] o.a.n.c.s.StandardControllerServiceNode Failed to invoke @OnEnabled method of HBase_1_1_2_ClientService[id=01571015-91d2-1dc1-f96e-a783702c1ea2] due to java.io.IOException: java.lang.reflect.InvocationTargetException
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache NiFi
10-24-2016
06:32 PM
nifi-bootstrap.log:2016-10-24 18:30:33,712 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: connecting to host "pop.x.com", port 110, isSSL false
nifi-bootstrap.log:2016-10-24 18:30:33,936 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: PIPELINING enabled
nifi-bootstrap.log:2016-10-24 18:30:33,936 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: authentication command trace suppressed
nifi-bootstrap.log:2016-10-24 18:30:34,090 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: authentication command succeeded
nifi-bootstrap.log:2016-10-24 18:30:38,072 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 1
nifi-bootstrap.log:2016-10-24 18:30:38,156 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 2
nifi-bootstrap.log:2016-10-24 18:30:38,229 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 3
nifi-bootstrap.log:2016-10-24 18:30:38,304 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 4
nifi-bootstrap.log:2016-10-24 18:30:38,380 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 5
nifi-bootstrap.log:2016-10-24 18:30:38,531 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 6
nifi-bootstrap.log:2016-10-24 18:30:38,607 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 7
nifi-bootstrap.log:2016-10-24 18:30:38,690 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 8
nifi-bootstrap.log:2016-10-24 18:30:38,769 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 9
nifi-bootstrap.log:2016-10-24 18:30:38,846 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 10
nifi-bootstrap.log:2016-10-24 18:30:38,925 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 11
nifi-bootstrap.log:2016-10-24 18:30:39,014 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 12
nifi-bootstrap.log:2016-10-24 18:30:39,090 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 13
nifi-bootstrap.log:2016-10-24 18:30:39,167 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 14
nifi-bootstrap.log:2016-10-24 18:30:39,242 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 15
nifi-bootstrap.log:2016-10-24 18:30:39,329 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 16
nifi-bootstrap.log:2016-10-24 18:30:39,395 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 17
nifi-bootstrap.log:2016-10-24 18:30:39,473 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 18
nifi-bootstrap.log:2016-10-24 18:30:39,551 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 19
nifi-bootstrap.log:2016-10-24 18:30:39,627 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 20
nifi-bootstrap.log:2016-10-24 18:30:39,701 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 21
nifi-bootstrap.log:2016-10-24 18:30:39,781 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 22
nifi-bootstrap.log:2016-10-24 18:30:39,856 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 23
nifi-bootstrap.log:2016-10-24 18:30:40,121 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 24
nifi-bootstrap.log:2016-10-24 18:30:40,196 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 25
nifi-bootstrap.log:2016-10-24 18:30:40,437 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: mail.pop3.apop.enable: false
nifi-bootstrap.log:2016-10-24 18:30:40,437 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: mail.pop3.disablecapa: false
nifi-bootstrap.log:2016-10-24 18:30:40,437 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: connecting to host "pop.agilemobiledeveloper.com", port 110, isSSL false
nifi-bootstrap.log:2016-10-24 18:30:40,663 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: PIPELINING enabled
... View more
10-24-2016
11:35 AM
is hbase running? do you have a firewall blocking it? what JDK are you using? perhaps an incompatible version? any other logs or details you can share?
... View more