Cloudera Community
TimothySpann
user rank
user rank
Master Guru

Re: Update JSON Element w/FlowFile Attribute - Nif...

by Contributor in Support Questions
‎01-15-2019 04:54 PM
1 Kudo
‎01-15-2019 04:54 PM
1 Kudo
@Andrew Watson Hi Andrew did you find a way out for this issue ... View more

Monitoring Your Containers with SysDig from HDF 3....

by Master Guru Master Guru in Community Articles
‎10-24-2016 10:13 PM
1 Kudo
‎10-24-2016 10:13 PM
1 Kudo
SysDig SysDig (Github) is an open source tool that allows for the exploration, analysis and trouble shooting of Linux systems and containers. It is well documented and very easy to install and use. It can be used for container and Linux system diagnostics, security analysis, monitoring and basic system information capture. Remember that sysdig can produces thousands of lines of messages and can continue doing so forever depending on the options selected. Check out the examples and read through all the options, you can monitor a ton of data really fast and also check for security anomalies. NiFi Ingesting SysDig Sysdig can produce amazing amounts of logs. I chose to ingest 1 second chunks as ASCII JSON. I selected those options and listed them below. The results are arrays of JSON. I decided it's best to save them as a large JSON files for now and convert them to ORC later for Hive analysis in Zeppelin. You could also split them into individual JSON rows and process those. I also save them to Apache Phoenix for fast queries. ExecuteProcess Command sysdig -A -j -M 1 --unbuffered I just wrap that in a shell script for neatness. HDF 2.0 / NIFI 1.0.0 Flow Event JSON from SysDig {"evt.cpu":6,"evt.dir":">","evt.info":"fd=7(<f>/usr/lib64/python2.7/lib-dynload/_elementtree.so) ","evt.num":111138,"evt.outputtime":1477313882635597873,"evt.type":"fstat", "proc.name":"python","thread.tid":14602} Apache Phoenix Table CREATE TABLE sysdigevents ( evtcpu varchar, evtdir varchar, evtinfo varchar, evtoutputtime varchar, evttype varchar, procname varchar, threadtid varchar, evtnum varchar not null primary key ); Links Sysdig Examples Sysdig Cheatsheet mapping to legacy tools Monitor Linux Server with sysdig NiFi Flow sysdig.xml ... View more
Labels:

Re: HDF 2.0 Error Message

by Super Guru Super Guru in Support Questions
‎10-24-2016 07:10 PM
‎10-24-2016 07:10 PM
To clarify because it's confusing otherwise: despite the missing class being named "org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory", this is an Apache Phoenix class. It has to be in the "org.apache.hbase..." package to access protected API. ... View more

Re: NIFI processors HTTP

by Rising Star in Support Questions
‎10-20-2016 02:18 PM
‎10-20-2016 02:18 PM
@Timothy : I got this flow I'm use for testing this curl curl -i -v -F file=@/var/opt/hosting/log/flume/flume-a1.log http://nifi011:10000/contentListener RouteOnAttribute has properties (abc ==> ${filename:contains('flume')} Why flume log not send tu HDFS ? ... View more

Re: ConsumePop3: Parsing Messages with Nifi 1.0.0

by Master Guru Master Guru in Support Questions
‎10-24-2016 06:32 PM
‎10-24-2016 06:32 PM
nifi-bootstrap.log:2016-10-24 18:30:33,712 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: connecting to host "pop.x.com", port 110, isSSL false nifi-bootstrap.log:2016-10-24 18:30:33,936 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: PIPELINING enabled nifi-bootstrap.log:2016-10-24 18:30:33,936 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: authentication command trace suppressed nifi-bootstrap.log:2016-10-24 18:30:34,090 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: authentication command succeeded nifi-bootstrap.log:2016-10-24 18:30:38,072 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 1 nifi-bootstrap.log:2016-10-24 18:30:38,156 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 2 nifi-bootstrap.log:2016-10-24 18:30:38,229 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 3 nifi-bootstrap.log:2016-10-24 18:30:38,304 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 4 nifi-bootstrap.log:2016-10-24 18:30:38,380 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 5 nifi-bootstrap.log:2016-10-24 18:30:38,531 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 6 nifi-bootstrap.log:2016-10-24 18:30:38,607 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 7 nifi-bootstrap.log:2016-10-24 18:30:38,690 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 8 nifi-bootstrap.log:2016-10-24 18:30:38,769 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 9 nifi-bootstrap.log:2016-10-24 18:30:38,846 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 10 nifi-bootstrap.log:2016-10-24 18:30:38,925 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 11 nifi-bootstrap.log:2016-10-24 18:30:39,014 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 12 nifi-bootstrap.log:2016-10-24 18:30:39,090 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 13 nifi-bootstrap.log:2016-10-24 18:30:39,167 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 14 nifi-bootstrap.log:2016-10-24 18:30:39,242 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 15 nifi-bootstrap.log:2016-10-24 18:30:39,329 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 16 nifi-bootstrap.log:2016-10-24 18:30:39,395 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 17 nifi-bootstrap.log:2016-10-24 18:30:39,473 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 18 nifi-bootstrap.log:2016-10-24 18:30:39,551 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 19 nifi-bootstrap.log:2016-10-24 18:30:39,627 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 20 nifi-bootstrap.log:2016-10-24 18:30:39,701 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 21 nifi-bootstrap.log:2016-10-24 18:30:39,781 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 22 nifi-bootstrap.log:2016-10-24 18:30:39,856 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 23 nifi-bootstrap.log:2016-10-24 18:30:40,121 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 24 nifi-bootstrap.log:2016-10-24 18:30:40,196 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: streaming msg 25 nifi-bootstrap.log:2016-10-24 18:30:40,437 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: mail.pop3.apop.enable: false nifi-bootstrap.log:2016-10-24 18:30:40,437 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: mail.pop3.disablecapa: false nifi-bootstrap.log:2016-10-24 18:30:40,437 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: connecting to host "pop.agilemobiledeveloper.com", port 110, isSSL false nifi-bootstrap.log:2016-10-24 18:30:40,663 INFO [NiFi logging handler] org.apache.nifi.StdOut DEBUG POP3: PIPELINING enabled ... View more

Re: Difference between apache knox and kerberos

by Rising Star in Support Questions
‎10-18-2016 03:18 PM
‎10-18-2016 03:18 PM
@Timothy Spann thanks for your answer ... View more

Re: NiFi Hive Connection Pool Error.

by New Member in Support Questions
‎06-12-2019 10:55 AM
‎06-12-2019 10:55 AM
@mkalyanpur do I have to copy the krb5.conf file from my Hive server to the NiFi server? ... View more

Re: KiteSDK HDP 2.5

by Master Mentor in Support Questions
‎10-21-2016 04:37 PM
‎10-21-2016 04:37 PM
@rmolina @Rafael Coss FYI ... View more

Ingesting JMS Messages to HDFS via HDF 2.0

by Master Guru Master Guru in Community Articles
‎10-14-2016 07:16 PM
2 Kudos
‎10-14-2016 07:16 PM
2 Kudos
Sometimes you have Java messages that you would like to easily ingest into HDFS or perhaps HDFS as raw files, Phoenix, Hive and other destinations. You can do that pretty easy with Apache NiFi 1.0.0 as part of HDF 2.0. For this simple example, I also added a REST gateway for bulk loading, testing and to provide another way to easily send JMS messages. ListenHTTP accepts HTTP POSTS on port 8099, which I made the listener port for that processor. It takes what you send and publishes that to a JMS queue. I am using ActiveMQ. I have a little Python 2.7 script that I found on github that makes fake log records and modified it to send 1,000 JSON messages via REST to our REST to JMS gateway in NIFI for testing. You can easily do this with shell script and CURL, Apache JMeter, Java code, Go script and many other open source REST testers and clients. url = 'http://server.com:8099/contentListener' r = requests.post(url, json={"rtimestamp": timestamp, "ip": random_ip(), "country": country, "status": status}) I installed an ActiveMQ JMS broker as my example JMS server, which is very simple on Centos 7. All you need to do is download the gziped tar and untar it. It's ready to run with a chmod. That download also includes the client jar that we will need on the HDF 2.0 server for accessing the message queue server. You must also have the port open. On ActiveMQ that defaults to 61616. ActiveMQ also includes a nice web console that you may want to unblock that port for viewing the status of queues and messages. In my simple example, I am running JMS via: bin/activemq start > /tmp/smlog 2>&1 &; I recommend changing your HTTP Listening Port, so you can run a bunch of these processors as needed. Processors used: ConsumeJMS, MergeContent and PutHDFS. You need to set Destination Name which is the name of the QUEUE in this case, but could also be the name of the Topic. I picked Destination Type of QUEUE since I am using a QUEUE in Apache ActiveMQ. It's very easy to add more output processors for sinking data into Apache Phoenix, HBase, Hive, Email, Slack and other NoSQL stores. It's also easy to convert messages into AVRO, ORC and other optimized big data file formats. As you see we get a number of jms_ attributes including priority, message ID and other attributes associated with the JMS message. Example Message ActiveMQ Screens References: https://community.hortonworks.com/articles/59349/hdf-20-flow-for-ingesting-real-time-tweets-from-st.html https://community.hortonworks.com/articles/59975/ingesting-edi-into-hdfs-using-hdf-20.html http://activemq.apache.org/uri-protocols.html http://activemq.apache.org/initial-configuration.html http://activemq.apache.org/version-5-getting-started.html http://www.apache.org/dyn/closer.cgi?filename=/activemq/5.14.1/apache-activemq-5.14.1-bin.tar.gz&action=download ... View more

Re: ListenSysLog

by Frequent Visitor in Support Questions
‎11-22-2018 01:27 PM
‎11-22-2018 01:27 PM
Hello @Bryan Bende @Timothy Spann I'm a newbee to syslog and i wanna use it with kafka. Can i have any idea how it works. Please, regards. ... View more
Contact Me
Online
Offline
Last Visited
‎05-20-2024 05:42 PM
Community Statistics
Member Since ‎01-07-2019 11:58 AM
Last Visited ‎05-20-2024 05:42 PM
Posts 1,973
Kudos received 1122