I am getting the following error on zeppelin: %pyspark df_single_recipes = df[df.Lotdf_nrecipes[df_nrecipes == 1].index Traceback (most recent call last): File "/tmp/zeppelin_pyspark-3154198105554295939.py", line 364, in <module> code = compile('\n'.join(stmts), '<stdin>', 'exec', ast.PyCF_ONLY_AST, 1) File "<stdin>", line 2 __zeppelin__._displayhook() ^ SyntaxError: invalid syntax Any ideas how to fix this? ... View more

Log Forwarding/Ingestion Patterns Log forwarding & ingestion is a key starting point for many logging initiatives such as log analytics, cyber security, anomaly & bot detection, etc etc. This article will focus few (not comprehensive) patterns for log forwarding/ingestion using NiFi. Commonly rsyslog is used to capture and ship log messages.“Rsyslog is an open-source software utility used on UNIX and Unix-like computer systems for forwarding log messages in an IPnetwork. It implements the basic syslog protocol, extends it with content-based filtering, rich filtering capabilities, flexible configuration options and adds features such as using TCP for transport.” More on how to configure rsyslog: here NiFi is able to ingest messages from rsyslog over TCP or UDP via ListenSysLog processor. This allows for little to no coding. Patterns Pattern A A minimalist design. Rsyslog is configured to simply forward log messages to a NiFi cluster. Rsyslog /etc/rsyslog.conf file needs to be configured to forward messages to a NiFi port identified in ListenSysLog processor. Pattern B A MiNiFi listen socket design. MiNiFi is installed on the server(s) leveraging ListenSysLog processor. This pattern offers end to end data linage along with more rich operational capabilities compared to Pattern A. MiNiFi via ListenSysLog will capture rsyslog messages and ship them to NiFi via S2S (site 2 site). Rsyslog is configured to simply forward log messages to a locally installed MiNiFi instance (localhost:port). Rsyslog /etc/rsyslog.conf file needs to be configured to forward messages to a the local MiNiFi port identified in ListenSysLog processor. This design will provide at least once message delivery guarantee. Pattern C A MiNiFi tail file design. MiNiFi is installed on the server(s) leveraging TailFile processor unlike Pattern B using ListenSyslog. Both pattern A and B offer end to end data linage and rich operational capabilities. MiNiFi will capture log messages by tailing a directory of files or a file and ship them to NiFi via S2S (site 2 site). Identify a log file to tail (ie /var/log/messages) or a directory for files, start MiNiFi and the log messages will start flow from the server(s) to NiFi. This design will provide at least once message delivery guarantee. These are a few but common pattens I have developed & implemented in the field with success. Happy log capturing! ... View more

During launch of HDP or HDF on azure via cloudbreak, if the following provisioning error is thrown (Check cloudbreak logs): log:55 INFO  c.m.a.m.r.Deployments checkExistence - [owner:xxxxx] [type:STACK] [id:2] [name:sparky] [flow:xxx] [tracking:] <-- 404 Not Found https://management.azure.com/subscriptions/xxxxxx/resourcegroups/spark. (104 ms, 92-byte body)/cbreak_cloudbreak_1 | 2018-09-05 14:25:22,882 [reactorDispatcher-24] launch:136 ERROR c.s.c.c.a.AzureResourceConnector - [owner:xxxxxx] [type:STACK] [id:2] [name:sparky] [flow:xxxxxx] [tracking:] Provisioning error: This means the instance type selected is not available within the region. Please change region where instance is available or change to instance type which is available within region. ... View more