Member since
02-05-2016
26
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
7545 | 09-30-2016 09:45 AM |
12-06-2021
02:31 PM
Many thanks for the offer, it is a great question, but this SHOULD be a pretty well understood process. I'm really looking for programming patterns, not consulting. I can't believe there aren't more examples of making this happen. We can create table properly and we've successfully written to Oracle with no issues. We're wondering about others experiences and best case examples.
... View more
12-02-2021
12:42 PM
I'm looking for examples on how to create tables and put data using schema registry into Oracle using NiFi. II searching I find many examples of read from Oracle and writing to HDFS, but I'm looking to read unstructured data and using a schema, build a Oracle table and write the data to said table. We can create the table using PutSQL, but we' worried about open/close/open connection performance or worse we open a cursor and Anyone have any examples? Still looking... WJS
... View more
Labels:
- Labels:
-
Apache NiFi
09-13-2021
06:00 AM
Just saw this reply... not alot of direction, but better than nothing. I'll have a look and reply.
... View more
08-23-2021
02:33 PM
I am looking for documentation, examples and other prior art for integrating Ranger logs with IBM QRadar. Our Security team uses QRadar for log filtering and monitoring and requires we send our audit logs to a QRadar event hub for audits.
... View more
Labels:
06-11-2020
08:33 AM
UPDATE - Apparently this is a Predictive Analytics (new feature in NiFI 1.11.4) bug per; JIRA NIFI-7516 where some of the linear algebra can become an issue in the predictive analytics modeling. When it does, it makes connecting through the NiFi canvas not possible. Looking forward to a fix.
... View more
06-03-2020
12:40 PM
We have recently upgraded to HDF 3.5.1 with NiFI v. 1.11.4.3.5.1.0-17.
Of five clusters, all with widely differing personalities, we've experienced an issue on three of the clusters where the canvas becomes unavailable and the error given when we try to log in is "Error Matrix is Singular.
In each case from the logs;
ERROR [Timer-Driven Process Thread-12] o.a.n.r.ambari.AmbariReportingTask AmbariReportingTask[id=3b80ba0f-a6c0-48db-b721-4dbc04cef28e] Error running task AmbariReportingTask[id=3b80ba0f-a6c0-48db-b721-4dbc04cef28e] due to org.apache.commons.math3.linear.SingularMatrixException: matrix is singular
it appears to be an issue with the Ambari Reporting Task. In one case the issue "resolved itself". In the other two instances, restarting the Controller Service "Ambari Reporting Task, the issue was cleared.
I'm still researching this, but I can find no Customer Support cases, nothing in Community Searches or the internet in general on this "Matrix is Singular: error.
Restarting the Reporting Task helps, but is not a fix. Is this a tuning issue or a setting that might need adjusting in the new version of NiFi? Does anyone have any experience with this issue?
... View more
Labels:
08-21-2019
05:59 AM
HW/Cloudera support has responded with "As per Apache NiFi documentation, EncryptContent process 1.9.0 does not supports DSA 1024 bit algorithm for encryption." and a link to the Apache EncryptContent processor Problem is we have a number of clients using DSA 1024 under pgp/gpg and we'd really prefer to encrypt with NiFI. Does anyone ( Andy Lopresto are you listening?) know if this is a fundamental incompatibility with DSA, a legal issue (see RSA patent fights of the 2000's), or simply an algorithm that has yet to be implemented?
... View more
- Tags:
- Encryption
- NiFi
02-20-2019
07:50 PM
I believe the ask was can we add "Custom Fields" eg; new fields other that the defined "valid" fields. How can I add pat or time stamp etc?
... View more
09-25-2018
07:49 PM
I'm having this exact issue also. Ever get a resolution?
... View more
09-25-2018
05:35 PM
Could you share the details of you HDP upgrade? I'm suing Ambari to manage HDF (Nifi) and am having a few issues getting the services (zookeeper at this point) to come up
... View more
09-25-2018
05:32 PM
I am in the process of performing an inplace upgrade RHEL6.7 to RHEL7.5. The OS has bee completed. Postgres has been completed. Python has been completed. TLS in ambari as bee set to v1.2 redhat7 repositories have been defined I am presently experiencing issue with the Zookeeper client install. I'll let you know how this plays out.
... View more
09-18-2018
02:09 AM
And probably should have found this earlier https://community.hortonworks.com/questions/178140/how-to-configure-nifi-to-maximize-the-usage-of-sys.html
... View more
09-18-2018
02:05 AM
OK Answer part of my own question In the Nifi interface NOT AMBARI configs.... Controller Settings; Maximum Event Driven Thread Count default: 10 - pushed up to 36 Maximum Timer Driven Thread Count default 5 - pushed up to 15 Running new flows. Will report how this works.
... View more
09-17-2018
08:56 PM
In Nifi my active thread count on one of our clusters won't go above 30. How can I increase the number of Active Threads? I have two clusters, that SEEM to be configured the same. On one the Active Thread count in Nifi won't go above 30. In the other we've seen as many as 70 Active Threads. Memory is the same in each cluster, but one cluster has more cores. Is this a setting or a function of available resources (possibly cores)?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache NiFi
09-06-2018
01:58 PM
There is a known issue in the PutGCSObject if running behind a secured network and accessing GCS through a proxy. The Proxy Host and Proxy Port added to the PutGCSObject will proxy the object itself, but DOES NOT proxy the GCPCredentialsControllerService, the service that authenticates your connection using yous Service AccountJSON file. At this time, this is a Community Supported Processor, and this issue exists in HFD 3.2 (Nifi v.1.7).
... View more
08-16-2018
05:48 PM
I agree on deleting the hbase data, however it's the only fix I've found so far. No argument from me. "medical kit logo" - can you clarify what do you mean by this?" --- Turns out it's the maintenance mode indicator. AMC was in maintenance mode. I'd never used Maintenance Mode and was unfamiliar with the indicator. I get the bozo award this week... Sigh.
... View more
08-16-2018
05:14 PM
OK OK... I'm a moron... Medical logo kit... Ambari Maintenance mode. but the corruption question stands.
... View more
08-16-2018
04:52 PM
We’re running Hortonworks Data Flow (HDF) Nifi actually. We have recently experienced Ambari Metrics Collector (AMC) HBase
corruption. As per the multitude of posted fixes we deleted hbase and hbase-tmp
and restarted. Great AMC is working again. But now I’ lost my history. This has
happened 3-4 times on different HDF clusters In addition in this instance Ambari displays what appears to
be a medical kit logo suggesting there are still issues. Is there any good way to recover from this corruption? Hbase hbc is not of much use although is does show that
/hbase does not exist in ZooKeeper even on good clusters. For the moment I’d like to get the suitcase to go away. Long
term does anyone have suggestions on avoiding AMC corruption?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache HBase
05-17-2018
01:58 PM
Heads up folks! The above responses are valid for previous version of HDF. For Ambari 2.6.1 with NIFI
1.5.0.3.1.1.0-35 or better the location changed to inside the Ambari interface. Please find Ambari GUI --> NIFI ---> Configs ----> Advanced nifi-toolkit-env ---> NIFI Toolkit Java Options I've found restarting the ambari-server and ambari-agents is no longer necessary as ambari-server pushes these options to the toolkit at runtime, overiding the configurations in the script and the configuration file.
... View more
05-17-2018
01:54 PM
Heads up folks! The above responses are valid for previous version of HDF. For Ambari 2.6.1 with NIFI 1.5.0.3.1.1.0-35 or better the location changed to inside the Ambari interface. Please find Ambari GUI --> NIFI ---> Configs ----> Advanced nifi-toolkit-env ---> NIFI Toolkit Java Options
... View more
12-04-2017
09:36 AM
I'm experiencing the same issue in 5.7.0. Can you offer any detail on the issue and why we might be sseing it in 5.7.0?
... View more
11-18-2016
07:18 AM
Harsh, In this thread you stated " but do look into if your users have begun creating too many tiny files as it may hamper their job performance with overheads of too many blocks (and thereby, too many mappers)." Too may tiny files is in the eye of the beholder if those files are what get you paid. I'm also seeing a block issue on wo of our nodes, but a rebalance to 10% has no effect. I've rebalanced to 8% and it improves, but I suspect we're running into a small files issue.
... View more
09-30-2016
09:45 AM
1 Kudo
This is apparently a know issue with JDK 8u91 and 8u92 and Oozie's TomCat environment. See JIRA OOZIE-2553 https://issues.apache.org/jira/browse/OOZIE-2533 Per the JIRA JDK 8u90 seeems to be fine, and there's a kludged work-a-round html page if you're in clined to implement it. (See the JIRA) or... you could always downgrade your Java to 8u90.
... View more