Member since
11-01-2019
28
Posts
3
Kudos Received
0
Solutions
10-11-2022
11:34 AM
I've attempt to set the above in the java.security file and in the bootstrap I set the runtime variables as java.arg.20=-Djava.security.networkaddress.cache.ttl=60 java.arg.21=-Dsun.net.inetaddr.ttl=60
... View more
09-27-2022
10:16 AM
Hi @araujo ! That seems to be cause, but I'm a bit confused on how I would set it up. I attempted to both set it in the bootstrap as well as via the jre, but still nothing seems to be working.
... View more
06-17-2022
09:13 AM
Kind of an update: A jank workaround was to force the Connection to close, but it's not the best solution. Looking to see why the client is ignoring the DNS TTL.
... View more
05-31-2022
12:30 PM
Version: NiFi 1.11.x Noticed a behavior where for this version of NiFi, when the remote url is a GSLB, it doesn't recognize when a host is removed from a GSLB until the processor is stopped. Any idea why? Any ideas for a solution?
... View more
Labels:
- Labels:
-
Apache NiFi
05-10-2022
06:10 AM
Apache NiFi is currently at version 1.16.x https://nifi.apache.org/ HDF is a frozen project where NiFi is at 1.11.4 Ambari seems to be attic'd. Is there a way to upgrade the version of NiFi packaged with HDF to a latter version of NiFi? Is there a way to combine the two standalone apache projects of NiFi and Ambari to get the functionality of Ambari's management but with a later version of NiFi?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache NiFi
02-01-2022
07:13 AM
Trying to determine if HDF/HDP is a valid choice going forward. Using this document (https://www.cloudera.com/downloads/paywall-expansion.html), it seems like these legacy tools require to an active subscription to download the binaries (as well as get support like bug-fixes and such). My question is if I set up a HDF/HDP cluster while I have an active subscription, then subscription lapses, what happens? Do I need to shutdown the cluster? Can I spin up local hosts and add them to a pre-existing cluster? What are the limitations? What about older clusters before this paywall was in place? EDIT: Adding this link here: https://www.cloudera.com/products/pricing/pricing-update.html "Subscription for binary access A key part of adopting the Red Hat model was that access to enterprise software binaries requires an active subscription. Cloudera requires a subscription to any compute and storage capacity assigned to run Cloudera software. All Cloudera software version releases, maintenance and security releases, and new product releases require a subscription." What does this bolded part mean?
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
01-18-2022
12:30 PM
Does the cloudera script handle the JndiLookup class for Log4j 1.x as well?
... View more
12-13-2021
01:47 PM
The TSB is not available unless you have a Knowledge Base subscription. Given the severity of the problem, will this information be made available to the public?
... View more
12-13-2021
11:15 AM
Agreed. Glad to see anything being done, but an official message needs to be put out before I destroy production lol.
... View more
12-13-2021
11:04 AM
Obviously, the best solution would be to replace all jars with the latest Log4j2 jars, but the way Cloudera does things now it might break things. In the long term, better to wait for them to make a statement. Here's a link that may help, look under workarounds: https://msrc-blog.microsoft.com/2021/12/11/microsofts-response-to-cve-2021-44228-apache-log4j2/
... View more
12-13-2021
08:26 AM
1 Kudo
Hive I believe is vulnerable and running 2.10.
... View more
09-01-2021
06:51 AM
Ok, I think I understand. I CAN install secure versions of these components, but that would be separate from Ambari and would sacrifice that level of control and maintenance. In order to get Ambari and these more secure components, I'll need to reach out to Cloudera for a private hotfix version or to upgrade off of HDP. Thank you for the clarification.
... View more
08-31-2021
05:45 AM
Hi @Shifu Thanks for the response! Regarding something you posted: " You can either install a component or you can upgrade to the next available HDP 3.X version but I can see you are in the latest 3.1.5 version." If I installed a later version of Zookeeper (for example), would ambari recognize that later version in it's management? Or would it exist in parallel with the version of Zookeeper packaged with 3.1.5? The current big security issues I see I've listed in the original question. Is there a contact form? Grafana is running v6.4.2, but has a major security issue that was patched in future releases: https://grafana.com/blog/2020/06/03/grafana-6.7.4-and-7.0.2-released-with-important-security-fix/ Infra Solr is running SOLR 7.7 and has a RCE vulnerability. This was patched in SOLR 8.3, which is not part of Ambari 2.7.5's InfraSolr. Zookeeper packaged is 3.4.6, but SSL implementation was added in 3.5.5
... View more
08-25-2021
07:56 AM
The components in HDP 3.1.5 is outdated and lack key security functionality. Grafana is running v6.4.2, but has a major security issue that was patched in future releases: https://grafana.com/blog/2020/06/03/grafana-6.7.4-and-7.0.2-released-with-important-security-fix/ Infra Solr is running SOLR 7.7 and haa a RCE vulnerability. This was patched in SOLR 8.3, which is not part of InfraSolr. Zookeeper packaged is 3.4.6, but SSL implementation was add in 3.5.5 I saw some questions talking about "Patch Upgrades" but is there a guide to upgrading individual components in a cluster via Ambari or however?
... View more
04-13-2020
08:05 AM
@stevenmatison Thanks for responding! I did think this was a file permissions issue on the start, but I ran some tests. Test 1: I chown'd/chmod'd the underlining files to match ORC files that presto could read from (those not written by PutHive3Streaming). Didn't work. Test 2: I ran Nifi's SelectHive3QL (which supports inserts). This wrote the data with file permissions and ownership similar to the other processor. Presto is able to read that data. Were you able to get to work? Additionally here's a snippet of puthive3streaming (minus the specifics like table, pathways, dbs). Using an avroreader to write.
... View more
04-10-2020
07:55 AM
What version of NiFi? Is it the Apache standalone or the HDF version? Also, are you saying you have other nifi nodes that work but the new one doesn't? What does the ui say? Is there errors in nifi-app.log?
... View more
04-09-2020
08:36 AM
So the title basically states it, but I'm currently running into an issue when leveraging Presto to ready from a Hive3 environment if the table is populated with ORC data by Nifi's PutHive3Streaming processor.
Presto is able to read ORC ACID tables if Hive 3 and populated via command line or other nifi processors. I attempted to write data using PutHive3Streaming from later versions of Nifi (1.11.4) to no avail.
Error:
io.prestosql.spi.PrestoException: Error opening Hive split hdfs://path/to/bucket (offset=0, length=29205493): rowsInRowGroup must be greater than zero
Versions: Nifi HDF 1.9 PrestoSQL 331/332
... View more
Labels:
- Labels:
-
Apache NiFi
04-09-2020
08:26 AM
So Presto now supports ACID tables, but only for Hive3. However, the subdirectory exception is from a configuration on the presto client side. In the hive.properties in presto's catalog directory, add " hive.recursive-directories=true"
... View more
04-09-2020
08:24 AM
@Ellyly No permanent solution to this collision type issue. The workaround I did was to split up the table into several smaller tables. That way no collisions occur. Not a great solution, but worked for my need.
... View more
11-14-2019
01:05 PM
@Shelton Hi thank you for the response. To clarify, I'm not talking about the data the tables are displaying. I'm talking about the actual metastore tables (TBLS, CDS, SDS, SERDES, etc.). These tables describe the table structure and if there is something wrong with the metadata, the tables won't function accordingly. The issue, I think, is when a table is dropped, the sql cmds to the metastore are fully executing (Not deleting from TBLS, but leaving the SD_ID null. The concern is the metadata itself, not the underlining csv/flatfile.
... View more
11-13-2019
06:45 AM
1 Kudo
If I can ask, why are you deleting? If you want to go through nifi, it sounds like an automated process. If it's a data retention issue, I think Hbase has a way to prune on timer. The TTL setting.
... View more
11-13-2019
06:41 AM
UPDATE: I'm wondering if this is happening on deletion, not creation. There is a describe that runs before the external table creation and it's throwing the same error. Checking binlogs in mysql for any clues. UPDATE2: After reading the binlog, I am more convinced the DROP TABLE <TABLE> cmd for hive isn't complete correctly and isn't delete from TBLS. -------------------------------------------------------------- This is a behavior I've experienced frequently in Hive 3, that occurred almost never in Hive. Hive Version: 3.0.0.3.1 Metastore is mysql. As part of an incremental data sqoop, I create two tables:One external, one ORC. Sometimes during the creation of these two tables, the metastore doesn't populate the the SERDES, SDS, CDS tables with the corresponding metadata and leaves the SD_ID null in the TBLS table. The table remain non-interactive (not even droppable) until I populate dummy data in the corresponding tables and update the TBLS's SD_ID field. The statements for table creation are as such: EXTERNAL create external table if not exists <DIFF EXTERNAL TABLE NAME> like <MAIN EXTERNAL TABLE NAME>; ORC create table <DIFF ORC TABLE> stored as orc as select * from <DIFF EXTERNAL TABLE NAME>; The logs in the metastore are a bit unclear: 2019-11-13T04:08:07,696 ERROR [pool-6-thread-187]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(201)) - java.lang.NullPointerException
2019-11-13T04:08:07,696 ERROR [pool-6-thread-187]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message. And the logs in the Hiveserver2 logs only highlight after the issue has happened.
... View more
Labels:
- Labels:
-
Apache Hive
11-11-2019
11:31 AM
This is a new type of issue, though. Is this for Hive3 ACID tables only that will solved for in the future? Regardless, it seems like partitioning the data (or just segmenting it) is the solution at this time. Thank you.
... View more
11-05-2019
07:39 AM
Basically, I need to know if this is a standard for hive going forward. We have a series of import that record the last value per table imported in a single table. However, this collision is breaking this process. The workaround solution I see is just splitting up these meta tables into multiple tables per import, but I don't like it. Any ideas or am I missing something?
... View more
11-04-2019
08:17 PM
1 Kudo
What does the nifi-bootstrap.log say?
... View more
11-04-2019
11:28 AM
I know this is like over a year old, but was the solution to this a yarn queue permission issue? To me, this line sticks out: User hive does not have permission to submit application_1526926153151_0001 to queue AdHoc I would check who can submit to the AdHoc queue in either the Yarn Scheduler view or in the yarn configuration tab in ambari.
... View more
11-04-2019
11:22 AM
We're currently looking to upgrade our production cluster to 3.1.0, but we're running into some pain points in a test environment that are stopping us from proceeding. Versions we're interacting with: HDFS 3.1.1.3.1 Hive 3.0.0.3.1 The problem: When attempting to run multiple updates on the same table, it fails due to a write conflict. This worked in previous versions, but in the new Hive 3 if the second update statement occurs during the first, it will throw this exception and fail (schema and table generalize). This is thru the new hive client (beeline). Error: Error while processing statement: FAILED: Hive Internal Error: org.apache.hadoop.hive.ql.lockmgr.LockException(Transaction manager has aborted the transaction txnid:306237872. Reason: Aborting [txnid:306237872,306237873] due to a write conflict on <SCHEMA>/<TABLE> committed by [txnid:306237871,306237872] u/u) (state=42000,code=12) The table properties are: TBLPROPERTIES( 'bucketing_version'='2','transactional'='true','transactional_properties'='default','transient_lastDdlTime'='1572894940')
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Tez