Member since
09-27-2019
48
Posts
0
Kudos Received
0
Solutions
01-04-2021
08:30 AM
@kalhan While it is possible to have a single ZK cluster to support multiple services, It is the recommendation that NiFi have its own dedicated ZK cluster. NiFi cluster stability is dependent on ZK and many of the NiFi processors that can be used depend on on Cluster state which is also stored in ZK. IF ZK becomes overburdened it can affect overall stability and performance of NiFi. If you found any of the answers provided on this query helped you, please select "accept solution" on each of them. Thank you, Matt Hope this helps.
... View more
09-02-2020
06:46 AM
Same error, different port (8080): the problem was fixed by deleting the /var/lib/nifi/state directory (you need to stop all Nifi instances first). You can have this problem everytime you switch from TLS to non-secure and viceversa.
... View more
06-01-2020
08:43 PM
Is this (now) considered a NiFi "anti-pattern"? Do you have any idea how to do this using NiFi Record serialization services? I'm under the impression that creating thousands of content files is not the best practice by today's standards, but I'm not sure how to use InvokeHTTP on a full set of records without splitting it into many flowfiles. Any ideas?
... View more
04-21-2020
01:29 AM
2 Kudos
@Cl0ck You can always use OLD database, this is not am issue. For disabling TLS for admin console from CLI you can use below method. 1. Determine Cloudera Manager Database cat /etc/cloudera-scm-server/db.properties 2. Make database backup Postgres Backup: ---------------- pg_dump scm > scm_db_.backup 3. Connect to database Postgres Connect: ------------------ psql -U scm -h localhost -p 7432 -d postgres Password for user scm: postgres=> \l postgres=> \c scm 4. Show TLS related rows select * from CONFIGS where attr like '%tls%'; 5. Update TLS for web_tls update CONFIGS set value = 'false' where attr = 'web_tls'; You can generate the key and password as well with same method you did previously. Else please for reference you can use this doc. https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/how_to_configure_cm_tls.html
... View more
04-17-2020
10:03 PM
Thank for you this reply! This has been quite difficult for me to troubleshoot, but I finally figured it out. These machines I've been using had chrony on them all along, but the previous machines I set up did not have chrony installed. Chrony and ntpd were both enabled, and ntpd was getting exited on reboot. Because the host monitor issues "ntpq -np", and ntpd was loaded but inactive, it would report a failure to query the server, even though chrony was running. I had no idea that chrony was installed, and thus, the whole problem could've been solved by just disabling/uninstalling ntpd. I spent WAY too many hours to come to such a simple solution. It may be very helpful to someone who doesn't understand network time protocols very well if there was a suggestion to explain potential conflicts between ntpd and chronyd in the documentation, or even to take a second to check which (if any) you already have installed. Maybe it won't be an issue for most people, but for me, assuming that I didn't have chrony already running cost me a bunch of time getting my cluster healthy. I would check, find ntpd dead, see no problems reported on Host Monitor, wonder why the hell ntpd died, kill ntpd, run ntpdate, restart ntpdate, restart scm-agent, and that would "fix" it, but on reboot it would go back to using chrony and exit ntpd, and host monitor would report failure to query ntp service, even though the machine was using chrony and synced just fine all along. I appreciate your help!
... View more
03-25-2020
12:27 PM
@MattWho Is HDF 3.5 already released? If not, do you know when it is planned to be released? I saw page with release notes, but repository locations are still not updated. https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.0/release-notes/content/hdf_repository_locations.html
... View more
01-05-2020
05:24 PM
@Cl0ck Glad that it is all resolved. cheers
... View more
12-29-2019
08:41 PM
@Cl0ck Each host's name is stored in CM's backend database with an UUID attached, please refer to table HOSTS. Example as below: HOST_ID: 12
OPTIMISTIC_LOCK_VERSION: 148
HOST_IDENTIFIER: bfaf4b71-01e2-4157-b46f-d1c13566b69a
NAME: host-xxx-xxx.xxx
IP_ADDRESS: xx.xx.xx.xx
RACK_ID: /default
STATUS: NA
CONFIG_CONTAINER_ID: 1
MAINTENANCE_COUNT: 0
DECOMMISSION_COUNT: 0
CLUSTER_ID: 1
NUM_CORES: 1
TOTAL_PHYS_MEM_BYTES: 1929342976
PUBLIC_NAME: NULL
PUBLIC_IP_ADDRESS: NULL
CLOUD_PROVIDER: NULL Where HOST_IDENTIFIER is the UUID, and is stored under /var/lib/cloudera-scm-agent/uuid on each host. Maybe you can try to update the table here for NAME field and see if that can help? Cheers Eric
... View more
12-04-2019
07:13 AM
@Cl0ck I see nothing you listed as having been removed as an issue. NiFi will create all of the following on startup if they do not exist: - All 4 repository directrories (content, flowfile, database, and provenance) and their contents provided the NiFi service user has permsissions to configured directory to do so. - flow.xml.gz - local state directory and its contents - users.xml (if needed based on authorizers.xml configuration) - authorizations.xml (if needed based on authorizers.xml configuration) I would start by digging in to your shutdown exception: java.lang.IllegalStateException: no valid keystore You'll want to get a verbose listing of your keystore and make sure the following exists: 1. Keystore contains only one PrivateKeyEntry 2. That PrivateKeyEntry has ExtendedKeyUsage (EKU) that supports both clientAuth and ServerAuth 3. That PrivateKeyEntry has one or more SubjectAlternativeName (SAN) entries. One of those SAN entries must match the hostname of the server on which this keystore is being used. 4. The password on the key inside the keystore must use the same password as the keystore itself. A verbose output of your truststore should show that it contains 1 or more TrustedCertEntry Also make sure that your NiFi service users owns and can navigate to the directory where the keystore and truststore files have been placed. Hope this helps, Matt
... View more
11-30-2019
09:48 PM
Any luck getting it working? I'm still working on it to this day, but I think I'm gonna finish it tonight. If so, I can help you when I know my method works.
... View more