Solved

We're doing a bit of research here and trying to understand if LLMs are only available by accessing them through git and huggingface or whether there are some in built LLMs accessible using CML. Thank you

Is it possible to conenct and query Nessie catalog using Hue editors/connectors?

I am doing an ETL project that is taking data from CSV files and loading into a postgresql database. The files can contain up to 1M rows, and I need to process many files.Using NiFI version 1.15.2.The flow that I have built can be simplified into "Fetch CSV" -> "Convert to Avro" -> "PutDatabaseRecord".But I am having an issue where any timestamp columns that have microsecond precision are being truncated to seconds when inserted into the database. The project's requirements are for no data loss, so losing the microseconds will fail the requirement.I have tried using both inferred and explicit schemas in the controller services, but both methods truncate the timestamps. I created a test flow and database to explore the issue. The database has columns for storing the timestamp in string, timestamp and as an epoch number. My schema for the controller service{"name": "timestampTest","namespace": "nifi","type": "record","fields": [{"name": "bom_stn_num", "type": "s

Hi, I am looking to fetch only new files added in the directory exactly one time and once file is picked it should not be picked again in apache nifi. I want to schedule this process to to every 3 hours. Please provide solution with screenshot the properties you used to do this process or which processors you are using. I am bit confused between listfile getfile and fetchfile and which properties to used. Any help in this issue will be greatly appreciated.Thank You!

Hello!all nodemanager down everyday withou FATAL, ERROR in logsIts all log 2024-06-28 00:11:50,491 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.143.10.2:46342:appattempt_1719446422443_24818_000001 (auth:SIMPLE) (DIGEST-MD5: digest response format violation. Mismatched response.)2024-06-28 00:11:50,492 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 38274: readAndProcess from client 10.143.10.2 threw exception [javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched response.]2024-06-28 00:11:50,748 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.143.10.2:46366:appattempt_1719446422443_24827_000001 (auth:SIMPLE) (DIGEST-MD5: digest response format violation. Mismatched response.)2024-06-28 00:11:50,748 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 38274: readAndProcess from client 10.143.10.2 threw exception [javax.security.sasl.SaslException: DIGEST-MD5: digest response for

Hi,It blow my mindI try to connect hive using Cloudera ODBC Driver for Apache Hive on ReportServer. We have kerberos autentication.When i test system DSN it SUCCESS And when i test connection on PowerBI ReportServer in upload powerbi report using this DSN (in menu : control>>data sources) it has success (windows autentificathion)But when i test connection on PowerBI ReportServer creating data source for SSRS report , using the same DSN , it have error (windows autentificathion) :ERROR [28000] [Cloudera][ThriftExtension] (9) Error occurred while authenticating via SASL. Error details: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Internal credentials cache error) ERROR [28000] [Cloudera][ThriftExtension] (9) Error occurred while authenticating via SASL. Error details: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Internal credentials cache error)

Hello there,I learned about nifi counters, but when I open my nifi instance and click on the counters button in the main menu, it just opens an empty list. Same when I call nifi-api/counters. It is just an empty list. I do have processors and processgroups running.So does anyone have an explanation why that is? I use nifi 1.23.2 on my local machine. Thanks in advance!

 nifi.python.command=python3nifi.flow.configuration.file=../data/flow.json.gznifi.flow.configuration.archive.enabled=falsenifi.flow.configuration.archive.dir=../data/archive/nifi.flow.configuration.archive.max.time=30 daysnifi.flow.configuration.archive.max.storage=500 MBnifi.flow.configuration.archive.max.count=nifi.flowcontroller.autoResumeState=truenifi.flowcontroller.graceful.shutdown.period=10 secnifi.flowservice.writedelay.interval=500 msnifi.administrative.yield.duration=30 secnifi.bored.yield.duration=10 millisnifi.queue.backpressure.count=10000nifi.queue.backpressure.size=1 GBnifi.authorizer.configuration.file=./conf/authorizers.xmlnifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xmlnifi.templates.directory=../data/templatesnifi.ui.banner.text=nifi.ui.autorefresh.interval=30 secnifi.nar.library.directory=./libnifi.nar.library.autoload.directory=./extensionsnifi.nar.working.directory=./work/nar/nifi.documentation.working.directory=./work/docs

Hello All,      When I try to restart or stop or start any services, the background operations in Ambari does not list the operations. It shows no services to show. It was working fine earlier and now all the previous operations are also missing in the list. Ambari - 2.7.3 HDP 3.1.0

Hello,We have custom syslog Cisco messages in the following format <117>2024-07-23T14:09:56Z XXXXXXXXX : %FTD-5-430003: EventPriority: Low, DeviceUUID: xxxxxxxxxxxxx, InstanceID: 2, FirstPacketSecond: 2024-07-23T14:09:56Z, ConnectionID: 32322, AccessControlRuleAction: Allow, SrcIP: A.B.C.D, DstIP: A.B.C.D, SrcPort: 42308, DstPort: 24224, Protocol: tcp, IngressInterface: XXX, EgressInterface: XXX, IngressZone: XXX, EgressZone: YYY, IngressVRF: Global, EgressVRF: Global, ACPolicy: AAA-BBB, We want to use NIFI to format to CEF ( common event format ). Any help which processors to use, please?

Hi,I’m experiencing an issue when multiple queries are executed simultaneously. When I run the query by itself, it succeeds. However, when multiple queries run concurrently, I encounter problems.Does anyone know how I should configure Impala to avoid this issue? I’ve attached the profile of the failed query.My mem_limit is set to 60GB for 5 workers and 90GB for the other 8.Thanks,

hello Cloudera Community,the process of creating a user and adding them to groups via the nifi WebUI mode works normally.i need to create a user and add them to a group directly using the NiFi API (post, put, get).how do I make this process 100% functional?

Hello, a customer has a CDH Express 6.3.1 setup and:- It appears that suddenly one (and just one!) of the hosts listed in Hosts --> All Hosts is now showing as having a complete FQDN. It never used to have it (I have screenshots of previous configurations where ALL hosts in the list are shown with just the hostname and no domain)- Also, there are Stale Configurations waiting to be applied. If I click on them, they are related to a number of Client Configuration files waiting to be deployed, and sure enough ALL the lines in ALL the files are showing a single difference for that particular host!Now, I don't know what happened here. Nobody is reporting a CM configuration change made by them.I don't have History/Rollback capabilities on this cluster, as it's an Express Edition. Also, I have no Audits as well to check what (if any?) parameters have been recently changedI just know that Sysadmins recently performed a change in FQDN names on those hosts at OS level, because a monitoring to

We are using Hortonworks (HDP-2.6.5.0) and by default, inter cluster communication uses non-SSL ports in this version of HDP. Is there any way to achieve SSL based communications within cluster for all the possible services and nodes in a running HDP cluster ? Any help you can provide in this matter will be highly appreciated.

We have a requirement to extract data from unstructured logs data and capture the results in a custom format. We have implemented the solution using the ExtractText and ReplaceText Processors but we are not getting the expected results,Please find the details of the input data, current implementation, current output, and expected output format below and help us with your expertise to improvise this using the NiFi Processors.Input File Content : <13>Jul 18 11:39:11 test234104.test.gmail.ae AgentDevice=WindowsLog    AgentLogFile=Security   PluginVersion=100.3.1.22  Source=Microsoft-Windows-Security-Auditing      Computer=mycomputer1 OriginatingComputer=102.123.33.1    User= Domain=     EventID=4688 EventIDCode=4688  EventType=8 EventCategory=13312{122}Aug 18 11:39:11 test234104.test.gmail.ae  PluginVersion=200.3.1.22  Source=Microsoft-Windows-Security-Auditing AgentDevice=WindowsLog    AgentLogFile=Security     Computer=mycomputer2.gmail.com OriginatingComputer=125.123.33.1   

Hello All,On CDP PB, like we send ranger audit logs of services thats support Ranger plugin to HDFS, can we also send logs of Stream Messaging Manager service to HDFS? I tried to look for config to achieve this, however, I could find only 2 configs (listed below) that allow us to configure the local path. However, there was nothing found to send these logs to HDFS.Ranger Streams Messaging Manager Plugin Audit Hdfs Spool Directory PathRanger Streams Messaging Manager Plugin Audit Solr Spool Directory PathAny help / guidance will be highly appreciated.Thankssnm1523

Good day,In SIT, I have manually create a hive table. Follow by manually create a partition on it. Then I copy the bucket file from other table and put inside the partition. At this moment, if I run select statement, I getting 0 result, I believe this is normal because data haven't write to meta store.Then I run the following commands:ANALYZE TABLE raw_zone_sit.jxtemp PARTITION (dl_created_yr='2023', dl_created_mth='12') COMPUTE STATISTICS;After I run this command, I able to select data from my table.After that I apply the same way in production, but I am still getting empty result.Currently I am trying to recover all the meta store in production, this is because 1 of the person accidently delete all the meta store in production. Thus now I am trying to recover on it.For your information, parquet tables working fine. But I have some ORC tables also, now pending ORC table to resolve.Please help.

Requirement:Building an open source Hadoop cluster. No Ambari (No support for RHEL 9) and No CM manager (My company is not ready). Server Information:I have 5 nodes in RHEL 9: 2 masters and 3 slaves. Apache Services to be installed as per my company requirement:- Hadoop- Hbase- ZookeeperOnline Resources:Overwhelmed to follow because Its tough for me to figure out how I will be able to interlinked everything together. Needed solution:Can someone please share the standard and most updated resources which i can follow to setup my cluster. And make those services talk to each other in a right direction. Like where to start ? As I know its hadoop but after that I got no clue. I did all the coding part in my previous project and now this admin part is a pain to me.Please someone help. Really appreciate. Thanks,Srikanth

After adding new datanodes to the existing cluster and updating topology file. I am not able to refresh the namenode to pick up rack of the new node. It puts under `/default-rack`.   Anyway to refresh the rack topology changes without restarting namenodes? 

We have a requirement to convert the CSV file content into a custom format. Please find the details of the input and expected output format below and help us with your expertise to achieve this using the NiFi Processors.Input File Content :username, first name, middle name, last nametest_user, test_FN, test_MN, test_LNtest_user2, test2_FN, test2_MN, test2_LNExpected Output Format :username = test_userfirst name = test_FNmiddle name = test_MNlast name = test_LNusername = test_user2first name = test2_FNmiddle name = test2_MNlast name = test2_LN

Hello, I m sending a http request to an API and facing some difficulties to configure the multipart specificities.I have to send 2 files as part of request. Is there any way to send 2 files ? Since there can be only one flow file that is possible  @SAMSALApache NiFi processor 'InvokeHTTP' POST with MultipartFile formdata to remote API --This works fine when one file is being sent. Any help on sending 2 of them ? 

I'm running oozie HA 5.2.1 on EMR and I have an issue with this temporary directory. I have a workflow which has start node -> action node -> end node. The job start running -> runs for 10-15 minutes -> is initially marked as successful -> is marked as failed. The error is JA008 File or directory not found for the /user/oozie/oozie-oozi/"oozie_job_id"/"ActionName"--java directory. I'm running in parallel a script which looks at this directory as well as the state of the workflow and while running this directory contains action.xml and launcher.xml file -> before marking it as SUCCEDED it adds the action-data.seq file -> after it marks it as succeeded it deletes this directory -> after 1-2 minutes marks it as failed with the error above. It looks like oozie is checking again for that directory after it previously deleted it.Although this jobs is marked as failed from oozie perspective in reality this job is successful (checking in the RM UI the jobs status and th

How to set up TeraGen & TeraSort performance testing in a CDP private cloud cluster ? 

HiI have an existing CDP 7.1.x cluster with Auto-tls enabled during the creation of the cluster. I followed the use case 2: https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/security-encrypting-data-in-transit/topics/cm-security-use-case-2.html, to use an existing Root CA. Recently the certificates expired and I'm trying to renew them. I've a couple of questions from the documentation.1. In the above page, it mentions "In this use case, rotation of the Auto-TLS certificate authority is not supported. Cloudera recommends creating an intermediate CA with a long lifetime. The host certificates can be rotated by using the generateHostCerts API." - Should I use this to generate the host certs. If so, can I get an example of the API call and it's usage.2. Or should I use this use case 3: https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/security-encrypting-data-in-transit/topics/cm-security-use-case-3.html. Generate the certificates myself and use the gener

We are on CDP 7.1.7 SP2 versionCan I ask what is the best strategy to backup the cluster? We are using 5 phyisical nodes (3 of which holds the actual HDFS data). What is a good strategy for backup?1. Create another cluster that will act as replica (lower CPU and RAM)2. In the same cluster, backup / to preserve the OS apps (CM etc.) and /data (HDFS)