Q&A

Hi, I am looking to fetch only new files added in the directory exactly one time and once file is picked it should not be picked again in apache nifi. I want to schedule this process to to every 3 hours. Please provide solution with screenshot the properties you used to do this process or which processors you are using. I am bit confused between listfile getfile and fetchfile and which properties to used. Any help in this issue will be greatly appreciated.Thank You!

Hello!all nodemanager down everyday withou FATAL, ERROR in logsIts all log 2024-06-28 00:11:50,491 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.143.10.2:46342:appattempt_1719446422443_24818_000001 (auth:SIMPLE) (DIGEST-MD5: digest response format violation. Mismatched response.)2024-06-28 00:11:50,492 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 38274: readAndProcess from client 10.143.10.2 threw exception [javax.security.sasl.SaslException: DIGEST-MD5: digest response format violation. Mismatched response.]2024-06-28 00:11:50,748 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 10.143.10.2:46366:appattempt_1719446422443_24827_000001 (auth:SIMPLE) (DIGEST-MD5: digest response format violation. Mismatched response.)2024-06-28 00:11:50,748 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 38274: readAndProcess from client 10.143.10.2 threw exception [javax.security.sasl.SaslException: DIGEST-MD5: digest response for

Hi,It blow my mindI try to connect hive using Cloudera ODBC Driver for Apache Hive on ReportServer. We have kerberos autentication.When i test system DSN it SUCCESS And when i test connection on PowerBI ReportServer in upload powerbi report using this DSN (in menu : control>>data sources) it has success (windows autentificathion)But when i test connection on PowerBI ReportServer creating data source for SSRS report , using the same DSN , it have error (windows autentificathion) :ERROR [28000] [Cloudera][ThriftExtension] (9) Error occurred while authenticating via SASL. Error details: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Internal credentials cache error) ERROR [28000] [Cloudera][ThriftExtension] (9) Error occurred while authenticating via SASL. Error details: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Internal credentials cache error)

my work on hdfs data at REST encryption.i m using hybrid encryption based on python language in ubuntu vmso plz guide me how we use this hybrid encryption based on python for provide security of data at REST

https://docs.cloudera.com/runtime/7.2.18/atlas-import-utility/topics/atlas-bulk-migration-import-hive-metadata.htmlI followed this document import-hive.sh errors out saying failing to parse the arguments.  Looking for any alternate options to bulk import database (having thousands of tables) from hive to atlas as it is going to be time consuming task, and may fail due to time out issue. is there export and import option? pls advise, currently its a challenge to import a database with thousands of tables in it. 

Good morning. I am attempting to automate a deployment of nifi.  I set all the properties I wished to dynamically configure as Parameters and uploaded the parameters through the API.  This included both normal and sensitive parameters.  This worked great.  Next I attempted to upload a flow into a process group.  All normal parameters worked great.  However all sensitive parameters did not get set.  For example. given a parameter my.password, it was set in the flow property as #{my.password}.  So entering it from the GUI one at a time works great, but Every sensitive property set by the import is incorrect.  Is there a fix for this or another approach to be able to set the sensitive properties through the api.  I would prefer not to call each processor which has sensitive properties individually. ThanksRich

Hi all - I am trying to create a Hive table from nested JSON parquet data.  The problem is one object is dynamic and I want to store it as a string since it changes. Example JSON: {"level1": {"level2": {"key1": "someString","level3": {"level4": {"key2": 1234,"level5": [{"this": 1,"changes": false,"each": 12345,"item": 0,},{"something": 12345,"new": true,"here": [123]}]}}}}} Here is what I have tried:CREATE EXTERNAL TABLE IF NOT EXISTS my_table (level1 STRUCT<,level2 STRUCT<,key1 STRING,level3 STRUCT<,level4 STRUCT<,key2 BIGINT,level5 ARRAY<STRING>>>>>) STORED AS PARQUETLOCATION '/my/json/parquet/'TBLPROPERTIES ("parquet.compression"="SNAPPY"); This will successfully create the table.  I can query down all levels except "level5" falls apart.  Is there a way I can cast the array in level5 into a string since it always changes??   FYI I have tried ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe', but this

Hello there,I learned about nifi counters, but when I open my nifi instance and click on the counters button in the main menu, it just opens an empty list. Same when I call nifi-api/counters. It is just an empty list. I do have processors and processgroups running.So does anyone have an explanation why that is? I use nifi 1.23.2 on my local machine. Thanks in advance!

 nifi.python.command=python3nifi.flow.configuration.file=../data/flow.json.gznifi.flow.configuration.archive.enabled=falsenifi.flow.configuration.archive.dir=../data/archive/nifi.flow.configuration.archive.max.time=30 daysnifi.flow.configuration.archive.max.storage=500 MBnifi.flow.configuration.archive.max.count=nifi.flowcontroller.autoResumeState=truenifi.flowcontroller.graceful.shutdown.period=10 secnifi.flowservice.writedelay.interval=500 msnifi.administrative.yield.duration=30 secnifi.bored.yield.duration=10 millisnifi.queue.backpressure.count=10000nifi.queue.backpressure.size=1 GBnifi.authorizer.configuration.file=./conf/authorizers.xmlnifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xmlnifi.templates.directory=../data/templatesnifi.ui.banner.text=nifi.ui.autorefresh.interval=30 secnifi.nar.library.directory=./libnifi.nar.library.autoload.directory=./extensionsnifi.nar.working.directory=./work/nar/nifi.documentation.working.directory=./work/docs

Hello All,      When I try to restart or stop or start any services, the background operations in Ambari does not list the operations. It shows no services to show. It was working fine earlier and now all the previous operations are also missing in the list. Ambari - 2.7.3 HDP 3.1.0

Hello,We have custom syslog Cisco messages in the following format <117>2024-07-23T14:09:56Z XXXXXXXXX : %FTD-5-430003: EventPriority: Low, DeviceUUID: xxxxxxxxxxxxx, InstanceID: 2, FirstPacketSecond: 2024-07-23T14:09:56Z, ConnectionID: 32322, AccessControlRuleAction: Allow, SrcIP: A.B.C.D, DstIP: A.B.C.D, SrcPort: 42308, DstPort: 24224, Protocol: tcp, IngressInterface: XXX, EgressInterface: XXX, IngressZone: XXX, EgressZone: YYY, IngressVRF: Global, EgressVRF: Global, ACPolicy: AAA-BBB, We want to use NIFI to format to CEF ( common event format ). Any help which processors to use, please?

Hi,I’m experiencing an issue when multiple queries are executed simultaneously. When I run the query by itself, it succeeds. However, when multiple queries run concurrently, I encounter problems.Does anyone know how I should configure Impala to avoid this issue? I’ve attached the profile of the failed query.My mem_limit is set to 60GB for 5 workers and 90GB for the other 8.Thanks,

Hello, I am trying to replicate ranger policies from one instance of ranger to another. I know there is an api call called - /plugins/policies/importPoliciesFromFile I found that's good for the initial migration of all of them but when it comes to the inserts, updates, and deletes after that what's the best way to handle that? If I use the same call, it will overwrite the previous policies which isn't what I want. 

hello Cloudera Community,the process of creating a user and adding them to groups via the nifi WebUI mode works normally.i need to create a user and add them to a group directly using the NiFi API (post, put, get).how do I make this process 100% functional?

Hello,I am using Apache Knox and am having trouble with some rewrites rules, to avoid those is there a way to tell Knox to just redirect the user to the service he asked for instead of proxying the requests on the KnoxUI on port :8443 ?Thanks for your answers

Hello, a customer has a CDH Express 6.3.1 setup and:- It appears that suddenly one (and just one!) of the hosts listed in Hosts --> All Hosts is now showing as having a complete FQDN. It never used to have it (I have screenshots of previous configurations where ALL hosts in the list are shown with just the hostname and no domain)- Also, there are Stale Configurations waiting to be applied. If I click on them, they are related to a number of Client Configuration files waiting to be deployed, and sure enough ALL the lines in ALL the files are showing a single difference for that particular host!Now, I don't know what happened here. Nobody is reporting a CM configuration change made by them.I don't have History/Rollback capabilities on this cluster, as it's an Express Edition. Also, I have no Audits as well to check what (if any?) parameters have been recently changedI just know that Sysadmins recently performed a change in FQDN names on those hosts at OS level, because a monitoring to

Hello,I am securing my Spark history UI with Apache knox and am having trouble with the proxied SparkhistoryUI as mentionned in this thread: https://lists.apache.org/thread/xwzo4g6qk2hqmn62t6p162b8t6l3hldqI managed to edit the rewrite rules to make the css load on the proxied sparkhistory UI but I am now facing another problem I don' t manage to overcome.When the pages loads the js code triggers a jquery call to the api: `$.getJSON(uiRoot + "/api/v1/applications", appParams`The problem is that when Knox is proxying Spark history UI this uiRoot is not the right one it's https://knox-host:8443/api/v1/applications instead of https://spark-ui-host:18080/api/v1/applicationsApache Knox rewrite rules don't seem to be able to fix that so this is why I am sending a message here to see if any of you have a better idea, maybe something related to the proxy setup.I tried to set `spark.ui.proxyBase` to the Spark History UI URL and port so that Knox tries to reach it butI end up with a 302 error `Cr

We are using Hortonworks (HDP-2.6.5.0) and by default, inter cluster communication uses non-SSL ports in this version of HDP. Is there any way to achieve SSL based communications within cluster for all the possible services and nodes in a running HDP cluster ? Any help you can provide in this matter will be highly appreciated.

Hi,It seems like the new version of Apache Nifi 2.0.0 M3 was just released. Per the release notes there are some changes to how python extension are packaged and changes to the UI as well . @pvillard Are you going to make new video to discuss those changes and other important ones? It will be highly appreciated. Im planning to download and start testing but having a video from someone like you would be helpful to set the expectation and avoid running into any issues.Thanks 

ERROR [python-log-97916] py4j.java_gateway There was an exception while executing the Python Proxy on the Python Side.Traceback (most recent call last):  File "C:\Nifi\nifi-2.0.0-M4\python\framework\py4j\java_gateway.py", line 2466, in _call_proxy    return_value = getattr(self.pool[obj_id], method)(*params)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "C:\Nifi\nifi-2.0.0-M4\.\python\framework\Controller.py", line 75, in createProcessor    processor = processorClass(jvm=self.gateway.jvm)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^TypeError: WriteHelloWorld.__init__() got an unexpected keyword argument 'jvm'2024-07-19 11:08:55,680 ERROR [Initialize Python Processor ca359d33-0190-1000-2b29-ff0f5ad6577a (WriteHelloWorld)] o.a.n.py4j.StandardPythonProcessorBridge Failed to load code for Python Processor ca359d33-0190-1000-2b29-ff0f5ad6577a (

We have a requirement to extract data from unstructured logs data and capture the results in a custom format. We have implemented the solution using the ExtractText and ReplaceText Processors but we are not getting the expected results,Please find the details of the input data, current implementation, current output, and expected output format below and help us with your expertise to improvise this using the NiFi Processors.Input File Content : <13>Jul 18 11:39:11 test234104.test.gmail.ae AgentDevice=WindowsLog    AgentLogFile=Security   PluginVersion=100.3.1.22  Source=Microsoft-Windows-Security-Auditing      Computer=mycomputer1 OriginatingComputer=102.123.33.1    User= Domain=     EventID=4688 EventIDCode=4688  EventType=8 EventCategory=13312{122}Aug 18 11:39:11 test234104.test.gmail.ae  PluginVersion=200.3.1.22  Source=Microsoft-Windows-Security-Auditing AgentDevice=WindowsLog    AgentLogFile=Security     Computer=mycomputer2.gmail.com OriginatingComputer=125.123.33.1   

Hi, I am using the free version of Cloudera for learning purpose, I have a need to upgrade the Spark version running in Cloudera to 2.4.0 to allow some functions work, only for me to be asked for password to authenticate to the site to be able to download and upgrade tom the newer version of Spark.Please can someone advise me how to go about this? It's quite urgent for me. I am not able to create a case with their support helpdesk as well.  Regards,Lewi      

Dear all, I have installed docker version of Cloudera QuickStart,  but file browser is missing, even when. it is totally green.I follow the guidelines in https://docs.cloudera.com/cdp-private-cloud-base/7.1.3/concepts/topics/sdx-workflow-2--view-hdfs-directory-structure-of-compute-clusters.htmlThe file browser should there,  but does not show up right now.

I am trying to install cloudera-manager offline and the above problem occurs when I install cloudera-manager-agent package.Installing cloudera-manager-agent packs...BEGIN yum --disablerepo=* --enablerepo=cloudera* list installed cloudera-manager-agentLoaded plugins：fastestmirror, langpacksInstalled packagescloudera-manager-agent.x86_64 6.3.1-1466458.el7 installedEND (0)BEGIN echo oracle-j2sdk1.8 cloudera-manager-agent cloudera-manager-daemons | grep cloudera-manager-agentEND (0)oracle-j2sdk1.8 cloudera-manager-agent cloudera-manager-daemonsBEGIN yum --disablerepo=* --enablerepo=cloudera* info cloudera-manager-agentLoaded plugins: fastestmirror, langpacksLoading mirror speeds from cached hostfileOne of the configured repositories failed (Unknown),and yum doesn't have enough cached data to continue. At this point the onlysafe thing yum can do is fail. There are a few ways to work "fix" this:1. Contact the upstream for the repository and get them to fix the problem.2. Reconfigure the base

Hi,I am getting this error while accessing one environment in Cloudera director... there is no issue with other environments.Also, I could not see the option "Update Environment Credentails" when I click choose the environment. I have tried to use the rest api to update the credentials but it didnt work. curl -u admin -X POST http://sampleweb.net:7189/api/d6.2/environments/testenv/provider/credentials -H 'Content-Type: application/json' -d '{"Access key ID":"username","Secret access key":"password"}'Enter host password for user 'admin':{"timestamp" : 1695055508171,"status" : 405,"error" : "Method Not Allowed","message" : "Request method 'POST' not supported","path" : "/api/d6.2/environments/testenv/provider/credentials"