Q&A

Hi everyoneWhen installing cloudera-scm-agent, the installation failed due to heartbeat check failure. However, the backend logs show no errors and even output INFO-level logs indicating successful heartbeat detection. What could be the root cause?agent.loginstalling logserver.log

Hi,I'm looking for a way to implement a process group level counter that I can read and update either with a processor, including ExecuteScript / ExecuteGroovyScript.I've looked at and tried:1. Counters: I can create one, set the value and increment it, but not read the value within a processor, that I've found as of yet.2. Parameter Context parameters: I can read the value, but not increment or reset it. It seems to be static once it's defined.Is there something I've missed with the above, or does anyone have another suggestion please?Thank you!

Hi All,I have created my processor group in a local instance of NiFi 2.0.0-M4 and I have downloaded the json definition file and also I can able to import that json manually to a target nifi system (production), but my ultimate goal is to deploy this json using NiFi RestAPI and I have accomplished the following tasks1. Generate the NiFi Token2. Create a ProcessorGroup to the target NiFi systemBut I could not find a way to import the json definition file through REST API and when I search on the apache NiFi documentation of API and in could not find any proper api call for NiFi version 2.0.0-M4 but there is only for version 1.28 supported Api endpoints for template concepts but as of NiFi version 2.0.0-M4 it was changed from xml template into json definition concept. Now how do I import the json file to the target system processGroup which i created through rest api.Please anyone can give your suggestion asap.ThanksVijay.AX

Does anyone here can share if we can perform stop and start of roles using commandline in Kafka backend if the Cloudera Manager is down? 

Hi,I'm still trying to get this to work:https://community.cloudera.com/t5/Support-Questions/Process-Group-level-counter/td-p/402648I have versions NiFi 2.0.0 and 2.2.0 at my disposal, and have tried to get simple examples working that I can then implement in my flow. I'm having difficulties as all the examples I find are with older versions of NiFi.First, Distributed Server and Client Services have been renamed:DistributedMapCacheServer has been renamed to MapCacheServerDistributedSetCacheServer has been renamed to SetCacheServerDistributedMapCacheClientService has been renamed to MapCacheClientService DistributedSetCacheClientService has been renamed to SetCacheClientServiceI've mostly gotten PutDistributedMapCache and FetchDistributedMapCache working as per the older example here:https://community.cloudera.com/t5/Support-Questions/how-to-put-data-in-PutDistributedMapCache/td-p/171803andhttps://www.youtube.com/watch?v=wjis28NrF90 In the Controller Settings under Management Controller

Hi,I have a use case where I have to read the JSON files from "/data/json_files/" location with schema enforced.For the completeness we want to mark the invalid records. The invalid records may be the ones where the mandatory field/s are null, data type mismatch or invalid json itself.I have tried below but nothing worked as of now. It would be nice if someone has already this use case and a solution for it or may be knowledgeable in this area. Example Schema:schema = StructType(    [        StructField(            "meta",            StructType(                [                    StructField("id", StringType(), False),                    StructField("timestamp", TimestampType(), False),                   

hola, estoy teniendo problemas con la descarga de CLOUDERA, alguien me puede dar las indicaciones por favor gracias

Hi All,This link has queries to get the custom utilization report -- https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/monitoring-and-diagnostics/topics/cm-cluster-utilization-report-metrics-queries.htmlIf I need 6 months data for each CPU utilization, memory utilization -- do we have a date column to specify the required date range.Where to execute these queries in CDSW ?.There are no table name specified in few queries. Will those queries fetch results ?Thank You.

Hello Everyone,I am part of the MSSP team, and we have received a request to support Cloudera Navigator. Based on the documentation, it appears to be part of the Cloudera Security Suite. However, I also came across other products like Atlas and Sentra.Could you please confirm if Navigator is still included in the Security Suite? Additionally, under which product does it fall, or is it available separately? Any specific details about Navigator, as well as insights into the suite or product structure, would be greatly appreciated.Thank you!

I am getting the following error. I have populated nifi.properties and bootstrap.conf using configmaps as mentioned below-------2025-02-11 06:48:50,240 ERROR [main] org.apache.nifi.runtime.Application Start Server failedjava.lang.IllegalStateException: Application Properties loading failedat org.apache.nifi.runtime.PropertiesProvider.readProperties(PropertiesProvider.java:65)at org.apache.nifi.runtime.Application.startServer(Application.java:121)at org.apache.nifi.runtime.Application.run(Application.java:78)at org.apache.nifi.runtime.Application.run(Application.java:60)at org.apache.nifi.NiFi.main(NiFi.java:42)Caused by: java.lang.reflect.InvocationTargetException: nullat java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:115)at java.base/java.lang.reflect.Method.invoke(Method.java:580)at org.apache.nifi.runtime.PropertiesProvider.readProperties(PropertiesProvider.java:63)... 4 common frames omittedCaused by: java.lang.NullPointerException:

i have issue in connecting external zookeeper and nifi cluster , i have 3 nodes like zookeepernode1 zookeepernode2 zookeepernode3 and nifinode1 and nifinode2nifiuser@zookeepernode3:~$ /opt/zookeeper/bin/zkServer.sh status/usr/bin/javaZooKeeper JMX enabled by defaultUsing config: /opt/zookeeper/bin/../conf/zoo.cfgmyid could not be determined, will not able to locate clientPort in the server configs.Client port found: 2181. Client address: localhost. Client SSL: false.Mode: followerroot@zookeepernode1:~# /opt/zookeeper/bin/zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: /opt/zookeeper/bin/../conf/zoo.cfgClient port not found in static config file. Looking in dynamic config file.grep: : No such file or directoryClient port not found in the server configsClient port found: 2181. Client address: localhost. Client SSL: false.Mode: followerroot@zookeepernode2:/opt/zookeeper/logs# /opt/zookeeper/bin/zkServer.sh statusZooKeeper JMX enabled by defaultUsing config: /opt/

HI Guys, I am having an issue running an Spark 3 jobs: Driver log: ERROR: cluster:YarnClusterScheduler: [dispatcher-CoarseGrainedScheduler] : Lost executor 1 on host_name: Unable to create executor due to unable to register with external shuffle service due to: java.lang.IllegalException: Unknown message type: -22Executor log: ERROR: Exception while bootstrapping client in 3 ms. Java.lang.RuntimeException: java.lang.IllegalException: Unknown message type: -22Executor making 3 attempts with 5 second waiting period, then exiting. From my research, I found recommendation on increasing the shuffle register timeout. Any recommendation is appreciated. 

Hello Cloudera folks,Good Day! I'm having a issue when i tried to connect the nifi putMarkLogic processor marklogic db, when I checked the roles in marklogic end the rest-writer and rest-read are already checked also but still encounter the issue below.from marklogic logs."GET /v1/internal/forestinfo?database=data-hub-STAGING HTTP/1.1" 404 0 - "okhttp/3.10.0"from nifi popup error. if you have any thoughts it's very helpful to us.Thanks,Vin 

Good day,There are NiFi processors that store state in memory, one example being the DetectDuplicate processor, which "remembers" FlowFiles based on a specific attribute. The duration for which this memory is retained is determined by the Age Off Duration parameter. I would like to know the default value for this parameter and whether it is possible to configure it to retain the memory indefinitely (i.e., with no expiration). Currently, I am using a value of '365000 days'.Additionally, I would like to know if there is a way to preserve the detected values even after a NiFi restart.

Dear Sirs, I have the following problem with my nifi cluster, it takes a long time to run a flow and when it does it presents the following error:Failed to properly initialize Processor. If still scheduled to run,NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed.Failure is due to org.apache.nifi.processor.exception.TerminatedTaskException: org.apache.nifi.processor.exception.TerminatedTaskExceptionNifi version: 1.11.4Zookeeper: external, apache-zookeeper-3.6.3I performed the following actions seen in this previous post:https://community.cloudera.com/t5/Support-Questions/Apache-Nifi-ListSFTP-Processor-Failed-to-properly-initialize/m-p/366854My state-managemnet.<cluster-provider><id>zk-provider</id><class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class><property name="Connect String">node-1:2181,node-2:2181,node-3:2181</property><property name="R

Hi all,I'm setting up a CML session with 64GB of RAM and 4 CPUs, then I set up a PySpark session with these configurationsspark = SparkSession.builder \.appName("OptimizedSparkSession") \.master("local[*]") \.config("spark.sql.execution.arrow.pyspark.enabled", "false") \.config("spark.sql.shuffle.partitions", "400") \.config("spark.executor.memory", "32g") \.config('spark.executor.instance', '10') \.config("spark.executor.cores", "4") \.config("spark.driver.memory", "32g") \.config("spark.network.timeout", "800s") \.config("spark.executor.heartbeatInterval", "200s") \.config("spark.driver.maxResultSize", "64g") \.getOrCreate()The transformations I do are quite simple. Basically loading two tables, one with entities of interest and another containing the additional features and other entities, so I join the second one to the first to keep only the records that are needed.table1 = spark.sql(f"""SELECT DISTINCT key1, key2FROM db1.table1""")table2 = spark.sql(f"""SELECT key1, key2, col1, c

Hello everyone,I have a question regarding to data provenance. I did read NiFi document about provenance repository that can store the provenance data, but I want to this provenance data in database for long-term. How can I do that in NiFi 2.2.0?

I am running Ranger Admin 2.4.0. Right now I have it synced with LDAP and am able to login to the Ranger Admin UI via my LDAP username and password. When I try upgrading to Ranger Admin 2.5.0 using the same configurations I am getting these errors in the logs.catalina.out - java.lang.RuntimeException: Failed to create user drew.nicolette in x_portal_user table. retryingat org.apache.ranger.biz.XUserMgr$ExternalUserCreator.createExternalUser(XUserMgr.java:3314)at org.apache.ranger.biz.XUserMgr$ExternalUserCreator.run(XUserMgr.java:3288)at org.apache.ranger.common.db.RangerTransactionSynchronizationAdapter.addRunnable(RangerTransactionSynchronizationAdapter.java:136)at org.apache.ranger.common.db.RangerTransactionSynchronizationAdapter.executeOnTransactionCommit(RangerTransactionSynchronizationAdapter.java:82)at org.apache.ranger.biz.XUserMgr.createServiceConfigUser(XUserMgr.java:2601)at org.apache.ranger.biz.XUserMgr$$FastClassBySpringCGLIB$$57c6d473.invoke(<generated>)at org

HiMy one liner question:How can we ensure that the DatabaseTableSchemaRegistry recognizes changes in the table structure and updates the corresponding table schema definition?Details:Our PutDatabaseRecord processors' readers are configured to use the DatabaseTableSchemaRegistry to parse incoming JSON. We have observed multiple instances where the NiFi flow fails when the target table structure is altered, such as when columns are added or removed.When new columns are added to the target table, the PutDatabaseRecord reader does not recognize the change and continues to create insert statements without the new columns. We are unsure what may have resolved the issue, but in some instances, it was automatically resolved after a day. We suspect there may be an internal cache that is rebuilt periodically. We have no clear solution for this issue and would greatly appreciate any assistance regarding it.Our NiFi system is a 3-node cluster (version 2.1) that uses Kubernetes-based leader electio

Hi everyone,I’m working on tracking and monitoring integration executions in Apache NiFi . I want to achieve the following:Display all the configured integrations available to users so they can check their execution status.Show all executions of a specific integration with details such as:Start timeEnd timeDurationStatus (success/failure)Total number of sub-items processed, including success and failure counts.Track all integrations that happened as part of a single execution and display:Start timeEnd timeDurationStatusAny relevant messages or error detailsI’d like to know the best way to implement this in NiFi. Are there recommended processors, templates, or logging strategies that can help achieve this? Should I be using provenance data, custom logging, or NiFi's built-in monitoring tools?Any insights, examples, or best practices would be greatly appreciated!Thanks in advance!

I have deployed an Apache NiFi instance (v2.1.0) and configured Azure AD authentication by following this guide:NiFi Authentication with Azure AD.Authentication is working correctly—I can log in, configure multiple users, and assign policies.Now, I have another application that also uses Azure AD authentication. I am trying to implement a scenario where a user first logs into my application, then gets redirected to NiFi using the same token generated for my application. However, this approach is failing, and I am encountering an error.I have attached the log below. Can anyone help me troubleshoot this issue?   2025-02-21 06:26:47,841 INFO [NiFi Web Server-117] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[kLM-4Eld2dZnX_dD3iB0df2fTvXQxa1J2ffdLoK-ozas], groups[] does not have permission to access the requested resource. Unable to view Process Group with ID fa3a74d9-0194-1000-85b2-d20263f3e54b. Returning Forbidden response.2025-02-21 06:26:47,841 DEBUG [NiFi Web Server-1

Urgent Help NeededI use open-source NiFi and currently handle deployments and upgrades manually. With multiple environments, this process is becoming time-consuming and inefficient.Current Setup:400 CPUs60 Nodes200 UsersRequirements:Automated (CI/CD) NiFi & Data Flow ManagementScheduled Deployments with History & Rollback24x7 Reliable Support Partner Despite extensive research, I have not found a single tool that meets all these needs. Any recommendations?

Hello, i am using ScriptedFilterRecord to filter empty records for one of the record filed. It is working as expected but when i try to push same records to QueryRecord to get list down those ids. NIFi push different set of records to query record processor. Please refer attached screenshot. Thank you.

Hi,I am trying to prepare a report listing Client IP addresses connecting to various Cloudera services. I know in Ranger Admin UI -> Audits -> Access, I can see this information. However, because of the huge list of records, it is not practical to copy each page from here. Hence, below are options i can think of to fetch the data:1. Ranger database: I am unsure which table in ranger DB stores this information from where Ranger Admin UI gets the data. 2. Solr: For our Ranger instance, source is configured as Solr. However, for some reason I am unable to query Solr via Solr UI. Possibly it could be coz of some permissions issues, as when attempted to get this data using Solr API, i get authentication error. Unsure where should I check for the permissions as in Solr Policy in Ranger, I have given all the reasonable permissions that I could think of to myself. Still no luck.Please suggestThankssnm1523

Hi,We are using Cloudera Private Cloud, and our customer wants to process SAS files using NiFi. However, I couldn't find a built-in NiFi processor that supports this.Is there a known solution or workaround for handling SAS files in NiFi? Any suggestions would be greatly appreciated.Thanks