Q&A

Hi,When I try to download custom parcel on Cloudera Manager, it cannot do it anymore.Older parcels can still be downloaded.We have a workaround on which we upload new parcels to bypass the issue.Following the error returned by LOG when we try to download custom parcel on CM.     Unable to dowload size for parcel : name.parcel     java.lang.NumberFormatException: null          at java.lang.Long.parseLong(Long.java:552)          at java.lang.Long.parseLong(Long.java:631)Do you have any idea to solve it ?

org.apache.hadoop.hdfs.server.datanode.DataNode: Unsuccessfully sent block report 0x687e2eff1ccff4e2 with lease ID 0x2e42f5718005b5ef to namenode: hadoop-standby/:8020, containing 1 storage report(s), of which we sent 0. The reports had 659503 total blocks and used 0 RPC(s). This took 83 msecs to generate and 56 msecs for RPC and NN processing. Got back no commands.2025-01-02 19:28:31,810 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: RemoteException in offerServiceorg.apache.hadoop.ipc.RemoteException(java.io.IOException): java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.runBlockOp(BlockManager.java:5558)at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1651)at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:182)at org.apache.hadoop.hdfs.protocol.proto.DatanodeP

I am using Nifi 1.23.2. I am using ListSFTP with Tracking Entities listing strategy. I'm using RedisDistributedMapCacheClientService as my Entity Tracking State Cache. The initial listing gets all files (around 500k files). The processor appears to run every minute as scheduled, but it never finds any more files. If I copy the processor and run it again, the initial listing will again get all files (now more than 500k files, since files have been added). But it never gets any more files.What am I missing?

while creating cluster , the following step succeeded then the following step failedI checked the logs:Installation completed successfully. View/tmp/scm_prepare_node.UTzwtsGZusing SSH_CLIENT to get the SCM hostname: 192.168.100.64 44748 22opening logging file descriptorStarting installation script...Acquiring installation lock...BEGIN flock 4END (0)Detecting root privileges...effective UID is 0Detecting distribution...BEGIN grep Tikanga /etc/redhat-releaseEND (1)BEGIN grep 'Scientific Linux release 5' /etc/redhat-releaseEND (1)BEGIN grep Santiago /etc/redhat-releaseEND (1)BEGIN grep 'CentOS Linux release 6' /etc/redhat-releaseEND (1)BEGIN grep 'CentOS release 6' /etc/redhat-releaseEND (1)BEGIN grep 'Scientific Linux release 6' /etc/redhat-releaseEND (1)BEGIN grep Maipo /etc/redhat-releaseEND (1)BEGIN grep Ootpa /etc/redhat-releaseEND (0)/etc/redhat-release ==> RHEL 8Detecting Cloudera Manager Server...BEGIN yum -y install bind-utilsRed Hat Enterprise Linux release 8.1 (Oot

I have a use case scenario coming up next week where I will need to process some Parquet to CSV.   I created a few demos with nifi 1.10 but unfortunately it is not possible to use this version of NiFi on the customers environment.   I know I can still satisfy the actions I need to take with 1.9 without the new 1.10 features for Parquet, but I know these new features would be more efficient.   Is it possible to drop the required artifacts into 1.9 and achieve Parquet Reader functionality?

Hello. I am using nifi cluster with 3 nodes. I tried to Impala Query with query, it was success.   But when I tried to use ListSFTP and FetchSFTP, It occurs Error. The Error was   ERROR [Timer-Driven Process Thread-6] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=c486c6ce-016e-1000-0000-000037351532] Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.IllegalArgumentException: A HostProvider may not be empty!: java.lang.IllegalArgumentException: A HostProvider may not be empty!java.lang.IllegalArgumentException: A HostProvider may not be empty!at org.apache.zookeeper.client.StaticHostProvider.init(StaticHostProvider.java:136)at org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:87)at org.apache.zookeeper.ZooKeeper.createDefaultHostProvider(ZooKeeper.java:1312)at org.apache.zooke

Hi Team,Good DayWe have a requirement to start the CDSW AI ML job from the API call. We are currently using Cloudera Machine Learning legacy REST API (Version V1) as per this link:https://docs.cloudera.com/machine-learning/1.5.4/jobs-pipelines/topics/ml-rest-apis.htmlcurl -v -XPOST http://cdsw.example.com/api/v1/<path_to_job> --user "<LEGACY_API_KEY>:"Ex: curl -XPOST https://***/api/v1/projects/bd***/s***/jobs/2/start  --user "***" --header "Content-type: application/json" --data "{}"We are working to upgrade the CDSW to CML and as part of this job rewrite exercise we also thought to use Cloudera Machine Learning REST API v2 (Version V2) for the curl command to Start the JobIn the below URL, We don't see a method to start the jobhttps://docs.cloudera.com/machine-learning/1.5.4/rest-api-reference/index.html#examples-CMLService-restartApplication-0-pythonPlease share your expertise on the below open queries :1) When is V1 getting depreciated? Is

Recently, we have started experiencing issues with our NiFi commits in the NiFi Registry failing due to read timeout exceptions. The error message displayed is "Failed to register flow with Flow Registry due to Read timed out." However, despite this error, the commit in the NiFi Registry is actually successful. We are using the Git flow provider on the NiFi Registry side, and I can see a commit after every attempt, despite NiFi showing a timeout error.Has anyone else encountered this issue, or does anyone have suggestions on how to resolve it? Any help would be greatly appreciated. Thanks in advance!Here are the complete logs of exception:2024-05-05 06:27:03,108 WARN [NiFi Web Server-599530] o.a.n.w.a.config.NiFiCoreExceptionMapper org.apache.nifi.web.NiFiCoreException: Failed to register flow with Flow Registry due to Read timed out. Returning Conflict response.org.apache.nifi.web.NiFiCoreException: Failed to register flow with Flow Registry due to Read timed outat org.apache.nifi.web

I'm currently facing an issue with implementing OpenID Connect (OIDC) authentication in Apache NiFi. And I am putting my NIFI in the kubernets. so, I am with the NIFI containers.Error Overview:I receive the following error in the NiFi logs:2025-01-02 10:29:05,167 INFO [NiFi Web Server-52] o.a.n.w.s.NiFiAuthenticationFilter Authentication Started 172.24.0.3 [<oncloudtemuser@sossourabh7687gmail.onmicrosoft.com><CN=172.24.0.3, OU=NIFI, O=NIFI, L=HYDRABAD, ST=TELANGANA, C=IN>] GET https://172.24.0.3:8443/nifi-api/flow/current-user2025-01-02 10:29:05,173 WARN [NiFi Web Server-52] o.a.n.w.s.NiFiAuthenticationFilter Authentication Failed 172.24.0.3 GET https://172.24.0.3:8443/nifi-api/flow/current-user [Untrusted proxy CN=172.24.0.3, OU=NIFI, O=NIFI, L=HYDRABAD, ST=TELANGANA, C=IN]2025-01-02 10:29:05,207 ERROR [NiFi Web Server-190] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.io.UncheckedIOException: Read Current User Entity failed.

Hello,In NiFi Toolkit 2.0, the nifi create-reg-client command does not provide a parameter to specify the registry client type (GitLabFlowRegistryClient 2.0.0 ). Are there any workarounds for setting up a registry client in the apache/nifi:2.0 Docker image for a CI pipeline?Best regards Wojtek  

My ML runtime comes preinstalled with python packages. e.g. databricks-cli, mlflow-CML-plugin etc. Is there any way to avoid installing them so that I can have a clean environment to start with?

Hi allWe are looking to enable user authentication using LDAP and access control enforcement using few LDAP groups. But we were suggested by our Cloudera admin that enabling LDAP group for user auth and ACL will enable LDAP auth even for Cloudera Service Ids.Is this true? Please reply

Hello, I'm planning to implement new cluster + Data services, I'm planning to use below versionsCM 7.11.3 CHF 11 Cloudera Runtime 7.1.9 SP1Cloudera Data services 1.5.4 SP1I have checked the compatibility matrix all the releases are compatible with each other, but when I read this article I can't find 7.1.9 SP1https://docs.cloudera.com/cdp-private-cloud-data-services/1.5.4/release-notes/topics/cdppvc-service-packs-1-5-4-SP1.html#ariaid-title2 Any one tried this path? 

Our customer needs to renew the license they purchased from HPE, but when following the instructions to access the link hpe.cloudera.com, it is not responding. We are concerned that this will affect the system renewal. Does anyone have a way to activate a license purchased from a third party?

We're still facing issues with RIT, and now there's an additional frustrating problem: several tables are not coming online.In the HBase meta table, the state is set to "\x08\x02", which indicates the table is "DISABLING" The region state is CLOSED, but modifying these states hasn't resolved the issue.I've tried addressing this by using put statements and the hbck2 tool, but the problem persists. The current HBase version is 2.0.2.

Hello,We have a nifi registry running under AWS ECS and integrated with GIT repo as persistence provider, we have to use GIT credentials for remote authentication username/password. Is there any way to have this done if we set an IAM role with access to that GIT repo (AWS CodeCommit) for the ECS/EC2 that hosting the NIFI application?We don't want to use any GIT credentials Could you please advise?

Hello sir,I am using the InvokeHTTP processor in Apache NiFi to call an API. I am writing the response using the PutFile processor. When the API returns a 200 status code, it works fine and writes the response. However, when the API returns a 400 or 500 status code, it writes the original input file instead. I want to capture and write the actual response body for 400 or 500 status codes instead of the input file. here is process:here is InvokeHTTP processor:Here is putfile processor:if status 200 it write the response . I want write all response from api instead of input file if it has any status from api.Please help me

Hi,I am trying to understand what is the optimal development lifecycle of Apache Nifi flows that should be ported from dev to qa to prod.I read some articles such as https://community.cloudera.com/t5/Support-Questions/Hortonworks-HDF-Apache-Nifi-how-to-deploy-a-flow-from-dev/m-p/158288  and I usually use export/import template in my flows but it's having some big drawback than I am not sure how they can be overcomed.In particular the issue is that the template/flow definition contains inside "code" and "logic" information together with "environment configurations" that shouldn't be exported and imported. With that I mean information such as number of threads per processor that should not be exported because obviously usualy prod environments have more resources so can support more threads and cron configurations that obviously can change between a dev/qa and prod environment.It seems like Apache Nifi was not designed well following the 12 factor principles https://1

When I install hue with cloudera manager it stops, when searching the log it shows me a permissions error. ======log cloudera manager======2021-04-19 10:30:55,858 INFO CommandPusher-1:com.cloudera.server.cmf.CommandPusherThread: Acquired lease lock on DbCommand:15463385272021-04-19 10:30:55,860 INFO CommandPusher-1:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'HueTestDatabaseConnection'2021-04-19 10:30:55,861 INFO CommandPusher-1:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 12021-04-19 10:30:55,861 INFO CommandPusher-1:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: + '[' syncdb = is_db_alive ']'+ '[' ldaptest = is_db_alive ']'+ exec /opt/cloudera/parcels/CDH-7.1.5-1.cdh7.1.5.p0.7431829/lib/hue/build/env/bin/hue is_db_aliveTraceback (most recent call last):File "/opt/cloudera/parcels/CDH-7.1.5-1.cdh7.1.5.p0.7431829/lib/hue/build/env/bin/hue", line 9, in <module>from pkg_resources import load_entry_pointFile

I have a processor group PG with below processors for exampleA -> B -> C       D-> E->FProcessor C  (Execute Stream Command) runs a Python code using multi threads  - generates filesProcessor D ( List File) will list the files generated by above processor C and transforms and processes  to F (PutElasticSearchJSON)Have a requirement to run this  processor group for one date at a time, start execution for next date only after capturing the status of previous date execution status (success/failure)I am ok  using Python/Shell Script. Guide me how can this be achieved.

Hello experts - I am using nifi 1.25 in Kubernetes.I have flow where it updates variable registry of a processor group .Invoking below url for the same - <baseNiFiUrl>/process-groups/${ProcessGroupId}/variable-registry/update-requestsIt works for some time and once in a while it starts hanging, where it is unable to update the variable registry at and status stuck at processor stopping.Even I tried running update variable registry after manually stopping the specific processor of processor group where updating variable is being referenced.Here is the below response I get for var registry update status, it does not change:{"request" : {"requestId" : "77422664-ad85-40d7-9464-4df7df1fe149","uri" : "https://nifi-0:9091/nifi-api/process-groups/fec6c2c9-ee27-31ea-6354-851d74803023/variable-registry/77422664-ad85-40d7-9464-4df7df1fe149","submissionTime" : "12/20/2024 09:43:32.941 UTC","lastUpdated" : "12/20/2024 09:43:32.941 UTC","complete" : false,"percentCompleted" : 0,"upda

 I get "Unexpected error. Unable to verify database connection" I checked cloudera-scm-server.log file, found the following ERROR:7345 MainThread redactor     ERROR    Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/57-HUE-test-db-connection------Details:[root@prw-cld-trial cloudera-scm-server]# pwd/var/log/cloudera-scm-server# more cloudera-scm-server.log2024-12-20 14:20:06,319 INFO CommandPusher-1:com.cloudera.server.cmf.CommandPusherThread: Acquired lease lock on DbCommand:812024-12-20 14:20:06,446 INFO CommandPusher-1:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'HueTestDatabaseConnection'2024-12-20 14:20:06,471 INFO CommandPusher-1:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 12024-12-20 14:20:06,472 INFO CommandPusher-1:com.cloudera.cmf.service.AbstractDbConnectionTestCommand:   from cryptography imp

Mates,Need help or references of how to configure REST API on NiFi 2.0, unable to generate token as we were doing on 1.x.  Have NiFi on plan Linux not on any cluster

Hi Team,I am using NIFI 2.0 M4 version-single node , I am trying to merge flow files using defragment merge strategy, Instead of merging all defragment flow files  , It is writing first fragment to merged and remaining to failure, Please suggest if i am missing some thing.NIFI Configuration: @MattWho ,please help to suggest.Thanks,

Hi,I can't seem to figure out why I'm unable to reach my NiFi GUI from computers on our intranet other than where it's installed.nifi.properties has these entries (nothing on the http settings)nifi.web.https.host=ourMacMini20nifi.web.https.port=9444nifi.web.https.network.interface.default=nifi.web.https.application.protocols=h2 http/1.1nifi.web.jetty.working.directory=./work/jettynifi.web.jetty.threads=200nifi.web.max.header.size=16 KBnifi.web.proxy.context.path=nifi.web.proxy.host=nifi.web.max.content.size=nifi.web.max.requests.per.second=30000nifi.web.max.access.token.requests.per.second=25nifi.web.request.timeout=60 secsnifi.web.request.ip.whitelist=nifi.web.should.send.server.version=truenifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i" "%{User-Agent}i"From the computer itself where the installation occurs this works:https://ourmacmini20:9444/nifi/For other computers (where certificates are installed) this message is shownThis site can’t be reachedourmacmini20