Solved

HiI am using many InvokeHTTP processors for sending APIs to a cloud service (complex flow in NIFI)I need to know how many requests NIFI is sending per minute (as I am paying by CallsPerMinutes). Besides the provenance window, where are the logs/files (on the Linux machine) showing those calls? I need to be able to read them and then I could analyze them. Thank you,Tamir 

I have a flow-file1. I need to generate an authentication token in the next step using a different flow-file2 through InvokeHttp processor which gives the response in another flow-file3. Now, using this token in this flow-file3, in the next step, I need to send flow-file1 to another HTTP endpoint using InvokeHttp. I am not able to figure out this flow. How to keep the flow-file1 and also handle other flow files which are completely for different purpose than the flow-file1? 

Cloudera Documentation - cdswctl and Using VS Code for Remote Editing describe how to create a ssh-endpoint using `cdswctl`. The command describes how to create a new session and associate a ssh-endpoint.   Is it possible to create a ssh-endpoint to an existing session? e.g. first list available sessions and then connect to one of them via its ID?

Hi, An external process creates an error.file on certain events and updates the process log file. The process that I'm creating starts when the error.file is created and need to read the last 10 - 15 lines of the log.file. My problem is that the log.file does not rollover and just grows continually and neither follows a log standard in order to use SysLog processors. How can I get that last 10 - 15 lines of the log file? Something like ( "tail -15 log.file" ). Thanks in advance.

Insufficient Permissions - Untrusted proxy CN=w-nifi-inf-wf02.dev.Company.com, OU=OCIO, O=Company -New 2-Node Cluster-SAML is working fine-Cluster is communicating fine-I have checked the user log and the DN is exactly the same as the authorizers below:<authorizers><userGroupProvider><identifier>file-user-group-provider</identifier><class>org.apache.nifi.authorization.FileUserGroupProvider</class><property name="Users File">./conf/users.xml</property><property name="Legacy Authorized Users File"></property><property name="Initial User Identity 1">user1</property><property name="Initial User Identity 2">CN=w-nifi-inf-wf02.dev.Company.com, OU=OCIO, O=Company</property><property name="Initial User Identity 3">CN=w-nifi-inf-wf03.dev.Company.com, OU=OCIO, O=Company</property></userGroupProvider><accessPolicyProvider><identifier>file-access-policy-provider</identifier>

What does it mean when a manually triggered job is in "Scheduling" job status? What happens behind the scenes during this Scheduling phase? Recently I have been noticing my job getting stuck in this phase for more than 10 minutes or so which is concerning.

I have a basic kafka 2.13_3.10 cluster with one broker, consumer and producer.I was testing if the "log.flush.interval.messages = 10"worked properly. It is supposed to write the kafka messages of the topics after 10 messages, but where are they saved in the disk by default??And, is there any way to specify the directory where you want data to be flushed overriding the default configuration?Thanks in advance.

I am using cloudera hdfs sink connector with parquetWriter.  "com.cloudera.dim.kafka.connect.hdfs.HdfsSinkConnector"   It seems to flush the data in kafka topic every 1 minute.  The number of partition of one of my kafka topics is 64.  64 (kafka partition) * 60 (minutes per hour) * 24 (hours per day) = 92160 files are sinked everyday in one directory. So, I created a job to delete files n days old. But this job is too slow because of the number of files of the directory where the parquet files are sinked by hdfs sink connector.   I have question about cloudera hdfs sink connector 1. Is it possible to sink the files in daily partition directory? ex) /blah/{topicName}/{yyyyMMdd} 2. Is there a way to change flush duration instead of every minutes?

I have a JSON document that I want to index into Hbase.   { "field1" : { "nestedfield" : "value1" } "field2" : "value2"}   In this case I want `nestedfield` to be the one used as a row identifier field, but I can't figure out how to specify this to the `PutHBaseJson` processor. I have tried multiple ways : - field1/nestedfiled - field1['nestedfiled'] - field1.nestedfield - $.filed1.nestedfiled None of them are working, the processor is saying that "the Row ID was null". Using `field2` for this purpose is working fwiw.

Hi Folks,After adding new columns in Kudu tables from impala, I had some issues computing stats for this table. Everything works fine until the Compute Stats command;The alter table command  ALTER TABLE myschema.mytable ADD COLUMNS (mycolumn TIMESTAMP DEFAULT now());  The issue when Compute Stats command. ImpalaRuntimeException: Error making 'updateTableColumnStatistics' RPC to Hive Metastore: CAUSED BY: MetaException: Column mycolumn doesn't exist in table mytable in database myschema  Someone can help me, please?

We are creating a PoC in a Cliente using CPD public on Azure.  Always when I try to create the Streams Messaging Light Duty.  The result I get is Failed: Orchestrator component went failed in 30 min(s), message: Status: 406 Not Acceptable Response: 406 Not Acceptable

Hello Team,   Spark job through Oozie is failing with the below exception in the Prod cluster. Note that the same job passes in the lower clusters (f.e. Dev)   22/03/24 05:05:22 INFO spark.SparkContext: Invoking stop() from shutdown hook22/03/24 05:05:22 ERROR scheduler.AsyncEventQueue: Listener EventLoggingListener threw an exceptionjava.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:477) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:627) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:583) at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:134) at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:145) at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:145) at scala.Option.foreach(Option.scala:257)

Hi, is there a way to login to NiFi registry via openid (ej google)?I'm thinking of how I can secure the instance without a domain like in NiFiThanks!

Hi All,   I am trying all means to find out a latest Cloudera sandbox for practicing purpose on mac m1 , yes i know and experienced many issues due to M1 not supporting most of the softwares with out any tweaks.   I have found few options   Option 1 - Virtual box -> not compatible for M1 Option 2 - Parallels -> not open source  Option 3 - VM ware fusion  -> not open source  Option 4 - UTM - again some issues keyboard no sync  Option 5 - Docker desktop -> This options works with tweaks   I have picked up Docker desktop ,since there is a software available for M1 , i have then explored the options  to find a suitable cloudera/hortonwork sandbox images   1. Docker - Hortonworks sandbox 3.0.1 -> failed Hortonworks 3.0.1 -> I have tried with some tweaks but not able to configure in spite of spending a lot of time fixing the issues ,no luck    Issue/error - Failed to get D-Bus connection: No such file or direc

Hi there... I was wondering if there is a way, that can report the failure of read eventhub failure while trying to read the eventhub. Suggestions would be appreciated.Thanks in advance 🙂

I have an issue with my JOLT transformation in NIFI:My INPUT:{"customerNumber": "40890036","customerOrderNumber": "TPACSAY4","orderType": "ES","orderEntryFlag": "createOrder","parts": [{"partNumberOrdered": "A0029864271","quantityOrdered": "2.00","customerInfo": "Cust information","chassisNo": "WDB45678901234567","additionalEntry": "ADDITIONAL ENTRY"},{"partNumberOrdered": "N000000001388","quantityOrdered": "10.00","customerInfo": "Cust information","chassisNo": "WDB45678901234567","additionalEntry": "ADDITIONAL ENTRY"}]} My current JOLT is below:[{"operation": "modify-default-beta","spec": {"CustNo1": "${CustNO}","Customer": "${CustNO}","ShippingSiteId": "0","SalesOrderOriginCode": "CFE","ShippingWarehouseId": "5559","LineDiscount": "${MCRHoldCode}","SALESORDERNUMBER": "","customerReqNo": "","OrderType": "${MCRHoldCode}","DELIVERYTERMSCODE": "${DELIVERYTERMSCODE}","ShippingDate_VO_Header": "=substring((${ShippingDate_VO}), 0, 20)","ShippingDate_EM_Header": "=substring((${Shipping

I'm trying to connect to dynamo db from nifi processor component "PutDynamoDB"I'm putting my secret key in the field Access Key IDsecret key in Secret Access Keyand the path of the credentials in the format: [default]aws_access_key_id=<key id>aws_secret_access_key=<access key> At the moment of make the validation i get this message: Perform Validation Component is invalid: '' validated against 'Access Key' is invalid because Cannot set both Credentials File and Secret Key/Access Key  Does someone have an idea about this? I haven't foun anything related

Version 1.15.3 Error: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate node CN=w-nifi-inf-wf03.dev.Company.com, OU=OCIO, O=Company to seed policies. Authorizers.xml:<userGroupProvider><identifier>file-user-group-provider</identifier><class>org.apache.nifi.authorization.FileUserGroupProvider</class><property name="Users File">./conf/users.xml</property><property name="Legacy Authorized Users File"></property><property name="Initial User Identity 1">user1</property><property name="Initial User Identity 2">CN=w-nifi-inf-wf02.dev.Company.com, OU=OCIO, O=Company</property><property name="Initial User Identity 3">CN=w-nifi-inf-wf03.dev.Company.com, OU=OCIO, O=Company</property></userGroupProvider><accessPolicyProvider><identifier>file-access-policy-provider</identifier><class>org.apache.nifi.authorization.FileAccessPolicyP

I'm getting same error again and again. Not able to resolve it. Can anyone help me in it. Nifi running in Linux server 

HiI've zip file that contains several other zips files.I want to extract the main ZIP and all other zips files (Recursive)Can anyone please advise how to do this using NIFI ?

Hi,   I am trying to add a custom runtime from gitlab private docker registry , and I am getting this error "The provided full version does not start with the short version concatenated with maintenance version." it might related to the path... I am trying to follow https://docs.cloudera.com/machine-learning/cloud/runtimes/topics/ml-distribute-the-runtime-image.html are there rules for the 'user' ?gitlab registry has some '/' , but I make sure that the last fields (sep by / ) are user , image-name:tag any idea? thanks

I am Trying running command in impala shell. And i am get error.I am trying Creating a New kudu table from impala CREATE TABLE my_first_table( id BIGINT, name STRING, PRIMARY KEY(id))PARTITION BY HASH PARTITIONS 16STORED AS KUDU; here log output[root@cloudera-master ~]# impala-shellStarting Impala Shell without Kerberos authenticationOpened TCP connection to master.cloudera-testing.io:21000Connected to master.cloudera-testing.io:21000Server version: impalad version 2.12.0-cdh5.16.2 RELEASE (build e73cce22064ef4972312d895bed2cdb8787a4215)***********************************************************************************Welcome to the Impala shell.(Impala Shell v2.12.0-cdh5.16.2 (e73cce2) built on Mon Jun 3 03:32:01 PDT 2019)Every command must be terminated by a ';'.***********************************************************************************[master.cloudera-testing.io:21000] > CREATE TABLE my_first_table > (

Hi,  we have a problem with agent installation on hosts which we want to add to the cluster.On the Install Agents page, the process fails with an error "Failed to copy installation files.  Examining cloudera-scm-server.log we got this (The process fails when trying to copy files). 2022-03-21 10:55:01,075 INFO NodeConfiguratorThread-7-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: Transitioning from MAKE_TEMP_DIR (PT0.197S) to COPY_FILES2022-03-21 10:55:01,081 INFO NodeConfiguratorThread-7-0:com.cloudera.server.cmf.node.NodeConfigurator: Using default key bundle URL2022-03-21 10:55:01,111 INFO NodeConfiguratorThread-7-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: Setting COPY_FILES as failed and done state2022-03-21 10:55:01,111 INFO NodeConfiguratorThread-7-0:net.schmizz.sshj.transport.TransportImpl: Disconnected - BY_APPLICATION2022-03-21 10:55:01,111 INFO NodeConfiguratorThread-7-0:com.cloudera.cmf.model.HostInstallArgs: Deprecated option

i have found my one of CDH has so many errors on every datanode, the error logs as below. who have this kind experience on this issue ? and give me some advises  2021-08-03 05:23:43,389 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2123011416-10.37.54.12-1457006347704:blk_3910061604_2849065475 src: /10.37.54.218:36088 dest: /10.37.54.218:10042021-08-03 05:23:43,700 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.37.54.218:36082, dest: /10.37.54.218:1004, bytes: 358, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-859199005_222, offset: 0, srvID: 44713da0-9f69-44ea-b6c0-8f7420a41f83, blockid: BP-2123011416-10.37.54.12-1457006347704:blk_3910061597_2849065468, duration: 597337782021-08-03 05:23:43,700 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2123011416-10.37.54.12-1457006347704:blk_3910061597_2849065468, type=HAS_DOWNSTREAM_IN_PIPELINE terminating2021-08-03 05:23:43,833 INFO org.apache.hadoop.hdfs.se

Solr process keeps stopping each 3 daysI am getting the below error22/02/27 05:15:21 ERROR handler.RequestHandlerBase: [qtp417301556-3371]: org.apache.solr.common.SolrException: Exception writing document id 6ae68838-4669-4f11-9544-53d22025085c-5 to the index; possible analysis error: number of documents in the index cannot exceed 2147483519Caused by: java.lang.IllegalArgumentException: number of documents in the index cannot exceed 2147483519 I have 1 solar server, Cloudera version 7.4.4, solar version 8.4.1what do you suggest to solve the problem?Thanks