Q&A

I am assigning a value to "reasonId" and later using it as a dynamic property in other processor(ExractText). Now, I want to assign this as an dynamic property so that I can store some API response into this. Means reasonId value and Api response both should change evertytime. Here I want this [${reasonId}=(.*)] to be like this:-6099 = Api response     (Everytime value of 6099 should change as it is a dynamic variable)But,I am getting it like this${reasonId}=Api response Why there is no 6099 instead of reasonId. Does it means Dynamic property assignment is not working properly? or I can use some jolt spec and all for this?  Basically I want reasonId to hold its value 6099 and Api response should be store in 6099 instead of ${reasonId}.    

How can I iterate on this array in NIfi, I am extracting this array from an API response in nifi but unable to extract every index value one by one for completing my whole flow.Example: Need 6100 in any variable so that I can use it for further process and in next iteration need 6101 in that variable and so on.Edit: This array is extracting like this only, it is not assigned to any named array such as arr[] or anything. [ "6100", "6101", "6102", "6103", "6104", "6105", "6106", "6107", "6108", "6423", "23382", "354548" ] 

Hello! I am new to NiFi. I hope someone here can advise me about my problem with time zone. I have these processors:ListDatabaseTables -> GenerateTableFetch -> ExecuteSQLRecord (writing to csv file by CSVRecordWriter) -> ... ... PutSQL (loading csv file to MySQL using Load Data command) The source DB is Oracle. CSVRecordWriter has the following properties:Schema Write Strategy -> Do Not Write SchemaSchema Access Strategy -> Inherit Record SchemaSchema Name -> ${schema.name}Schema Text -> ${avro.schema}Date Format -> yyyy-MM-ddTime Format -> HH:mm:ssTimestamp Format -> yyyy-MM-dd HH:mm:ss My source DB and the target DB are both in US east time zone. However, I noticed that the output of ExecuteSQLRecord having time values converted to UTC (added to 5 hours). That results in the wrong time values in the target DB. There may be some ways to convert each date/time column individually, but that will require a huge amount of development effort.&nb

Dears,     My usecase is to extract user name entered and store it in an attribute and load it into a column of a database. How can I extract user name entered by user while logging in. Thankyou!

HelloI have created a JSON body but I need to reconstruct it so I can send it in a REST API.Input : { "systemId" : "4", "systemName" : "Main", "items.nid" : "123", "items.birth_date" : "1978-01-01", "items.last_name" : "Willson", "items.nationality" : "GN", "formId": 1}  Output : { "formId": 1, "systemId" : "4", "systemName" : "Main", "items" { "nid" : "123", "birth_date" : "1978-01-01", "last_name" : "Willson", "nationality" : "GN" }}  I used JoltTransformJSON but with no luck always get null as a result, I can't figure how to regroup them based on the name "items.*" . how can I regroup my items or is there a better way to solve this point ?

Greetings Cloudera Community!! Text4shell vulnerability is impacting the apache application which is using commons-text version 1.5 to 1.9 and our application Nifi version 1.16.2 hosted on linux server (Red Hat Enterprise Linux Server 7.9) is using commons-text version 1.8 jar file in lib folder. Can anyone please help to figure out the best solution to handle this vulnerability in our production servers: We have few queries for the vulnerability: 1:Is the nifi version 1.16.2 application is affected by this vulnerability? 2: In Nifi configuration files, we are not using any calls related to StringSubstitutor API. Are we still     vulnerable to test4Shell? 3: If nifi version 1.16.2 vulnerable then can we just replace the commons-text jar file from 1.8 to 1.10 in nifi 1.16.2. Is there any impact of this in our prod servers? Please do let us know on this vulnerability for nifi 1.16.2. If it is impacting nifi 1.16.2 version then what would be th

Hi,I'm running a reconciliation application using Spark. This application needs to connect a table from 2 different clusters Hbase table and compare the data. My application is able to connect and get the data from where the application is running on a cluster. but for remote cluster I'm getting below error failed on local exception:failed on local exception:exception is Failed after attempts=4, exceptions: 2023-04-19T17:11:13.491Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to Host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options 2023-04-19T17:11:14.505Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options 2023-04-19T17:11:16.522Z, RpcRetryingCaller{gl

Hello everybody Trying to connect NIFI Server to Kafka brokerUse kerberous and SSL_SASLGet error from java : Failed to process session due to Failed to construct kafka producer; Processor Administratively Yielded for 1 sec: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user↳ causes: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user↳ causes: org.apache.kafka.common.KafkaException: Failed to construct kafka producer  Already checked some number of Topics here on cloudera but cant find anything that work for me Someone can help? log error

User trying to connect via beeline but connection itself taking long time. Please suugest what is the issue.

Using Sqoop export on CDP cluster (hdfs), I'm trying to export data geometry to Postgresql table but I'm facing a Sqoop ERROR orm.ClassWriter: No Java type for SQL type 1111 for column 'geom'.The 'geom' column definition on PostGis is : geometry (Point,3945).Here is my Sqoop export query :```sqoop export --connect "jdbc:postgresql://$database_server/$database_source_name" \--username $username --password $password_alias \--table $database_table_name \--export-dir "${DIR_OUT_HDFS}" \--fields-terminated-by '|' --lines-terminated-by "\n" \-- --schema $database_table_name 1>>${LOG_FILE} 2>>${LOG_FILE}```Here is Sqoop output error :```ERROR orm.ClassWriter: No Java type for SQL type 1111 for column geomERROR orm.ClassWriter: No Java type for SQL type 1111 for column geomERROR orm.ClassWriter: No Java type for SQL type 1111 for column geomERROR orm.ClassWriter: No Java type for SQL type 1111 for column geomERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerExcep

Dears, In Apache nifi, the flow is stored in the backend as an xml file . That file contains all the configuration and connection information for the nifi flow, including processors, input and output ports,.But we are unable to find that XML file in the backend directory .Could you please suggest the directory path in the nifi server  that where exactly  we find the XML files? Thank you!    

HI ,Hi,I was using NIFI 1.12.0 version and now shiting a job into NIFI 1.18.0 version because of some security concerns. I am facing challenges in that the NIFI queue is not getting empty after successfully data transfer to EventHub. Below is the NIFI  

we are facing space issue on following directories. it is safe to remove or move these directories to another directory or make a tar of it. i need space . PLease help[root@mprl509 lib]#pwd/var/lib/[root@mprl509 lib]# du -sh cloudera-*1.3G cloudera-host-monitor300K cloudera-scm-agent20M cloudera-scm-eventserver4.6M cloudera-scm-headlamp3.0G cloudera-scm-server272M cloudera-scm-server-db5.5G cloudera-service-monitor

Is it possible...does it make sense  to place an AWS application load balancer in front of a NiFi cluster?  

Hi Community Team, I have a Private Cloud Base Platform 7.1.7.SP1 with hive 3.1. I need to connect my SAP server (Windows server) to hive. I installed Cloudera ODBC driver 2.6.13 and when I configured the DSN, shows me an error "ERROR 1100 SSL certificate verification failed because the certificate is missing or incorrect. - I have the AUTO TLS/SSL installed in the cluster. Hive has enabled TLS/SSL.My Questions: - How I generate a certificate .jks to this server that is not part of the cluster CDP.- What other steps I do to resolve this. Thank for your support. 

I have a NiFi flow that is exporting Ranger Policies from one Cloudera Cluster and another NiFi flow that is importing them into another NiFi cluster. The API call to export them is ${ranger.local.baseurl}/service/plugins/policies/exportJson? and the API call to import them is ${ranger.local.baseurl}/service/plugins/policies/importPoliciesFromFile?Everything works great but once in a blue moon when trying to import the Policies I get - {"statusCode":1,"msgDesc":"XXPolicy not found","messageList":[{"name":"DATA_NOT_FOUND","rbKey":"xa.error.data_not_found","message":"Data not found","objectId":295}]}The best solution I have right now is to restart Ranger Admin and after it works. Does anyone know why this happens or how it gets into this state?

Hello, I have a 3 node NiFi cluster.  I would like to add new users to the cluster that would be administrators i.e., a new user with all of the capabilities of the initial admin identity user (in my case, the initial admin identify is user nifi). I am using PKI certs as the means of identifty/authenticating users and this is working well. So, my question is, can I create a handful of users that are administrators that have a policy set equivalent to the initial admin identity? Is this possible?  

hello guys, I need to copy hbase data from replication set cluster to ec cluster, i have tried hbase snapshot export and hadoop distcp ,they look like throw the same error: Not replicated yet, i think this was caused by when they transporting ,they will check if destination cluster replication num was same as local, but ignore if destination cluster used replication set. how can i do it ,thx. There is all the errors.snapshot export:2023-04-19 19:44:10,199 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK* blk_-9223372036854751104_9943 is COMMITTED but not COMPLETE(numNodes= 5 >= minimum = 3) in file /hbase/archive/data/default/test/09748ed90d0d58c0fe7ac4b3c08f3cd4/cf/e35fadd88d244766800728318ccea5082023-04-19 19:44:10,200 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020, call Call#31 Retry#0 org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 172.31.0.146:52210org.apache.hadoop.hdfs.server.namenode.NotRepl

ConsumerKafka2.6 I connect to kafka using ssl I added a keystore and a triac from kafka servera I placed the same ones on the nifi server but in the nifi logs I get this error.ConsumeKafka_2_6[id=9da42b1a-0187-1000-ffff-ffffb41254ef] Exception while interacting with Kafka so will close the lease org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@62e55d3b due to SSL handshake failed: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed- Caused by: javax.net.ssl.SSLHandshakeException: No subject alternative names matching IP address my settings StandardSSLContextServiceKeystore Filename/var/ssl/nifi/kafka_broker.keystore.jksKeystore PasswordSensitive value setKey PasswordSensitive value setKeystore TypeJKSTruststore Filename /var/ssl/nifi/kafka_broker.truststore.jksTruststore PasswordSensitive value setTruststore TypeJKSTLS ProtocolTLS My settings consumerKafka2.6Security Protocol SSLSASL Mechanism SCRAM-SHA-512Username

Below are the version detailsCDP 7.1.8CM : 7.8.1HIVE : 3.1.3We are trying to insert data into partitioned table using ORC file. It contains approx 50,000 rowsUsing below command to load dataLOAD DATA INPATH '/user/test/r_14_5.orc' INTO TABLE va_offer_16;we see that all the map and reducer task are completed withing 500s.----------------------------------------------------------------------------------------------VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED----------------------------------------------------------------------------------------------Map 1 .......... container SUCCEEDED 7 7 0 0 0 0Reducer 2 ...... container SUCCEEDED 33 33 0 0 0 0----------------------------------------------------------------------------------------------VERTICES: 02/02 [==========================>>] 100% ELAPSED TIME: 2601.60 s----------------------------------------------------------------------------------------------However further there is no progress for next 2000 sec be

Hi Team,I am having two issues with cloud era Hbase connection:1. Connecting Hbase through the Phoenix driver is not working.2. Somehow I am able to connect hbase through hbase shell and inserted data. But I have a problem while reading data from Phoenix driver.The steps I followed are below:created habse Table using sqlline and inserting data through hbase shell put API from spark java application. But my consumer app reading data through phoenix driver.can you please help me any table configuration helps me to use hbase table from hbase shell as we all phoenix driver. Issue with existing table is: I am able to query data properly through hbase shell command, But when I am querying data through Phoenix driver rowkey value is getting truncated (only the first letter) and other columns are good.While creating a table, using the following configurations:Column_encoded_bytes=0, slatbucket=88, Compression=snappy, data_block_encoding= fast_DIFF

I am using QueryRecord to translate data from one format to another, but once in a while, I am getting this error here. Not sure what to do about and how to handle it. I am using Nifi 1.19.1[Timer-Driven Process Thread-15] o.a.n.c.r.StandardProcessSession Failed to asynchronously commit session StandardProcessSession[id=25336651] for QueryRecord[id=dba21f2d-ea2b-327b-3692-5eefbb53cd52] java.lang.IllegalStateException: Cannot commit session while reading from FlowFile at org.apache.nifi.controller.repository.StandardProcessSession.validateCommitState(StandardProcessSession.java:245) at org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:274) at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:556) at org.apache.nifi.controller.repository.StandardProcessSession.commitAsync(StandardProcessSession.java:510) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28) at org

Hi ,I am trying to run a "alter table drop partition" command using spark sql which prints the metastore lost connection stack trace : 23/04/19 06:47:53 WARN metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect.org.apache.thrift.transport.TTransportException: SASL authentication not completeat org.apache.thrift.transport.TSaslTransport.write(TSaslTransport.java:472)at org.apache.thrift.transport.TSaslClientTransport.write(TSaslClientTransport.java:37)at org.apache.hadoop.hive.thrift.TFilterTransport.write(TFilterTransport.java:72)at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:178)at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:106)at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:70)at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_get_partitions_ps_with_auth(ThriftHiveMetas

Hello,with this topic i need a clue. I have a text based data, this is an example row: CA20230405112337BOL006057 20230405V1260     2023021000409I     311003000102504CA20230405112337BOL006057 20230405V1260     2023021000509I     311003000102504 I need to convert to json but:1  - i need to remove all spaces in between2 - i need to separate fields with a separator in order to create a schema and make a conversionExample:CA;20230405;bla bla blaCA;20230405;bla bla bla How can i achieve this?Is there any other way to do that? Many Thabks 

Hello,I am using jolt to change the structure of a payload to another form, and it must be dynamic, especially the top level elements "test" which can be multiple. I have tried but not getting it. I would appreciate your kind help.Here is my input json:{ "test1": "test output", "test2": "test output", "Details": { "0": { "id": "first", "name": "the first one" }, "1": { "id": "second", "name": "the second one" } }} and my output json is :{ "Details" : [ { "test1" : "test output", "test2" : "test output", "id" : "first", "name" : "the first one" }, { "test1" : "test output", "test2" : "test output", "id" : "second", "name" : "the second one" } ]} my current jolt spec is:[ { "operation": "shift", "spec": { "Details": "Details", "*": "outer.&" } }, { "operation": "shift", "spec": { "Details": { "*": { "*": "Details[&1].&", "@(2,outer)": "Details[&am