Solved

I have a flow file that I want to enrich with the contents from a list of JSON values based on the "id".  Basically perform an inner join of the flow file with this look-up data set.  However, I need the look-up dataset to be in memory, then using the "id", query the look-up dataset and append the results to the current flow file.  Here is an example: Incoming flow file: { "id": "abc123", "fname": "The", "lname": "Rock"}  Contents of the look-up data set: [ { "id": "abc123", "dob": "03/09/1977", "phone": "987-654-0001" }, { "id": "def765", "dob": "04/08/1976", "phone": "789-654-0001" }, { "id": "hij765", "dob": "05/06/1975", "phone": "456-654-0001" }]  Enriched flow file:{ "id": "abc123", "fname": "The", "lname": "Rock", "dob": "03/09/1977", "phone": "987-654-0001"} I need to be able to look up the correct record in the look-up dataset based on the "id" then append the

My service is to forcibly interrupt the impala query, but the interrupt requires queryId, impala driver does not find the corresponding method, jdbc does not see queryId return, driver log level LogLevel=6, and queryId is not seen

I want to stop a processor on the same machine  with restApi but I get an authorization error when I call the rest api.I can solve this problem with bearer token, but since the token has expired, I need to change the bearer  token manually again.Is there a way I can authorize automatically?I am using the executeScript processor when calling the API.Script engine is Groovy.My Nifi version 1.23.2 and Nifi is secure  and not clustered.I login with ldap on UI. This is the part where I call the API in the script:def post = new URL(NifiBaseUrl.value + NifiComponentId.value).openConnection();post.setSSLSocketFactory(sslContext.getSocketFactory())post.setRequestMethod("PUT")post.setDoOutput(true)post.setRequestProperty("Content-Type", "application/json")post.getOutputStream().write(content.getBytes("UTF-8"));post.getResponseCode() Error:java.io.IOException: Server returned HTTP response code: 401 for URL

Hi all, in my test cluster I am noticing a slowdown in the execution of acid queries. Analyzing in detail I noticed that the compactions remain stuck at "ready for cleaning" and there are many delta files.  I also tried to manually launch the compaction without any result. hive.metastore.housekeeping.threads.on and hive.metastore.housekeeping.threads.on is true only in 1 hive metastore host. This is a table properties:bucketing_version 2transactional truetransactional_properties defaulttransient_lastDdlTime 1720453037In the development cluster with the identical configuration I do not have this problem. Do you have any suggestions?I'm running in CDP 7.1.9Thanks, Lorenzo

I am using Nifi 1.14.1, Kafka 2.13-2.7.1 and the processor ConsumeKafkaRecord_2_6 to process messages from a topic where the key and the value where both serialized using avro - schemas for the key and value are stored in the confluent schema registry. But the processor fails to parse the message because there is not a way - that I can see - to specify that both key and value are avro serialized with schemas stored in the confluent schema registry. The convention for naming the schema is usually [topic name]-value and [topic name]-key. I can read the messages just fine using kcat, formerly kafkacat using: kcat -b broker1:9092,broker2:9092,broker3:9092 -t mytopic -s avro -r http://schema-registry_url.com -p 0Is there a way to read such messages or am I supposed to add my own processor to nifi? Here's an error trace:[Timer-Driven Process Thread-9] o.a.n.p.k.pubsub.ConsumeKafkaRecord_2_6 ConsumeKafkaRecord_2_6[id=f9933326-017e-1000-60be-e5077975f031] Failed to parse message fconfigur

I just register cloudera on public cloud 5 days trial. I want to assess dataflow if it's suit on my usecase. When i follow the "Hello world" project there is deploy button that i can't click which needs to have admin account. Stated on https://community.cloudera.com/t5/Support-Questions/CDP-trial-version-admin-rights/m-p/348347#M235350 i need to contact support (create case) to give me admin access, however i can't access it due to access restriction, how come???  

When I attempt to' join' tables or even use 'group by', I get the following error: Hortonworks Hive DSN -- [Hortonworks][Hardy] (35) Error from server: error code: '1' error message: 'Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask'.(HY000,35) Yet, I am able to do a simple 'select-from-where' query with no problem. I am using the IDE Rapid SQL.  Any idea of what is going on?

Hello, I have a simple use case. I have an incoming file having say user name and user address in each line. User name is from char 1 to char 5 and user address is from char 6 to char 15. How to use ExtractText processor for it. I tried using Search Value as ^(.{5})(.{10}) and in replacement value as $1,$2Issue I am having is that for address it is capturing all the chars from 6th char to last and not necessarily upto 15th char. What should I modify?Just for experimentation I tried doing ^(.{5})(.{10})(.{1}) and $1,$2,$3 and this is able to capture properly from 6th char to 15th char. Please help.

Hello, I want to copy record from one table to another table that has the same structure in Oracle 12c+.One not-null column in source has white space. When trying to insert record to the target table, I got error "cannot insert NULL", so I added UpdateRecord processor to replace Null with white space, but still have the same error.It works if I replace Null with any other character like "_". Is there any way to keep white space when inserting to the target table?Below is my Flowfile and the UpdateRecord properties. Thank you 

Atlas is unable to start since it is not being able to connect to Hbase region server 

I'm new to NiFi, and I've been struggling with wrapping square brackets around a specific JSON object. Below is my JSON input, my desired JSON output and my attempted JoltTransformJSON shift operation spec. I'm really close, but I'm getting a duplicate of the "financialSystemFields" object.Please advise, and many thanks in advance.JSON Input:[{"type" : "APPROPRIATION","subType" : "ADMINISTRATIVE", "duration" : "1","retiredDate" : null,"expiredDate" : "2023-09-30 00:00:00.0","closedDate" : "2028-09-30 00:00:00.0","association" : null,"financialSystemFields" : {      "FundId" : "01234NFWJ",      "fundedProgram" : null,      "projectDescription" : "NO FUN WITH JOLT"   }}]Current JSON Output:[{"type": "APPROPRIATION","subType": "ADMINISTRATIVE","duration": "1","retiredDate": null,"expiredDate": "2023-09-30 00:00:00.0","closedDate": "2028-09-30 00:00:00.0","association": null,"financialSystemFields": [{      "FundId"

Hello everybody, basically there was an electric problem and the cluster was suddently shutdown.After restarting everything Hbase results to have all the Region Servers online (but with 0 regions each) and the Region Server with the same names are shown in Dead Region Servers.Everytime i restart hbase, new rows are add in the Dead Region Server .This already happened to me long time ago and the problem was related to zookeeper, but i can't find the old post.Do you know what i can do? ThanksP.S. my cluster is kerberized, hbase version 2.0

Hi All,I am trying to write my spark dataframe to hive table using HWC - Hive Warehouse connector. My spark application is in pySpark and i have 5 concurrent spark application running at same time and trying to write to same hive table probably at same time. I am getting following issue Error- caused by: java.lang.RuntimeException: Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.FileNotFoundException: File hdfs://tablespace/managed/hive/my_db_name.db/hive_table_name/.hive-staging_hive_2024-08-05_15-55-26_488_5678420092852048777-45678Does HWC won't allow concurrent write to same hive table? or its limitation with hive table?  

We have a streaming requirement where the data is received continuously and we store them in HDFS by merging 10,000 records to avoid multiple IO operations.We generate the batchID using the below calculation,${now():format('yyyyMMdd')}${now():format("HH"):toNumber():multiply(60):plus(${now():format('mm'):toNumber()}):divide(15)}So this batch will change every 15 minutes and in Nifi sometimes there are 4 flows or 5 flows based on the number of records received during 15-minute time intervals. For example, if we receive 50,000 records in 15 minutes then this flow will be executed 5 times but if there are 40,000 records in 15 minutes then the entire NiFi flow will be executed 4 times.Once the batch execution is completed fully (after 4 flows or 5 flows as per the incoming data volume as explained in the above example) we want this batchID to be stored in the next table for further processing.So, we are working on some logic/condition in NiFi to invoke the "RouteOnAttribute" to check the b

Dear! Once I installed and started PQS on Ambari, the service was successfully installed and started, but a second later I encountered PQS automatically stopped. I checked var/log/phoenix-hase-query-server.log found that, ... Unrecoverable service error : shutting down.  Anybody used to encounter such this issue?Regard!

IHi All,I am getting one issue while updating my existing entity using PUT calls.I have downloaded json by using GET and i just trying to update my entity (i have tried updating description also tried not updating anything and PUT as it is ) using PUT operation and its failing with same reason  -  "errorCode":"ATLAS-400-00-023","errorMessage":"Attribute null not found for type hive_table"   could you please advise how did you update the entity using PUT command ?   curl --negotiate -u : -X PUT   -H "Content-Type: application/json"   https://myURL:portNumber/api/atlas/v2/entity/guid/<myGuid>-d @downloaded_entity.json i have referred this post , however its not showing how to perform PUT operation.   Solved: Re: PARTIAL UPDATE APACHE ATLAS ENTITY GUI - Cloudera Community - 297845 so my question is - Does atals REST PAI allows updating entity ?  my usecase is i want to update an hive table enti

Hello Community,Back again with different version same issue ...added the proxy settings, properly, this is reaching out the Runtime repos but not the CM repo...don't know how to set this to make the installer to work, this is exhausting...where is this taking the info from??? Checking https://[userpass]@archive.cloudera.com/p/cm7/7.11.3.0/repodata/repomd.xmlthis URL is not existing!and then is creating  cloudera.repo file emptywhy is so dificult to make this installer to work? and yes, there is connectivity froem the serrvers to CLOUDERA:anyone knows how to install this CM 7.11.3 and CDP 7.1.9?  

Hi Everyone,I’m new to Apache Ranger. I’ve created an HDFS policy and configured it to store audit logs in Solr. When I create or delete directories in HDFS, the audit logs are generated and visible in the Ranger UI. However, when I perform operations as the superuser (hdfs), no audit logs are generated.As shown in below screenshot, rangertest1 and rangertest2 users audit logs are shown on UI. But any operations performed using hdfs user, those logs not going in Audit DB. Does Ranger not support audit logging for superuser actions?I have already checked all the configs and there are no exclusions to any user.Best regards,Aditya

After upgrading our cloudera cluster to CDP 7.1.9 from CDP 7.1.7snappy is no longer available when running hadoop checknative and our apps getting error : nnative snappy library not available: this version of libhadoop was built without snappy supportHadoop version we have is Hadoop 3.1.1.7.1.9.14-2 Native library checking:hadoop: true /opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p14.53489573/lib/hadoop/lib/native/libhadoop.so.1.0.0zlib: true /lib64/libz.so.1zstd : true /opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p14.53489573/lib/hadoop/lib/native/libzstd.so.1bzip2: true /lib64/libbz2.so.1openssl: true /lib64/libcrypto.soISA-L: false Loading ISA-L failed: Failed to load libisal.so.2 (libisal.so.2: cannot open shared object file: No such file or directory)Please advice 

Hi all, I am trying to integrate sending emails from NiFi using the put email processor. Below is my configuration:When I start the processor I get this error:Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.NullPointerException: java.lang.NullPointerException Do you have any advice regarding the configuration? Thanks

HI allI've been using this community support a LOT lately as I try and get into NiFi and the ETL power it has, and I think it's incredible, but today I find myself instead asking for help, after spending far too long trying to work something out on my own 😕I'm not too familiar with JOLT and so what I'm trying to do is take a data flow I have which is in the following format: [ { "CODE": "1234", "PRODUCT": "product 1", "Valid": null, "Zone 1": "1.00 USD", "Zone 2": "1.00 USD", "Zone 3": null, "Zone 4": null, "Zone 5": null, }, { "CODE": "9876", "PRODUCT": "product 2", "Valid": null, "Zone 1": "18.00 USD", "Zone 2": "12.00 USD", "Zone 3": null, "Zone 4": null, "Zone 5": null }, { "CODE": "3745", "PRODUCT": "product 3", "Valid": null, "Zone 1": "51.00 USD", "Zone 2": "21.00 USD", "Zone 3": null, "Zone 4": null, "Zone 5": null }]and try to convert it to something like this:[ { "code": "1234", "produc

I have a requirement to read say name of users from a file(all in new lines) and insert to Database(oracle). I am using GetFile->SplitText->ExtractText->PutSql in NiFi. All works fine for less records(around 10 records). But when I try it for 50 or 100 plus records, the connection between ExtractText and PutSql, I am getting "A flowfile is currently penalized and data can not be processed at this time". Also all flowfiles remain in queue.Note in PutSQL I am using simple sql (insert into users_table(user_name) values ('user123'); It works properly for 10, 20 records but when I provide 100+ records then it gets stuck.

Hello Team ,We are using HDP-3.1.0We have executed import-hive.sh script to import already existing hive tables in Atlas. It got successfully executed. Now we can see all hive databases and tables in Atlas, but we are not able to see data linage of that imported tables.If we create external table on any hdfs path then we can see lineage in Atlas.Also, If we create any managed tables in Atlas, we are not able to see lineage of that old tables. But we still can see lineage of that new tables. Why we are not getting lineage of older tables?Please suggest. We are stuck now.Thanks,

After get the entity gui, I sent the request for the partial Update, but i get this error curl "http://xxxxxxxxxx:21000/api/atlas/v2/entity/guid/5a71e80b-c610-4a0f-bb30-f196225114d6" \-i -X PUT --negotiate \-H 'Content-Type: application/json' \-H 'Accept: application/json' \-u username:password -d @entidad.json{"errorCode":"ATLAS-400-00-023","errorMessage":"Attribute null not found for type LBK_Dataset"}[  please any suggestion??

Hi,Do I need a license key for CDP Private base trial installation and if yes where can I obtain it?