Q&A

Atlas is unable to start since it is not being able to connect to Hbase region server 

I'm new to NiFi, and I've been struggling with wrapping square brackets around a specific JSON object. Below is my JSON input, my desired JSON output and my attempted JoltTransformJSON shift operation spec. I'm really close, but I'm getting a duplicate of the "financialSystemFields" object.Please advise, and many thanks in advance.JSON Input:[{"type" : "APPROPRIATION","subType" : "ADMINISTRATIVE", "duration" : "1","retiredDate" : null,"expiredDate" : "2023-09-30 00:00:00.0","closedDate" : "2028-09-30 00:00:00.0","association" : null,"financialSystemFields" : {      "FundId" : "01234NFWJ",      "fundedProgram" : null,      "projectDescription" : "NO FUN WITH JOLT"   }}]Current JSON Output:[{"type": "APPROPRIATION","subType": "ADMINISTRATIVE","duration": "1","retiredDate": null,"expiredDate": "2023-09-30 00:00:00.0","closedDate": "2028-09-30 00:00:00.0","association": null,"financialSystemFields": [{      "FundId"

I am unable to use date related queries in getmongo , following simple query is working mongo but its not filtering anything in the getmongo and selecting the complete collection in nifi{"createdOn" : {"$gt":{"$date" : "2024-08-19T11:39:46.308Z"}}}

so i am working on NIFI and we are converting and sorting our .asn file into json file so right now for this i am using some processors this all processors i am using , now i want to change  name of existing variable into input file means i am having 2 resultcode in my input file and output is coming like this resultcode = {2001,2001} now i want the second value to come under resultcodemscc what processor i can use here ? my input after jolt looks like                                           "sessionStopTime" : "09/07/2024 06:47:19","resultCode" : [ "2001", "2001" ],"recordOpeningTime" : "09/07/2024 06:44:22" this is all live data i cant hardcode the value , pls suggest something ASAP !

In nifi I am using PutDatabaseRecord. When i am inserting data to the db then why is 0 being omitted like if it is 0123 then in db only 123 is inserting in the table. Note: If I use PutSql then it is not omitting leading 0. How to configure PutDatabaseRecord to not omit leading zeros.

Hello everybody, basically there was an electric problem and the cluster was suddently shutdown.After restarting everything Hbase results to have all the Region Servers online (but with 0 regions each) and the Region Server with the same names are shown in Dead Region Servers.Everytime i restart hbase, new rows are add in the Dead Region Server .This already happened to me long time ago and the problem was related to zookeeper, but i can't find the old post.Do you know what i can do? ThanksP.S. my cluster is kerberized, hbase version 2.0

Hi,I have a requirement where I need to update the Application-Name of a Yarn  Application Master (AM) that I use to execute a  task.  The  same Yarn AM is reused  to run  multiple tasks and  and I need to update the Application-Name of the Yarn AM  based on the task I  run currently. Example.: If I am running  a  sort task, then  ' yarn application -list' command should show: sh-4.4$ yarn application -list23/06/23 00:34:24 INFO client.RMProxy: Connecting to ResourceManager at abc.def.hig.com/xx.xx.xx.xx:8032Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):2Application-Id Application-Name Application-Type User Queue State Final-State Progress Tracking-URLapplication_1686581388019_0752 sort_job1 YARN dsadm default RUNNING UNDEFINED 0% N/A  And later, If I am running  a  delete task using the same AM then  ' yarn appli

Hi All,I am trying to write my spark dataframe to hive table using HWC - Hive Warehouse connector. My spark application is in pySpark and i have 5 concurrent spark application running at same time and trying to write to same hive table probably at same time. I am getting following issue Error- caused by: java.lang.RuntimeException: Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.exec.MoveTask. java.io.FileNotFoundException: File hdfs://tablespace/managed/hive/my_db_name.db/hive_table_name/.hive-staging_hive_2024-08-05_15-55-26_488_5678420092852048777-45678Does HWC won't allow concurrent write to same hive table? or its limitation with hive table?  

We have a streaming requirement where the data is received continuously and we store them in HDFS by merging 10,000 records to avoid multiple IO operations.We generate the batchID using the below calculation,${now():format('yyyyMMdd')}${now():format("HH"):toNumber():multiply(60):plus(${now():format('mm'):toNumber()}):divide(15)}So this batch will change every 15 minutes and in Nifi sometimes there are 4 flows or 5 flows based on the number of records received during 15-minute time intervals. For example, if we receive 50,000 records in 15 minutes then this flow will be executed 5 times but if there are 40,000 records in 15 minutes then the entire NiFi flow will be executed 4 times.Once the batch execution is completed fully (after 4 flows or 5 flows as per the incoming data volume as explained in the above example) we want this batchID to be stored in the next table for further processing.So, we are working on some logic/condition in NiFi to invoke the "RouteOnAttribute" to check the b

Dear! Once I installed and started PQS on Ambari, the service was successfully installed and started, but a second later I encountered PQS automatically stopped. I checked var/log/phoenix-hase-query-server.log found that, ... Unrecoverable service error : shutting down.  Anybody used to encounter such this issue?Regard!

IHi All,I am getting one issue while updating my existing entity using PUT calls.I have downloaded json by using GET and i just trying to update my entity (i have tried updating description also tried not updating anything and PUT as it is ) using PUT operation and its failing with same reason  -  "errorCode":"ATLAS-400-00-023","errorMessage":"Attribute null not found for type hive_table"   could you please advise how did you update the entity using PUT command ?   curl --negotiate -u : -X PUT   -H "Content-Type: application/json"   https://myURL:portNumber/api/atlas/v2/entity/guid/<myGuid>-d @downloaded_entity.json i have referred this post , however its not showing how to perform PUT operation.   Solved: Re: PARTIAL UPDATE APACHE ATLAS ENTITY GUI - Cloudera Community - 297845 so my question is - Does atals REST PAI allows updating entity ?  my usecase is i want to update an hive table enti

Hello Community,Back again with different version same issue ...added the proxy settings, properly, this is reaching out the Runtime repos but not the CM repo...don't know how to set this to make the installer to work, this is exhausting...where is this taking the info from??? Checking https://[userpass]@archive.cloudera.com/p/cm7/7.11.3.0/repodata/repomd.xmlthis URL is not existing!and then is creating  cloudera.repo file emptywhy is so dificult to make this installer to work? and yes, there is connectivity froem the serrvers to CLOUDERA:anyone knows how to install this CM 7.11.3 and CDP 7.1.9?  

Hi team I want a jolt spec where I want the output to be completely flattened json. this is my input json and assuming where ever there is nested array it should be mapped with other nested array as well. this is my input json - {"unitOfMeasure": "unit:piece","linkedDemandSeries": [{"loadFactor": 3.5,"materialNumberCustomer": "MNR-7307-AU340474.002","materialNumberSupplier": "MNR-8101-ID146955.001","customerLocation": "{{CATENAX-CUSTOMER-BPNS}}","demandCategory": {"demandCategoryCode": "0001"}},{"loadFactor": 4.0,"materialNumberCustomer": "MNR-7307-AU340474.002","materialNumberSupplier": "MNR-8101-ID146955.001","customerLocation": "{{CATENAX-CUSTOMER-BPNS}}","demandCategory": {"demandCategoryCode": "0002"}}],"linkedCapacityGroups": ["be4d8470-2de6-43d2-b5f8-2e5d3eebf3fd","ab4d8470-2de6-43d2-b5f8-2e5d3eebf3fd"],"unitOfMeasureIsOmitted": false,"capacityGroupIsInactive": true,"demandVolatilityParameters": {"rollingHorizonAlertThresholds": [{"sequenceNumber": 1,"absoluteNegativeDeviat

Hi,I use following in my cdp cluster. While I create a simple parquet table from Hue and store it in external table on s3 express bucket, it shows 'EOFException'.1. Hive 3.1.3000.7.2.18.200-39, Hadoop 3.1.1.7.2.18.200-392. Data hub CM version 7.12.0.200, CM runtime version: 7.2.18-1.cdh7.2.18.p200.54625612I have tried following setting but didn't help. Could anyone help here? Thank you.1. Data cluster -> CM -> hdfs -> add fs.s3a.experimental.input.fadvise=sequential2. Data cluster -> CM -> spark3_on_yarn -> add spark.sql.parquet.enableVectorizedReader=false INFO : Compiling command(queryId=hive_20240816153040_ebe87870-3c90-4f07-84dc-b2b6e354520c): select * from smallINFO : Semantic Analysis Completed (retrial = false)INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:small.id, type:string, comment:null)], properties:null)INFO : Completed compiling command(queryId=hive_20240816153040_ebe87870-3c90-4f07-84dc-b2b6e354520c); Time tak

Hi there!!I'm trying to insert into Hive managed table from HBase which is the external table made with SerDe and I'm getting the error WARN org.apache.hadoop.hbase.ipc.RpcServer: Expected HEADER=HBas but received HEADER=\x0A\x04hi from xx.xxx.xx.xxx:39720The insert into external table runs well, I mean the insert persist the information in the final HBase table but trying to run insert overwrite table managed_ml select key, cf1_id , cf1_name from c_0external_ml; The error is shown in HBase logs. Any idea what's wrong? HBase log:2024-08-13 21:53:30,583 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for hive/HOST@REALM (auth:KERBEROS)2024-08-13 21:53:30,584 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from xx.xxx.xx.xxx:55106, version=2.2.3.7.1.7.0-551, sasl=true, ugi=hive/HOST@REALM (auth:KERBEROS), service=ClientService2024-08-13 21:53:30,584 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationMan

Hi Everyone,I’m new to Apache Ranger. I’ve created an HDFS policy and configured it to store audit logs in Solr. When I create or delete directories in HDFS, the audit logs are generated and visible in the Ranger UI. However, when I perform operations as the superuser (hdfs), no audit logs are generated.As shown in below screenshot, rangertest1 and rangertest2 users audit logs are shown on UI. But any operations performed using hdfs user, those logs not going in Audit DB. Does Ranger not support audit logging for superuser actions?I have already checked all the configs and there are no exclusions to any user.Best regards,Aditya

recently HIVE 4 has been released, please advise about the plans to make it available in cloudera 

After upgrading our cloudera cluster to CDP 7.1.9 from CDP 7.1.7snappy is no longer available when running hadoop checknative and our apps getting error : nnative snappy library not available: this version of libhadoop was built without snappy supportHadoop version we have is Hadoop 3.1.1.7.1.9.14-2 Native library checking:hadoop: true /opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p14.53489573/lib/hadoop/lib/native/libhadoop.so.1.0.0zlib: true /lib64/libz.so.1zstd : true /opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p14.53489573/lib/hadoop/lib/native/libzstd.so.1bzip2: true /lib64/libbz2.so.1openssl: true /lib64/libcrypto.soISA-L: false Loading ISA-L failed: Failed to load libisal.so.2 (libisal.so.2: cannot open shared object file: No such file or directory)Please advice 

Hi ,I have apache atlas setup in place and hive hooks are up and running. All the hive create/update , lineages etc sowing in atlas.Now I have dropped one hive table from Hive and  this same hive table entity is still showing up in apache atlas as ACTIVE. can someone suggest is this something expected or do we need to explicitly delete this entity from Atlas? 

Hi all, I am trying to integrate sending emails from NiFi using the put email processor. Below is my configuration:When I start the processor I get this error:Failed to properly initialize Processor. If still scheduled to run, NiFi will attempt to initialize and run the Processor again after the 'Administrative Yield Duration' has elapsed. Failure is due to java.lang.NullPointerException: java.lang.NullPointerException Do you have any advice regarding the configuration? Thanks

HI allI've been using this community support a LOT lately as I try and get into NiFi and the ETL power it has, and I think it's incredible, but today I find myself instead asking for help, after spending far too long trying to work something out on my own 😕I'm not too familiar with JOLT and so what I'm trying to do is take a data flow I have which is in the following format: [ { "CODE": "1234", "PRODUCT": "product 1", "Valid": null, "Zone 1": "1.00 USD", "Zone 2": "1.00 USD", "Zone 3": null, "Zone 4": null, "Zone 5": null, }, { "CODE": "9876", "PRODUCT": "product 2", "Valid": null, "Zone 1": "18.00 USD", "Zone 2": "12.00 USD", "Zone 3": null, "Zone 4": null, "Zone 5": null }, { "CODE": "3745", "PRODUCT": "product 3", "Valid": null, "Zone 1": "51.00 USD", "Zone 2": "21.00 USD", "Zone 3": null, "Zone 4": null, "Zone 5": null }]and try to convert it to something like this:[ { "code": "1234", "produc

I have a requirement to read say name of users from a file(all in new lines) and insert to Database(oracle). I am using GetFile->SplitText->ExtractText->PutSql in NiFi. All works fine for less records(around 10 records). But when I try it for 50 or 100 plus records, the connection between ExtractText and PutSql, I am getting "A flowfile is currently penalized and data can not be processed at this time". Also all flowfiles remain in queue.Note in PutSQL I am using simple sql (insert into users_table(user_name) values ('user123'); It works properly for 10, 20 records but when I provide 100+ records then it gets stuck.

Hello Team ,We are using HDP-3.1.0We have executed import-hive.sh script to import already existing hive tables in Atlas. It got successfully executed. Now we can see all hive databases and tables in Atlas, but we are not able to see data linage of that imported tables.If we create external table on any hdfs path then we can see lineage in Atlas.Also, If we create any managed tables in Atlas, we are not able to see lineage of that old tables. But we still can see lineage of that new tables. Why we are not getting lineage of older tables?Please suggest. We are stuck now.Thanks,

After get the entity gui, I sent the request for the partial Update, but i get this error curl "http://xxxxxxxxxx:21000/api/atlas/v2/entity/guid/5a71e80b-c610-4a0f-bb30-f196225114d6" \-i -X PUT --negotiate \-H 'Content-Type: application/json' \-H 'Accept: application/json' \-u username:password -d @entidad.json{"errorCode":"ATLAS-400-00-023","errorMessage":"Attribute null not found for type LBK_Dataset"}[  please any suggestion??

 [09/Jul/2024 14:42:27 +0800] 276182 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/yarn/yarn.sh application-diagnostic-data -use_cli true -applicationIds job_1720485812953_0005[09/Jul/2024 14:42:27 +0800] 276182 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 276204[09/Jul/2024 14:42:27 +0800] 276205 MainThread redactor INFO Re-exec redactor: /opt/cloudera/cm-agent/bin/cm redactor --fds 3 5[09/Jul/2024 14:42:27 +0800] 276205 MainThread redactor INFO Started redactorTue Jul 9 14:42:27 CST 2024+ source_parcel_environment+ '[' '!' -z /opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/meta/cdh_env.sh ']'+ OLD_IFS=' '+ IFS=:+ SCRIPT_ARRAY=($SCM_DEFINES_SCRIPTS)+ DIRNAME_ARRAY=($PARCEL_DIRNAMES)+ IFS=' '+ COUNT=1++ seq 1 1+ for i in `seq 1 $COUNT`+ SCRIPT=/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.15945976/meta/cdh_env.sh+ PARCEL_DIRNAME=CDH-7.1.7-1.cdh7.1.7.p0.15945976+ . /opt/clo