Solved

Hey everyone, i get a csv file as record from an api call and don't need all columns and want to convert the datetime columns to a time format. For example from 15.09.2022 00:15:00 to 00:15 This is my csv recordInterval startInterval stopCamera serial numberCounter namePedestrians coming inPedestrians going out15.09.2022 00:00:0015.09.2022 00:15:00000000111123400  This is my setting in the Query record ProcessorQuery SELECT #{TodayAxisV2Param} as "date", "Interval start" as "start time", -- Expected FORMAT(HH:mm) "Interval stop" as "end time", -- Expected FORMAT(HH:mm) "Pedestrians going out" as "out"FROM FLOWFILE I expect this result for the time fields date start timeend timeout2022091500:0000:150 Does anyone have an idea how I can do this with the Query Record Processor?

Hello everyone. I am trying to train a ML model on a cluster by using a docker image and spark-submit with yarn.I already tried to follow this process before on a training cluster I made and I succeeded. But when I run it this time, yarn prompts that one of the mounts is invalid.Of course I tried with Kerborise and without, both didn't work and both runs didn't imply any problems related to connection so we are good from this side.  This is what I tried:spark-submit \--master yarn \--deploy-mode cluster \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=image:v1 \--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS="/etc/passwd:/etc/passwd:ro,/etc/hadoop:/etc/hadoop:ro,/opt/cloudera/parcels/:/opt/cloudera/parcels/:ro,/data01/yarn/nm/:/data01/yarn/nm/:ro,/data02/yarn/nm/:/data02/yarn/nm/:ro,/data03/yarn/nm/:/data03/yarn/nm/:ro,/etc/krb5.conf:/etc/krb5.conf:ro" \--conf s

hello to all,I have many connections in time_wait on the ipc port 1019 of the datanode:More than 600 time_wait and about 250 established.Is that normal?I’m afraid that’s why index write closed on solr errors (the index is on hdfs).The servers are downloaded and the datanode does not saturate the jvm heap.I couldn’t find any max connection configuration for port 1019Any ideas? Environment:HDP 3.1.5.0-152 with HDFS 3.1.1 Thanks in advance

Hi community,hope someone can help me on below. My Flow File content is something like this:  [ {"version" : "5","shipment_number" : "2022065597","situation_code" : "MLV","justification_code" : "CFM","situation_date" : "202206071253","new_appointment_date" : "202301081253",}, {"version" : "5","shipment_number" : "2022065597","situation_code" : "MLV","justification_code" : "CFM","situation_date" : "202206071252","new_appointment_date" : "",} ]  I need to add a Query Record processor in order to grab only one JSON record for each Flow File.The condition is: Get a single result with highest value of situation_date or new_appointment_date. Just to explain better, the logic behind should be the following:Grab the max of situation_date, the max of new_appointment_date and compare each other, then take the highest of the comparison and get entire JSON record. In MYSQL or T-SQL it could be easier, but if i understand well Query Record is based on Apache Calci

Hi everyone,i had a .csv file converted to json with the convert csv to json processor and configured the csv reader. In my csv file i have names with german umlauts. I have use character set UTF 8 or ISO-8859-1 but the output is not the right for german umlauts.  the csv file has the following character set from the source system This are my setting of the csv reader controller serviceDoes anyone have an idea for the correct setting.Thanks

Hi Everyone, i would like to add an attribute to the following json file using the jolt transfomr processor. Unfortunately my json file remains unchanged with the following transformation Original Json File     { "logs": [], "ranges": [ { "day": "2022-08-31T22:00:00.000Z", "manager": { "costByRemunerationId": [], "total": 0, "totalHours": 0, "ancillaryCost": 0 }, "crew": { "costByRemunerationId": [ { "remunerationId": "salary", "remunerationUnit": 2, "compensation": 76.66666666666667, "worktimeInH": 0, "durationInDays": 0, "quantity": 0, "ancillaryCost": 15.854666666666667 } ], "total": 76.66666666666667, "totalHours": 0, "ancillaryCost": 15.854666666666667 }, "trainee": { "costByRemunerationId": [], "total": 0, "totalHours": 0, "ancillaryCost": 0 },

Hy everyone, I need help to extract a part from a url with the extratct text processor. /organisations/60910c2f18572ebe8aa7a677/detailedLaborcosts? (Note the ID may change)I expect this result 60910c2f18572ebe8aa7a677 Thanks for suggestions

Hi. I have just started on a project, where we want to set up a landscape of nifi instanses, connected to nifi registry. And we are planning on making a dev. environment a test and a prod.The question started, when we made versioned nifi flows with versioned nifi flows inside, for code reuse and easier maintenance. But that cannot be moved easily between nifi registry instanses? Do we connect all of the nifi instanses to one registry or what is the best practice? Or what are we doing wrong?

I want to filter my rest api with date range. The rest api has dateFrom and dateTo parameters. I want to set dateFrom to 5 days ago and dateTo to now. How do i set the parameters in NiFi ?My api looks like the below URL:https://api.aa/reports/api/order_report/?format=json&dateFrom=&dateTo=

Hi, All !   I am having the same problem as below. Drop empty Impala partitions - Stack Overflow Impala external table partitions still show up in stats with row count 0 after deleting the data in HDFS and altering (like ALTER TABLE table RECOVER PARTITIONS) refreshing (REFRESH table) and invalidation of metadata. Trying to drop partitions one by one works, but there are tens of partitions which should be removed and it would be quite tedious. Dropping and recreating the table would also be an option but that way all the statistics would be dropped together with the table. Is there any kind of other options in impala to get this done? Is this only a workaround using Hive below? Found a workaround through HIVE. By issuing MSCK REPAIR TABLE tablename SYNC PARTITIONS then refreshing the table in impala, the empty partitions disappear. Also, I could not find an Issue in Impala JIRA.If anyone knows of an Issue in Impala JIRA, please let me know.   My environment is below.

Hello,Is there a way to retrieve all HDFS groups and respective users, in Cloudera, using Java or Scala/Spark code?In other words, I’m looking for Java/Scala code equivalent to "hdfs groups <user1>, <user2>, ...".Ideally, the code would run in Zeppelin.Many thanks. 

Hi Experts, We are planning to implement BDR in our project. As a back up and recovery solution I have proposed the BDR cloudera cluster where in the HDFS and Hive data will be backed up, But I want to know if there is an alternative way by backing up the data in the cold storage, Not sure if this approach can be the best as per cost perspective since we want the options available to reduce cost. If any one share a use case and best solution for this. Thanks.

Hi,when I switched to Fair scheduler however I still can't start the Resource Manager.I'm using CDP 7.4.4Following steps listed here   https://community.cloudera.com/t5/Support-Questions/Unable-to-start-Node-Manager/td-p/285976I made the following changes via UI and verified the deployed filesIn yarn_site.xml I can see from the resource manager that fair scheduler is loaded2022-09-04 10:39:43,377 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService: Loading allocation file file:/run/cloudera-scm-agent/process/1546336134-yarn-RESOURCEMANAGER/fair-scheduler.xml [root@10-222-53-95 1546336089-yarn-RESOURCEMANAGER]# cat /run/cloudera-scm-agent/process/1546336134-yarn-RESOURCEMANAGER/fair-scheduler.xml<?xml version="1.0"?><allocations><queue name="sample_queue"><minResources>10000 mb,0vcores</minResources><maxResources>90000 mb,0vcores</maxResources><maxRunningApps>50</maxRunningAp

Hi Everyone,I have a pipeline that reads data from Postgres database and putts in to another Postgres database unchanged, I need the data to be identical, however Timestamps after being migrated missing microseconds.2022-08-29 22:03:13.620961-05 -> Source2022-08-29 22:03:13-05 -> TargetI have noticed that flow files contain correct format before reaching PutDatabaseRecord processor, but then on Target database microseconds are gone.Can you please help me to understand how I can migrate timestamps with the microseconds?

We are planning to upgrade from HDP to CDP version.How many users have logged-in to the Hadoop system (or accessed the HIVE database) since the beginning, broken down by months.We need to report for this Hadoop system for better utilization and performance.we are using Amabari HDP 2.6.5 version.Could you please help me on this issue.

Hi, I need to compare several csv columns with values from my db. To do so I use the LookupRecord with concat various csv columns and the corresponding concat of sql columns in the DatabaseRecordLookupService. This works fine as long as I do not have any NULL values in my db.If there is a NULL value the Lookup Processor always routes those datasets to unmatched, as an empty string in the csv does not match the NULL values from the db.What needs to be written in the corresponding csv column so that it matches the db NULL value?I already tried to insert 'null', 'NULL' and 0 in the csv but it never matched the db value.Thanks for your help!

Hi,  Please i need your kind help. SCENERIO: INSERT CONTENT OF AN XML FILE AS A FIELD INTO A DATABASE - I am trying to read the content of an xml file into a flowfile attribute - I want to use the attribute as variable in sql query statement to insert the content into a DB ISSUES: - I tried to use both Xpath & Xquery but no success.   DATA: The data i am trying to convert is as below   <?xml version='1.0' encoding='utf-8'?><!DOCTYPE database><database xmlns='http://www.lotus.com/dxl' version='9.0' maintenanceversion='1.0' replicaid='8525886700702AFD' path='BStravel2.nsf' title='BSTravel_Expenses_and_Vacations' maintainunread='false' optimizetablebitmaps='true' increasemaxfields='true' compressdesign='true' compressdata='true' markmodifiedunread='false' logtransactions='false' uselz1='true'><databaseinfo dbid='85256700702AFD' odsversion='43' diskspace='261440' percentused='96.2421875' numberofdocuments='13'><datamodified><datetime dst=

HiI'm looking for a list of supported ODBC functions in the HIVE ODBC driver.. I have a particular issue with  { fn EXTRACT(YEAR FROM "datecolumn")  } which is not recognised  but a comprehensive list would be great.Thanks

Does anyone know what does mean the text like "IO729 05:56:15.224205 4764 CatalogServiceCatalog,java:1344] Table db_name.table_name (version=10446022, lastSeen=10446022) is skipping topic update (10448033, 10448495] due to lock contention" in the Impala catalog service log file?Thank you in advance!

Hi Team, We enabled LDAP authentication on Kafka cluster and added below LDAP DN template so that it allows all the users from its domain. We are trying to allow users present in other domain as well but couldn't find any templates. Can any one help to achieve this use-case. Current setup(working): ldap.auth.user.dn.template = {0}@domain1.org.comRequired setup : ldap.auth.user.dn.template = {0}@domain1.org.com + {0}@domain2.org.com We tried adding the other domain with comma & space separated but in vain. CDP 7.1.7, Kafka 2.5 Thanks,Uday

I want to be able to read binary Avro data from Kafka using ConsumeKafka Processor. And I am able to do that, but the content type of the file is in "application/octet-stream". Not able to view it as well, it says "No viewer is registered for this content type". I am not even able to convert this avro data to json, since the content type is octet-stream. But when i use "kafka-avro-console-consumer" on the console the data is in json. How to get this json data into Nifi?

I am trying to read a kafka topic that contains a csv file with no headers. The file is read as it should be when the csv contains two rows or more (no headers) but when I use a csv file with only one row (no headers) the processor Read from Kafka doesn't do anything (no error message, and no file ingested. (0 in 0 out)The controller services show below are : - CSVReaderCSVRecordSetWriter : 

Hello,we want to migrate data from an open source Atlas 1.2.0 to an Atlas in version 2.1.0 included in our Cloudera CDP 7.1.7 cluster.In addition, the versions of solr and hbase of the source cluster are Solr 5.5.1 and Hbase 1.1.0.To migrate Atlas data, is it sufficient to perform a backup of the hbase tables from the source hadoop cluster and a restore on the target CDP cluster hbase?Is it necessary to backup and restore Solr collections as well? I know that the Solr 5.5.1 of the source cluster is not compatible with the Solr 8.4.1 of the target cdp cluster, so if possible I would like to avoid the export/import of the collections and perform an indexing of solr after the migration of the Hbase tables . Is this the correct procedure to follow?Thanks in advance.Ivo

Hi,  We are migrating data from one db to another using NIFI. We are using GenerateTableFetch processor for the incremental fetch by setting a column in Max-Value columns. For logging purpose, i need to extract the state of the GenerateTableFetch  processor and insert in a table. By state I mean the max value of the 'Max-Value column' that is captured as the state of the processor. Can someone please help with how can i extract the processor state? Thank you.  

Hey everyone,I would like to add an attribute to a json file. The json file is a result of an api query with the Invoke HTTP procesor. As a next step I want to add an attribute to the json file to the existing fieldsI expected this:[ {"_id" : "5d55246b9ac36012c14b5e7c","gtin" : "4009602235038","target.sku : "59218_5_champagner", <- added attribute value"articleNumber" : "59218-376","styleGroupId" : "59218","colorGroupId" : "376","brand" : {"_id" : "57f75950a56ab89d1973ee36"}, Does anyone have any ideas?