Find answers, ask questions, and share your expertise
Recently active
Hi Everyone, i would like to add an attribute to the following json file using the jolt transfomr processor. Unfortunately my json file remains unchanged with the following transformation Original Json File { "logs": [], "ranges": [ { "day": "2022-08-31T22:00:00.000Z", "manager": { "costByRemunerationId": [], "total": 0, "totalHours": 0, "ancillaryCost": 0 }, "crew": { "costByRemunerationId": [ { "remunerationId": "salary", "remunerationUnit": 2, "compensation": 76.66666666666667, "worktimeInH": 0, "durationInDays": 0, "quantity": 0, "ancillaryCost": 15.854666666666667 } ], "total": 76.66666666666667, "totalHours": 0, "ancillaryCost": 15.854666666666667 }, "trainee": { "costByRemunerationId": [], "total": 0, "totalHours": 0, "ancillaryCost": 0 },
Hy everyone, I need help to extract a part from a url with the extratct text processor. /organisations/60910c2f18572ebe8aa7a677/detailedLaborcosts? (Note the ID may change)I expect this result 60910c2f18572ebe8aa7a677 Thanks for suggestions
Hi. I have just started on a project, where we want to set up a landscape of nifi instanses, connected to nifi registry. And we are planning on making a dev. environment a test and a prod.The question started, when we made versioned nifi flows with versioned nifi flows inside, for code reuse and easier maintenance. But that cannot be moved easily between nifi registry instanses? Do we connect all of the nifi instanses to one registry or what is the best practice? Or what are we doing wrong?
I want to filter my rest api with date range. The rest api has dateFrom and dateTo parameters. I want to set dateFrom to 5 days ago and dateTo to now. How do i set the parameters in NiFi ?My api looks like the below URL:https://api.aa/reports/api/order_report/?format=json&dateFrom=&dateTo=
All hosts have the same issue : Installation failed. Failed to copy installation files. Details /tmp/scm_prepare_node.9aRx6niZFailed to download package. Status code: 404 I am using the free trial version. Log error:2021-04-23 01:03:08,424 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: (1 skipped) Invalid parcel in manifest : CDH-7.1.6-1.cd$2021-04-23 01:03:11,178 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Failed to download manifest. Status code: 404 URI: htt$2021-04-23 01:03:12,739 INFO ParcelUpdateService:com.cloudera.cmf.paywall.PaywallHelper: (2 skipped) License incomplete; unable to use it for authentication $
Hi, All ! I am having the same problem as below. Drop empty Impala partitions - Stack Overflow Impala external table partitions still show up in stats with row count 0 after deleting the data in HDFS and altering (like ALTER TABLE table RECOVER PARTITIONS) refreshing (REFRESH table) and invalidation of metadata. Trying to drop partitions one by one works, but there are tens of partitions which should be removed and it would be quite tedious. Dropping and recreating the table would also be an option but that way all the statistics would be dropped together with the table. Is there any kind of other options in impala to get this done? Is this only a workaround using Hive below? Found a workaround through HIVE. By issuing MSCK REPAIR TABLE tablename SYNC PARTITIONS then refreshing the table in impala, the empty partitions disappear. Also, I could not find an Issue in Impala JIRA.If anyone knows of an Issue in Impala JIRA, please let me know. My environment is below.
Hello,Is there a way to retrieve all HDFS groups and respective users, in Cloudera, using Java or Scala/Spark code?In other words, I’m looking for Java/Scala code equivalent to "hdfs groups <user1>, <user2>, ...".Ideally, the code would run in Zeppelin.Many thanks.
Hi all, I've been struggling to deploy HDP 2.6.5 via docker.I downloaded and ran the deploy script but kept receiving this error for this line of command: + docker exec -t sandbox-hdp sh -c rm -rf /var/run/postgresql/*; systemctl restart postgresql;Failed to get D-Bus connection: No such file or directory Calling systemctl thru the container CLI gives the same error. I used this script to replace the container's /bin/systemctl but was unable to start postgresql ( systemctl start -v postgresql yieldsWARNING:systemctl:No PIDFile for forking '/usr/lib/systemd/system/postgresql.service'WARNING:systemctl:forking start not active)which resulted in unsuccessful start of ambari-server and localhost:1080 kept giving 502 bad gateway. Any idea how to solve this? Thanks!
Hi Experts, We are planning to implement BDR in our project. As a back up and recovery solution I have proposed the BDR cloudera cluster where in the HDFS and Hive data will be backed up, But I want to know if there is an alternative way by backing up the data in the cold storage, Not sure if this approach can be the best as per cost perspective since we want the options available to reduce cost. If any one share a use case and best solution for this. Thanks.
Hello community, After a recent Domain Controler update, our fully kerberized Cloudera cluster 3.1.1 does not allow us to login the web interfaces of Oozie and YARN (the underneath services are working fine, apps are up and running) When authenticating, we receive in the browser the following message: HTTP 403 - Authentication exception: java.lang.IllegalArgumentException. We have tried with different browsers, and configured them to work with the Kerberos authentication as explained in other threads. It is important to mention, that the authentication worked fine before the Domain Controller update. Any ideas on how to solve this? thanks for your support.
I'm getting the error "message:java.lang.UnsupportedOperationException: Storage schema reading not supported" I've found this online and similar articles to it:https://knowledge.informatica.com/s/article/579038?language=en_US That says to "To resolve this issue, add the following property to the hive_site.xml file in the cluster configuration and restart the Data Integration Service: metastore.storage.schema.reader.impl=org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader"I'm running Hive 3.1.5. I have hive-site.xml files on all of the hive clients in the cluster as well as on the server. I don't see an option to add that value to the Hive configs in the Ambari GUI. Also, when I look at the Ambari GUI, there's no Data Integration Service for Hive. So, to fix that issue, do I add that line to all of the client hive-site.xml files then stop/start certain Hive servers on the Ambari GUI? Or is this all command line and outs
Hi,when I switched to Fair scheduler however I still can't start the Resource Manager.I'm using CDP 7.4.4Following steps listed here https://community.cloudera.com/t5/Support-Questions/Unable-to-start-Node-Manager/td-p/285976I made the following changes via UI and verified the deployed filesIn yarn_site.xml I can see from the resource manager that fair scheduler is loaded2022-09-04 10:39:43,377 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService: Loading allocation file file:/run/cloudera-scm-agent/process/1546336134-yarn-RESOURCEMANAGER/fair-scheduler.xml [root@10-222-53-95 1546336089-yarn-RESOURCEMANAGER]# cat /run/cloudera-scm-agent/process/1546336134-yarn-RESOURCEMANAGER/fair-scheduler.xml<?xml version="1.0"?><allocations><queue name="sample_queue"><minResources>10000 mb,0vcores</minResources><maxResources>90000 mb,0vcores</maxResources><maxRunningApps>50</maxRunningAp
Hi, I am using a QueryDatabase Table processor to get data from Oracle database to transfer data.I use custom query like below.Both table_a and table_b has incremental columns that I can use for maximum-value columns.My question is, can we track changes from both tables in a single QueryDatabase processor.When any column from table_a or table_b changes, I want to transfer below query result to another Oracle database. Select a.column_1, a.column_2, a.column_3, b.column_1, b.column_2, b.column_3from table_a a table_b bwhere a.id=b.id Thanks
Hi Everyone,I have a pipeline that reads data from Postgres database and putts in to another Postgres database unchanged, I need the data to be identical, however Timestamps after being migrated missing microseconds.2022-08-29 22:03:13.620961-05 -> Source2022-08-29 22:03:13-05 -> TargetI have noticed that flow files contain correct format before reaching PutDatabaseRecord processor, but then on Target database microseconds are gone.Can you please help me to understand how I can migrate timestamps with the microseconds?
We are planning to upgrade from HDP to CDP version.How many users have logged-in to the Hadoop system (or accessed the HIVE database) since the beginning, broken down by months.We need to report for this Hadoop system for better utilization and performance.we are using Amabari HDP 2.6.5 version.Could you please help me on this issue.
Hi, I need to compare several csv columns with values from my db. To do so I use the LookupRecord with concat various csv columns and the corresponding concat of sql columns in the DatabaseRecordLookupService. This works fine as long as I do not have any NULL values in my db.If there is a NULL value the Lookup Processor always routes those datasets to unmatched, as an empty string in the csv does not match the NULL values from the db.What needs to be written in the corresponding csv column so that it matches the db NULL value?I already tried to insert 'null', 'NULL' and 0 in the csv but it never matched the db value.Thanks for your help!
Hi, Please i need your kind help. SCENERIO: INSERT CONTENT OF AN XML FILE AS A FIELD INTO A DATABASE - I am trying to read the content of an xml file into a flowfile attribute - I want to use the attribute as variable in sql query statement to insert the content into a DB ISSUES: - I tried to use both Xpath & Xquery but no success. DATA: The data i am trying to convert is as below <?xml version='1.0' encoding='utf-8'?><!DOCTYPE database><database xmlns='http://www.lotus.com/dxl' version='9.0' maintenanceversion='1.0' replicaid='8525886700702AFD' path='BStravel2.nsf' title='BSTravel_Expenses_and_Vacations' maintainunread='false' optimizetablebitmaps='true' increasemaxfields='true' compressdesign='true' compressdata='true' markmodifiedunread='false' logtransactions='false' uselz1='true'><databaseinfo dbid='85256700702AFD' odsversion='43' diskspace='261440' percentused='96.2421875' numberofdocuments='13'><datamodified><datetime dst=
HiI'm looking for a list of supported ODBC functions in the HIVE ODBC driver.. I have a particular issue with { fn EXTRACT(YEAR FROM "datecolumn") } which is not recognised but a comprehensive list would be great.Thanks
Does anyone know what does mean the text like "IO729 05:56:15.224205 4764 CatalogServiceCatalog,java:1344] Table db_name.table_name (version=10446022, lastSeen=10446022) is skipping topic update (10448033, 10448495] due to lock contention" in the Impala catalog service log file?Thank you in advance!
Hi Team, We enabled LDAP authentication on Kafka cluster and added below LDAP DN template so that it allows all the users from its domain. We are trying to allow users present in other domain as well but couldn't find any templates. Can any one help to achieve this use-case. Current setup(working): ldap.auth.user.dn.template = {0}@domain1.org.comRequired setup : ldap.auth.user.dn.template = {0}@domain1.org.com + {0}@domain2.org.com We tried adding the other domain with comma & space separated but in vain. CDP 7.1.7, Kafka 2.5 Thanks,Uday
I want to be able to read binary Avro data from Kafka using ConsumeKafka Processor. And I am able to do that, but the content type of the file is in "application/octet-stream". Not able to view it as well, it says "No viewer is registered for this content type". I am not even able to convert this avro data to json, since the content type is octet-stream. But when i use "kafka-avro-console-consumer" on the console the data is in json. How to get this json data into Nifi?
I am trying to read a kafka topic that contains a csv file with no headers. The file is read as it should be when the csv contains two rows or more (no headers) but when I use a csv file with only one row (no headers) the processor Read from Kafka doesn't do anything (no error message, and no file ingested. (0 in 0 out)The controller services show below are : - CSVReaderCSVRecordSetWriter :
Hello,we want to migrate data from an open source Atlas 1.2.0 to an Atlas in version 2.1.0 included in our Cloudera CDP 7.1.7 cluster.In addition, the versions of solr and hbase of the source cluster are Solr 5.5.1 and Hbase 1.1.0.To migrate Atlas data, is it sufficient to perform a backup of the hbase tables from the source hadoop cluster and a restore on the target CDP cluster hbase?Is it necessary to backup and restore Solr collections as well? I know that the Solr 5.5.1 of the source cluster is not compatible with the Solr 8.4.1 of the target cdp cluster, so if possible I would like to avoid the export/import of the collections and perform an indexing of solr after the migration of the Hbase tables . Is this the correct procedure to follow?Thanks in advance.Ivo
Hi, We are migrating data from one db to another using NIFI. We are using GenerateTableFetch processor for the incremental fetch by setting a column in Max-Value columns. For logging purpose, i need to extract the state of the GenerateTableFetch processor and insert in a table. By state I mean the max value of the 'Max-Value column' that is captured as the state of the processor. Can someone please help with how can i extract the processor state? Thank you.
Hey everyone,I would like to add an attribute to a json file. The json file is a result of an api query with the Invoke HTTP procesor. As a next step I want to add an attribute to the json file to the existing fieldsI expected this:[ {"_id" : "5d55246b9ac36012c14b5e7c","gtin" : "4009602235038","target.sku : "59218_5_champagner", <- added attribute value"articleNumber" : "59218-376","styleGroupId" : "59218","colorGroupId" : "376","brand" : {"_id" : "57f75950a56ab89d1973ee36"}, Does anyone have any ideas?
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.