Q&A

Hi Experts, We are planning to implement BDR in our project. As a back up and recovery solution I have proposed the BDR cloudera cluster where in the HDFS and Hive data will be backed up, But I want to know if there is an alternative way by backing up the data in the cold storage, Not sure if this approach can be the best as per cost perspective since we want the options available to reduce cost. If any one share a use case and best solution for this. Thanks.

Hello community, After a recent Domain Controler update, our fully kerberized Cloudera cluster 3.1.1 does not allow us to login the web interfaces of Oozie and YARN (the underneath services are working fine, apps are up and running) When authenticating, we receive in the browser the following message: HTTP 403 - Authentication exception: java.lang.IllegalArgumentException. We have tried with different browsers, and configured them to work with the Kerberos authentication as explained in other threads. It is important to mention, that the authentication worked fine before the Domain Controller update. Any ideas on how to solve this? thanks for your support.

I'm getting the error "message:java.lang.UnsupportedOperationException: Storage schema reading not supported" I've found this online and similar articles to it:https://knowledge.informatica.com/s/article/579038?language=en_US That says to "To resolve this issue, add the following property to the hive_site.xml file in the cluster configuration and restart the Data Integration Service: metastore.storage.schema.reader.impl=org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader​"I'm running Hive 3.1.5.  I have hive-site.xml files on all of the hive clients in the cluster as well as on the server.  I don't see an option to add that value to the Hive configs in the Ambari GUI.  Also, when I look at the Ambari GUI, there's no Data Integration Service for Hive.   So, to fix that issue, do I add that line to all of the client hive-site.xml files then stop/start certain Hive servers on the Ambari GUI?  Or is this all command line and outs

Hi,when I switched to Fair scheduler however I still can't start the Resource Manager.I'm using CDP 7.4.4Following steps listed here   https://community.cloudera.com/t5/Support-Questions/Unable-to-start-Node-Manager/td-p/285976I made the following changes via UI and verified the deployed filesIn yarn_site.xml I can see from the resource manager that fair scheduler is loaded2022-09-04 10:39:43,377 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService: Loading allocation file file:/run/cloudera-scm-agent/process/1546336134-yarn-RESOURCEMANAGER/fair-scheduler.xml [root@10-222-53-95 1546336089-yarn-RESOURCEMANAGER]# cat /run/cloudera-scm-agent/process/1546336134-yarn-RESOURCEMANAGER/fair-scheduler.xml<?xml version="1.0"?><allocations><queue name="sample_queue"><minResources>10000 mb,0vcores</minResources><maxResources>90000 mb,0vcores</maxResources><maxRunningApps>50</maxRunningAp

Hi,  I am using a QueryDatabase Table processor to get data from Oracle database to transfer data.I use custom query like below.Both table_a and table_b has incremental columns that I can use for maximum-value columns.My question is, can we track changes from both tables in a single  QueryDatabase processor.When any column from table_a or table_b changes, I want to transfer below query result to another Oracle database. Select a.column_1, a.column_2, a.column_3, b.column_1, b.column_2, b.column_3from table_a a table_b bwhere a.id=b.id Thanks  

Hi Everyone,I have a pipeline that reads data from Postgres database and putts in to another Postgres database unchanged, I need the data to be identical, however Timestamps after being migrated missing microseconds.2022-08-29 22:03:13.620961-05 -> Source2022-08-29 22:03:13-05 -> TargetI have noticed that flow files contain correct format before reaching PutDatabaseRecord processor, but then on Target database microseconds are gone.Can you please help me to understand how I can migrate timestamps with the microseconds?

We are planning to upgrade from HDP to CDP version.How many users have logged-in to the Hadoop system (or accessed the HIVE database) since the beginning, broken down by months.We need to report for this Hadoop system for better utilization and performance.we are using Amabari HDP 2.6.5 version.Could you please help me on this issue.

Hi, I need to compare several csv columns with values from my db. To do so I use the LookupRecord with concat various csv columns and the corresponding concat of sql columns in the DatabaseRecordLookupService. This works fine as long as I do not have any NULL values in my db.If there is a NULL value the Lookup Processor always routes those datasets to unmatched, as an empty string in the csv does not match the NULL values from the db.What needs to be written in the corresponding csv column so that it matches the db NULL value?I already tried to insert 'null', 'NULL' and 0 in the csv but it never matched the db value.Thanks for your help!

Hi,  Please i need your kind help. SCENERIO: INSERT CONTENT OF AN XML FILE AS A FIELD INTO A DATABASE - I am trying to read the content of an xml file into a flowfile attribute - I want to use the attribute as variable in sql query statement to insert the content into a DB ISSUES: - I tried to use both Xpath & Xquery but no success.   DATA: The data i am trying to convert is as below   <?xml version='1.0' encoding='utf-8'?><!DOCTYPE database><database xmlns='http://www.lotus.com/dxl' version='9.0' maintenanceversion='1.0' replicaid='8525886700702AFD' path='BStravel2.nsf' title='BSTravel_Expenses_and_Vacations' maintainunread='false' optimizetablebitmaps='true' increasemaxfields='true' compressdesign='true' compressdata='true' markmodifiedunread='false' logtransactions='false' uselz1='true'><databaseinfo dbid='85256700702AFD' odsversion='43' diskspace='261440' percentused='96.2421875' numberofdocuments='13'><datamodified><datetime dst=

HiI'm looking for a list of supported ODBC functions in the HIVE ODBC driver.. I have a particular issue with  { fn EXTRACT(YEAR FROM "datecolumn")  } which is not recognised  but a comprehensive list would be great.Thanks

Does anyone know what does mean the text like "IO729 05:56:15.224205 4764 CatalogServiceCatalog,java:1344] Table db_name.table_name (version=10446022, lastSeen=10446022) is skipping topic update (10448033, 10448495] due to lock contention" in the Impala catalog service log file?Thank you in advance!

Hi Team, We enabled LDAP authentication on Kafka cluster and added below LDAP DN template so that it allows all the users from its domain. We are trying to allow users present in other domain as well but couldn't find any templates. Can any one help to achieve this use-case. Current setup(working): ldap.auth.user.dn.template = {0}@domain1.org.comRequired setup : ldap.auth.user.dn.template = {0}@domain1.org.com + {0}@domain2.org.com We tried adding the other domain with comma & space separated but in vain. CDP 7.1.7, Kafka 2.5 Thanks,Uday

I want to be able to read binary Avro data from Kafka using ConsumeKafka Processor. And I am able to do that, but the content type of the file is in "application/octet-stream". Not able to view it as well, it says "No viewer is registered for this content type". I am not even able to convert this avro data to json, since the content type is octet-stream. But when i use "kafka-avro-console-consumer" on the console the data is in json. How to get this json data into Nifi?

I am trying to read a kafka topic that contains a csv file with no headers. The file is read as it should be when the csv contains two rows or more (no headers) but when I use a csv file with only one row (no headers) the processor Read from Kafka doesn't do anything (no error message, and no file ingested. (0 in 0 out)The controller services show below are : - CSVReaderCSVRecordSetWriter : 

Hello,we want to migrate data from an open source Atlas 1.2.0 to an Atlas in version 2.1.0 included in our Cloudera CDP 7.1.7 cluster.In addition, the versions of solr and hbase of the source cluster are Solr 5.5.1 and Hbase 1.1.0.To migrate Atlas data, is it sufficient to perform a backup of the hbase tables from the source hadoop cluster and a restore on the target CDP cluster hbase?Is it necessary to backup and restore Solr collections as well? I know that the Solr 5.5.1 of the source cluster is not compatible with the Solr 8.4.1 of the target cdp cluster, so if possible I would like to avoid the export/import of the collections and perform an indexing of solr after the migration of the Hbase tables . Is this the correct procedure to follow?Thanks in advance.Ivo

Hi,  We are migrating data from one db to another using NIFI. We are using GenerateTableFetch processor for the incremental fetch by setting a column in Max-Value columns. For logging purpose, i need to extract the state of the GenerateTableFetch  processor and insert in a table. By state I mean the max value of the 'Max-Value column' that is captured as the state of the processor. Can someone please help with how can i extract the processor state? Thank you.  

Hey everyone,I would like to add an attribute to a json file. The json file is a result of an api query with the Invoke HTTP procesor. As a next step I want to add an attribute to the json file to the existing fieldsI expected this:[ {"_id" : "5d55246b9ac36012c14b5e7c","gtin" : "4009602235038","target.sku : "59218_5_champagner", <- added attribute value"articleNumber" : "59218-376","styleGroupId" : "59218","colorGroupId" : "376","brand" : {"_id" : "57f75950a56ab89d1973ee36"}, Does anyone have any ideas?

Hello all,   I am a student taking a course in SQL.  The final module of this course has us installing Oracle VMWare and Cloudera to utilize Hadoop.     I went through a detailed instruction manual for the VMWare installation, including an expansion pack install and guest account install within the VMWare environment.  I ran into no problems with this installation process (linked below for reference)   Installation_ClouderaQuickstartVirtualMachine    Afterwards, we were to follow a Cloudera basic tutorial guide to get started (linked below).    Cloudera Quickstart Beginners Tutorial   However, when attempting to input the first script into Terminal as provided, I began running into errors.  The script and errors received are shown below.         [cloudera@quickstart ~]$ sqoop import-all-tables \> m 1 \> connect jdbc:mysql://quickstart:3306/retail_db \> username=retail_dba \> password=cloude

Hi EveryoneWe trying to integrate LDAP to our existing nifi server single node. After configuring ldap details, Nifi fails with timeout connection error. There is no firewall block and we are able to telnet the ldap server. Please see our setup ldap below. LDAP server :CN=Service Account\, nifi_ldap,OU=Service Accounts,OU=Enterprise Dev,DC=dev,DC=abcdeldap://ldap.dev.abcde:389password: 5$qwerty! nifi.properties *Previously our application was running as http but we change it to https*nifi.authorizer.configuration.file=./conf/authorizers.xmlnifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml# web properties ############################################### For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface.# Be aware that changing these properties may affect how your instance can be accessed without any restriction.# We recommend configuring HTTPS instead. The administrators guide provides in

Within my NiFi flow I have a number of datasets, with various schemas where the 'key' of key value pairs are on camelCase but i want any incoming keys to be outputted in snake case, does anyone know what jolt spec i can use to achieve this? Example input:{"aRandomFieldName":"Random Value"} Required Output:{"a_random_field_name":"Random Value"} Been struggling to achieve this using Jolt or replaceText processors, any assistance would be great. Thanks in advanceAndy.

Hello, one of our cluster nodes (a 3 node cluster v1.16.3 on Kubernetes) ran out of disk space. After increasing the disk space, we now get the above mentioned exception in the logs and the node does not start.What can we do to fix this and get the node to start again? Thanks a lot! Related logs:{"@timestamp":"2022-09-02T07:49:50.865Z","log.level": "WARN","message":"Encountered unexpected End-of-File when reading journal file ../flowfile_repository/journals/474192833.journal; assuming that NiFi was shutdown unexpectedly and continuing recovery","ecs.version": "1.2.0","service.name":"nifi-app.log","service.node.name":"nifi-prod-1","event.dataset":"nifi-app.log","process.thread.name":"main","log.logger":"org.apache.nifi.wali.LengthDelimitedJournal"}{"@timestamp":"2022-09-02T07:49:50.866Z","log.level": "INFO","message":"Successfully recovered 0 updates from journal ../flowfile_repository/journals/474192833.journal","ecs.version": "1.2.0","service.name":"nifi-app.log","servi

I have a csv with 50 columns, is there a way to grab a specific column by it's header name and move it in a separate flow file for further processing?

Hi,     Basically, I am trying to import a database table from MySQL to hive but it turns out not importing to hive instead it gives me the following error. can anyone please help me with a solution which would be very much appreciated. [cloudera@quickstart ~]$ sqoop import --connect jdbc:mysql://localhost:3306/sqoop --username root --password cloudera --table customer --hive-importWarning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.Please set $ACCUMULO_HOME to the root of your Accumulo installation.19/11/27 23:11:33 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.13.019/11/27 23:11:33 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.19/11/27 23:11:33 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override19/11/27 23:11:33 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.19/11/27 23:11:33 INFO manager.MySQLManager: Preparing to use a MySQL s

Hi All, We are planning to upgrade our Nifi application(Linux platform) from 1.9.2 version to 1.16.2.Before proceeding with this major version jump , we wanted to check for any possible issues and backward compatibility . We do not see entry for version jump from 1.9.x to 1.16.x in below :  https://cwiki.apache.org/confluence/display/NIFI/Migration+GuidanceWe request your guidance for anything we need to be aware of before we make jump from Nifi 1.9.2 to 1.16.2 Thanks,Alvin

Resources we have in place:7 nodes with each having 250 gb memoryvcpu = 32 per each nodeconfiguration specified in spark-defaults.conf :spark.executor.memory = 100gspark.executor.memoryOverhead = 49gspark.driver.memoryOverhead=200gspark.driver.memory = 500gQuery tried to execute :hive_context.sql("select * from 5mcsv CROSS JOIN 2mcsv").show(8000000) So, we are facing below issue when trying to fetch 8 million rows with the above mentioned query. However , we are not facing any issue with fetching 7 million rows Traceback (most recent call last): File "/root/hivespark.py", line 29, in <module> hive_context.sql("select * from 5mcsv CROSS JOIN 2mcsv").show(8000000) File "/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/lib/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 381, in show File "/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__ File "/opt/cl