Solved

Hi experts,I am trying to upgrade my current hive version from 1.2 to 2.3.8.I am able to connect to hive via hiveserver2 but when I do "show databases", there are no databases showing up. 0: jdbc:hive2://test02.domain.com:100> show databases;+----------------+| database_name |+----------------++----------------+No rows selected (0.124 seconds)0: jdbc:hive2://test02.domain.com:100> When accessing the mysql database I can see the databases in hive.mysql> use hive;Reading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -ADatabase changedmysql> select * from DBS;+-------+-----------------------+-------------------------------------------+---------+------------+------------+| DB_ID | DESC | DB_LOCATION_URI | NAME | OWNER_NAME | OWNER_TYPE |+-------+-----------------------+-------------------------------------------+---------+------------+------------+| 1 | Default Hive database | hdfs://test/apps/hiv

Hello,I need to increase the query timeout (backend_client_rpc_timeout_ms) from current 5 minutes to 30 minutes. Could anyone guide me from where in Impala Configurations I can do this change. I did some checking but cannot find any related, appreciate any assistance / guidance. Thanks Amn

CDSW version 1.7I need to add a file to every image that cdsw builds.jupter_notebook_config.py will need to be placed in the in the .jupyter directory of every project.This is because our organization needs the cdsw sessions to be culled after inactivity and jupyter notebooks prevents the built in idle timeout from functioning. 

There is a Regionserver that is not in a dead state, but it doen not have any regions and has not received any Request.This situation occurred after a memstore ERROR, and then the status of RS on the HBase Master UI was always 0. Then this RS does not show any ERROR Log anymore.Also tried to delete this RS and reinstall it, but it still shows 0 on the HBase Master UI.And still No ERROR Log... memstore error: HBase Master UI: Hadoop Release: HDP 3.1.0

We have CDH 6.1.1 cluster and Kafka cluster. I want to make possible to select data from Kafka topics using Hive. Is there any way to do it? Where I can find Kafka storage handler version for Hive 2.1.1 ?

This is just a knowledge sharing article. I had faced this issue in production and took me a day to resolve it.The workaround that I'm sharing will help you in getting your table back online in "Enabled" without deleting Zookeeper Hbase table znode or any data as a matter of fact..Here are steps to resolve it.1. Run a "get" command against hbase:meta for the affected tablehbase(main):003:0> get 'hbase:meta', '<AFFECTED_TABLE_NAME>', 'table:state' COLUMN                                   CELL  table:state                             timestamp=1551456805377, value=\x08\x022. Notice the above "value". Its pointing to \x08\x02 which is wrong. The value should either be \x08\x00 (Enabled) or \x08\x01 (Disabled)3. Edit the value manually.hbase(main):003:0> put 'hbase:meta','<AFFECTED_TABLE_NAME>','table:state',"\b\0"

I heard that Apache Flume has been removed with the update from CDH to CDP. With the update to CDP, who will do Apache Flume's functions?Help me please....

For instance: http://localhost:18080/nifi-registry/#/explorer/grid-listIs there a way to remove the `#` and still have the url work?

Hello,We are seeing concerning alert on one of our data node related to File Descriptor (Concerning: Open file descriptors: 16,410. File descriptor limit: 32,768. Percentage in use: 50.08%. Warning threshold: 50.00%.) Would appreciate any help/ guidance to fix this before it goes out of hand. [user1@myserver ~]$ ulimit -acore file size (blocks, -c) 0data seg size (kbytes, -d) unlimitedscheduling priority (-e) 0file size (blocks, -f) unlimitedpending signals (-i) 1030544max locked memory (kbytes, -l) 64max memory size (kbytes, -m) unlimitedopen files (-n) 1024pipe size (512 bytes, -p) 8POSIX message queues (bytes, -q) 819200real-time priority (-r) 0stack size (kbytes, -s) 8192cpu time (seconds, -t) unlimitedmax user processes (-u) 4096virtual memory (kbytes, -v) unlimitedfile locks (-x) unlimited [user1@myserver ~]$ cat /proc/sys/fs/file-max26161091[user1@myserver ~]$ cat /proc/sys/fs/file-nr80400 0 26161091 Thanks Amn

Hi,I have a test cluster running HDP3.15 and Ambari2.7.5 where, using Ambari config groups, I've confgured 3 datanodes with the following values of "dfs.datanode.data.dir":- dn-1: "[SSD]file:///dn_vg1/vol1_ssd" - dn-2: "[SSD]file:///dn_vg1/vol1_ssd,[SSD]file:///dn_vg2/vol2_ssd" - dn-3: "[DISK]file:///dn_vg1/vol1_disk,[SSD]file:///dn_vg3/vol3_ssd,[DISK]file:///dn_vg2/vol2_disk"   "dfs.replication" is set to 1 and the storage policies are all default (HOT).   Most of my attempts to "hdfs dfs -put" a file into HDFS fail with:   2021-03-11 14:58:33,315 WARN blockmanagement.BlockPlacementPolicy (BlockPlacementPolicyDefault.java:chooseTarget(432)) - Failed to place enough replicas, still in need of 1 to reach 1 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlaceme

What browsers are supported by CDP 7.1.4?help me!

So we're running Cloudera 6.2.1 which has Hbase 2.1.2  We'd like to provide row and cell level security. Looking at this guide...https://docs.cloudera.com/documentation/enterprise/6/6.2/topics/admin_hbase_security.html#id_v2w_bv3_tw ...I have to set a brunch of properties. hbase.security.exec.permission.checks=truehbase.security.access.early_out=falsehfile.format.version=3 Also I need to toggle on the 'Enable Hbase authorization' flag. I assume I also have to configure the (the VisibilityController and AccessController) coprocessors?  There is this vague statement in the document mentioned about.---Optionally, search for and configure HBase Coprocessor Master Classes and HBase Coprocessor Region Classes.---Also I see in Cloudera 6.2.1 manager Hbase configuration toggles for 'Enable Row level authorization' and 'Enable Cell ACLs'.  Do I need to toggle those on too?  There is no mention of these in the setup. It's a bit confusing. Ri

I have a Ranger (v1.1.0) and UserSync integration with LDAP. All of a sudden I am seeing user-sync error and new user/groups are not getting syn'd. Looks like something is timing-out but not sure what and how to fix that.Error logs when default log4j.rootLogger = info,logFile 10 Mar 2021 01:50:55 INFO UnixAuthenticationService [main] - Starting User Sync Service!10 Mar 2021 01:50:55 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.username.regex10 Mar 2021 01:50:55 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.groupname.regex10 Mar 2021 01:50:55 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder created10 Mar 2021 01:50:55 INFO UserGroupSync [UnixUserSyncThread] - initializing sink: org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder10 Mar 2021 01:50:56 WARN NativeCodeLoader [UnixUserSyncThread] - Unable to load native-hadoop library for your platform... using builti

Hi all,I'm having trouble during the provisioning of an environment via cloudera manager cloud console.I followed the quick start, https://docs.cloudera.com/management-console/cloud/azure-quickstart/topics/mc-azure-quickstart.html and the guide on the repository https://github.com/cpv0310/cdp-azure-tools, but the problem remains the same:hdfs can't write to storage abfs: // data @ xxx I tried to create the managed identity both through the template and through the script provided, but I have not had any changes. The only different thing is that in the guide, step 6, it says to assign both assumer identity and data identity, but in the form I only have the possibility to assign the assumer identity. Same thing when I go to assign the identity logger: I only have one slot and I can't assign the identity ranger.  In the logs I see that the creation of the data lake stops trying to create the first folder on HDFS (abfs) and the error is on the "slave" node which through

We have a valid license and and login to the paywall, which I have tested by using parcel URLs in a browser and from the command line (Ubuntu) with wget. Both show I can obtain a directory listing from behind the paywall. However, despite entering this URL in the configuration for the Parcel window in CM, I still do not see new available parcels. We are on 6.3.1 on one cluster and 6.3.2 on another, it's the same for both, no new parcels listed. We presumably should see at least 6.3.4, if not 6.3.3 as well. I first tried with the URL format given by the documentation (obviously replacing username:password with our actual info), and a slightly different URL I was given in a support ticket I raised. I get no errors, but still no new parcels showing as available. Has anyone else experienced this?

Hi,This is my first post here! I recently started working on Cloudera Work Bench.I'm working on Cloudera Work Bench with a python code that imports pyspark. This code works fine outside of Cloudera workbench (for example on the node of a cluster) but I am struggling to make it work there. It seems I have problems with python UDF function, the web seems filled with people that have similar problems with no answers. The code runs and loads spark dataframes in memory successfully until it calls a .take(1) on a Spark dataframe which functions inside contain an udf function.   I get the error message pasted below. It seems it does not find the module called myPersonalModule, but until that moment the code had lots of import myPersonalModule and had lots of functions used from it. I tried to pass the module as a source file to the workers with  spark.sparkContext.addPyFile("myPersonalModule.zip") but then I get the same error with Pandas (I paste it later). This time the

I am running a Spark action on Oozie in a Cloudera 7.1.4 cluster.The action itself completes successfully but in the logs there is a stack trace showing the atlas hook failed. It looks like the application is looking for the Spark SQL Kafka 0.10 Jar. <<< Invocation of Main class completed <<<2021-03-08 09:31:00,968 [SparkExecutionPlanProcessor-thread] WARN com.hortonworks.spark.atlas.sql.SparkExecutionPlanProcessor - Caught exception during parsing eventjava.lang.ClassNotFoundException: org.apache.spark.sql.kafka010.KafkaRelation at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at com.hortonworks.spark.atlas.utils.ReflectionHelper$.classForName(ReflectionHelper.scala:113) at org.apache.spark.sql.kafka01

Can anyone please help me a JOLT spec to convert my matrix type json to denormalized json. Please find the below my input json and my expected josn output.Input Json:[ { "attributes": [ { "name": "brand", "value": "Patriot Lighting" }, { "name": "color", "value": "Chrome" }, { "name": "price", "value": "49.97 USD" } ] }, { "attributes": [ { "name": "brand", "value": "Masterforce" }, { "name": "color", "value": "Green" }, { "name": "price", "value": "99.0 USD" } ] }]Expected Json output:[ { "brand": "Patriot Lighting", "color": "Chrome", "price": "49.97 USD" }, { "brand": "Masterforce", "color": "Green", "price": "99.0 USD" } ]I was trying to build JOLT spec to convert this json. But challenge is the json which I have multiple tables with "attributes" tag. @Json @NIFI

Hi @Nifi !I created a flow in local nifi, now i ve to do the deployment in virtual machine azure.I use docker compose to do that and it works but the ListFile processor to ingest the file doesn't accept the directory path of virtual machine.I insert the path that i find from virtual machine terminal (with the command pwd) but doesn't work.How can i ingest the file that i ve load on virtual machine?thanks 

Ambari : 2.7.4HDP : 3.1.4.0-315 Hi everyone,Recently after kerberizing my cluster, I am unable to access my Resource Manager UI. Whenever I hit on the resource manager link on Ambari, it shows up the following error :  Please help me out to resolve this issue as soon as possible.@KuldeepK @Shelton @jsensharma  

I have folders with date as their name: ./20141002./20141009./20141016./20141023 How do I get files from the latest folder based on the names above? I know this command find . ! -path . -type d | sort -nr | head -1works but how do I do it with a NiFi processor?

Hello,I'm somewhat confused about the hdfs block count threshold configuration.As a rule of thumb, I read about setting  the threshold like 1 GB Java Heap Size of NameNode ~= 1M Blocks so with our setting of 10GB it would mean the threshold would be 10M Blocks but then I read the replication factor ( x3 standard ) isn't included ?So does this mean I have to set the threshold to 3,3M Blocks if I set the Java Heap Size of NameNode to 10GB ?thx Martin

What are the steps for Solr Replication between two CDP PvC 7.1.4 clusters for disaster recovery (DR), across two data centers.

Hello,    We've recently migrated CDH from 5.16.2 to 6.3.3 and we now have the following message when we create a table using Impala JDBC driver (we are using the latest version 2.6.15.1017). errorMessage:ImpalaRuntimeException: Error creating Kudu table 'impala::pdo.discoveries'CAUSED BY: IllegalArgumentException: table owner must not be null or empty   When the same "create table" statement is executed in impala-shell, it works perfectly well. The search for this error message in google did not find much result.   Any idea about the root cause ?  

Hello, I would like to test ANY version of CDH but I need a username/password to download files. eg:https://username:password@archive.cloudera.com/p/cm7/7.2.4/redhat7/yum/cloudera-manager.repo But when I want to get one, I'm able to find where to get it ? I just want to install a trial version. If I navigate until Download webpage, I get : ======================================Access RestrictedYour MyCloudera account is not currently connected with a CDH subscription which is required for CDH 6.3.3 and later. Please contact Cloudera Support for assistance if you believe you should have access. We apologize for any inconvenience.Older Versions of CDHOlder versions of CDH and Cloudera Manager can be downloaded directly from the URLs in our public documentation:CDH 6.xCDH 5.x====================== And I want CDH 5.x Thanks for your help Regards Cédric