Q&A

Hello,We are seeing concerning alert on one of our data node related to File Descriptor (Concerning: Open file descriptors: 16,410. File descriptor limit: 32,768. Percentage in use: 50.08%. Warning threshold: 50.00%.) Would appreciate any help/ guidance to fix this before it goes out of hand. [user1@myserver ~]$ ulimit -acore file size (blocks, -c) 0data seg size (kbytes, -d) unlimitedscheduling priority (-e) 0file size (blocks, -f) unlimitedpending signals (-i) 1030544max locked memory (kbytes, -l) 64max memory size (kbytes, -m) unlimitedopen files (-n) 1024pipe size (512 bytes, -p) 8POSIX message queues (bytes, -q) 819200real-time priority (-r) 0stack size (kbytes, -s) 8192cpu time (seconds, -t) unlimitedmax user processes (-u) 4096virtual memory (kbytes, -v) unlimitedfile locks (-x) unlimited [user1@myserver ~]$ cat /proc/sys/fs/file-max26161091[user1@myserver ~]$ cat /proc/sys/fs/file-nr80400 0 26161091 Thanks Amn

Hi,I have a test cluster running HDP3.15 and Ambari2.7.5 where, using Ambari config groups, I've confgured 3 datanodes with the following values of "dfs.datanode.data.dir":- dn-1: "[SSD]file:///dn_vg1/vol1_ssd" - dn-2: "[SSD]file:///dn_vg1/vol1_ssd,[SSD]file:///dn_vg2/vol2_ssd" - dn-3: "[DISK]file:///dn_vg1/vol1_disk,[SSD]file:///dn_vg3/vol3_ssd,[DISK]file:///dn_vg2/vol2_disk"   "dfs.replication" is set to 1 and the storage policies are all default (HOT).   Most of my attempts to "hdfs dfs -put" a file into HDFS fail with:   2021-03-11 14:58:33,315 WARN blockmanagement.BlockPlacementPolicy (BlockPlacementPolicyDefault.java:chooseTarget(432)) - Failed to place enough replicas, still in need of 1 to reach 1 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlaceme

What browsers are supported by CDP 7.1.4?help me!

So we're running Cloudera 6.2.1 which has Hbase 2.1.2  We'd like to provide row and cell level security. Looking at this guide...https://docs.cloudera.com/documentation/enterprise/6/6.2/topics/admin_hbase_security.html#id_v2w_bv3_tw ...I have to set a brunch of properties. hbase.security.exec.permission.checks=truehbase.security.access.early_out=falsehfile.format.version=3 Also I need to toggle on the 'Enable Hbase authorization' flag. I assume I also have to configure the (the VisibilityController and AccessController) coprocessors?  There is this vague statement in the document mentioned about.---Optionally, search for and configure HBase Coprocessor Master Classes and HBase Coprocessor Region Classes.---Also I see in Cloudera 6.2.1 manager Hbase configuration toggles for 'Enable Row level authorization' and 'Enable Cell ACLs'.  Do I need to toggle those on too?  There is no mention of these in the setup. It's a bit confusing. Ri

I have a Ranger (v1.1.0) and UserSync integration with LDAP. All of a sudden I am seeing user-sync error and new user/groups are not getting syn'd. Looks like something is timing-out but not sure what and how to fix that.Error logs when default log4j.rootLogger = info,logFile 10 Mar 2021 01:50:55 INFO UnixAuthenticationService [main] - Starting User Sync Service!10 Mar 2021 01:50:55 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.username.regex10 Mar 2021 01:50:55 INFO AbstractMapper [UnixUserSyncThread] - Initializing for ranger.usersync.mapping.groupname.regex10 Mar 2021 01:50:55 INFO LdapUserGroupBuilder [UnixUserSyncThread] - LdapUserGroupBuilder created10 Mar 2021 01:50:55 INFO UserGroupSync [UnixUserSyncThread] - initializing sink: org.apache.ranger.unixusersync.process.PolicyMgrUserGroupBuilder10 Mar 2021 01:50:56 WARN NativeCodeLoader [UnixUserSyncThread] - Unable to load native-hadoop library for your platform... using builti

Hi all,I'm having trouble during the provisioning of an environment via cloudera manager cloud console.I followed the quick start, https://docs.cloudera.com/management-console/cloud/azure-quickstart/topics/mc-azure-quickstart.html and the guide on the repository https://github.com/cpv0310/cdp-azure-tools, but the problem remains the same:hdfs can't write to storage abfs: // data @ xxx I tried to create the managed identity both through the template and through the script provided, but I have not had any changes. The only different thing is that in the guide, step 6, it says to assign both assumer identity and data identity, but in the form I only have the possibility to assign the assumer identity. Same thing when I go to assign the identity logger: I only have one slot and I can't assign the identity ranger.  In the logs I see that the creation of the data lake stops trying to create the first folder on HDFS (abfs) and the error is on the "slave" node which through

We have a valid license and and login to the paywall, which I have tested by using parcel URLs in a browser and from the command line (Ubuntu) with wget. Both show I can obtain a directory listing from behind the paywall. However, despite entering this URL in the configuration for the Parcel window in CM, I still do not see new available parcels. We are on 6.3.1 on one cluster and 6.3.2 on another, it's the same for both, no new parcels listed. We presumably should see at least 6.3.4, if not 6.3.3 as well. I first tried with the URL format given by the documentation (obviously replacing username:password with our actual info), and a slightly different URL I was given in a support ticket I raised. I get no errors, but still no new parcels showing as available. Has anyone else experienced this?

Hi,This is my first post here! I recently started working on Cloudera Work Bench.I'm working on Cloudera Work Bench with a python code that imports pyspark. This code works fine outside of Cloudera workbench (for example on the node of a cluster) but I am struggling to make it work there. It seems I have problems with python UDF function, the web seems filled with people that have similar problems with no answers. The code runs and loads spark dataframes in memory successfully until it calls a .take(1) on a Spark dataframe which functions inside contain an udf function.   I get the error message pasted below. It seems it does not find the module called myPersonalModule, but until that moment the code had lots of import myPersonalModule and had lots of functions used from it. I tried to pass the module as a source file to the workers with  spark.sparkContext.addPyFile("myPersonalModule.zip") but then I get the same error with Pandas (I paste it later). This time the

I am running a Spark action on Oozie in a Cloudera 7.1.4 cluster.The action itself completes successfully but in the logs there is a stack trace showing the atlas hook failed. It looks like the application is looking for the Spark SQL Kafka 0.10 Jar. <<< Invocation of Main class completed <<<2021-03-08 09:31:00,968 [SparkExecutionPlanProcessor-thread] WARN com.hortonworks.spark.atlas.sql.SparkExecutionPlanProcessor - Caught exception during parsing eventjava.lang.ClassNotFoundException: org.apache.spark.sql.kafka010.KafkaRelation at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at com.hortonworks.spark.atlas.utils.ReflectionHelper$.classForName(ReflectionHelper.scala:113) at org.apache.spark.sql.kafka01

Can anyone please help me a JOLT spec to convert my matrix type json to denormalized json. Please find the below my input json and my expected josn output.Input Json:[ { "attributes": [ { "name": "brand", "value": "Patriot Lighting" }, { "name": "color", "value": "Chrome" }, { "name": "price", "value": "49.97 USD" } ] }, { "attributes": [ { "name": "brand", "value": "Masterforce" }, { "name": "color", "value": "Green" }, { "name": "price", "value": "99.0 USD" } ] }]Expected Json output:[ { "brand": "Patriot Lighting", "color": "Chrome", "price": "49.97 USD" }, { "brand": "Masterforce", "color": "Green", "price": "99.0 USD" } ]I was trying to build JOLT spec to convert this json. But challenge is the json which I have multiple tables with "attributes" tag. @Json @NIFI

Hi @Nifi !I created a flow in local nifi, now i ve to do the deployment in virtual machine azure.I use docker compose to do that and it works but the ListFile processor to ingest the file doesn't accept the directory path of virtual machine.I insert the path that i find from virtual machine terminal (with the command pwd) but doesn't work.How can i ingest the file that i ve load on virtual machine?thanks 

Ambari : 2.7.4HDP : 3.1.4.0-315 Hi everyone,Recently after kerberizing my cluster, I am unable to access my Resource Manager UI. Whenever I hit on the resource manager link on Ambari, it shows up the following error :  Please help me out to resolve this issue as soon as possible.@KuldeepK @Shelton @jsensharma  

I have folders with date as their name: ./20141002./20141009./20141016./20141023 How do I get files from the latest folder based on the names above? I know this command find . ! -path . -type d | sort -nr | head -1works but how do I do it with a NiFi processor?

Hello,I'm somewhat confused about the hdfs block count threshold configuration.As a rule of thumb, I read about setting  the threshold like 1 GB Java Heap Size of NameNode ~= 1M Blocks so with our setting of 10GB it would mean the threshold would be 10M Blocks but then I read the replication factor ( x3 standard ) isn't included ?So does this mean I have to set the threshold to 3,3M Blocks if I set the Java Heap Size of NameNode to 10GB ?thx Martin

What are the steps for Solr Replication between two CDP PvC 7.1.4 clusters for disaster recovery (DR), across two data centers.

Hello,    We've recently migrated CDH from 5.16.2 to 6.3.3 and we now have the following message when we create a table using Impala JDBC driver (we are using the latest version 2.6.15.1017). errorMessage:ImpalaRuntimeException: Error creating Kudu table 'impala::pdo.discoveries'CAUSED BY: IllegalArgumentException: table owner must not be null or empty   When the same "create table" statement is executed in impala-shell, it works perfectly well. The search for this error message in google did not find much result.   Any idea about the root cause ?  

Hello, I would like to test ANY version of CDH but I need a username/password to download files. eg:https://username:password@archive.cloudera.com/p/cm7/7.2.4/redhat7/yum/cloudera-manager.repo But when I want to get one, I'm able to find where to get it ? I just want to install a trial version. If I navigate until Download webpage, I get : ======================================Access RestrictedYour MyCloudera account is not currently connected with a CDH subscription which is required for CDH 6.3.3 and later. Please contact Cloudera Support for assistance if you believe you should have access. We apologize for any inconvenience.Older Versions of CDHOlder versions of CDH and Cloudera Manager can be downloaded directly from the URLs in our public documentation:CDH 6.xCDH 5.x====================== And I want CDH 5.x Thanks for your help Regards Cédric 

Hello!I installed Nifi, Nifi toolkit.I have an error "Unknown user with identity 'CN=admin, OU=people, DC=nifi, DC=com'. Contact the system administrator." or "Unable to locate initial admin CN=admin,OU=people,DC=nifi,DC=com to seed policies".Ldap config -> ldap.pngnifi.properties:# security properties #nifi.sensitive.props.key=nifi.sensitive.props.key.protected=nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSLnifi.sensitive.props.provider=BCnifi.sensitive.props.additional.keys=nifi.security.keystore=./conf/keystore.jksnifi.security.keystoreType=jksnifi.security.keystorePasswd=C952nQSnvjyqA38F4Z5WHB80qzzCGPjB8qCrfIUTXWsnifi.security.keyPasswd=C952nQSnvjyqA38F4Z5WHB80qzzCGPjB8qCrfIUTXWsnifi.security.truststore=./conf/truststore.jksnifi.security.truststoreType=jksnifi.security.truststorePasswd=0f9v05dN9t15lVRklQ3B6jGphb5fbtfYuEKjefQyb7Anifi.security.needClientAuth=yesnifi.security.user.authorizer=managed-authorizernifi.security.user.login.identity.provider=ldap-provide

Hi,Can someone please tell me what this error means when trying to connect to hiveserver2 via beeline and how to solve this? [root@test1 ~]# beelineBeeline version 1.2.1000.2.6.1.0-129 by Apache Hivebeeline> !connect jdbc:hive2://test.domain.com:10000/test;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2Connecting to jdbc:hive2://test.domain.com:10000/test;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2Enter username for jdbc:hive2://test.com:10000/test;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: nameEnter password for jdbc:hive2://test.domain.com:10000/test;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: ******************21/03/01 10:16:28 [main-SendThread(test.com:10000)]: WARN zookeeper.ClientCnxn: Session 0x0 for server test.domain.com/10.49.0.42:10000, unexpected error, closing socket connection and attempting reconnectjava.io.IOException: Packet len67108864 is out of range!at org.apache.zookeeper.ClientCnxnSocke

Hello, I have a 18.04 bionic cluster using the Cloudera Manager 7.1.3 and CDH 7.1.3 parcels.   I need to install CDS3 (Spark 3.0.0) on my cluster but I can't find any parcels for the bionic version, only xenial, wheezy and trusty.     Does it exist? Can you also publish it for bionic?   Thank you !!

Hi,I've enabled HA successfully. My directory for the three journal node is not the same accross that 3 journal node. There are alert of required JournalNode Default Group box. Is it safe to I put anything here? as my journal node location is not the same? It is only for managing that 3 journal node loaction right? Thanks 

Hi,I was wondering if there is anyway to upgrade our HIVE from an existing HDP cluster?So we have deployed previously a HDP cluster, but the hortonworks public url is currently not accessible, so I was wondering if there was any other way to upgrade our HIVE. Can someone help please point me in the right direction? Thanks,

Hello. I'm a data analyst who was unfortunately tasked to fix an issue with hive. I triedRestarting all HDFS Components - successRestarting all Hive Components - failed.Restart HiveServer2 has failed 1 / 4 nodes have failed. The node that failed is not the master nodeOther components was started succesfully.

Hi,I have json flowfile,  [{"prediction":"Test2"},{"prediction":"Test2"}]. I am want extract the 1st part of the array ie prediction":"Test2  my destination is flowfile-attribute using EvaluateJsonPath . I have added a custom attribute mlresult --->$.prediction[0], to extract the "Test2" into mlresult, but I am getting empty string  for the mlresults for the flow file attribute section. flow file : [{"prediction":"Test2"},{"prediction":"Test2"}]EvaluateJsonPath configuration:     How to get the value(only one)  Test2 to mlresult? Thanks--Murali   

Hi,Can someone please help me and let me know what I did wrong or am I missing something? [root@sandbox ~]# yarn log -applicationId application_1614345689259_0002Picked up _JAVA_OPTIONS: -Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=trueJava HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0Error: Could not find or load main class log[root@sandbox~]#  Thanks,