Solved

I typically upload csv files into Cloudera Data Science Workbench, but I wonder if there is a way to programmatically read in a csv file from a shared server drive while in yarn mode? Using the below code, I get an error. Any Tips? df = spark.read.format('csv').load('Q:\\project\\data_folder\\file.csv', header=True) The error I get is: IllegalArgumentException: 'java.net.URISyntaxException: Relative path in absolute URI: Q:%5Cproject%5Cdata_folder/file.csv'

Hi I am trying to install CDP 7.1.7 SP1 production licensed product. Can you please confirm if the following link is the correct document I should follow to Install CDP7.1.7 SP1? https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/installation/topics/cdpdc-prod-installation.html Thanks

We have an AWS loadbalancer setup for NIFI cluster. Authentication works fine when accessing the NIFI UI using the loadbalancer url. While trying to configure Site-to-Site, authorization errors are encountered. On checking the logs, seems like the NIFI API is trying to authenticate loadbalancer using the user "anonymous" which doesnot exist. Nifi UI AWS LB Url : https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi Nifi API AWS LB Url : https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api Loadbalancer : HTTPS Listener on Port 8443 Why NIFI is trying to access the api url "https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api" using "anonymous" user Snippet : nifi-app.log 2018-07-11 05:26:44,822 WARN [Timer-Driven Process Thread-7] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api due to org.apache.nifi.remote.util.SiteToSiteRestApiClient$HttpGetFailedException: response code 401:Unauthorized with ex

Hello Team, We are setting the Hive replication through Cloudera Manager. The replication policy looks as follows. Note that we also enabled the snapshot on the source cluster for the path - /warehouseHowever, when we press save policy then we get the below notification.  We looked into the Cloudera Manager logs and we can see the below error. Can you please help us to get the correct configuration to resolve the issue?2022-04-19 15:51:07,848 ERROR scm-web-1686:com.cloudera.server.web.cmf.WebController: getHiveWarehouseSnapshotsEnabledjavax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org

I'm very new to Cloudera and Spark. I need to know the specifications of the cluster I'm using, so I can better optimize my Spark session configuration (executors/cores/ram/etc). I'm trying to view these details in Cloudera Manager, but I'm not sure where to look. I want to know on the specific cluster I'm using, how many nodes it has, and how many cores and how much memory per node? Forgive my naivety, but some guidance would be appreciated!

wget https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hive/hive-serde/0.13.1-cdh5.3.0/hive-serde-0.13.1-cdh5.3.0.pom--2022-04-19 09:47:22-- https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hive/hive-serde/0.13.1-cdh5.3.0/hive-serde-0.13.1-cdh5.3.0.pomResolving repository.cloudera.com (repository.cloudera.com)... 54.164.28.72, 54.227.239.175, 54.81.245.86, ...Connecting to repository.cloudera.com (repository.cloudera.com)|54.164.28.72|:443... connected.ERROR: cannot verify repository.cloudera.com's certificate, issued by ‘CN=DigiCert TLS RSA SHA256 2020 CA1,O=DigiCert Inc,C=US’:Unable to locally verify the issuer's authority.To connect to repository.cloudera.com insecurely, use `--no-check-certificate'.

Hi all, we are planing to use Vmware VMs for our production cluster for CDP 7.1.7. Is there any recommendation on the configurations, capacity and architecture from Cloudera on this? Any link,whitepaper or documentation will be helpful.What is the difference in performance in using VMs than using physical servers? Which one is preferred and why? 

Hi,I have successfully installed CDP private cloud trial version on AWS using below instructions.https://www.cloudera.com/tutorials/how-to-create-a-cdp-private-cloud-base-development-cluster.htmlQuestion:How to access the CDP components (spark, hive, etc) from my local laptop through CLI?As part of installation, 4 hosts were created in aws, but I do not know how to access those hosts through CLI from my laptop. I am using ubuntu on my laptop.Also I am able to access all the 4 aws hosts through ssh using below command, but after getting into the aws host I am not able to invoke pyspark because I am getting below error.ssh command:ssh -i '/home/<localuser>/.ssh/cdp-trial-key.pem' centos@<aws_ip_address>Error:org.apache.hadoop.security.AccessControlException: Permission denied: user=centos, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-xMy objective:I need to learn all components in CDP, write few lines of code using python and spark, access HDFS, etc. But I am new to t

Here is the code.create external table testtable1 (code string, codesystem string)ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'WITH SERDEPROPERTIES ( "input.regex" = "(.{27)(.{50})" )LOCATION '/data/raw/testtable1';The error message is:ParseException: Syntax error in line 3:undefined: ROW FORMAT SERDE 'org.apache.hadoop.hiv... ^ Encountered: IDENTIFIER Expected: DELIMITED CAUSED BY: Exception: Syntax errorIt looks like Impala table only accepts "Row Format Delimited".Then how can I create an hive table with fixed width layout? Should I just do it outside Impala, bu through Hive, and then do other data operation on this table via Impala?Thanks.

can we delete the files that exist in the directory configured in the "Heap Dump Directory" option of the services in CDH/CM?

Hello,I receive a json from a HTTP Request. It looks like this :  { "accessToken" : "token", "expiresIn" : "12600", "refreshToken" : "token", "tokenType" : "Bearer", "usagePointsId" : "22516914714270, 22516914714270, 22516914714270"} I have the usagePointsId key containing a string with multiple values. I will need each values to include them in different API calls.I would therefore like to be able to separate the string at the comma level to extract each value.Then the idea is to create a new unique flow file for each value. This value would be associated with a "usagePointId" attribute which would be used in my API calls. Also, the others json key would be added as Attributes for each flow filesI don't know if it's possible to create multiple differents flow file based on a split attributes. I tried to use the ExecuteScript (with ECMASCRIPT) to split the string and create Flow Files with something like this : flowFile = session.get()if (!flowFi

we are having a problem with hbase, the following error is displayed: INFO client.RpcRetryingCallerImpl: Call exception, tries=6, retries=11, started=4175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.PleaseHoldException: Master is initializingat org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2982)at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1018)at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304), details=, see https://s.apache.org/timeout hbase and zookeeper have already been restarted several times and still the above error is presented the zooke

we need to know which is the responsible file that keeps the zookeeper settings, which one would it be?we need to validate if the "maxClientCnxns" option is the same as what was defined by the CDM in the zookeeper service.

Hello,Is it possible to send dynamic values using some external commands like "curl" so that values will be used in the process?For example we will be send date dynamically then it can be used in SQL query.

I am facing the following error on the cmd:   command: grunt> A =  LOAD 'C:\data\myfile.txt'; or  grunt> a = load 'c:\data\myfile.txt' using PigStorage(' ') as (f1:int,f2:int, f3:int); error:<line 1, column 14>  Unexpected character 'D'2020-03-15 20:56:50,153 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1200: <line 1, column 14>  Unexpected character 'D'Details at logfile: C:\deploy\logs\pig_1584281243336.log     the error trace is as follows-  Pig Stack Trace---------------ERROR 1200: <line 1, column 14>  Unexpected character 'D' Failed to parse: <line 1, column 14>  Unexpected character 'D' at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:246) at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:182) at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1792) at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1765

When I am connecting via Cloudera ODBC Driver the process fails due to following error: [Cloudera][Support] (50170) Creation of new thread failed.  Does anyone know what the issue is and can help here? I did not find anything concerning this error.

I'm having a problem trying to create a nifi cluster with kubernetes.Error displayed: "The specified run.as user ec2-user does not exist. Exiting."The error appears when I use /opt/nifi/nifi-current/conf folder mappingIf I remove this mapping from the nifi folder, the cluster is created without problems.But I need this folder to keep the data flows if the container dies, come back with the datathe folder in the instance is in /home/ec2-user/user/1.16I already gave permission 777 on the folder to test and it still didn't workI already changed the user to upload with root user and nifi (pid 0 and 1000) and it didn't work

Hello Community, I'm running into a problem when I submit a job in Hue editor, from Hue home page I navigate to Query > Editor > Java. I point to the path of my jar file and then add the class name called Hortonwork.SparkTutorial.Main. Then after I hit run I get this error   "Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf".   I imagine it has to with not having the jar file SparkConf, I'm using IntelliJ for this project and I've added some jars to my project, I've taken some screen shots of what the jar files look like and have attached them to the post. Curious if any of these jars look familiar to anyone, I may not even be adding the right ones.    Thank you all for the taking the time to read my post and for your help.    Full list error is:    >>> Invoking Main class now >>>INFO: loading log4j config file log4j.properties.INFO: log4j config file log4j.properties loaded successfully.Setting [

I have literally used the VM out of the box. Trying to train on materials and can't get anything working. Been trying to resolve this issue for a  week. Just finished a Codementor session and they couldn't get it either. if the password is not default (and I literally haven't touched anything) what is next?  Hive access denied.  Someone suggested changing something in ranger. I don't know if that's true or not. Asked a guy on Codementor. Just trying to get data into the table I created.Delete this VM and try again or?  I have logged in and ran hive as root, maria_dev, hive and hdfs. (btw, don't know what I am doing, not being trained properly, have zero tech support and am expected to learn this for work clients and am not supposed to have any problems that need solving. (shrugs)  

Hi, I am trying to perform a scripted start / stop of HDP services and its components. I need to get this done service by service, because there are few components like Hive Interactive / Spark Thrift server / Kafka broker, which does not get started / stopped in proper time. Most of the times it is observed that even if HSInteractive is still starting (visible in the background operations panel on Ambari), command moves to next service to start / stop and ultimately HSI fails. Hence, I also want to ensure that previous service is completely stopped / started, before attempting to stop / start next service in the list. To achieve, below is the script I have written (this is to stop, I have similar for start). However, when it reaches to the point for Hive or Spark or Kafka services, even though the the internal components like Hive Interactive or Spark Thrift server or Kafka broker are not started / stopped, it moves to the next service. Most of the times the start of Spark T

Hi Everyone!   I've been receiving the following error on my Cloudera Manager.   "603.1 MiB of swap memory is being used by this role's process. Warning threshold: 200.0 MiB."     I have tried removing Solr collections by going to the following location and running the following query.   http://<HOSTNAME>:8993/solr/#/ranger_audits/documents<delete><query>evtTime:[* TO NOW-0DAYS]</query></delete>   But it's not having any effect on the Swap memory usage.      P.S. Swappiness is set to 1 as when I run  cat /proc/sys/vm/swappiness The output generated is 1.      Cluster Details :  Solr: 8.4.1 Hadoop 3.1.1.7.1.6.0-297 CDH-7.1.6

Hi everyone   I want to add 2 servers with KTS and 2 servers with KMS to my cluster. The question is : Do I need licenses for each of this additional servers?   Thanks

We would like to understand two behaviors that Kudu is showing. 1. When restarting the Kudu Tablet, it takes between 5 and 10min for this Kudu Tablet to communicate with the Kudu Master again, why is this happening? 2. There are 5 Kudu Tablets and two of those Kudu Tablets is using more than 90% of the memory that was set in "memory_limit_hard_bytes", why is this happening?

recently i have set up a new CDH cluster with all SSD disk. after this cluster goes live , i found the namenode log always output some WARNING log, as below:  2022-03-26 06:00:57,688 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 3 to reach 3 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology2022-03-26 06:00:57,688 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 3 to reach 3 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]}, newBlock=true) For more information, please en

We are planning to automate our hive schema changes as part of our CI/CD deployment. Currently we are using Gitlab to manage our repo and execute deployment pipeline. One of the tools we're considering is Liquibase. This blog detailed out the way to implement this https://docs.liquibase.com/install/tutorials/impala-hive.html. It explains that in order to use Liquibase with Hive, it needs an impala/hive extensions. This approach seems promising. However I'm concerned that the extension is outdated and may impose security risk in our system since the last update of it was 5 years ago (https://github.com/eselyavka/liquibase-impala).   I was wondering if anyone here have the same use case. What approach/options did you choose to implement the solution? I appreciate any comments or suggestion from this groups.   Thank you!