Q&A

Impala 3.4 When impala-shell is started, impalad.ERROR:authentication.cc:177] SASL message (Kerberos (external)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No key table entry found for impala@node1) I don't understand why it got the wrong principal. I passed in the correct principal and keytab in the startup file. Can someone help me? Thank you very much. 

I have a JSON data with DateTime as 0000-00-00 00:00:00 & it is not been accepted in SQLPlease Support to resolve this issue for me 

Hello Team, In our customer cluster, we are testing the HDFS replication through Cloudera Manager. The replication policy looks as follows.All the other configuration is the default. The replication is hung in the below state for a long time.We looked into the Cloudera Manager logs and we can see the below error repeatedly occurring. Can you please help us to resolve the issue?2022-04-21 12:27:57,199 ERROR CommandPusher-1:com.cloudera.cmf.service.AgentResultFetcher: Exception occured while handling tempfile com.cloudera.cmf.service.AgentResultFetcher@618eac09 Best RegardsSayed Anisul Hoque 

select concat SPZ301FPL , a meter primary id as pillarid, b locationid as zoneidnumber, b remarks as zonename, f wardid as division, d sectionname as section, concat e streetname, , , e landmark as location, f latitude as pillarlatitude, f longitude as pillarlongitude, g ontime as scheduledontime, g offtime as scheduledofftime, f numberoffittings as numberoflightsconnected, f totalkw as basewattage from meter primary master a inner join locationmaster b on a locationid b locationid inner join polemaster f on a meter primary id f poleid inner join sectionmaster d on d sectionid f sectionid inner join addressmaster e on e addressid f addressid inner join currentreading g on g meter primary id a meter primary id where b locationid in 1, 2, 3, 4, 5 order by a meter primary id br br br Response Array:Array 0 { 1 access token 2 : 3 10350925 2a0d 4bbe b044 7393ae1426c6 4 , 5 token type 6 : 7 bearer 8 , 9 refresh token 10 : 11 eca77863 e681 4a5c abfb b90ea6a436f8 12 , 13 expires in 14 :9348, 1

I saved thousands of small json files in SequenceFile format to resolve the "small file issue". I use the following pyspark code to parse the json data from saved sequence files.  reader= sc.sequenceFile("/mysequencefile_dir", "org.apache.hadoop.io.Text", "org.apache.hadoop.io.Text")rdd=reader.map(lambda x: x[1])mydf=spark.read.schema(myschema).json(rdd)mydf.show(truncate=False)  The code worked. However, I do not know how to put the key value from the sequence file, which is actually the original json file name, into the mydf dataframe. Please advise. Thank you. Regards,

I have tried to connect local NIFI to Local mysql, I have been getting an error on the processor "cannot create poolableconnectionfactory communications link failure nifi". JDBC Url and Driver are correct. i assume there may be an error while connecting with Mysql. And also i have tried to Connect local NIFI in mac with local Mysql, i am facing an error while i enable the DBCP connection pool the error is mentioned in the image. Thank you

field to add "quality" input json :{"json_data": {"id": "1234","items": [{"id": "000000","metadata": {"newId": "82896888047600"}}]}} output json : {"json_data": {"id": "1234","items": [{"id": "000000","metadata": {"newId": "82896888047600","quality" :"good"}}]}}  

Hi All,I have a requirement where there are some empty curl braces in my JSON output which needs to be removed and replaced with empty string. The regex seems to work fine to find the braces when tested outside.Sample can be found here:https://regex101.com/r/AtTPv1/2 I tried to use this regex in nifi using replaceText, but it's not doing the replace. Below is my config for replaceText.  Can anyone suggest if there are any options to change to make it work or any other better alternatives for working with JSON?Thanks in advance!

I typically upload csv files into Cloudera Data Science Workbench, but I wonder if there is a way to programmatically read in a csv file from a shared server drive while in yarn mode? Using the below code, I get an error. Any Tips? df = spark.read.format('csv').load('Q:\\project\\data_folder\\file.csv', header=True) The error I get is: IllegalArgumentException: 'java.net.URISyntaxException: Relative path in absolute URI: Q:%5Cproject%5Cdata_folder/file.csv'

Hi I am trying to install CDP 7.1.7 SP1 production licensed product. Can you please confirm if the following link is the correct document I should follow to Install CDP7.1.7 SP1? https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/installation/topics/cdpdc-prod-installation.html Thanks

Team, facing issue connecting DbVisualiser to kerberised hive (hdp), An error occurred while establishing the connection:Long Message:Bad URL format. Hostname not found in authority part of the url: <hiveserevr2 host>:10000;. Are you missing a '/' after the hostname ?FYI: i followed this kb article,https://community.cloudera.com/t5/Community-Articles/Support-Video-How-to-connect-DbVisualizer-to-Hive-with/ta-p/278424 any suggestions pls?

We have an AWS loadbalancer setup for NIFI cluster. Authentication works fine when accessing the NIFI UI using the loadbalancer url. While trying to configure Site-to-Site, authorization errors are encountered. On checking the logs, seems like the NIFI API is trying to authenticate loadbalancer using the user "anonymous" which doesnot exist. Nifi UI AWS LB Url : https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi Nifi API AWS LB Url : https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api Loadbalancer : HTTPS Listener on Port 8443 Why NIFI is trying to access the api url "https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api" using "anonymous" user Snippet : nifi-app.log 2018-07-11 05:26:44,822 WARN [Timer-Driven Process Thread-7] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api due to org.apache.nifi.remote.util.SiteToSiteRestApiClient$HttpGetFailedException: response code 401:Unauthorized with ex

Hello Team, We are setting the Hive replication through Cloudera Manager. The replication policy looks as follows. Note that we also enabled the snapshot on the source cluster for the path - /warehouseHowever, when we press save policy then we get the below notification.  We looked into the Cloudera Manager logs and we can see the below error. Can you please help us to get the correct configuration to resolve the issue?2022-04-19 15:51:07,848 ERROR scm-web-1686:com.cloudera.server.web.cmf.WebController: getHiveWarehouseSnapshotsEnabledjavax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org

I'm very new to Cloudera and Spark. I need to know the specifications of the cluster I'm using, so I can better optimize my Spark session configuration (executors/cores/ram/etc). I'm trying to view these details in Cloudera Manager, but I'm not sure where to look. I want to know on the specific cluster I'm using, how many nodes it has, and how many cores and how much memory per node? Forgive my naivety, but some guidance would be appreciated!

wget https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hive/hive-serde/0.13.1-cdh5.3.0/hive-serde-0.13.1-cdh5.3.0.pom--2022-04-19 09:47:22-- https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hive/hive-serde/0.13.1-cdh5.3.0/hive-serde-0.13.1-cdh5.3.0.pomResolving repository.cloudera.com (repository.cloudera.com)... 54.164.28.72, 54.227.239.175, 54.81.245.86, ...Connecting to repository.cloudera.com (repository.cloudera.com)|54.164.28.72|:443... connected.ERROR: cannot verify repository.cloudera.com's certificate, issued by ‘CN=DigiCert TLS RSA SHA256 2020 CA1,O=DigiCert Inc,C=US’:Unable to locally verify the issuer's authority.To connect to repository.cloudera.com insecurely, use `--no-check-certificate'.

Hi all, we are planing to use Vmware VMs for our production cluster for CDP 7.1.7. Is there any recommendation on the configurations, capacity and architecture from Cloudera on this? Any link,whitepaper or documentation will be helpful.What is the difference in performance in using VMs than using physical servers? Which one is preferred and why? 

Hi,I have a flow where I make an HTTP request, and I save as an attribute the last ID value with EvaluateJsonPath and ExtractText processors. However, the InvokeHTTP processor doesn't read the saved attribute. What should I change?  I attach an image of this part of the flow. Thank you very much in advance. I'm stuck at this point and I don't know what else to try.

How do you configure a PutMongo NiFi Processor with ID as the primary key used to select the document and updating 2-3 fields inside the document with user defined attributes? What I have so far and it errors out : Update Query--> {my_id:${my_id},$set:{['a']['b']['exists_in_file']:"True"}}

we need to know if nifi is responsible for the file below: /run/cloudera-scm-agent/process/24750-nifi-NIFI_NODE/supervisor.conf because this file contains information that we are not finding in the nifi settings in the CDM

Hi,I have successfully installed CDP private cloud trial version on AWS using below instructions.https://www.cloudera.com/tutorials/how-to-create-a-cdp-private-cloud-base-development-cluster.htmlQuestion:How to access the CDP components (spark, hive, etc) from my local laptop through CLI?As part of installation, 4 hosts were created in aws, but I do not know how to access those hosts through CLI from my laptop. I am using ubuntu on my laptop.Also I am able to access all the 4 aws hosts through ssh using below command, but after getting into the aws host I am not able to invoke pyspark because I am getting below error.ssh command:ssh -i '/home/<localuser>/.ssh/cdp-trial-key.pem' centos@<aws_ip_address>Error:org.apache.hadoop.security.AccessControlException: Permission denied: user=centos, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-xMy objective:I need to learn all components in CDP, write few lines of code using python and spark, access HDFS, etc. But I am new to t

Here is the code.create external table testtable1 (code string, codesystem string)ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'WITH SERDEPROPERTIES ( "input.regex" = "(.{27)(.{50})" )LOCATION '/data/raw/testtable1';The error message is:ParseException: Syntax error in line 3:undefined: ROW FORMAT SERDE 'org.apache.hadoop.hiv... ^ Encountered: IDENTIFIER Expected: DELIMITED CAUSED BY: Exception: Syntax errorIt looks like Impala table only accepts "Row Format Delimited".Then how can I create an hive table with fixed width layout? Should I just do it outside Impala, bu through Hive, and then do other data operation on this table via Impala?Thanks.

can we delete the files that exist in the directory configured in the "Heap Dump Directory" option of the services in CDH/CM?

Hello,I receive a json from a HTTP Request. It looks like this :  { "accessToken" : "token", "expiresIn" : "12600", "refreshToken" : "token", "tokenType" : "Bearer", "usagePointsId" : "22516914714270, 22516914714270, 22516914714270"} I have the usagePointsId key containing a string with multiple values. I will need each values to include them in different API calls.I would therefore like to be able to separate the string at the comma level to extract each value.Then the idea is to create a new unique flow file for each value. This value would be associated with a "usagePointId" attribute which would be used in my API calls. Also, the others json key would be added as Attributes for each flow filesI don't know if it's possible to create multiple differents flow file based on a split attributes. I tried to use the ExecuteScript (with ECMASCRIPT) to split the string and create Flow Files with something like this : flowFile = session.get()if (!flowFi

we are having a problem with hbase, the following error is displayed: INFO client.RpcRetryingCallerImpl: Call exception, tries=6, retries=11, started=4175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.PleaseHoldException: Master is initializingat org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2982)at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1018)at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304), details=, see https://s.apache.org/timeout hbase and zookeeper have already been restarted several times and still the above error is presented the zooke

we need to know which is the responsible file that keeps the zookeeper settings, which one would it be?we need to validate if the "maxClientCnxns" option is the same as what was defined by the CDM in the zookeeper service.