Find answers, ask questions, and share your expertise
Recently active
I have one Json file and I have validate it, Its a valid json.I am creating Hive table on that file and then select data but its showingError: java.io.IOException:org.apache.hadoop.hive.serde2.SerDeException: Row is not a valid JSON Object- JSONException: A JSONObject text must end with '}' at 2 [character 3 line 1](state=,code=0)when I tried to set below property.ALTER TABLE test_rahul SET SERDEPROPERTIES ( "ignore.malformed.json" = "true");Then It showing Null values for all the fields. Can someone Kindly Help me.json File:{"buyer": {"legalBusinessName": "test1 Company","organisationIdentifications": [{ "type": "abcd","identification": "test.bb@tesr"},{"type": "TXID","identification": "12345678"}]},"supplier": {"legalBusinessName": "test Company","organisationIdentifications": [{"type":"abcd","identification": "test28@test"}]},"paymentRecommendationId": "1234-5678-9876-2212-123456","excludedRemittanceInformation": [],"recommendedPaymentInstructions": [{"executionDate": "2022-06-12","
Hi I see that Cloudera Manager and CDP Private Cloud Base versions 7.6.7 and 7.1 can be install on redhat 8.4. I have servers redhat 8.4 but I did a yum update and now are redhat 8.5, my question is, can I install CDP in this linux version? redhat 8.5?Thanks
I'm having difficulty with this processor and making sure I get all the security stuff correct. I've got no problem with everything downstream, but when it comes to Keystore/Truststore stuff I'm like a potato in a forgotten cellar. For now, just to test, I have this (nifi 1.15.3): And the SSL Context:Where my jks was generated using (with testing values)keytool -genkey -alias nifi-ftps -keyalg RSA -keypass MyKeyPass312 -storepass MyStorePass312 -keystore nftp.jks When I log on with Filezilla it gets to this point:Status: Connecting to my.server.ip...Status: Using username "nifitest".Status: Access deniedError: Authentication failed.Error: Critical error: Could not connect to server Which I assume means the ports etc are all good and it can connect to the Processor - but I'm missing something in my SSL settings. I have tried changing FileZilla to use a key file but it only uses PPK or PEM files. This is my java version: sudo java -versionopenjdk
I have written python function and run in pyspark - cloudera shell but it throwing error while running it. But when I run the same function in my pyspark - local shell it's working perfectly fine. Complete Code:from pyspark.sql.functions import *df=spark.createDataFrame([('America/New_York','2020-02-01 10:00:00'),('Europe/Lisbon','2020-02-01 10:00:00'),('Europe/Madrid','2020-02-01 10:00:00'),('Europe/London', '2020-02-01 10:00:00'),('America/Sao_Paulo', '2020-02-01 10:00:00')],["OriginTz","Time"])df2=spark.createDataFrame([('Africa/Nairobi', '2020-02-01 10:00:00'),('Asia/Damascus', '2020-02-01 10:00:00'),('Asia/Singapore', '2020-02-01 10:00:00'),('Atlantic/Bermuda','2020-02-01 10:00:00'),('Canada/Mountain','2020-02-01 10:00:00'),('Pacific/Tahiti','2020-02-01 10:00:00')],["OriginTz", "Time"])df.createOrReplaceTempView("test")df2.createOrReplaceTempView("test2")tables = ["test", "test2"]frames = list(range(0,2))def hive_read_func(tables, frames):for table, frame in zip(tables, frame
Hello Team, can you please advise why connection to Kafka fails? Is the issue with keytab? Keytab is working fine on Linux machine.sasl.jaas.config= com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="D:\services.kerberos.keytab" principal="services/rb-hadoop-06.mtg.local@INNOV.LOCAL"; org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClientat org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:540)at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)at io.conduktor.kafka.plugins.PluginsAwareKafkaAdmin$.$anonfun$create$2(PluginsAwareKafkaAdmin.scala:17)at io.conduktor.plugins.PluginsResources$$anon$1.executeWithPlugins(PluginsResources.scala:74)at io.conduktor.kafka.plugins.PluginsAwareKafkaAdmin$.$anonfun$create$1(PluginsAwareKafkaAdmin.scala:17)at scala.util.Try$.apply(Try.scala:210)at io.conduktor.kafka.plugins.PluginsAwareKafkaAdmin$.create(PluginsAwareKafkaAdmin.scala:17)at io.condukt
Is there a limit to the size of an array that can be copied using INSERT INTO SELECT FROM?I have several source Hive EXTERNAL tables, stored as ORC files, that have ARRAY<DOUBLE> columns. One such table has an array with up to 8192 values. Another table has an array with exactly 1024 values in all rows. SELECTing from either table gives the expected results. INSERT INTO another_table SELECT FROM source_table succeeds for the table with 1024 values, and fails with an "ArrayIndexOutOfBoundsException: 1024" for the table that has up to 8192 values in some rows. I'm on HDP 3.1.4.0-315 with HIVE 3.1.0 Stack track snippet:Caused by: java.lang.ArrayIndexOutOfBoundsException: 1024at org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.storePrimitiveRowColumn(VectorDeserializeRow.java:568)at org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.storeComplexFieldRowColumn(VectorDeserializeRow.java:778)at org.apache.hadoop.hi
NIFI Version 1.15.3Java Version 8.0.72 2 node cluster 2022-04-21 10:33:08,447 ERROR [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Unable to save flow controller configuration due to: org.apache.nifi.encrypt.EncryptionException: Failed to get Cipher for Algorithm [AES/GCM/NoPadding]org.apache.nifi.encrypt.EncryptionException: Failed to get Cipher for Algorithm [AES/GCM/NoPadding]
Dear Team, kindly advise on why connection from our CDC tools to Kudu is not working? What does KuduWriterException: End of file: unable to send message: Other end of pipe was closed (error 0) mean? at java.lang.Thread.run(Thread.java:750)com.webaction.common.exc.ConnectionException: Couldn't establish connection with Kudu ( RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.m
Impala 3.4 When impala-shell is started, impalad.ERROR:authentication.cc:177] SASL message (Kerberos (external)): GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No key table entry found for impala@node1) I don't understand why it got the wrong principal. I passed in the correct principal and keytab in the startup file. Can someone help me? Thank you very much.
I have a JSON data with DateTime as 0000-00-00 00:00:00 & it is not been accepted in SQLPlease Support to resolve this issue for me
Hello Team, In our customer cluster, we are testing the HDFS replication through Cloudera Manager. The replication policy looks as follows.All the other configuration is the default. The replication is hung in the below state for a long time.We looked into the Cloudera Manager logs and we can see the below error repeatedly occurring. Can you please help us to resolve the issue?2022-04-21 12:27:57,199 ERROR CommandPusher-1:com.cloudera.cmf.service.AgentResultFetcher: Exception occured while handling tempfile com.cloudera.cmf.service.AgentResultFetcher@618eac09 Best RegardsSayed Anisul Hoque
select concat SPZ301FPL , a meter primary id as pillarid, b locationid as zoneidnumber, b remarks as zonename, f wardid as division, d sectionname as section, concat e streetname, , , e landmark as location, f latitude as pillarlatitude, f longitude as pillarlongitude, g ontime as scheduledontime, g offtime as scheduledofftime, f numberoffittings as numberoflightsconnected, f totalkw as basewattage from meter primary master a inner join locationmaster b on a locationid b locationid inner join polemaster f on a meter primary id f poleid inner join sectionmaster d on d sectionid f sectionid inner join addressmaster e on e addressid f addressid inner join currentreading g on g meter primary id a meter primary id where b locationid in 1, 2, 3, 4, 5 order by a meter primary id br br br Response Array:Array 0 { 1 access token 2 : 3 10350925 2a0d 4bbe b044 7393ae1426c6 4 , 5 token type 6 : 7 bearer 8 , 9 refresh token 10 : 11 eca77863 e681 4a5c abfb b90ea6a436f8 12 , 13 expires in 14 :9348, 1
I saved thousands of small json files in SequenceFile format to resolve the "small file issue". I use the following pyspark code to parse the json data from saved sequence files. reader= sc.sequenceFile("/mysequencefile_dir", "org.apache.hadoop.io.Text", "org.apache.hadoop.io.Text")rdd=reader.map(lambda x: x[1])mydf=spark.read.schema(myschema).json(rdd)mydf.show(truncate=False) The code worked. However, I do not know how to put the key value from the sequence file, which is actually the original json file name, into the mydf dataframe. Please advise. Thank you. Regards,
I have tried to connect local NIFI to Local mysql, I have been getting an error on the processor "cannot create poolableconnectionfactory communications link failure nifi". JDBC Url and Driver are correct. i assume there may be an error while connecting with Mysql. And also i have tried to Connect local NIFI in mac with local Mysql, i am facing an error while i enable the DBCP connection pool the error is mentioned in the image. Thank you
field to add "quality" input json :{"json_data": {"id": "1234","items": [{"id": "000000","metadata": {"newId": "82896888047600"}}]}} output json : {"json_data": {"id": "1234","items": [{"id": "000000","metadata": {"newId": "82896888047600","quality" :"good"}}]}}
Hi All,I have a requirement where there are some empty curl braces in my JSON output which needs to be removed and replaced with empty string. The regex seems to work fine to find the braces when tested outside.Sample can be found here:https://regex101.com/r/AtTPv1/2 I tried to use this regex in nifi using replaceText, but it's not doing the replace. Below is my config for replaceText. Can anyone suggest if there are any options to change to make it work or any other better alternatives for working with JSON?Thanks in advance!
I typically upload csv files into Cloudera Data Science Workbench, but I wonder if there is a way to programmatically read in a csv file from a shared server drive while in yarn mode? Using the below code, I get an error. Any Tips? df = spark.read.format('csv').load('Q:\\project\\data_folder\\file.csv', header=True) The error I get is: IllegalArgumentException: 'java.net.URISyntaxException: Relative path in absolute URI: Q:%5Cproject%5Cdata_folder/file.csv'
Hi I am trying to install CDP 7.1.7 SP1 production licensed product. Can you please confirm if the following link is the correct document I should follow to Install CDP7.1.7 SP1? https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/installation/topics/cdpdc-prod-installation.html Thanks
Team, facing issue connecting DbVisualiser to kerberised hive (hdp), An error occurred while establishing the connection:Long Message:Bad URL format. Hostname not found in authority part of the url: <hiveserevr2 host>:10000;. Are you missing a '/' after the hostname ?FYI: i followed this kb article,https://community.cloudera.com/t5/Community-Articles/Support-Video-How-to-connect-DbVisualizer-to-Hive-with/ta-p/278424 any suggestions pls?
We have an AWS loadbalancer setup for NIFI cluster. Authentication works fine when accessing the NIFI UI using the loadbalancer url. While trying to configure Site-to-Site, authorization errors are encountered. On checking the logs, seems like the NIFI API is trying to authenticate loadbalancer using the user "anonymous" which doesnot exist. Nifi UI AWS LB Url : https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi Nifi API AWS LB Url : https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api Loadbalancer : HTTPS Listener on Port 8443 Why NIFI is trying to access the api url "https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api" using "anonymous" user Snippet : nifi-app.log 2018-07-11 05:26:44,822 WARN [Timer-Driven Process Thread-7] o.a.n.r.util.SiteToSiteRestApiClient Failed to get controller from https://dev-nifi-lb.dev-data.mytestdomain.io:8443/nifi-api due to org.apache.nifi.remote.util.SiteToSiteRestApiClient$HttpGetFailedException: response code 401:Unauthorized with ex
Hello Team, We are setting the Hive replication through Cloudera Manager. The replication policy looks as follows. Note that we also enabled the snapshot on the source cluster for the path - /warehouseHowever, when we press save policy then we get the below notification. We looked into the Cloudera Manager logs and we can see the below error. Can you please help us to get the correct configuration to resolve the issue?2022-04-19 15:51:07,848 ERROR scm-web-1686:com.cloudera.server.web.cmf.WebController: getHiveWarehouseSnapshotsEnabledjavax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org
I'm very new to Cloudera and Spark. I need to know the specifications of the cluster I'm using, so I can better optimize my Spark session configuration (executors/cores/ram/etc). I'm trying to view these details in Cloudera Manager, but I'm not sure where to look. I want to know on the specific cluster I'm using, how many nodes it has, and how many cores and how much memory per node? Forgive my naivety, but some guidance would be appreciated!
wget https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hive/hive-serde/0.13.1-cdh5.3.0/hive-serde-0.13.1-cdh5.3.0.pom--2022-04-19 09:47:22-- https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hive/hive-serde/0.13.1-cdh5.3.0/hive-serde-0.13.1-cdh5.3.0.pomResolving repository.cloudera.com (repository.cloudera.com)... 54.164.28.72, 54.227.239.175, 54.81.245.86, ...Connecting to repository.cloudera.com (repository.cloudera.com)|54.164.28.72|:443... connected.ERROR: cannot verify repository.cloudera.com's certificate, issued by ‘CN=DigiCert TLS RSA SHA256 2020 CA1,O=DigiCert Inc,C=US’:Unable to locally verify the issuer's authority.To connect to repository.cloudera.com insecurely, use `--no-check-certificate'.
Hi all, we are planing to use Vmware VMs for our production cluster for CDP 7.1.7. Is there any recommendation on the configurations, capacity and architecture from Cloudera on this? Any link,whitepaper or documentation will be helpful.What is the difference in performance in using VMs than using physical servers? Which one is preferred and why?
Hi,I have a flow where I make an HTTP request, and I save as an attribute the last ID value with EvaluateJsonPath and ExtractText processors. However, the InvokeHTTP processor doesn't read the saved attribute. What should I change? I attach an image of this part of the flow. Thank you very much in advance. I'm stuck at this point and I don't know what else to try.
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.