Solved

Hi,Please, is there a way to reference a key part of JSON file that can come in different structures.For example i want to get the value of "requestUid" from these different JSON structures just with a single processor. So, irrespective of JSON formats or structures sent, i would want to get the value of requestUid.Thanks.For example:1){"responseMetaData": {"type": "ACK","userId": "SYSTEM","requestUid": "1742963e-f02f-4db6-88e0-6af8c4d4825b",,"systemId": "CIF","serviceId": "FR_CUS"},"responseDataObject": {"result": "Accepted","responseCode": "SUC","responseMessage": "","execTime": "1216"}} 2){"responseMetaData": {"type": "ACK","sendTime": "23/03/2019 09:16:25.305277","timeZone": "4","userId": "SYSTEM","branchCompanyId": "784100","branchLocationId": "784101","systemId": "CIF","serviceId": "FR_CUS"},"responseDataObject": {"result": "Accepted","responseCode": "SUC","requestUid": "yywhhnan56289200200","execTime": "1216"}}

Hi, I'm trying to setup the data engineering experiences (ECS) on top of a CDP base installation.These are licensed installations, so we first installed and configured a base cluster and we have been working on it.We want to setup the data experiences so we can move some work between the cdp public cloud and the cdp private cloud experiences since the control planes are similar. The goal is to work from the CDP private ECS enabled cluster and not use the base cluster unless any components are needed for the ECS/experiences cluster from the base. Trying to comb through the documentation to find the relationship between CDP base and CDP ECS enabled experiences cluster. What roles/manager/services are being used from base cluster in the experiences cluster? Understand a base "core" (manager/runtime etc.) is needed but not sure if I understand the dependencies between the two. The documentation says start with a base cluster and install ECS - well there a many flavors for base te

Hi, I am unfamiliar with Cloudera and tasked with enforcing a log retention policy on all services, currently working in NIFI. Our NIFIis part of CFM-2.2.3.2.I have changes to make to logback.xml, but they are overwritten when a new process directory is created. What is the source of the config.zip that is downloaded to create the process directory and how can I rebuild config.zip to include a copy of logback.xml with my changes? I couldn't find this in the docs I had access to, nor clearly in the cloudera logs I examined.  Many thanks in advance!

We have a NIFI cluster which have 3 nodes. All the nodes were working fine earlier. But now if all 3 nodes are connected to cluster and we right click on NIFI UI canvas and select "Configure" then we are getting error - "An unexpected error has occurred. Please check the logs for additional details."   Below are more details fetched from network tabs on inspection on chrome. The same error can be found if we access NIFI UI using any node's hostname - Request URL:https://<node1>:<port>/nifi-api/flow/process-groups/{process_group_id}/controller-servicesRequest Method:GETStatus Code:500 Internal Server ErrorIf we disconnect node 3 from cluster then we are able to click on "Configure" and list all controller services on NIFI UI. Issue seems to be with node 3. We have synced flow.xml.gz file from node2 to other node ie.node1, node3 and also restarted the NIFI. But still the issue is there.We have got this error in nifi-user.log -"o.a.nifi.web.api.config.

Hi community, I am a newbie to Nifi and unfortunately, We don't have a system admin with the knowledge of Nifi installation for production and I  have to build a 3 node secured cluster without Zookeeper. I understood that with the NiFi 0.x baseline, the cluster has a cluster manager which serves a similar role and did not use a zookeeper at all. But I am not sure how to use NiFi 0.x baseline, If someone can help me with steps/guidance to configure using NiFi 0.x baseline to set up a three-node cluster that will be great. Best,Shanoj

Hello! Sorry for my english. I am completely new to nifi and I am learning SplitText processor. So here's the case. I have the comma separated txt file, something like this:KeyWord, SomeInformation   <---1st line is schema.KeyWord1, "information"KeyWord2, "information"KeyWord1, "another information"KeyWord2, "another information" and so on. So the question is how can I split this file on a few files based on KeyWord? So the every line with KeyWord1 go to one file, every line with KeyWord2 go to another file and so on? Thank you beforehand!

Hi All, We did the disk partition activity on one of our nodes on which NN services were running. After rebooting Active NN goes down. Need urgent solution on this. Please help ---Version---Ambari version -> 2.6OS -> Linux (centos 7.9.2009)HDP-2.6.5.1100Attaching the required NN log 2022-04-30 06:48:30,683 WARN namenode.NameNode (NameNodeRpcServer.java:errorReport(1548)) - Fatal disk error on DatanodeRegistration(10.0.3.26:1019, datanodeUuid=a500e7d8-3052-4aa8-a18d-e768a15d22da, infoPort=1022, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-dc49d075-7a70-4ddf-b66b-b39b4b445af0;nsid=897785728;c=0): DataNode failed volumes:/grid1/hadoop/hdfs/data/current; 2022-04-30 06:48:34,123 ERROR namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(410)) - Error: flush failed for (journal JournalAndStream(mgr=FileJournalManager(root=/grid1/hadoop/hdfs/namenode), stream=EditLogFileOutputStream(/grid1/hadoop/hdfs/namenode/current/edits_inprogres

Hi all, i saw this guide: https://community.cloudera.com/t5/Support-Questions/how-to-change-password-for-ambari-db-post-install/td-p/126685 but it was for Ambari 2.2.0 and i was wondering if do you think is the same for Ambari 2.7.4.I have in ambari.properties: - server.jdbc.user.passwd=/etc/ambari-server/conf/password.datDoing "cat /etc/ambari-server/conf/password.dat " i see the default password (bigdata) in plain, so NO PASSWORD ENCRYPTION ENABLED.I'm using Postgres 9.3Thanks!!

Hi Team, I am ex-employee of CLoudera/Hortonworks and completed HDPCA certification in Dec'19. Registered mail ID was smahendrada@cloudera.com. I missed to save my certificate and I dont have the certificate copy. Could you please help me. Thanks,Sruthi

Hi Guyz, please help me to build the python script for cleaning HDFS dir which are older than 3 days.give me the suggestions 

hello all,I am currently running a python version of 3.6.8, Can I still install spark 3 on my cdp cluster?

Hi,I am not able to permanently delete tombstone in Hbase.Tombstone markers are flushed for non-MOB column family but it is not flushed for the MOB column family. Any solution to delete the same so that we save some hdfs space. Regards,Satya

Hi,every one:    I have a question.I found there are many "cmd=get_database" log in hadoop-cmf-hive-HIVEMETASTORE.out file.But I haven't done this much.Is it because there are so many other actions that trigger it? 

I have one Json file and I have validate it, Its a valid json.I am creating Hive table on that file and then select data but its showingError: java.io.IOException:org.apache.hadoop.hive.serde2.SerDeException: Row is not a valid JSON Object- JSONException: A JSONObject text must end with '}' at 2 [character 3 line 1](state=,code=0)when I tried to set below property.ALTER TABLE test_rahul SET SERDEPROPERTIES ( "ignore.malformed.json" = "true");Then It showing Null values for all the fields. Can someone Kindly Help me.json File:{"buyer": {"legalBusinessName": "test1 Company","organisationIdentifications": [{ "type": "abcd","identification": "test.bb@tesr"},{"type": "TXID","identification": "12345678"}]},"supplier": {"legalBusinessName": "test Company","organisationIdentifications": [{"type":"abcd","identification": "test28@test"}]},"paymentRecommendationId": "1234-5678-9876-2212-123456","excludedRemittanceInformation": [],"recommendedPaymentInstructions": [{"executionDate": "2022-06-12","

Hi I see that Cloudera Manager and CDP Private Cloud Base versions 7.6.7 and 7.1 can be install on redhat 8.4. I have servers redhat 8.4 but I did a yum update and now are redhat 8.5, my question is, can I install CDP in this linux version? redhat 8.5?Thanks

I have written python function and run in pyspark - cloudera shell but it throwing error while running it. But when I run the same function in my pyspark - local shell it's working perfectly fine. Complete Code:from pyspark.sql.functions import *df=spark.createDataFrame([('America/New_York','2020-02-01 10:00:00'),('Europe/Lisbon','2020-02-01 10:00:00'),('Europe/Madrid','2020-02-01 10:00:00'),('Europe/London', '2020-02-01 10:00:00'),('America/Sao_Paulo', '2020-02-01 10:00:00')],["OriginTz","Time"])df2=spark.createDataFrame([('Africa/Nairobi', '2020-02-01 10:00:00'),('Asia/Damascus', '2020-02-01 10:00:00'),('Asia/Singapore', '2020-02-01 10:00:00'),('Atlantic/Bermuda','2020-02-01 10:00:00'),('Canada/Mountain','2020-02-01 10:00:00'),('Pacific/Tahiti','2020-02-01 10:00:00')],["OriginTz", "Time"])df.createOrReplaceTempView("test")df2.createOrReplaceTempView("test2")tables = ["test", "test2"]frames = list(range(0,2))def hive_read_func(tables, frames):for table, frame in zip(tables, frame

Is there a limit to the size of an array that can be copied using INSERT INTO SELECT FROM?I have several source Hive EXTERNAL tables, stored as ORC files, that have ARRAY<DOUBLE> columns.  One such table has an array with up to 8192 values.  Another table has an array with exactly 1024 values in all rows.  SELECTing from either table gives the expected results. INSERT INTO another_table SELECT FROM  source_table succeeds for the table with 1024 values, and fails with an "ArrayIndexOutOfBoundsException: 1024" for the table that has up to 8192 values in some rows.  I'm on HDP 3.1.4.0-315 with HIVE 3.1.0 Stack track snippet:Caused by: java.lang.ArrayIndexOutOfBoundsException: 1024at org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.storePrimitiveRowColumn(VectorDeserializeRow.java:568)at org.apache.hadoop.hive.ql.exec.vector.VectorDeserializeRow.storeComplexFieldRowColumn(VectorDeserializeRow.java:778)at org.apache.hadoop.hi

NIFI Version 1.15.3Java Version 8.0.72 2 node cluster 2022-04-21 10:33:08,447 ERROR [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Unable to save flow controller configuration due to: org.apache.nifi.encrypt.EncryptionException: Failed to get Cipher for Algorithm [AES/GCM/NoPadding]org.apache.nifi.encrypt.EncryptionException: Failed to get Cipher for Algorithm [AES/GCM/NoPadding]

Dear Team, kindly advise on why connection from our CDC tools to Kudu is not working? What does KuduWriterException: End of file: unable to send message: Other end of pipe was closed (error 0) mean? at java.lang.Thread.run(Thread.java:750)com.webaction.common.exc.ConnectionException: Couldn't establish connection with Kudu ( RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.mtg.local:7051,RB-HADOOP-03.mtg.local:7051,RB-HADOOP-04.mtg.local:7051,RB-HADOOP-05.m

I have a JSON data with DateTime as 0000-00-00 00:00:00 & it is not been accepted in SQLPlease Support to resolve this issue for me 

Hello Team, In our customer cluster, we are testing the HDFS replication through Cloudera Manager. The replication policy looks as follows.All the other configuration is the default. The replication is hung in the below state for a long time.We looked into the Cloudera Manager logs and we can see the below error repeatedly occurring. Can you please help us to resolve the issue?2022-04-21 12:27:57,199 ERROR CommandPusher-1:com.cloudera.cmf.service.AgentResultFetcher: Exception occured while handling tempfile com.cloudera.cmf.service.AgentResultFetcher@618eac09 Best RegardsSayed Anisul Hoque 

select concat SPZ301FPL , a meter primary id as pillarid, b locationid as zoneidnumber, b remarks as zonename, f wardid as division, d sectionname as section, concat e streetname, , , e landmark as location, f latitude as pillarlatitude, f longitude as pillarlongitude, g ontime as scheduledontime, g offtime as scheduledofftime, f numberoffittings as numberoflightsconnected, f totalkw as basewattage from meter primary master a inner join locationmaster b on a locationid b locationid inner join polemaster f on a meter primary id f poleid inner join sectionmaster d on d sectionid f sectionid inner join addressmaster e on e addressid f addressid inner join currentreading g on g meter primary id a meter primary id where b locationid in 1, 2, 3, 4, 5 order by a meter primary id br br br Response Array:Array 0 { 1 access token 2 : 3 10350925 2a0d 4bbe b044 7393ae1426c6 4 , 5 token type 6 : 7 bearer 8 , 9 refresh token 10 : 11 eca77863 e681 4a5c abfb b90ea6a436f8 12 , 13 expires in 14 :9348, 1

I saved thousands of small json files in SequenceFile format to resolve the "small file issue". I use the following pyspark code to parse the json data from saved sequence files.  reader= sc.sequenceFile("/mysequencefile_dir", "org.apache.hadoop.io.Text", "org.apache.hadoop.io.Text")rdd=reader.map(lambda x: x[1])mydf=spark.read.schema(myschema).json(rdd)mydf.show(truncate=False)  The code worked. However, I do not know how to put the key value from the sequence file, which is actually the original json file name, into the mydf dataframe. Please advise. Thank you. Regards,

field to add "quality" input json :{"json_data": {"id": "1234","items": [{"id": "000000","metadata": {"newId": "82896888047600"}}]}} output json : {"json_data": {"id": "1234","items": [{"id": "000000","metadata": {"newId": "82896888047600","quality" :"good"}}]}}  

Hi All,I have a requirement where there are some empty curl braces in my JSON output which needs to be removed and replaced with empty string. The regex seems to work fine to find the braces when tested outside.Sample can be found here:https://regex101.com/r/AtTPv1/2 I tried to use this regex in nifi using replaceText, but it's not doing the replace. Below is my config for replaceText.  Can anyone suggest if there are any options to change to make it work or any other better alternatives for working with JSON?Thanks in advance!