Q&A

Hi Everyone,I am parsing data from JIRA portal, after all cleanup I am pushing the data in database. However my mapping running fine when I am loading the small amount of data, but its get failed when I am loading two months data which is around 800 rows (360 only loaded). however I am encountering an error java.lang.ClassCastException: null and in failure, I am not able to find the rows and reason which causing this issue.  2024-03-02 21:07:48,707 ERROR [Timer-Driven Process Thread-10] o.a.n.p.standard.PutDatabaseRecord PutDatabaseRecord[id=f10515e5-3b10-1c3c-802a-83b2ee68649a] Failed to put Records to database for StandardFlowFileRecord[uuid=0e2ec0fc-ebc5-4734-8e59-868a2aa0e6d5,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1709399268684-21833, container=default, section=329], offset=4, length=115143],offset=0,name=a016a8e7-82ae-4ed4-a51d-934d976fd173,size=115143]. Routing to failure.java.lang.ClassCastException: null2024-03-02 21:07:50,049 INFO [Cl

Hi.I'm trying to use Python extensions with Nifi.I've uncommented the line   nifi.python.command=python3   in nifi.properties but Nifi always crash when I try to run it. When I re-comment the line, Nifi starts and works fine. I've read this thread  https://community.cloudera.com/t5/Support-Questions/Nifi-2-0-0-M1-Installation-error-with-python/m-p/381430and tried to follow the instructions in the solutions.It still crashes. I'm posting here because the problem is on Windows instead of Linux. Thanks for your help! OSWindows 10 home editionNifi versionnifi-2.0.0-M2Java installed version21.0.2Python installed version3.11.8   nifi.propertiesIn   nifi.properties   I changed the line from   nifi.python.command=python3   to   nifi.python.command=python VenvI made sure the python venv package is installed by issuing this command   python -m venvwhich gave this messa

Hi everyone, I would like some clarification. Thank you.If a customer is already using active OCP (OpenShift) and wants to use CDP Private Cloud Data Services, will this impact other services performed on OCP? If so, which services of CDP Private Cloud Data Services will affect OCP?For example,If CDP Private Cloud Data Services are installed, the scheduler will default to using Yunikorn. So, if the settings are not changed, it won't use the default scheduler.For instance, if I apply a pod and want to use the default scheduler, one method is to adjust the Yunikorn ConfigMap settings to bypass the admission controller.

I'm just installing Apache Nifi on linux server with version 1.25 .this is my nifi.properties :#nifi.security=none#nifi.security.autoreload.enabled=false#nifi.security.autoreload.interval=10 secs#nifi.security.keystore=#nifi.security.keystoreType=PKCS12#nifi.security.keystorePasswd=#nifi.security.keyPasswd=#nifi.security.truststore=#nifi.security.truststoreType=PKCS12#nifi.security.truststorePasswd=#nifi.security.user.authorizer=nifi.security.allow.anonymous.authentication=falsenifi.security.user.login.identity.provider=file-login-provider#nifi.security.user.jws.key.rotation.period=PT1H#nifi.security.ocsp.responder.url=#nifi.security.ocsp.responder.certificate=this is my users.xml :<?xml version="1.0" encoding="UTF-8" standalone="yes"?><tenants> <users> <!-- Add more users as needed --> <user identifier="ronald"> <identity>ronald</identity> <!-- Replace CREDENTIAL_VALUE with the hashed password -->

Every so often we get the following error from Impala:[HY000] [Cloudera][ThriftExtension] (11) Error occurred while contacting server: EAGAIN (timed out). The connection has been configured to use a binary mechanism for authentication. This error might occur if the server is not using a binary mechanism for authentication. (11) (SQLDriverConnect) Our software makes up to 3 attempts to run the action, if the first (and second) attempts fail.The attempt fails on this error sometimes, but not always.This (the fact that the error occurs only some of the times) leads us to think that it is not a configuration issue, as suggested by the wording of the error. 1. What causes this error ?2. How should we deal with this ?3. Can we prevent the problem from happening ? Thanks  Lior H.

Hello Team, we have requirement as below:1. Create table db1.t1 (partition loading_date string) as select * from db2.t1 where 1=2;   creating table as step1 or execute full create table statements with update location2. alter table db1.t1 set location <new path name>3. Add partition for every dayalter table db1.t1 add  partition (loading_date='20220101') location "hdfs dir path for each partition"4. refresh table;5. compute stats table_name; Have executed above steps in impala, data not loading from partitions files.

hello cloudera community, do I need to insert a new datanode in the hdfs and with that I need to know how much size will be used from this new datanode after performing a rebalance on the hdfs? for example, I have the following scenario: datanode 1 has 36tb total to use, 30tb is being used, containing about 2.5 million blocksdatanode 2 has 36tb total to use, 30tb is being used, containing about 2.5 million blocksdatanode 3 has 36tb total to use, 30tb is being used, containing about 2.5 million blocksdatanode 4 has 36tb total to use, 30tb is being used, containing about 2.5 million blocksdatanode 5 has 36tb total to use, 30tb is being used, containing about 2.5 million blocksdatanode 6 has 36tb total to use, 30tb is being used, containing about 2.5 million blocks i will insert the "datanode 7" which has 36tb total, after performing the rebalance in hdfs, how much will this new datanode 7 receive data?

Hello everyone, first of all I apologise for my English.I'm facing a big problem between IBM DataStage and HortonWorks Let me first explain IBM DataStage: It's an ETL tool that's some connection types for importing/exporting data from a number of DataSource types.I'm trying to load data from IBM DataStage 11.7 into Hive using the Hive connector, but I'm encountering some strange behavior:There are a couple of configurations for the Hive connector, the most important of which is - as I suspected - :. Record count=2000. Batch size=2000for a dataset with 8 columns and almost 1000 rows, data inserted into Hive.For a dataset with 200 columns and 20 million rows, it behaves strangely:for 10 columns, works.For more than 10 columns, the multiplication of the stack size propertiesfails - I mean for 2000, 4000 or 20000 rows - with 'IIS-CONN-DAAPI -00099 Hive_Connector_7,0: java.lang.StringIndexOutOfBoundsException: String index out of bounds: 0 at java.lang.String.substring (String.java: 2667)'I

I have an HDFS path including two different file format1- Avro2-JSONI want to build a hive external table on that path but only for Avro files. I do not have the option to have each file format in a separate location and I could not find a way to pick those file based on the file extension for the hive external tablethe suggested solutions was to build a custom class which will fitter the files based on the targeted type which is extended for the main avro input class AvroContainerInputFormat. package com.company.hadoop.customavroinputformat;import org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat;import org.apache.hadoop.mapred.FileSplit;import org.apache.hadoop.mapred.InputSplit;import org.apache.hadoop.mapred.JobConf;import org.apache.hadoop.mapred.Reporter;import org.apache.hadoop.mapred.RecordReader;import java.io.IOException;import java.util.ArrayList;import java.util.List;import org.apache.avro.generic.GenericRecord;import org.apache.avro.mapred.AvroWrapper;import

Hi all, my team are currently testing on HDFS rebalancing in a HDFS cluster with two DataNode. I have read the documentation https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/scaling-namespaces/topics/hdfs-step-1-storage-group-classification.htmlWe have a few question about how the file moving size is calculated.1. How is cluster average utilization calculated. Do the calculation use Configured capacity or Present capacity?2. How is individual utilization calculated. Do the calculation use configured capacity or the sum of DFS used and remaining?3. When looking at the balancer log, it said need to move 4.98GB to make the cluster balanced. However, afterwards, the log show that the balancer decided to move 18.98GB from one host to another host. Why there is difference?Balancer log4. We have set the DataNode balancing bandwidth to 200MiB, but as per observation, the bandwidth consumption can exceed 200MiB. Why did it happen?Please help me with my questions. Thank you in advance

Earlier after performing this replacement, a month back. Data wiped and it stood at 7.7 GB ever a month, data consistently increased now standing at 326 GB./dev/sde1                                           7.1T  326G  6.8T   5% /u05We usually don't perform, decommissioning. So, before replacement data wasn't transferred to any datanode. If such case, after the activity, the data was in wiped state.Kindly help with Cloudera review

Hello,I am trying to add key: value in a json after SplitJSON. I was using ReplaceText processor for achieving so. But problem my regex expr is coming twice my json object which causing the invalid json. What would be the appropriate way to do so ?Current Structure:Replace Text Processor looks like :Any suggestions will be of great help!!

Hi everyone, We faced some issue when using Flink version 1.16.2.CSA.1.11.0.x, this is the error message2024-02-19 16:52:42java.lang.IllegalStateException: Failed to commit KafkaCommittable{producerId=1077051, epoch=0, transactionalId=payload_migs-TEST_mig_2-0-3} at org.apache.flink.streaming.runtime.operators.sink.committables.CommitRequestImpl.signalFailedWithUnknownReason(CommitRequestImpl.java:77) at org.apache.flink.connector.kafka.sink.KafkaCommitter.commit(KafkaCommitter.java:119) at org.apache.flink.streaming.runtime.operators.sink.committables.CheckpointCommittableManagerImpl.commit(CheckpointCommittableManagerImpl.java:126) at org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmit(CommitterOperator.java:176) at org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.commitAndEmitCheckpoints(CommitterOperator.java:160) at org.apache.flink.streaming.runtime.operators.sink.CommitterOperator.initializeState(CommitterOperator.java:

Hey there, we're running a CDP PC Base Cluster in our datacenter. In the last months we saw serveral disks failing due to theire age. The Problem we had, is that, when a disk fails, HDFS DN and YARN NM creating directories in the root-vg of the node. During normal operations the DN has it directories on the grids (/grid/[0-16]/*).Is there a parameter to prevent them of writing into the root-vg, when a grid points to the root-vg and not it's physical device under /dev/sd*?Regards, Timo

Hello Team, I have one column and did some calculation on it and wanted to make another copy of same column to create flag to check whether records are empty or not and retain the same column also.I tired to using pulling another processor and  wrote calculation New_field , Old_column which did not work.However, this approach did not yield the desired outcome.   

Hello,I am getting the data from GetMongo Processor then i am evaluating the JSON Path using EvaluateJSONPath where i am getting series_id and then I am doing in InvokeHTTP.  I am able to send as json but i am not able to send as form-data. Can you please suggest me what i am doing wrong here.Before invokeHTTP the data flow is     

Hi all, I have a cluster in CDP 7.1.8 private cloud based. Today I tried migrating CM server and following the instruction in https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/managing-clusters/topics/cm-moving-the-cm-server-new-host2.html After I manually restore the cluster following the instruction in https://docs.cloudera.com/cdp-private-cloud-base/7.1.3/configuring-clusters/topics/cm-api-import-configuration.html, the cluster cannot restart. Found that in the Parcel page in CM Console, the parcels are activating but keep on loading forever. Here are the log message related to parcel from cloudera-scm-agent.log: [02/May/2023 03:29:55 -0400] 6258 MainThread parcel ERROR Error while attempting to modify permissions of file '/opt/ cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller'. File "/opt/clo

Hi Team,Mysql service is not getting start and log is also not showing . Can someone help? [root@]# service mysqld startRedirecting to /bin/systemctl start mysqld.serviceJob for mysqld.service failed because a configured resource limit was exceeded. See "systemctl status mysqld.service" and "journalctl -xe" for details.[root@devtest mysqld.log]# RegardsShamim 

C#- i am using below parameterized query for getting the data from hiveQuery- select * from builds_changelogs where array_contains(deliveryconstructuid,?)Parameter-OdbcParameter paramdc = new OdbcParameter("@deliveryconstructuid", OdbcType.NVarChar);paramdc.Value = "1E4272C7D9E64DF8B3DADA3D056F35A1"; When i am executing the query it is throwing below error-ERROR [42000] [Hortonworks][Hardy] (80) Syntax or semantic analysis error thrown in server while executing query. Error message from server: Error while compiling statement: FAILED: ParseException line 1:48 cannot recognize input near '?' 'and' 'array_contains' in expression specification   

hi all I'm looking to create a table to show all table names and information within a database as im having capacity issues  something simple like  username,size,tablename,last updated date  hoping this is an easy ask but i had no luck so far 

Hello Team,I have JSON like following[{"INTEGRATION_ID": "1","ISSUE_STATUS_CAT": "Done","SPRINT_LIST": ["24-w1-2","24-w1-3","24-w1-4"]},{"INTEGRATION_ID": "2","ISSUE_STATUS_CAT": "Done","SPRINT_LIST": "24-w1-2"}]My requirement is to check whether SPRINT_LIST is array or not. so I am using UpdateRecord processor with following formula ${field.value:startsWith(" [ ")}I was getting the correct output earlier, where it returned 'True' when the value was a list and 'False' when it was null or not a list. However, after making some changes and then reverting back to the original settings, I started encountering issues. "Cannot convert value [true] of type class java.lang.String to Object Array for field error"

When my job streaming is running, i go to yarn cluster ui and i choose the application that im running. I click to the Tracking URL: ApplicationMaster, it bring me to the page "ip:port/proxy/application_id".It said HTTP ERROR 403 User yarn is not authorized to access this page.Is this because of yarn manager queue authorization or some missing configs for yarn.

Have enyone had a deal with Impala parametrs such as topic_update_tbl_max_wait_time_ms and catalog_max_lock_skipped_topic_updates. Should I try to change them and what will it affect on?May the reduce of the topic_update_tbl_max_wait_time_ms improve the time of loading table's metadata to the impala daemon from catalog?If you know some recomendation or more detail information please share it.Thank you in advance!

Hi,Our yarn spark jobs trigger with oozie workflow. there are situations where we need to set a high priority for some of the jobs. I noticed there is a property mapreduce.job.priority to set a priority. how to set a job priority in oozie workflow so that yarn schedules our job with high priority over default from other jobs?

Hi Everyone,I'm trying to use ValidateRecord processor where I have a json input with date field as follows:  { "name": "sam", "age": "20", "dob": "01/01/2004", "id": "001"}  In the validaterecord Record Writer Im using JsonRecordSetWriter with the following configuration:Where I'm using the following avro schema in the Schema Text property:   { "name": "person_rec", "type": "record", "namespace": "nifi.com", "fields": [ { "name": "name", "type": "string" }, { "name": "age", "type": "int" }, { "name": "dob", "type": "int", "logicalType": "date" }, { "name": "id", "type": "string" } ]}  Also I'm setting the Date Format to : MM/dd/yyyyI though based on the description that when I set the Date Format it will be able to parse the date when reading\writing date fields. I thought by date field it means using logicalType "date" which means the type must be "int" according to avro schema. My expectation is that my validate