Ask the Community

I am using CSVReader and CSVRecordSetWriter using the Infer schema setting. But when I have values such as "030", even though the all the values has double quotes enclosed. When I write the schema into avro.schema, I see it is considered as type 'int'. But I want to treat it as string. Because of this our output after the processor looks like 30 and the first 0 is omitted. I want to use Infer schema property only because, I want to read the csv files dynamically without hardcoding the schema and use that.

Hi team,I'm struggling while converting nested JSON to Flat JSON using JOLT.The Nested JSON is in the below format: { "data": { "getCarListing": { "edges": [ { "node": { "id": "1001", "identifier": 500001, "carClusterIds": [ { "element": { "name": "SEDAN1000", "businessRelations": [ { "countries": [ { "countryCode": "US" } ], "code": "US01" }, { "countries": [ { "countryCode": "CZ" }, { "countryCode": "SK" } ], "code": "CZ01" }, {

I have recently upgraded from CM 7.6.7 to CM 7.11.3 and CDP 7.1.7 SP2 to CDP 7.1.7 SP3.HDFS Datanode, Impala Daemon, Yarn Resource Manager, and Hbase Region Server are showing unhealthy web server on Cloudera as shown below. After checking one of the agents log, I found the following error.  18/Nov/2024 09:09:09 +0100] 2414 GM IMPALAD throttling_logger ERROR Error fetching metrics at 'https://host.domain.com:25000/jsonmetrics?json'Traceback (most recent call last): File "/data/cloudera/cm-agent/lib/python3.8/site-packages/cmf/monitor/generic/metric_collectors.py", line 224, in _collect_and_parse_and_return opened_url = urlopen_with_retry_on_authentication_errors( File "/data/cloudera/cm-agent/lib/python3.8/site-packages/cmf/util/url.py", line 339, in urlopen_with_retry_on_authentication_errors return function() File "/data/cloudera/cm-agent/lib/python3.8/site-packages/cmf/monitor/generic/metric_collectors.py", line 244, in _open_url return self._urlopen_ca

Hi Community,I have recently upgraded from CM 7.6.7 to CM 7.11.3 and CDP 7.1.7 SP2 to CDP 7.1.7 SP3.There are many services that are showing web server error on Cloudera as shown below. One of those services is HDFS. When I checked the cloudera-scm-agent log, I found the following error.  [21/Oct/2024 09:09:09 +0100] 2414 GM IMPALAD throttling_logger ERROR Error fetching metrics at 'https://host.domain.com:25000/jsonmetrics?json'Traceback (most recent call last): File "/data/cloudera/cm-agent/lib/python3.8/site-packages/cmf/monitor/generic/metric_collectors.py", line 224, in _collect_and_parse_and_return opened_url = urlopen_with_retry_on_authentication_errors( File "/data/cloudera/cm-agent/lib/python3.8/site-packages/cmf/util/url.py", line 339, in urlopen_with_retry_on_authentication_errors return function() File "/data/cloudera/cm-agent/lib/python3.8/site-packages/cmf/monitor/generic/metric_collectors.py", line 244, in _open_url return self._urlopen_call

Hi Everyone, I have the below Input JSON and would like to flatten the data into the below mentioned output JSON format. I am new to the JOLT, I tried to transform (its incomplete) but could not achieve the result. Kindly help me to get the desired output, thanks in advance. Input JSON:           { "data": { "getItemListing": { "edges": [ { "node": { "id": "2940126", "identifier": 3119650, "parentItems": [ { "id": "364270", "parentIdentifier": 371153 }, { "id": "364258", "parentIdentifier": 371147 } ], "treatmentClusterIds": [ { "metadata": [ { "treatmentClusterIDs": "528" } ], "element": { "name": "GOAT02", "labelingClusters": [

In Apache Spark, spark_shuffle and spark2_shuffle are configuration options related to Spark's shuffle operations, which can be set to start auxiliary services within the Yarn NodeManager. But what is the difference between these two?

Hello everyone, I am facing issue with creating an custom python processor in Nifi 2.0.0. It is linux based server with installed java 22 and python 3.9.18. I also set the nifi.properties command to the python3.9.Currently I am just trying to get the simplest processor such as from nifiapi.flowfiletransform import FlowFileTransform, FlowFileTransformResultclass WriteHelloWorld(FlowFileTransform): class Java: implements = ['org.apache.nifi.python.processor.FlowFileTransform'] class ProcessorDetails: version = '0.0.1-SNAPSHOT' def __init__(self, **kwargs): pass def transform(self, context, flowfile): return FlowFileTransformResult(relationship = "success", contents = "Hello World", attributes = {"greeting": "hello"})This is picked up by nifi and it seems that successfuly but in the GUI the process still has the warning 'Processor' is invalid because initializing runtime environment for the Processor.From the logs I saw that there is

2024-05-04 15:27:08,590 WARN [RpcServer.default.FPBQ.Fifo.handler=26,queue=1,port=16000] assignment.AssignmentManager: rit=OPENING, location=l230-n2.<SERVER>,16020,1714828981071, table=<NAMEAPACE>:<TABLE>, region=275bbfd7b7044afcba2deeeebf622b07 reported OPEN on server=l230-n2.<SERVER>,16020,1714828981071 but state has otherwise AND NO procedure is runningWe have about 300 RIT and there alle stuck under the procedures & locks (<hbase master>:16010/procedures.jsp) we have some in state waiting_timeout.We all ready tried restarting the cluster (hbase/zk) but got only whorse. There are 14 nodes in the cluster 3 are not assigned and we have a couple (about 4) that do all the work.Where still on hbase 2.0.2  so we can't use hbck2 (2.0.3 >) and in hbase 2.0.2 hbck repair is disabled. Our average region count per server is about 330.1. how do i get my regions unstuck? 

My pyspark yarn-client application got killed by cluster because of this setting, yarn.scheduler.capacity.root.default-application-lifetime. What config should I use to declare my application lifetime and avoid getting killed?

I need to authenticate to a Kafka Broker using OAuth. I am looking at the Apache NiFi issues pages and see this URL to add support for SASL/OAUTHBEARER to the Kafka 3 processors which now have an allowable values list of SASL mechanisms - https://issues.apache.org/jira/browse/NIFI-7421Does anyone know if this will ever be implemented?Otherwise, does anyone have any list of ideas I could use to achieve this requirement?@MattWho @SAMSAL 

Hello Experts,I am using Wait - notify processors for my flow, once in a while I am seeing this error in Notify processor.java.lang.RuntimeException: Unable to communicate with cache when processing.The screen shot attached. Any idea what is the reason and how to resolve this ?Thanks in advance.Mahendra 

Hello Team, I am Sachin Duggal , and I am the owner of a bags manufacturing business. I am encountering an “Out of memory heap” error while processing a large text file (26k+ rows) through SFTP. The file contains | separated values, and I'm using the following processors for data manipulation. However, the process halts after the splitJSON processor. Has anyone else faced a similar issue or has any suggestions on how to optimize the memory usage or address this issue? I would appreciate any insights or alternative approaches that could help resolve this. Looking forward to your suggestions! Sachin Duggal           

Hello,Recently I setup NiFi 2.0.Everytime I start NiFi, I get below warning messages in /var/log/messages. Dec 13 12:07:58 myserver nifi.sh[208489]: WARNING: A restricted method in java.lang.foreign.Linker has been calledDec 13 12:07:58 myserver nifi.sh[208489]: WARNING: java.lang.foreign.Linker::downcallHandle has been called by org.apache.lucene.store.PosixNativeAccess in an unnamed moduleDec 13 12:07:58 myserver nifi.sh[208489]: WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for callers in this moduleDec 13 12:07:58 myserver nifi.sh[208489]: WARNING: Restricted methods will be blocked in a future release unless native access is enabled Is there any way to supress these messages?Here are product versions.Red Hat Enterprise Linux 8.8NiFi 2.0.0Java OpenJDK 23.0.1.0.11Regards,

2024-08-25 02:47:07,116 DEBUG org.apache.hadoop.net.NetworkTopology: Choosing random from 0 available nodes on node /, scope=, excludedScope=null, excludeNodes=[192.168.1.81:9866, 192.168.1.125:9866, 192.168.1.8>2024-08-25 02:47:07,116 DEBUG org.apache.hadoop.net.NetworkTopology: chooseRandom returning null2024-08-25 02:47:07,116 DEBUG org.apache.hadoop.net.NetworkTopology: No node to choose.2024-08-25 02:47:07,116 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: [Datanode None is not chosen since required storage types are unavailable for storage type DISK.2024-08-25 02:47:07,116 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Not enough replicas was chosen. Reason: {NO_REQUIRED_STORAGE_TYPE=1}2024-08-25 02:47:07,116 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 1 (unavailableStorages=[], storagePolicy=BlockStorage>2024-08-25 02:47:07,116 WARN

hi community cloudera, i'm experiencing strange hive drive behaviour:my app consequently performing `select` queries on multiple tables successfully. But at some point it gets an error: java.sql.SQLException: [Cloudera][HiveJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: Socket is closed by peer. FetchResults for query "SELECT `iq***". and when the app retries to reconnect (because java.sql.Connection.isValid() == false)it constantly gets  [Cloudera][JDBC](10060) Connection has been closed.java.sql.SQLNonTransientConnectionException: [Cloudera][JDBC](10060) Connection has been closed.at com.cloudera.hiveserver2.exceptions.ExceptionConverter.toSQLException(Unknown Source)at com.cloudera.hiveserver2.jdbc.common.SConnection.closeConnection(Unknown Source)at com.cloudera.hiveserver2.jdbc.common.SConnection.abortInternal(Unknown Source)at com.cloudera.hiveserver2.jdbc.common.SConnection.close(Unknown Source) Since connection

Hi Team, Good Day.. I'm beginner in Node.JS trying to connect Hive DB  using hive-driver npm from Node.JS code. I couldn't establish the db connection. Any help would be much appreciated... Note: Our Hive DB has ssl connection, jks trust store password, HTTP protocol , httppath=cliservice   NODE.JS  SAMPLE CODE: client.connect({host: 'jdbc:hive2://abc.com:2181/dev_db;ssl=true;sslTrustStore=/etc/hive-jks/hivetrust.jks;trustStorePassword=pro@!23;transportMode=http;httpPath=cliservice;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2',port: 2181},new hive.connections.HttpConnection(),new hive.auth.PlainHttpAuthentication({username: 'username',password: 'password'})).then(async client => { console.log(client); const session = await client.openSession({client_protocol: TCLIService_types.TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V10}); }).catch(error => {console.log(error);});   when I console.log client It's printing but I tried to open client.o

Hi Team, I am implementing the CSFLE(Client Side Field Level Encryption) in the Nifi for the Kafka data. I have written Groovy script to get the logic like Consume/deserialize/Encrypt the data. The script is working as standalone. However, if i place it in ExecuteScript processor ,i am getting Caused by: javax.security.auth.login.LoginException: No LoginModule found for org.apache.kafka.common.security.plain.PlainLoginModule Error. It seems it is compatibility issue in the kafka-client jar file. However, i could not find the fix. Please provide your input on this.My groovy script is below.  import org.apache.kafka.clients.consumer.ConsumerRecordimport org.apache.kafka.clients.consumer.ConsumerRecordsimport org.apache.kafka.clients.consumer.KafkaConsumerimport org.apache.kafka.common.serialization.StringDeserializerimport java.time.Durationimport org.apache.kafka.common.security.plain.PlainLoginModuleimport javax.security.auth.login.Configurationimport javax.security.auth.logi

Hi Team, HDFS was working before kurberos enabled , noww post enabling kurberos HDFS is not working. Getting error in HDFS GUI, HDFS is running in cloudera manager.Name node is showing, Test of whether the NameNode is in safe mode.```Bad : This NameNode is in safe mode.```Error:```HTTP ERROR 502 Process information not available.URI:STATUS:MESSAGE:SERVLET:/cmf/process/1546341404/logs502Process information not available.Spring MVC Dispatcher Servlet```

In your article:https://docs.cloudera.com/management-console/cloud/environments-azure/topics/mc-using_azure_database_for_postgresql_flexible_server.htmlIn the middle of this article, you mentioned:"In general, when registering an Azure environment and creating a Data Hub, you can choose to use Flexible Server, Flexible Server with Private Link, Flexible Server with Delegated Subnet (deprecated), Single Server (deprecated), or Single Server with Private Link (deprecated),..."I believe that the Flexible Server with Delegated Subnet is NOT deprecated.

I have 2 NIFI cluster, one NIFI cluster which is running as helm in EKS and another NIFI cluster which is deployed  with 2 linux servers, We want to enable Site to Site connection between EKS NIFI cluster to Remote NIFI clusterwhat is the best way to connect via Remote Processor group, what configuration to be set for RPG

Hi everyone,I am facing an issue while running a Sqoop import job. The process gets stuck at:INFO mapreduce.Job: map 0% reduce 0%The job does not progress further. Additionally, I see the following message in the logs:INFO conf.Configuration: resource-types.xml not found. INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.Despite this, there are plenty of available resources in the cluster, so resource allocation should not be the problem. I also have 3 healthy nodes. I’ve tried various troubleshooting steps, but I haven’t been able to resolve the issue.Could the "resource-types.xml not found" warning be causing this problem?If so, where should the resource-types.xml file be placed in a Cloudera setup?Any help or suggestions would be greatly appreciated!

Hi Team,We have 2 nodes cluster, second node we added recently, now we ran hdfs balancer but output is showing the cluster is already balanced but in actual the data is not balanced. Please suggest here.. ThanksIrshankhan

All OS disks on the cluster server are broken,Therefore, I need to install a new OS and set up hdfsHowever, all data disks are considered availablename node has "edits" file and "fsimage" file,There are "blk" and "blk meta" files in the data node.Is it possible to restore hdfs data based on block data in this situation?

I used the count command to view the HDFS directory of the cluster, and there is still 9T of storage space left. However, when I upload a 2MB file, I will report that the space is insufficient. Has anyone encountered this situation before

Hello Team, I am Darshan Hiranandani, I recently performed an upgrade from CDP 7.1.6 to CDP 7.1.8, and since the upgrade, we are encountering difficulties starting the HDFS NFS Gateway role. Previously, on CDP 7.1.6, we were able to start the NFS Gateway by running the command hdfs --daemon start portmap and then manually starting the HDFS NFS Gateway role from Cloudera Manager. However, after the upgrade, this approach no longer works, and the HDFS NFS Gateway role fails to start even though the portmap service appears to be running. Additionally, we are facing restrictions that prevent us from starting the rpcbind service due to security concerns. Given this, we would like to know: Is there a workaround or alternative method to start the HDFS NFS Gateway role without the need to enable or start the rpcbind service? Are there any known changes or updates in CDP 7.1.8 that could be causing the issue with the NFS Gateway startup process? Could the issue be related to the way the mount