Solved

I am observing following error for one of my nifi data flow:o.a.nifi.processors.kite.InferAvroSchema InferAvroSchema[id=9fa25adc-91da-31e3-97ff-d5012c68588c] Failed to infer Avro schema for StandardFlowFileRecord[uuid=a8cf241e-8804-4177-837f-f832657sa596,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1713770110449-11, container=default, section=11], offset=0, length=44],offset=0,name=iSAM_ontEthPort.csv,size=44] due to java.lang.NullPointerException: No content to infer schema: java.lang.NullPointerException: No content to infer schemajava.lang.NullPointerException: No content to infer schema What to make out of it?

Hoping someone can help me with a JOLT spec. I have the following json:{"schemaVersion": "3.2","uniqueIdentifier": "ri.nzRNR5a3ad1AzOkv_2FU9WN6Z_2FluxpEl4eXjgUnYpgkvk_3D","entityType": "CAR"} and I need to use JOLT to transform to this: {"_id": "ri.nzRNR5a3ad1AzOkv_2FU9WN6Z_2FluxpEl4eXjgUnYpgkvk_3D", "schemaVersion": "3.2","uniqueIdentifier": "ri.nzRNR5a3ad1AzOkv_2FU9WN6Z_2FluxpEl4eXjgUnYpgkvk_3D","entityType": "CAR"} I cannot figure out the correct spec. Thanks!

For example, the spring framework of the official nar package of nifi 1-25.0 is 5.3.31. I changed it to 5.3.33 and I can modify these files /home/servers/nifi-1.25.0/lib/properties/spring-core-5.3 .31 .jar/home/servers/nifi-1.25.0/lib/properties/spring-aop-5.3.31.jar/home/servers/nifi-1.25.0/lib/properties/spring-context-5.3.31.jar/home/servers/nifi-1.25.0/lib/properties/spring-jcl-5.3.31.jar/home/servers/nifi-1.25.0/lib/properties/spring-beans-5.3.31.jar/home/servers/nifi-1.25.0/lib/properties/spring-express-5.3.31.jarIs /home/servers/nifi-1.25.0/lib/properties/spring-web-5.3.31.jar implemented? Is it possible to modify the spring framework in these nar packages? I tried to modify these, but the result was that the apache nifi service could not be started./home/servers/nifi-1.25.0/work/nar/framework/nifi-framework-nar-1.25.0.nar-unpacked/NAR-INF/bundled-dependencies/spring-core-5.3.31.jar/home/servers/nifi-1.25.0/work/nar/framework/nifi-framework-nar-1.25.0.nar-unpacked/NAR-INF/b

  I have a remote server and servers authenticated Hadoop environment.I want to copy file from Remote server to Hadoop machine to HDFSPlease advise efficient approach/HDFS command to copy files from remote server to HDFS.Any example will be helpful.as ordinary way to copy file from remote server to server itself isscp -rp file remote_server:/tmpbut this approach not support copy directly to hdfs

Hi,I have Nifi 2.0-M2 and Apache Ignite 2.16 database running on my localhost.http://localhost:8443/nifijdbc:ignite:thin://localhost:10800  (I am able to connect to Ignite localhost db from Dbeaver)I have copied ignite-core-2.16.0.jar file to nifi lib directory (before starting nifi) and tried to create DBCPConnectionPool controller service in Nifi to connect to Ignite database.  DB Controller service returns the below error.Failed to invoke @OnEnabled method: java.lang.NoClassDefFoundError: Could not initialize class org.apache.commons.dbcp2.BasicDataSource - Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.NoClassDefFoundError: Could not initialize class org.apache.ignite.IgniteJdbcThinDriver [in thread "Timer-Driven Process Thread-10"]Can you let me know what am I missing here?  Please let me know if more details are required.Screenshot of DB controller service     

I read the article But I can't understand a few things1. I havea Hadoop (3.1.3)MIT KDC cluster on a separate serverActive Directory forserver users with hadoop is not included in the domain2. All hadoop NN, DN, JN, etc. services are launched using the keytabs created for them and are working successfully3. any requests from the user from AD fall with an error2024-01-27 15:56:07,949 DEBUG security.SaslRpcClient: Sending sasl message state: NEGOTIATE2024-01-27 15:56:07,950 DEBUG security.SaslRpcClient: Get token info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB info:@org.apache.hadoop.security.token.TokenInfo(value=class org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSelector)2024-01-27 15:56:07,951 DEBUG security.SaslRpcClient: tokens aren't supported for this protocol or user doesn't have one2024-01-27 15:56:07,951 DEBUG security.SaslRpcClient: Get kerberos info proto:interface org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolP

hellomy cluster is cdh5.14   centos7.4   JAVA7We have been receiving alerts for Impala. The health test result for IMPALAD_QUERY_MONITORING_STATUS has become bad: There are 0 error(s) seen monitoring executing queries, and 1 errors(s) seen monitoring completed queries for this role in the previous 5 minute(s). Critical threshold: any.The following picture shows cloudera-scm-agent error logshow to solve this error? 

Hi Everyone,I am trying to perform some data transformations using UpdateRecord processor.The input format is a nested JSON, and I am able to perform shifting of some fields using RecordPath.But facing issues when preparing a field using some conditional logic from a nested list.Here is my scenario:Input JSON[ { "orderId": "1234", "orderName": "Order 1", "orderItems": [ { "orderItemSeqId": "01", "itemStatusId": "ITEM_APPROVED", "orderItemQuantity": 1, "unitPrice": 149, "goodIdentifications": [ { "goodIdentificationTypeId": "SHOPIFY_PROD_ID", "idValue": "43623074824370" }, { "goodIdentificationTypeId": "SKU", "idValue": "2510524-4BB-S" }, { "goodIdentificationTypeId": "UPCA", "idValue": "2510524-4BB-S" } ] }, { "orderItemSeqId": "02", "itemStatusId": "ITEM_APPROVED", "orderItemQuant

I use the PutIcebergProcessor to write data to my data lake. Therefore I need to specify a HiveCatalogService. This Service needs HadoopConfigurationResources. This parameter is a path to an xml file containing the credentials to the S3 where the Iceberg files are stored.My Problem with this, that some content of this file is supposed to be secret to the users interacting with the nifi ui. However, as soon as a UI user knows this path, he can simply use ExecuteProcess Processor to retrieve those information. Is there any way to keep those information safe? Reference:HiveCatalogService PutIceberg 

I'd like to have a process for automatically exporting the data flow definition and configuration without needing to go into the CDF UI.Right now, if I wanted the configuration, I'd click here in the UI:My thought was to use the CDP CLI, but under the df section, there doesn't appear to be a command for this.Any suggestions?

Hi All community/Support, I would like to ask how to extend this self-signed certificate validity to 6 months. As per default was 60 days. May i know which section of the code that setting this 60 days validity during start-up?  2024-04-02 07:32:08,803 INFO [main] org.apache.nifi.bootstrap.Command Generating Self-Signed Certificate: Expires on 2024-06-012024-04-02 07:32:11,796 INFO [main] org.apache.nifi.bootstrap.Command Generated Self-Signed Certificate SHA-256: 4XXXXXXXXXXXXXXXXXXXXXXX Appreciate if someone could help point out. 

Hey Everyone,We installed CDP Private Cloud Data Services i.e. ECS 1.4.1. After registering environment through management console, when we are trying to create CDE (Cloudera Data Engineering) cluster we are getting aws_key_id error although we have not deployed servers through AWS. Please find the logs below.2023-10-18 08:06:48: Create: Cluster: Data-engineering-1, user:   <admin@cdp.example>, event: Create, properties: [internal-actor-crn=crn:altus:iam:us-west-1:altus:user:__internal__actor__]2023-10-18 08:07:04: Log: Cluster provisioning initiated, cloud platform: STANDARD2023-10-18 08:07:04: Log: Started provisioning a managed cluster, provisionerID: liftie-9bylvhxh2023-10-18 08:07:04: LogDebug: Updating cluster status from ClusterProvisioningInitiated to ClusterProvisioningInProgress2023-10-18 08:07:04: LogDebug: Checking underlying cluster status2023-10-18 08:07:04: Log: Cluster provisioning completed2023-10-18 08:07:04: LogDebug: Updating cluster status from Clus

We are seeing an error / alert and unable to schedule / run jobs in CDE. We are on CDP - PCAny guidance  in resolving this Tks Wert

I have tried this code on jupyter of Cloudera Machine Learning.============================================================import cml.data_v1 as cmldatafrom pyspark import SparkContext#Optional Spark ConfigsSparkContext.setSystemProperty('spark.executor.cores', '4')SparkContext.setSystemProperty('spark.executor.memory', '8g')#Boilerplate Code provided to you by CML Data ConnectionsCONNECTION_NAME = "go01-dl"conn = cmldata.get_connection(CONNECTION_NAME)spark = conn.get_spark_session()# Sample usage to run query through sparkEXAMPLE_SQL_QUERY = "show databases"spark.sql(EXAMPLE_SQL_QUERY).show()=================================================  I cheched enable spark, as well when I tried to create sessionHowever the result was failed with error message 'No data connection named go01-dl found'    While I am trying this, I thought I have to get information of spark.BUT I CANNOT. Where Can I get the connection name of SPARK?What should I do to do ru

Dear Team, I am new to Cloudera. We are planning to use cloudera data platform. Is it possible to manage Data Governance activities with cloudera itself? Actually we are using informatica suite for data governance.  Please advice. I like to have a call with Middle East Cloudera Technical Team. Is it possible to connect with me?   Thanks! Regards, S.Ramakrishnan

We recently upgraded to CDE 1.20.3. After the upgrade we encountered issues related to resource allocation. I suspect that the yunikorn is not properly functionining because jobs are waiting time even if there are available resources to distribute to the spark jobs.Is there a yunikorn UI available from the CDE where we can access to monitor so we can easily monitor the yunikorn pods?

After deployed Impala Virtual data warehouse of Data service.I faced Error "401 client Error: Unauthorized for url: http://coordinator:28000/cliservice"Sometimes It works well.  To solve this problem what should I check? 

Hi ,In my profile, I could see company as "London Stock Exchange Group PLC" and it needs to be change to "Refinitiv US LLC".  Thanks & Regards,Vijendar

my cdh version is 6.3.2my hbase version is 2.1.0-cdh6.3.2How can I modify the hbase. online. schema. update. enable parameter of hbase on CDH

Hi all,We have a Nifi community version and we noted that the service is using 46GB of ram on a virtual server that has 47GB.We used these settings to limit the amount of ram to useJava. Ag. 2=-Xms21gJava. Ag. 3=-Xmx42gBut the issue remains.Can you suggest how to address this issue?Regards,Fortunato

I am fetching file(s) from an HDFS path and transferring them to an SFTP server using Nifi. The HDFS file list is created by a Sqoop job, and the HDFS directory may contain one file or more than one file.Here is the list of processors I am using right now:RouteOnAttribute --> GetHDFSFileInfo --> RouteOnAttribute --> UpdateAttribute --> FetchHDFS --> PutSFTP --> UpdateAttributeMy data flow starts from a single FlowFile produced by a Sqoop job, which then becomes many FlowFiles after executing the GetHDFSFileInfo Processor (based on the number of HDFS files). However, I require only a single FlowFile post PutSFTP for downstream processing of job completion.Could you please suggest some solutions to execute the processors after PutSFTP only once? Do we need to create any separate processor group from GetHDFSFileInfo to PutSFTP?My Dataflow looks like below 

Stuck in 'upgrade control plane' step while upgrading ECS to 1.5.3 from 1.5.2.Casuses: cdp-release-post-upgrade-hook-job1 failed.container logs are =========================================****************************************************https://console-cdp.apps.cdppvc1.hadoop.com/etc/cdp/cdp_keys.jsonWhile list environments: Something is wrong with output, Output JSON:____ERROR__WHILE__CALLING__LIST__ENVIRONMENTS__COMMAND____ /opt/app-root/lib64/python3.9/site-packages/urllib3/connectionpool.py:1103: InsecureRequestWarning: Unverified HTTPS request is being made to host 'console-cdp.apps.cdppvc1.hadoop.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warningswarnings.warn(An error occurred: 5 NOT_FOUND: No access key found with name: 1b098f81-ce2c-4342-ba02-f0e607b0d83c (Status Code: 401; Error Code: ; Service: environments; Operation: listEnvironments; Request ID: 80bb6513-3fe8-41b9-bf2f-e0a6

Hi everyone, I would like some clarification. Thank you. During the installation of ECS, we encountered some errors that prevented successful installation. One of the issues was as follows: the Certificate Signing Request (CSR) remained in an "approved" state instead of being "certificated," causing it to remain stuck in a loop.   # /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get csrNAME AGE SIGNERNAME REQUESTOR REQUESTEDDURATION CONDITIONecs-tolerations-webhook-csr 55m kubernetes.io/kubelet-serving system:admin <none> Approved   How should we address this? Should we consider a reinstallation? Or is there an underlying cause behind this issue? Thank you. Related issue: Failure to Install Embedded Container Service (ECS), encountering "Failed to reconcile with temporary etcd" after manual uninstallation 

Dear all, I want to ask the difference between `insert entire record` and `insert record fields` for the record result contents.When i use `insert record fields`, i got error:but returned ID is what I want:  When i use `insert entire record`, i did not get any errors, but in stead i have this output How can I get the correct output that I want without any errors? 

Hello,I have a Controller Service for DistributedMapCacheServer, when i tried to disable it so i can change it's properties, it shows an error of a missing referencing component. Why does my controller service still detects a component that is already deleted and how do i solve this so i can change my controller service properties?Thanks,Regards,Adhitya