Q&A

Hi,Do I need a license key for CDP Private base trial installation and if yes where can I obtain it?

Hello everyone.So I've been finding on How to Perform Word Replacement in a csv file by using a Dictionary File as the Guide.I've yet unable to succeed in doing this. I was hoping to get it done with just a single dataflow and just by using processors (without script) but I don't know what processors to use. I've tried a few but nothing seems to work. Hopefully I can get help.The input csv file i have is named try.csv which contains the words that need to be processed (word replacement takes place).While the dictionary file is named syno_street.csv which have a couple of rows and 3 columns:column1: original word (need to be searched in input csv)column2: original word (need to be searched in input csv)column3: replacement word (the output after processing)syno_street.csv:try.csv: (input csv for testing)Thank you in advance.

When we try to get YARN log from cloudera manager, it's shopiwng following  Logs are unavailable because Application Timeline Service seems unhealthy and could not connect to the JobHistory server. Please help anyone?   

Hello Everyone,We are developing a data lakehouse using Hive for the banking and financial sector. We would appreciate your insights on the following:1. Which data modeling approach is recommended for this domain?2. Are there any sample models available for reference?3. What best practices should we follow to ensure data integrity and performance?4. How can we efficiently manage large-scale data ingestion and processing?5. Are there any specific challenges or pitfalls we should be aware of when implementing a lakehouse in this sector?Your expertise and guidance would be greatly appreciated. 

I am trying to connect to impala using jdbc from one of the BI tools however ,The connection being successful sometimes disconnects for a while with below errorCaused by: java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500593) Communication link failure. Failed to connect to server. Reason: java.net.SocketException: Connection or outbound has closed. ExecuteStatment for query "SELECT a,b from test limit 101".at com.cloudera.impala.hivecommon.api.HS2Client.handleTTransportException(Unknown Source)at com.cloudera.impala.hivecommon.api.HS2Client.executeStatementInternal(Unknown Source)at com.cloudera.impala.hivecommon.api.HS2Client.executeStatement(Unknown Source)at com.cloudera.impala.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.executeNonRowCountQueryHelper(Unknown Source)at com.cloudera.impala.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.executeQuery(Unknown Source)at com.cloudera.impala.hivecommon.dataengine.HiveJDBCNativeQueryExecutor.<init>(Unknown Source)at com.

In the development cluster, the NiFi nodes do not show data or connection to Cloudera Manager, but if I access them via WebUI, the nodes appear properly connected and functional. Attached are NiFi logs and screen prints as evidence Previously I was instructed to review the Ranger policies as described in the Cloudera documentation, but the problem persisted, attached are the current Ranger settings. I also found the following error curious: Cookie rejected [hadoop.auth=""u=nifi&p=nifi/v321d186.prevnet@DEVDATALAKE.CDPDTP&t=kerberos&e=1721689709429&s=qJPbU6hDNqLtkL1BokMp...", version:0, domain:v321d175.prevnet, path:/solr/ranger_audits_shard1_replica_n1, expiry:null] Domain attribute "v321d175.prevnet" violates the Netscape cookie specification However, in the prod cluster the same error appears in the logs and I don't have this problem

Hi,I have a json data as follow.INPUT :{  "vlineid" : "9PIA",  "type" : "M",  "catg" : "C",  "nhhigh" : 100.0,  "nhigh" : 50.0,  "nlow" : null,  "nllow" : null,  "uom" : "sec",  "vsectionid" : "PEPO60",  "vparamdesc" : "CYCLE TIME",  "vpartid" : "-",  "vflgipc" : "IPC02",  "vwctid" : "P9PIA0",  "vmachineid" : "IMM04",  "vparam" : "CYCLE-TIME",  "nvalue" : 0.0,  "t" : 1722833150160,  "vcrea" : "NIFI",  "vmodi" : "NIFI",  "dmodi" : "2024-08-05 11:49:17",  "dcrea" : "2024-08-05 11:49:17",  "t_epoch" : 1}I tried to remove the null field with this JOLT SpecificationJOLT :[{    "operation": "shift",    "spec": {      "*": "temp.&"    }  },  {    "operation": "modify-overwrite-beta",    "spec": {      "*": "=recursivelySquashNulls"  &nbsp

When trying odbc connection it is successful here. But when trying to connect from Qliksense, it is throwing an error "Error: Connector connect error: SQL##f - SqlState: 28000, ErrorCode: 9, ErrorMsg: [Amazon][ThriftExtension] (9) Error occurred while authenticating via SASL. Error details: SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Matching credential not found)" What might be the issue?is it issue with kerbeors? then what is the solution?it is happening with all the conenctions.

Hello,I have a 5-node cluster, running nifi 1.19.1. Recently, I got multiple alerts saying CPU usage is too high (more than 380% in a 4-core server). When I tailed nifi-app.log, I found the node was kept trying to join the cluster. An weird thing I found was as following fragment:2024-08-16 10:21:44,490 INFO [Process Cluster Protocol Request-15] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 8bd938b4-6e77-4353-8eef-1e00662efd82 (type=HEARTBEAT, length=5905 bytes) from mnprdlgmsyslog04.ds.cpr.ca:9443 in 145 millis2024-08-16 10:21:44,501 INFO [Clustering Tasks Thread-2] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2024-08-16 10:21:41,904 and sent to mnprdlgmsyslog04.ds.cpr.ca:11443 at 2024-08-16 10:21:44,501; determining Cluster Coordinator took 105 millis; DNS lookup for coordinator took 0 millis; connecting to coordinator took 2334 millis; sending heartbeat took 146 millis; receiving first byte from response took 11 millis; receiving full response t

We want to setup some Impala monitoring and look specifically for queries that have a "Waiting for client" state.  I am pretty familiar with the Cloudera Manager API calls to get Impala query information, but can't seem to figure out how to get this particular state.  Can someone shed light on how Cloudera Manager itself does it, since I see that status in the Cloudera Manager UI when looking at Impala queries?  Thanks, I appreciate any help in advance.Mark

Basically, I'm attempting to create a standalone Yarn application that runs code on hdfs files on a block level for validation purposes. First I will need to get where the different blocks are distributed (hdfs file metadata), and will run validation code on every block and will combine the results to determine if the file is valid or not. I want to minimize/eliminate network overhead by making the application validation code run on the same node on which the block resides (data locality).So, for instance, a 1 GB file could be divided into 10 blocks. In order to run the same validation code on all blocks in parallel, I would want to run 10 different instances.My thinking is, I would launch a single ApplicationMaster with 10 containers.Does my approach make sense, and if not, what do you suggest I change in my approach? And if it does make sense, how do I launch 10 different instances while (possibly) determining 10 different hosts for every container.

I need to inject data read from elastic in Json format to two different tables having parent child relationship.i.eUser tableName, Age, RoleDavid,35,1Tay,23,2John,27,2Role table:1,admin2,userMy requirement is to insert in user table first get the Role ID and then include that id in next insert to dump data in Role tableHow can i achieve this thing Apache Nifi?@mburgess

In Apache NiFi, We are interacting to SOAP api through powershell. As API expect attachment with MTOM.Now if we load 10000 request we can see only one server is acting on those requeset while other remainnig 2 server are sitting idle. Is there way to share the load acrooss all three servers?

Hi,I am new to NIFI, and I was wondering if I could change the specific column in the csv file.for example, if the column has numerical values it is supposed to be converted to string type like if the column had a numerical value 1 it should be converted to apple in the same columnI tried doing it by referring to websites but still I was not successful. Looking forward for some response for this problem. Thanks in advance   

I'm trying to connecting to a remote hbase from a java application.The remote hbase version is 2.1.0, such as my local hbase-client.The code is working well with another cloudera environment, the only difference is that this environment is protected with kerberos, but I get a successfull login with keytab in the log.In the RpcServer log I found "Expected HEADER=HBas but received HEADER=\x00\x00\x01\x0B from :61866".I can't find any post on internet and I don't know what to check.Any help on what should I check?

Hi All,I need to assess a Cloudera platform for a particular application of a Bank and create a utilization report. How to approach this task. I couldn't find any sample reports for reference. I found steps to download utilization report through Cloudera manager, but how to interpret it. What does each metric mean ? -- Does the  CPU / memory utilization percentage we get in the report are w.r.t the total resources that is allocated to that particular application or the overall available resources. What is the difference between pool vs users... etc.I haven't worked on this before. Any help is much appreciated.Thank You.

Hello!I'm trying to insert into a managed table from an external table created with hbase structure. insert overwrite table managed_ml select key, cf1_id , cf1_name from c_0external_ml;My CDP contains 3 nodes so my question is If I need to make a copy of hbase-site.xml into /etc/hive/conf so Hive is able to make the connection to HBase in order to populate my Hive table?Currently the insert is failing and the log shows that sentence is trying to call a different node instead the one where the tables are located. Thanks in advance!! 

I am trying use LookupRecord LookupRecord from Oracle table and i am using Avro reader/writer in LookupRecord . BUt i am receiving below errorAny suggestion @mburgess 2024-08-13 18:09:09,214 ERROR [Timer-Driven Process Thread-4] o.a.n.processors.standard.LookupRecord LookupRecord[id=3a641d1a-1b3c-1a47-2c18-b3541562e3ea] Could not determine schema to use for enriched FlowFilesorg.apache.avro.InvalidAvroMagicException: Not an Avro data file.

My application is streaming data to NIFI continuosly. During that time If NIFI Restartsthen the data flow in progress between any processors lost it's content with below error.Note: This is not related to processor, it happens for any processor.Before Restart:List queue->View-> I can able to see my streaming record.Now After doing NIFI RestartList Queue ->View or Download->My flowfile atrributes are there and other details are available but the flowfile content is not there.Kindly help me to solve this issue. All relevant idea's are welcome.

Hi,I have been struggling for sometime now trying to figure out if Nifi cluster can be deployed using windows docker-desktop where each node lives on a different host. I managed to do it using Ubuntu system but since Ubuntu is not supported by our IT I'm trying to see if the same can be implemented in the windows env.I knew there will be challenges with windows since it has some limitations specially to containers networking  over Unix. One of those limitations is that if you are trying to use host networking the IP address of the host doesnt bind to the container , instead it uses what is called a loop back. When I try to set the  nifi host property to the machine fqdn or ip I keep getting the error:  java.net.BindException: Cannot assign requested address  Which I imagine that is a result of the IP already being reserved to the host and not accessible to the container.After doing some research to work around this problem, I came across Reverse Proxy. The

hi cloudera community i am using the "ExecuteScript" processor and executing python code using the "python" engine (python 2.7 (jython 2.7.3)) and when running the code, I get the error below:"org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: ImportError: No module named <name_module> in <script> at line number 3- Caused by: javax.script.ScriptException: ImportError: No module named <name_module> in <script> at line number 3- Caused by: Traceback (most recent call last):File "<script>", line 3, in <module>ImportError: No module named <name_module>" i would like to know how to install libs in this python that the processor is using?

I'm running Apach Hive in a Docker container and using the following image:  https://github.com/big-data-europe/docker-hiveI want to add kerberos authorization to connect to Apach Hive via JDBC from a Java application.everything I could find:hive-site.xml   <property> <name>hive.server2.authentication</name> <value>KERBEROS</value></property><property> <name>hive.server2.authentication.kerberos.principal</name> <value>hive/hive-server.example.com@EXAMPLE.COM</value></property><property> <name>hive.server2.authentication.kerberos.keytab</name> <value>/home/mykeytab.keytab</value></property>   I generated a valid keytab file but the container start and after a while stoped.There is only one configuration file in the container repository:ivysettings.xml   <ivysettings> <!--name of the defaultResolver should alway

Hello,We are in process of adding new servers to existing CDP PB cluster. These servers will have master and worker roles along with Kafka brokers distributed / assigned to them. As part of the process, we will also decommission few servers which will be replaced by these new ones.Questions are:1. What will be the functional impact of this change on data engineers / analysts / developers of team?2. What changes they will need to do in their codes / applications / integrations / connections, etc? I am aware of few like:   - Zookeeper ensemble will need to be modified   - Teams will need to be informed of new brokers   - Custom Knox topologies will need to be modifiedWhat are the other changes that will need to be done by the user / engineers group?3. I am aware that connection strings will need to be modified. Which ones, just Hive or any other as well?Services that will be impacted of this change are:HDFSYARNHIVEIMPALAKAFKAHBASEAlso, we have Tableau and ot

Hello, I have recently built a CDP 7.1.7 SP1 cluster and have all the services running except YARN QueueManager WebApp. QM Store is running with no errors. Even in logs for WebApp, only entry i see is:WARN WebApp Properties Reading conf/webapp.properties. I am almost clueless what to do to bring this up.Any suggestions please. Thankssnm1523

Need to save Json data to multiple child tables. How should i do?