Q&A

ip:9866:DataXceiver error processing WRITE_BLOCK operation src: /ip:30458 dst: /ip:9866java.io.EOFException: Unexpected EOF while trying to read response from server at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:539) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:846) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:291) at java.lang.Thread.run(Thread.java:748) ip:9866:DataXceiver error processing WRITE_BLOCK operation src: /ip:46217 dst: /ip:9866java.io.IOException: Not ready to serve the block pool, BP-711568456-10.13.6.7-1597913882399. at org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAndWaitForBP(DataXceiver.java:1402) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.checkAccess(DataXceiver.

There have some rows data with header in txt file, like this: test_a|test_b|test_c|test_d|test_ea|b|3.0|4.0|5.0a|b|3.0|4.0|5.0a|b|3.0|4.0|5.0 now , i want  remove the value after test_c and test_d  decimal point,  the result change this: test_a|test_b|test_c|test_d|test_ea|b|3|4|5.0a|b|3|4|5.0a|b|3|4|5.0 how could i do? thanks.

hi there,I have encountered a common error as follows:Memory left in process limit: 180.41 GBMemory left in query limit: -467.80 KBQuery(7e4794a439e5f26d:748d4c7c00000000): memory limit exceeded. Limit=1.00 GB Reservation=635.75 MB ReservationLimit=819.20 MB OtherMemory=388.71 MB Total=1.00 GB Peak=1.00 GBI don't quite understand the difference between the "process limit" and "query limit", and what's the reservation memory for?many thanks!

Our sales team are extracting the months sales opportunities, actuals etc. Within the one CSV file there are potentially 100+ rows. Now, we need to create new flow files (array?) containing individual sales people. We have 20+ sales people each working on many opportunities.salesOpId,salesRepId, salesOpStage, .....142435,135235,inception ....142436,572856,inception ....142437,135235,pipelining ....142435,572856,inception....149468,135235,contract....149464,135653,contract.... As you can seesalesRepId 135235 is working against 3 opportunities, (142436,142437,149468)salesRepId 572856 is working against 2 opportunities (142436, 142435)salesRepId 135653 is working on a single opportunity (149464) For this example I need 3 FlowFiles created (which will be sent to their respective managers); I'm capable of reading the CSV changing it to Json; but now I'm completely stumped on the next step - split the json? Any help would be greatly appreciated. Ron M. &nbs

Hi, We have a CDH cluster with version 5.15 and we have migrated the db used by Cloudera from postgres to Oracle. We have followed the migration steps and took the cluster down. When we tried to start the cloudera server again it's failing with the below error.  022-01-13 09:53:19,028 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists.2022-01-13 09:53:19,029 INFO main:com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 51300.2022-01-13 09:53:19,029 INFO main:com.cloudera.enterprise.dbutil.DbUtil: Current database schema version: 51300WARN main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Failed to determine prior version. This is expected if you are starting Cloudera Manager for the first time. Please also ignore any error messages about missing tables. Moving ahead assuming no upgrade: org.hibernate.MappingException: No Dialect mapping for JDBC type: -92022-01-13 09:53:19,043 ERROR main:com.cloudera.server.cmf.bootst

Hello Everyone,  I want to use hive class library name "org.apache.hadoop.hive.ql.udf". Could you tell me where I can download it? Regards,

Hello! I am trying to use PutDatabaseRecordProcessor to execute an Insert statement on my table. However, I have encountered an issue. The type of the column in the SQL Table is "uuid", which is definitely unsupported by the apache Nifi. The record is of type string, previously generated by the UpdateRecord Processor. I have tried some more moves, using logical types in the Record Writer. One more time unsuccessful. Is there any case, where i can convert mine string type record to uuid type, which is presented in my table? Thanks in advance  

we have 3 Kafka brokers on Linux RHEL 7.6 ( 3 linux machines )kafka version is 2.7.Xbrokers ID's are - `1010,1011,1012`from kafka described we can see the followingTopic: __consumer_offsets Partition: 0 Leader: none Replicas: 1011,1010,1012 Isr: 1010Topic: __consumer_offsets Partition: 1 Leader: 1012 Replicas: 1012,1011,1010 Isr: 1012,1011Topic: __consumer_offsets Partition: 2 Leader: 1011 Replicas: 1010,1012,1011 Isr: 1011,1012Topic: __consumer_offsets Partition: 3 Leader: none Replicas: 1011,1012,1010 Isr: 1010Topic: __consumer_offsets Partition: 4 Leader: 1011 Replicas: 1012,1010,1011 Isr: 1011Topic: __consumer_offsets Partition: 5 Leader: none Replicas: 1010,1011,1012 Isr: 1010from Zookeeper cli we can see that broker `id 1010` not defined[zk: localhost:2181(CONNECTED) 10] ls /brokers/ids[1011, 1012]and from the log - `state-change.log`we can see the following[2021-12-16 14:15:36,170] WARN [Broker id=1010] Ignoring LeaderAndIsr request from controller 1010 with correlation id 485 e

Hi,  I am currently trying to count the number of files in the directory. The command that I keyed to count in the windows command prompt is "dir /a:-d /s /b "D:\Testing\UploadFile" | find /c ":""I am trying to use NiFi to apply the above command using the ExecuteStreamCommand processor. Currently I am facing the "Access Denied error" as shown in the picture. Execute Stream Command Processor ErrorExecute Stream Command Processor Properties Appreciate if anyone could help with an example and explanation. Thanks.Note: the fileslocation set in the properties is the path to the file.

Hive Table:Internal tableROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' STORED AS TEXTFILEData ingestion is successfulFacing memory issues while querying!Input file: JSON fileWhile researching on memory configuration, found some information here to calculate recommended memory configuration against available resources.Our POC cluster size: 4 node cluster - 8 GB, 2 cores and 2 disks on each nodePlease refer below information and let me know if memory needs to be extended or do TEZ memory configuration needs to be changed to overcome "out of memory" issue?Calculated memory configuration uisng yarn-utils.py:$ python yarn-utils.py -c 2 -m 8 -d 2 -k True Using cores=2 memory=8GB disks=2 hbase=True Profile: cores=2 memory=5120MB reserved=3GB usableMem=5GB disks=2 Num Container=4 Container Ram=1024MB Used Ram=4GB Unused Ram=3GB yarn.scheduler.minimum-allocation-mb=1024 yarn.scheduler.maximum-allocation-mb=4096 yarn.nodemanager.resource.memory-mb=4096 mapreduce.map.memory.mb=1

I am using CDP 7.1.7 and the cluster has not enabled Kerbores yet. Ranger is not enabled either. I followed the step in this post https://community.cloudera.com/t5/Support-Questions/CDP-7-1-3-Zepplin-not-able-to-login-with-default-username/td-p/303717 to be able to log in as admin. But this "admin" account has no permission to access the configuration or interpreter page.  According to CDP documentation, https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/configuring-zeppelin/topics/enabling_access_control_for_interpreter__configuration__and_credential_settings.html, to configure shiro.ini for Zeppelin security, I have to go through Zeppelin web UI.  What should I do? Regards, 

how to use ListFTP and FetchFTP processors get all files two hours before the current time in FTP server.There will be more and more files in the FTP server named by dateThese filename in the FTP contains a date as follows: xxxx-xxxx-xxxx-xxxx-20211231143000-xxxx-xxxx.csvxxxx-xxxx-xxxx-xxxx-20211231153000-xxxx-xxxx.csv......    

Hi,   our CDH CM is down (difficult issue, related to certificates renewal), so we would like to access everything related to the workbench (sessions, projects, ...) from terminal(I guess there is a database on the CDSW host related to that...)  thanks for help

Hi,I'm trying to pass an output value of a processor dynamically into another and use it as a new property , here is my flow:ReplaceText (gives the output)-> UpdateAttribute (i created a property which value is referencing output) ->  EvaluateXpath ( i want to pass this value in a new property here) . I tried passing it by ${property_name} and $property_name but nothing works. Is it impossible to have a property referencing another in the EvaluateXpath processor ?Thanks.

I know the ListSftp processor have 'Minimum File Age' property, but the ListFtp have not  'Minimum File Age' property, do you know how to add 'Minimum File Age' property  in ListFtp properssor. thanks! when i add 'Minimum File Age' in ListFTP processor,  it not work, like this:it displays 'Minimum File Age' is not a supported property or has no Validator associated with it 

How does nifi delay scanning the files on the ftp server, which processor to choose,thank you

How to convert String Into Int in the SQL table

While creating table , its failing with below error.Query: CREATE TABLE REPORTKUDUPRIMARY KEY(Marketing_Cloud_Visitor_ID)PARTITION BY HASH(Marketing_Cloud_Visitor_ID) PARTITIONS 10STORED AS KUDU TBLPROPERTIES('kudu.master_addresses'='X.X.X.X','kudu.num_tablet_replicas'='1')AS SELECT * FROM REPORTN  WARNINGS: Error applying Kudu Op.: Incomplete: buffer size limit is too small to fit operation: required 52236317, size limit 10485760Key already present in Kudu table 'impala::db.REPORTKUDU'. (1 of 9006 similar)

Hello  I'm new to the community and to cloudera/big data in general,  I am having issues with hive performance I have for example a table of 600 records and when I use a select * it runs in .05 seconds but if I use for example a count(*) or any function it runs in like 17 seconds, do any have any tip or trick to check performance or what parameter to check/modify in order to improve this execution time? My enviroment are CDH 6.1.0 withHive 2.1.1-cdh6.1.0 Thank you in advance  Ulises Rangel

Hello Guys, We have 3 node NiFi cluster and the NiFi version is 1.12.1nifi-hatest-01nifi-hatest-02nifi-hatest-03 All the 3 nodes are in cluster, but not in UI.In all the 3 NiFi nodes, we see this 2 nodes in cluster mode:Now, as we see nifi-hatest-02 is the primary and cooridantor. And nifi-hatest-03 is not being shown here. In nifi-hatest-01, we see nifi-hatest-01 is connecting to nifi-hatest-022022-01-19 16:02:09,840 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2022-01-19 16:02:09,820 and sent to nifi-hatest-02:9088 at 2022-01-19 16:02:09,840; send took 20 millis In nifi-hatest-03, we see nifi-hatest-03 is also connecting to nifi-hatest-022022-01-19 16:02:09,840 INFO [Clustering Tasks Thread-1] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2022-01-19 16:02:09,820 and sent to nifi-hatest-02:9088 at 2022-01-19 16:02:09,840; send took 20 millis In nifi-hatest-02

Hi Team ,  We have upgraded from CDH  to CDP 7.4.4,Atlas has been installed and Navigator also running parallely .Is it safe to remove and clean up databases of cloudera navigator and from CM ?Please share the steps involved. Thanks,Syed

Hi,We are not able to start the metrics collectors. We get the following alerts on ambari :  Every time we tried to start the metrics collector without any success,  We are getting the following logs:stderr:   /var/lib/ambari-agent/data/errors-90083.txtPython script has been killed due to timeout after waiting 1200 secs stdout:   /var/lib/ambari-agent/data/output-90083.txt...2022-01-17 16:25:00,758 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/ams.collector.keytab amshbase/xx-xxx-x1-xx03.xxxxx.xx@XXXX.XXXXX.XX; /usr/sbin/ambari-metrics-collector --config /etc/ambari-metrics-collector/conf --distributed start'] {'user': 'ams'} /var/log/ambari-metrics-collector/ambari-metrics-collector.log...2022-01-18 07:32:51,941 WARN Timed out waiting to close instance java.util.concurrent.TimeoutException at java.util.concurrent.FutureTask.get(FutureTask.java:205) at org.apache.phoenix.jdbc.PhoenixDriver$1.run(PhoenixDriver.java:101)2022-01-18 07:35:26,793 WAR

Hello,   I wanted to ask if there's a page / instructions / info regarding the recent log4j2 vulnerability (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228) and how it can affect Cloudera CDH setups? If it does affect, what are the recommended mitigations on it?   Thanks, Mor

Is Apache NiFi HIPAA certified?

Hello Experts, I have a 3 node nifi cluster with host name as nifi-hatest-01, nifi-hatest-02, nifi-hatest-03. nifi-hatest-01 is primary node at the moment and I can see (in logs) it is receiving heartbeats from 02 & 03.And if I create flows in any one of the 3 nodes it gets sycned to all nodes. But the problem now is the 3rd node (nifi-hatest-03) does not show up in the "cluster" tab of Nifi UI.  Any idea what could be the issue?any suggestion would be much appreciated. @MattWho @GangWar @smdas  ThanksMahendra