Solved

my cluster is CDH 5.16.2 Please help ##log## [xxx@hdpwxxx1.true.care]:/reserve1>hdfs dfs -put 30.img /tmp/41.img22/08/12 17:49:55 INFO hdfs.DFSClient: Exception while adding a blockorg.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException): Not replicated yet: /tmp/41.img._COPYING_at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3688)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3477)at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:694)at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:219)at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:507)at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.

I'm installing CDH 6.2 on rhel7 and using Oracle19C as metastore and backend db for all components wherever required. All services start normally, however when trying to login to hue, the page throws error. CDH Parcel Version: CDH-6.2.0-1.cdh6.2.0.p0.967373Oracle DB: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - ProductionOracle Client: 19.3.0.0.0 Tried with three cx_Oracle versions: cx_Oracle-5.2.1-py2.7 / cx_Oracle-5.3-py2.7 / cx_Oracle-7.2.0-py2.7 Environment:Request Method: GETRequest URL: http://<URL>:8888/hue/accounts/login?next=/Django Version: 1.11Python Version: 2.7.5Installed Applications:['django.contrib.auth', 'django_openid_auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'django.contrib.sites', 'django.contrib.staticfiles', 'django.contrib.admin', 'django_extensions', 'babeldjango', 'desktop', 'axes', 'about', 'beeswax', 'filebrowser', 'help', 'impala', 'jobbrowser', 'jobsub', 'metastore', 'oozie', 'pig', 'proxy', 'rdbms'

Hey everyone,I want to make an API request for every single day in a period. I have defined a start date and an end date in the Generate Flow File Procesorthe api structure is as followshttps://example.com/interfaces?session=${SessionId}&date_start=${since}&date_end=${since}and now I want to loop with route on attribute and update attribute to request all days in this time period.now the question how can I, the attribute of the start date always increase by one day. with the following expression in the update attribute processore I get an empty field as result${since:toNumber():plus(86400000):format('yyyy-MM-dd')}ResultDoes anyone have an idea what adjusted we 

Hi all, I want to access Nifi REST API from a different client machine which is in a different domain. The REST API is up and running but gives me a CORS error when I try accessing the API from a client in a different domain. Please see below -Access to XMLHttpRequest at '...' from origin 'null' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. I understand that I need to change Jetty's web.xml to allow cross origin requests. I can't seem to find web.xml in my nifi install. This is a nifi docker container.Any help I can get in this is much appreciated. Please also let me know if there's a better solution than changing web.xml. I tried CORS browser plugins but haven't been able to get them to work. Thanks in advance.

Which versions and distributions of Hadoop are compatible with Spark 3.3?

Hi team I am new to CDP , I have a problem with the ldap changes in the cloudera manager service.It requests a restart of the service after applying the changes , this restart is applied but the changes are not applied.Can you help me , to identify the problem .

Hello,I'm new to nifi so my problem may be dumb. Sorry if it seems really basic.I'm trying to use nifi to :1/ Read a JSON response from a REST GET endpoint.2/ Parse the JSON and split some JSON array into JSON objects3/ insert objects in SQL.I'm stuck at Step 2 because my SplitJSON doesnt return anything. No error as well.Here is a sample of my JSON : { "context":"q%3D%2523all%26q.restriction%3Dclass%253Acr_hashtag%26of%3Djson", "nhits":1000000, "nmatches":1000000, "start":0, "ellql":"#query{nbdocs=26966678, text_relevance.expr=\"@term.score*@proximity+@b\", noproximity, term.score=NO_RANKING}(#and(#true() #category(categories,\"Top/datamodelclass/cr_hashtag\") ))", "executor":null, "estimated":false, "autocorrected":false, "spellCheckSuggestions":{ "origStr":null, "suggestions":[ ] }, "hits":[ { "did":1, "url":"ID=10055&", "buildGroup":"bg0", "source":null, "slice":3, "score":0,

Hello, I would like to create Hive tables  using Zeppeling and I found the the below document that use HWC:https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/integrating-hive-and-bi/topics/hive-hwc-reading.htmlI'm not sure what means "livy configuration file"? Is "%livy2" a new Interpreter? May I create a new Interpreter with only indicated configurations? Thanks!  

Hello, please kindly help. i am trying to convert a datetime(attribute) from one format to the other but it keeps giving me empty value on the attribute:SCENERIO:Flowfile is json----->extract date from Json flowfile using Evaluatejsonpath---->format date using UpdateattributeActual Date: "2022-08-01 00:00:00.0"Desired date: "01/08/2022 00:00:00.000000"I have tried to use this on the updateattribute, but it is not working:${DATE_TIME:toNumber():toDate('yyyy-MM-dd hh:mm:ss.S'):format('dd/MM/yyyy HH:mm:ss.SSSSSS')} Thank you.

Is there any way to update the flow description in the NiFi Registry after the initial creation of the flow that does not require restarting version control?  I checked the REST API endpoints as well and didn't see anything. I would like to be able to clean up descriptions (or even just correct a typo) without going through a cumbersome export versions, import flow, import versions, delete and re-import PGs process. Thanks.

good morning cloudera community, we are using apache ambari in version 2.6.2.2 and in this version we are using HDP 2.6.5, with that, we need to configure the parameter below in HDFS, but unfortunately we are not finding this option in the HDFS configuration, so we need to know how configure this parameter in HDFS? parameter: dfs.client.block.write.replace-datanode-on-failure.policy=ALWAYS PS: would it be in the "custom hdfs-site" option by clicking on "add property" and adding the parameter in the "properties" box? 

if I have 5 zookeeper nodes and from that 5 zookeeper nodes one 1 zookeeper nodes goes down which was leader node then how zookeeper will elect the new leader from the remaining 4 zookeeper node?

Hi,  I am new to nifi and stuck with two issues for which i need help:- The task is to migrate a set of tables from source to target database. The data will have to be filtered as well based on a field before migrating. The pipeline has to migrate historical data as well as perform the incremental fetch.  1. For incremental fetch, I used the GenerateTableFetch processor and set the max-value column to a date field in the table. I also used the partitioning feature to get the data in chunks by setting the partitioning size and column. It all works well. But considering it is a stateful processor, i have received feedback to go stateless as we can loose the state in case of node crash. How can i achieve the incremental fetch and partitioning feature in a stateless manner? 2. The set of tables to be migrated have a parent child relationship. The incremental fetch is required for all the tables. However child tables don't have any such column that can be used

Hi Team, We have CDH - 5.16.2 in our environment, recently renewed Cloudera -license.But still since today we are not able to create a case on support portal.Please find the attached snapshot. 

Hello All, I am using the mysql CDC processor in Nifi, to get incremental changes into DWH, in this processor fo the username field, which permissions should be granted to the user so that it can access cdc files ?should be the one with highest privileges/ root user ?because it is not mentioned anywhere in the help section (https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-cdc-mysql-nar/1.5.0/org.apach... ) And when i provide a user credentials who has GRANT ALL PRIVILEGES ON *.*  then it is able to access CDC bin logsBut getting permission denied when trying to access with a user who has only read permissions on *.* Since the processor has nothing to do with write, i thought only read permissions were enough, but looks like its not the case..  Because I need to create new users in the source to access these CDC files.. and I dont want to give them Grant ALL, as it seems riskier.. Please help with your suggestions and th

Hi, We have got this issue post java upgradation. Kindly let us know whether this error occurred because of java upgrade or else please suggest a solution.  spark-shellSetting default log level to "WARN".To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).22/08/03 15:34:49 ERROR SparkContext: Error initializing SparkContext.org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:89)at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:164)at org.apache.spark.SparkContext.<init>(SparkContext.scala:500)at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2498)at org.apache.spark.sql.SparkSession$

当前Hive版本：3.1.3000.7.1.7.0-551 创建hive外部表： CREATE TABLE IF NOT EXISTS `alter_test` (a string,b string) STORED AS ORC TBLPROPERTIES("transactional"="false");转hive内部表：ALTER TABLE alter_test SET TBLPROPERTIES('external'='false');重命名：ALTER TABLE alter_test RENAME TO alter_test_bak;注：external.table.purge 这个配置是一直为true的此时，表alter_test_bak的location目录名称依然是alter_test。同样的操作步骤，我在Hive版本 3.1.3000.7.1.1.0-565是可以同步更改location目录名的，请问这是怎么回事呢

I have 3 nodes NiFi and 1 node Registry instances running. Registry - SSL enabled and AD-integrated  NIfi  - SSL enabled and AD-integrated    how can I integrate NiFI with Registry, so I can start using Registry.   @MattWho please advice for any recommendations     

Hello,is it possible to alter compression.type on a kafka topic while having applications on the topic running? What should be expected hard drive saving factor for text records like json (2KB size) and compression.type=gzip. In CM, I see a global setting. Does it apply to KafkaMirrorMaker only? Are producer and consumer applications somehow affected, when the global parameter changes.Best regardsJaro

Hi, I am getting error when I am querying my table in impala editor.The simple query:select * from table GeoIP2_ISP_Blocks_IPv4;The error:The table creation query is give below:CREATE External TABLE GeoIP2_ISP_Blocks_IPv4 (Client string,ID int,Number int,Name string,)ROW FORMAT DELIMITED FIELDS TERMINATED BY ','WITH SERDEPROPERTIES ("separatorChar" = "\,","quoteChar" = "\"")LOCATION 'hdfs:///user/hive/warehouse/tips/'tblproperties ("skip.header.line.count"="1"); The same error is coming when I am creating external tables from tab delimited files. All this is happening after latest CDP Update. Please someone help me it would be really helpful.

Hi,is there a way to find and remove duplicate entries in two flowfiles?I have one flowfile generated with SQL Processor with entries from my database. The other one contains new and "old" entries. So in order to only write the new entries in the database, I have to detect and remove the entries that already exist in the other flowfile.I already tried the HashContentProcessor but it hashes the content of the whole file. I would need a processor that hashes line for line (and then compares all hashes with each other).Thanks for your help!

We are trying to determine the correct latest version of Cloudera Runtime to use, CM is offering 2 possibilities for CDH 7.1.7-1: 7.1.7-1.cdh7.1.7.p74.210577657.1.7-1.cdh7.1.7.p1000.24102687 What is the best way to determine the latest version? Thanks in advance.    

When using the XMLReader service, how do you retain the outermost parent element of an XML document so when I convert it to JSON I have the outermost element in the JSON version too? For example, given the following XML document how do I capture the "<a>" element rather than just the "<b>" element?   <a> <b>45</b></a> What I want:{"a": {"b":45}} What I get:{"b":45} 

Hi, I want to connect connect two different MySQL database services(mysql 5.7 and Mysql 8.0 ). I am using two  two CaptureChangeMySQL Processors for this. Mysql 5.7 is install in my local machine and CaptureChangeMySQL Processors connecting successfully.  Mysql 8.0 is install on docker container and on host machine port 3310 like (sudo docker run --name mysql-8.0-container -p 127.0.0.1:3310:3306 -e MYSQL_ROOT_PASSWORD=password -d mysql:8.0). CaptureChangeMySQL connecting with mysql 8.0 is giving  error. Can any suggest why i am getting this error and how to resolve it.Note: I am able to connect mysql 8.0 using mysql workbench.  Error:2022-08-03 15:59:58,863 ERROR [Timer-Driven Process Thread-4] o.a.n.c.m.processors.CaptureChangeMySQL CaptureChangeMySQL[id=352daba9-0182-1000-a02a-faf160ab1329] Processing failedorg.apache.nifi.processor.exception.ProcessException: Could not connect binlog client to any of the spec

As per the accepted solution posted in this question, export_snapshot is not supported ( or may not work ) between major versions of Hbase. However, this CDP documentation mentions the supportability of export_snapshot from CDH 5 (Hbase 1.2.0) to CDP 7 (HBase 2.2.3).  We are working on migrating Hbase tables from Hbase 1.2 to Hbase 2 and would like to know if "export_snapshot" work in this scenario.