Q&A

Hi,I have gone through multiple community support link, where it is mainly saying  to start cluster, say, 50MB/s And 1000 events/sec sustained throughput for average Flow is:3 nodes each with:CPU: 8+ cores (16 is preferred)Memory: 8+ GBDisk: 6 disks, each 1TB disks (could be spinning or SSD)Referred link: https://community.cloudera.com/t5/Community-Articles/NiFi-Sizing-Guide-Deployment-Best-Practices/ta-p/246781My Nifi (version 1.23.2) cluster maintaining below hardware configuration, where Disk is a concern in my environment. Only one disk and all the repository is using the same default Nifi configuration. What will be expected throughput in these case ?3 nodes each with:CPU: 8+ cores (16 is preferred)Memory: 8+ GBDisk: 1 disk of 500GB (could be spinning or SSD) 5 nodes each with:CPU: 8+ cores (16 is preferred)Memory: 8+ GBDisk: 1 disk of 500GB (could be spinning or SSD) 7 nodes each with:CPU: 8+ cores (16 is preferred)Memory: 8+ GBDisk: 1 disk of 500GB (could be

When I try 'Activate' Environment to create virtual datawarehouse of CDWThe error occurred below image.This is Authentication pageThis is environmentThis is role of cm_admin accoutThe cm_admin account has all permission on ActiveDirectory.And all ports related LDAP of Activedirectory server. Please let me know what shoul I have to do.     

Hello sir,I am using invokedHttp to post request. But it did not get response from invokeHttp. I get request file . But I want to get the response invokeHttp and write to remote server. Here is the diagramhere is invokeHttp properties.Request has been posted to server successfully.How to get response from invokeHttp and write in remote server?Please help me

Dears team, I have been using PySpark 3.4.2 with the following syntax:  sql_query = "INSERT OVERWRITE TABLE table_1 PARTITION(partition_date = {YYYYMMDD})SELECT table_1.a , table_1.b , table_2.cFROM table_2 change_capture_viewFULL OUTER JOIN ( SELECT * FROM table_1 WHERE WHERE partition_date = {YYYYMMDD_D_1}) current_view ON change_capture_view.a <=> current_view.aWHERE change_capture_view.a IS NULL"  and use spark.sql(sql_query ). And encountered the error:  File "/usr/hdp/3.1.0.0-78/spark3/python/lib/pyspark.zip/pyspark/sql/session.py", line 1440, in sqlFile "/usr/hdp/3.1.0.0-78/spark3/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__File "/usr/hdp/3.1.0.0-78/spark3/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 175, in decopyspark.errors.exceptions.captured.AnalysisException: Cannot overwrite a path that is also being read from. In essence, I am trying to retrieve data from

I am trying run query via HUE api of cloudera data warehouse on azure. However I couldn't get token for authentication.  I tried enter ID and password started with csso_ and machine user.Tried enter token made from knox as well. How could I get token for access Hue API??

Nifi version is 1.23.2 and kafka version 2.6.I use ConsumeKafka_1_0 1.23.2 to read messages from kafka.DetectDuplicate 1.23.2 processor is available, which deduplicates stream files on the same stream.However, stream read the same recording twice  and DetectDuplicate processor could not deduplicate the same message.I understand that it reads the same message because the offset  id value is same in two message.What could be the reason why the ConsumeKafka_1_0  processor reads the same message and DetectDuplicate  processor cannot detect same message?    

Hi guys,I am new in Nifi and I have some problem with authentication-I am using HandleHttpRequest to coolect some data from other system. I need to do some basic authorization. I know that it something like http.header.authentication but I have a few question about this:1.How can get it to this authorization header? My first processor is HandleHTTPRequest so it is enough that my second processor will be  EvaluateJsonPath and I do something like this: $.http.headers.Authorization or I need do something else?2. I heard that this authorization data are in Base64. How can I decode it and compere with my user and password when for example my user is user1 and password is password1I will be grateful for tips because I don't know how to ho about it 

Hello,I have nested JSON objects coming in a flowfile, looking like this (example): { "a" : 1, "b" : {"asd" : 156, "bed" : "lalala"}, "c" : "abba", "d" : {"add" : 872, "bla" : "bebebe"} } I'd like to put double quotes around every nested json object and also escape any double quote inside of that nested json. The result would look like this: { "a" : 1, "b" : "{\"asd\" : 156, \"bed\" : \"lalala\"}", "c" : "abba", "d" : "{\"add\" : 872, \"bla\" : \"bebebe\"}" } I tried doing a replacetext with a regex like this: (?<=:.*\{[^}]*)"However this is utilizing a negative lookbehind with varying length and it seems this isn't supported in java/nifi. What would be the best approach here? Thanks!

I have an existing iceberg table and I'm trying to evolve schema by adding a new column:`alter table my_db.my_table add column insert_time timestamp`When I execute the above statement, I get a 'New column(s) have been added to the table.' but the column doesn't appear. I tried running a REFRESH, an INVALIDATE METADATA, and I tried restarting Impala but it had no effect. I tried the same operation on a test table and it worked. I'm using Impala as a query engine and Azure for storage. Impala is running in a Data Hub Cluster.

Hello,I'm new to using oozie and I've been trying to figure out how to use the output of an action as the input of the next action in Oozie. I'm actually coming from using traditional ETLs, where the output of one Step can be used as arguments in the next Step. My question is whether the following points are possible with Oozie:Use the output of one action as the input of the next actionLet a subsequent action be executed "n" times according to the output of the first action. Like sub jobs or something.Actually, in order to know if this is possible, we are trying to run a very basic dump example where we read and insert with two hive actions, one for read a table and another one to write. Something like the following example created with Hue Oozie Editor.<workflow-app name="Workflow" xmlns="uri:oozie:workflow:0.5">  <start to="hive-04f4"/>  <kill name="Kill">  <message>Error al realizar la acción. Mensaje de error [${wf:errorMessage(wf:l

We have a client tool that we use for query log ingestion.  For CDH, we were able to provide our audit logs.  Is there something comparable to /var/log/impalad/audit/impala_audit_event_log_1.0-* in CDW?I tried looking in PREFIX + “app=impala-executor-log” based on this article: https://docs.cloudera.com/cdw-runtime/cloud/impala-troubleshooting/topics/impala-logs-s3.html, but did not see what I was looking for.

Hi,Whenever I am trying to create any external table with any path, the error comes up FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Illegal character in path at index 17: hdfs://namenode1/ apps/sysDSPPRS) Don't know from where it is taking the space between "namenode1/" and "apps".DDL:  hive> CREATE EXTERNAL TABLE mydb.ext_apps (a string, b string) LOCATION 'hdfs://namenode1/apps/sysDSPPRS/sysuser/ext_apps'; There is no path in HDFS as "hdfs://namenode1/ apps", but "hdfs://namenode1/apps" does exist [sysuser@server dev]$ hadoop fs -ls /Found 5 itemsdrwxrwxrwx - hdfs supergroup 0 2022-03-04 09:15 /appsdrwx--x--x - hbase hbase 0 2022-02-02 17:33 /hbasedrwxr-xr-x - hdfs supergroup 0 2020-09-18 12:57 /systemdrwxrwxrwt - hdfs supergroup 0 2022-03-11 07:29 /tmpdrwxr-xr-x - hdfs supergroup 0 2022-03-08 08:05 /userWe are using CDH version

Hi Team ,We are trying to add timestamp to Kafka message to get more details of message travel time from producer to landing   . I wanted to check if we can add any broker properties  from Cloudera manager  . RegardsBharad

we make database connect to apache hive and it's 'ok , but , the following issues when try to browse tables when we try to create dataset  2024-02-06 09:58:58,682:DEBUG:pyhive.hive:TExecuteStatementReq(sessionHandle=TSessionHandle(sessionId=THandleIdentifier(guid=b'\x9d\xfc\x17\xd3acJ\x81\x87\xf2<\x92mk\x92C', secret=b'\x1b\x8f\x9cX_NIP\x96\xe55\xce\x16\x0e\xf1:')), statement='SHOW VIEWS IN hrdb', confOverlay=None, runAsync=False, queryTimeout=0)12024-02-06 09:58:58,688:DEBUG:pyhive.hive:TExecuteStatementResp(status=TStatus(statusCode=3, infoMessages=["*org.apache.hive.service.cli.HiveSQLException:Error while compiling statement: FAILED: ParseException line 1:5 cannot recognize input near 'SHOW' 'VIEWS' 'IN' in ddl statement:28:27", 'org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:374', 'org.apache.hive.service.cli.operation.SQLOperation:prepare:SQLOperation.java:136', 'org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOp

We have a dataframe with 5 columns with nearly 1.5 Billion records.We are need to write that a single line (single records) as json.We are facing two issuesdf.write.format('json') is writing all as single line or single record but there is second line (empty line is coming) we need to avoid itdf.write.format('json').save('somedir_in_HDFS') is giving errorWe want the save the file as single file so that the downstream application will read itHere is the sample code. from pyspark.sql import SparkSessionimport pyspark.sql.functions as ffrom pyspark.sql.types import *schema = StructType([StructField("author", StringType(), False),StructField("title", StringType(), False),StructField("pages", IntegerType(), False),StructField("email", StringType(), False)])# likewise we have 1.5 billion recordsdata = [["author1", "title1", 1, "author1@gmail.com"],["author2", "title2", 2, "author2@gmail.com"],["author3", "title3", 3, "author3@gmail.com"],["author4", "title4", 4, "author4@gmail.com"]]&nb

Hello.We are running Cloudera CDP 7.1.7 and Cloudera Manager 7.4.4 but when we access Hue UI we can only browse Jobs :Do we have to follow links given in the article ?https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/hue-introduction/topics/hue-about-query-processor.htmlIs it for CDP 7.1.8 and higher ?Because when I look for adding a new service to our cluster, I can add DAS :but Query Processor is not listed :After installing and enabling Query Processor, will we be able to Query information or Timeline like in this article for CDW ?https://docs.cloudera.com/cdw-runtime/1.5.1/using-hue/topics/hue-view-query-information.htmlhttps://docs.cloudera.com/cdw-runtime/1.5.1/using-hue/topics/hue-view-query-timeline.htmlThanks for your help.

Hello.In Yarn Queue Manager, there is 4 queues defined :but when we submit queries (using Hue or SAS Software to load data), only the default queue is used.And we can go very high :From what we undestand, Default queue is defined for using 20% of the cluster but can use up to 100%, so 70% is normal.But our cluster is 365 GB / 15 vCores and some applications can use more :7 / 7 / 6 / 5 / 3 vCores in the graph = 28 vCore !How is it possible ?Thanks for your help 

I purchased an exam a few days ago and I could not provide information for billing with which I can refund my exam costs through my company. Can you provide me with some actual help to provide me with a bill? I tried emailing the certification@cloudera.com email a few days (every day from that) but got no reply and it is getting frustrating...

Hi,Input is getting from jolt transforms and data is something like :{"dv_sys_id" : [ "23ec0223138d73006877", "983f853edb2c77007166f", "6a3d0a23138d73006877b3", "805d5339db8dd050f66" ]}output should go in invokehttp processor somthing like: dv_sys_id: 23ec0223138d73006877, 983f853edb2c77007166f, 6a3d0a23138d73006877b3, 805d5339db8dd050f66 remote url : https://<url>${dv_sys_id} How can i achieve this requirement, Pls help me out.    

Hi, I'm struggling with the error below while trying to run Oozie workflow containing ssh action:[FNF], Message [FNF: Required Local file /tmp/oozie-oozi2214367252678167938.dir/ssh/ssh-base.sh not present Can anyone please suggest a potential cause and resolution?  Thank you!

Hello experts,We are facing an issue, where preprod Nifi server is able to read a kafka topic correctly.It is able to read all 3 partitions data.At the same time UAT Nifi server is connecting to same topic with different consumer group id and which is reading only the partition 0 & partition 1 but not partition 2.Screen shot from Lenses abt the same consumer group which is facing issue Any idea what could be the possible reasons?ThanksMahendra

we have HDP Hadoop cluster with two name-node services ( one active name-node and the secondary is the standby name-node )due of unexpected electricity failure , the standby name-node failed to start with the flowing exception , while the active name-node starting successfully2024-02-02 08:47:11,497 INFO common.Storage (Storage.java:tryLock(776)) - Lock on /hadoop/hdfs/namenode/in_use.lock acquired by nodename 36146@master1.delax.com2024-02-02 08:47:11,891 INFO namenode.FSImage (FSImage.java:loadFSImageFile(745)) - Planning to load image: FSImageFile(file=/hadoop/hdfs/namenode/current/fsimage_0000000052670667141, cpktTxId=0000000052670667141)2024-02-02 08:47:11,897 ERROR namenode.FSImage (FSImage.java:loadFSImage(693)) - Failed to load image from FSImageFile(file=/hadoop/hdfs/namenode/current/fsimage_0000000052670667141, cpktTxId=0000000052670667141)java.io.IOException: Premature EOF from inputStreamat org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:204)at org.apache.hadoop.hdfs.se

I have enabled https on nifi registry properties and generated key store and trust store using tls toolkit and added in the properties file. I am able to start nifi registry on port 18443 and host 0.0.0.0 as it showing in logs that jetty server started on https://0.0.0.0:18443/nifi registrymy nifi registry is hosted as aks pod. when I enable ingress for it. It’s returning me as 502 bad request.are there any changes to be done to make my nifi registry run https. Goal : i want to enable 2 way ssl authentication so that nifi could connect to nifi registry.

Hi,recently I have been trying to track my spark application using SparkListener.Now I have a problem:          There is a application, the result of its execution is SUCCESSED on Yarn WebUI,but it's actually a failed application.         And my listener class can not get the error message,the application log only show the Start events for failed jobs and tasks.Some can help me ?  

Where is the location for the Spark driver and executor logs when the Spark job is executed in YARN cluster mode?Does yarn logs -applicationId CMD capture the logs? I can't find log4j-active.log for the driver.