Q&A

Hello!I'm using CM api v19I'm trying to get only critical alerts which I'm looking at right now (RED ones).Tried https://cloudera.github.io/cm_api/apidocs/v19/path__events.html  /api/v19/events?query=category==HEALTH_EVENT;severity=CRITICAL   The output consists of too many not relevant events. What should I do to get only critical events displayed on the dashboard?Thank you.

Hello,a question regarding application design in NiFi. We have a REST resource which provides data changes since a unix timestamp, which is passed inside a json structure to be sent as http POST.The major question is how and where to keep the state of the last unix timestamp and how this state is accessible to operation.We should be able to reset the timestamp to any value to repeat calls. Also we should be able set increment, which es added to the timestmap in the next call. Similary, we have a related issue ina use case, which access a database. We generate a query in GenerateTableFetch and set the property "Maximum-value Columns".Here the state is maintened by the processor, but it pretty much unclear how to set the state to specific stating value. To clear the state is obviously not appropriate. Best regardsJaro

Hi, Company I'm working for is paying subscription to Cloudera and I need to use Cloudera support on their behalf.  When I created my cloudera account with company mail I don't see any subscription or asset. My colleague did the same and his account was paired automatically.  What's proper way how to link my account to company I'm working for ? Can I trigger this manually ?  Thank you for any help. 

Hello, Apache druid cannot be added as a separate service in CDP but I can see its binaries under the CDH-7.1.7 parcel folder. Is Druid available and supported in CDP? 

Hello all, I am trying to connect to Kafka from my Flink flow. I am using Flink version 1.14.3 and Kafka connector version: flink-connector-kafka-0.11_2.11:jar:1.11.6 (latest version in Maven repo). I am using FlinkKafkaConsumer011 in my code to create Kafka consumer to consume my kafka topics. However, when running Flink and deploying my flow, I see the below error thrown in logs: java.lang.NoSuchMethodError: org.apache.flink.api.common.functions.RuntimeContext.getMetricGroup()Lorg/apache/flink/metrics/MetricGroup;at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:762)at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:67)at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:323) Appreciate any help on this. - Prabu.

upgradation

Hi All. I'm relatively new to NiFi and would like a little help in designing a data flow that looks like this: SELECT from a SQL data sourcesplit/convert to JSONoptionally transform JSONmerge to 500 record batchesPOST to target via REST API (requires session token from API auth POST)I know how to create this flow in NiFi. The part I'm having trouble with is including a step that gets a session token via the REST API auth endpoint. I can get the session token using InvokeHTTP no problem. I just don't know how to add the session token to the flow file(s) containing the data to POST to the API. Thanks for your help.

Hi All, I want to append cluster name in the 'Cloudera Alert mail subject'. I am not getting any configuration for same. Please suggest how to do that. Current Subject: [Cloudera Alert] 4 Alerts since 2:31:41 AM Want to change as below (assuming cluster name as ABC):[Cloudera Alert - ABC ] 4 Alerts since 2:31:41 AM Regards,Banshi.

在CDP集群hive的外部仓库目录权限默认为1777，而内部仓库目录为0770，这样会出现一个问题，hive的外部仓库目录任何用户都可以写入数据问题。是设计就是这样还是起塔问题？

Hi guys,   we added solr srever in solrCloud，when I want to achieve to HA in solrCloud. but it is failed in transfer. when one is stop , the other one is living and used by Atlas.     

Hi,We have noticed on ambari that after a maintenance that both hbase masters are on "Standby" mode:We tried without success to restart each HBase master to see if they were going to change into "Active". Could you please help use

Hello all,Please bear with me if I ask a simple/basic question. I am new to NiFi and appreciate your patience and support.I am setting up secure NiFi in 2 clusters. I am using NiFi 1.15.3 version in my setup. My cluster 1 has nodes: node1, node2, node3 and cluster 2 has nodes: node4, node5, node6.I have created certificates in my organization by setting node1 as primary node and other nodes as alternative nodes. I use the same certificates in both clusters since I added all the nodes to the certificate when creating it.My cluster 1 starts and I can able to login to web ui with admin user without any issues. However, my cluster 2 starts without any issues but when I try to login, it throws:Insufficient permission. Untrusted proxy CN=node1, O=xxxx, L=yyyy, ST=zzzz, C=USand in logs:o.a.n.w.s.NiFiAuthenticationFilter Authentication Failed GET https://node4:8443/nifi-api/flow/current-user [Untrusted proxy CN=node1, O=xxxx, L=yyyy, ST=zzzz, C=US]As a part of troubleshooting, I added the node

I can't connect to db2/as400I have tried with db2jcc4.jar, db2jcc.jar, jt400.jarjt400.jarErrorWARN [Timer-Driven Process Thread-4] o.a.n.controller.tasks.ConnectableTask Administratively Yielding ExecuteSQL[id=3732162d-1fac-1df4-52b9-e9190b60af34] due to uncaught Exception: java.lang.AbstractMethodError: Receiver class com.ibm.as400.access.AS400JDBCConnection does not define or inherit an implementation of the resolved method 'abstract boolean isValid(int)' of interface java.sql.Connection.

Hello, A simple create table as select fails for some tables in my original_db and I don't know why.The query I use is below, and the error HIVE gives out is really long, but the main theme is "Vertex Failed", "Error while processing row", "No enum constant" - as you can see below. However, when ran without "STORED AS PARQUET" option, the query runs fine.Any ideas what I can try to maybe create the tables as compressed parquet? Thanks   beeline -e "SET parquet.compression=SNAPPY;SET hive.exec.compress.output=true;SET parquet.output.codec=snappy;set hive.tez.container.size = 8192;DROP table if exists DB.table;CREATE table DB.tableSTORED AS parquetAsSELECT * FROM ORIGINAL_DB.table;" ERROR : Status: FailedERROR : Vertex failed, vertexName=Map 1, vertexId=vertex_1640013186854_27946_1_00, diagnostics=[Task failed, taskId=task_1640013186854_27946_1_00_000004, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1640

Hello,   I did enough google before coming to this forum.  I have table  Employee with following column Names (there are 40 columns)   Employee_ID,  Employee_Type, Employee_Tenure   I want to use these column names in CASE Statement as part of HIVE Query to assign value.    E.g.   Case when Column Name = 'Employee_Type' then 1 else 0 when Column Name = 'Employee_Tenure' then 2 else 0 end as Rank   I know I can "show columns in Employee" to get all column names. But I don't know how to refer output of this query in CASE statement   I also try to save results of Query into variable ( I could be wrong doing this )   set hivevar:col_names=(show columns in employee) select '${hivevar:col_names}'    but I get below error : Error while processing statement: Variable substitution depth is deeper than 40 for expression    Any help appreciated. If my question is too naive or stupid - I apologize. 

How can I transform a string to an integer in NiFi? I tried JoltTransform, but the string is transformed to a negative (??) integer. In the example, below, the timestamp_start original string value 1677720088000 is transformed to -1612124736. What am I missing in my Jolt specification? Any help is appreciated. Is there an alternative to the JoltTransform - another processor? Looks like this is a known issue: https://github.com/bazaarvoice/jolt/issues/749 Original JSON Transformed JSON  Jolt Specification [ { "operation": "modify-overwrite-beta", "spec": { "timestamp_start": "=toInteger" } }, { "operation": "default", "spec": {} }]   

I have recently upgraded Private Cloud Base components (CM 7.5.4, CDH 7.1.7). I am unable to locate a Spark 3 or Spark3 LIVY service after following the CDS 3.2, installation steps. ( https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cds-3/topics/spark-install-spark-3-parcel.html ) I am able to download, distribute, and activate the Spark 3 parcel but there are no options to add it as a service. I did add the `csd` jars to the directories with no luck. logs indicate conflicts with the csd (I tried with both the page listed .49 and also the unlisted .1001). Removing the csd jars removes the warning but the service is still not available.  "...2022-02-11 12:26:19,206 WARN main:com.cloudera.csd.components.CsdRegistryImpl: Found multiple CSDs with non-homogeneous versions for service type SPARK3_ON_YARN. Latest version is 7.5.42022-02-11 12:26:19,208 INFO main:com.cloudera.csd.components.CsdRegistryImpl: Ignoring CSD: bundle name [SPARK3_ON_YARN-3.2.0.3.2.7170

I started a apache phoenix query server, phonenix version is apache-phoenix-5.0.0-HBase-2.0-binAll other functions work well in the cmd line of sqlline-thin.py. But when I tried to get the primary key of a table, its just listed nothing. 0: jdbc:phoenix:thin:url=http://localhost:876> !primarykeys FN2.TESTYL2+------------+--------------+-------------+--------------+----------+----------+| TABLE_CAT | TABLE_SCHEM | TABLE_NAME | COLUMN_NAME | KEY_SEQ | PK_NAME |+------------+--------------+-------------+--------------+----------+----------++------------+--------------+-------------+--------------+----------+----------+ when I tried in phoenix cmd line sqlline.py, it works well.do you know why the thin phoenix query server cannot find the primary key? thanksYang

I just sow a video with a rather weird claim. It stated that if nifi fell, the offset of the kafkarecordreader will be lost. This however, seems to me a bit weird, since the offset should be kept by kafka to the specific consumer group ID, AFAIK. Am I m

I have a WF which runs daily on Real and Beta.I found that there is a problem running my WF in beta today.The WF in real takes around 90-120 minutes.But in beta, the WF takes about 8 hours.I don’t understand the 8 hours run time in beta because the real data is much larger than the beta data.Can you please tell me how to reduce WF execution time in beta?

I've encountered this error log at cloudera HDFS sink connector. (I erased only hostname) org.apache.kafka.connect.errors.ConnectException: org.apache.kafka.connect.errors.ConnectException: java.io.IOException: DestHost:destPort Failed on local exception: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client is configured to only allow secure connections.  As this log says, the destination HDFS uses SIMPLE authentication. (hadoop.security.authentication=simple)Is there any way to connect cloudera HDFS sink connector to HDFS with simple authentication?

I've already got Atlas working and different hooks configured in other services in my Kerberized CDP Private Cloud Base 7.1.6. I still don't have as many entities as I would expect to see. I tried to sync the Hive Metastore but can't pass an authentication error and cannot find which log to look to get more information.I've the superuser ticket generated with kinit and when I run the import-hive.sh script in the server where the Hive Metastore is installed, the following message is printed:[root@<metastore-server> ~]# /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/atlas/hook-bin/import-hive.sh -d inventariosUsing Hive configuration directory [/etc/hive/conf]/etc/hive/conf:/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels

Hi Is there a way to get the number of the files in the input then assign the count value to an attribute and assign a number value to each file I have files to merge using MergeContent so I should assign fragment.index for each file and fragment.count as total count of files to merge.

HiI want to merge 6 CSV files into 1I useListHDFS >> FechHDFS >> UpdateAttribute >> MergeContent >> QueryRecord >> ... ListHDFS >> FechHDFS >> UpdateAttribute is repeated as the number of files to merge ( 6 times)because I shoud to give for each file the fragment.index parameter and an allias ( used later for the join query in QueryRecord ) The UpdateAttribute for one of the files:  Is there a way to avoid multiple processors to get the files ListHDFS >> FechHDFS >>  UpdateAttribute How to reduce is into one ListHDFS >> FechHDFS >> UpdateAttribute and give a different fragment.index for each different file which shloud be between 0 and 6 (max number of files) ? I tried NextInt() to attribute a new fragment.index value but it is incremental, not suitable for multiple executions. Thanks in advance.

Hi, can someone help me to find solution for execute plsql oracle procedure after I finish putin record in my table in database. I have flow like thisand when all job is done I have to execute procedure with max and min ID from this flow... Thanks in advance