Q&A

Hi,I have downloaded version 1.16 which is the latest. Im trying to secure nifi with TLS and LDAP. However I keep getting the followning message : Unknown user with identity 'CN=nifi_admin, OU=NIFI'. Contact the system administrator My Nifi,properties has the following set :    nifi.security.user.authorizer=managed-authorizer    nifi.security.user.login.identity.provider=ldap-provider If I have set as follows it works and it accepts the cert & authentication:     nifi.security.user.authorizer=single-user-authorizer   nifi.security.user.login.identity.provider=single-user-provider My Authorizer file has the identity set as follows: <userGroupProvider>...<property name="Initial User Identity 1">CN=nifi_admin, OU=NIFI</property></userGroupProvider> <accessPolicyProvider>...<property name="Initial Admin Identity">CN=nifi_admin, OU=NIFI</property><proper

I am trying to process a CSV file and convert it to a JSON in a specific format. My CSV file is as followsSTARTPI,0010002,25,king,address,phonePE,3.2,company1PE,1.9,company2STOPSTARTPI,0010003,25,prince,address,phonePE,3.2,company1PE,1.9,company2STOPI want help in extracting records from START till STOP. I was able to do the JSON conversion.

I have a problem very similar to this one which has been marked as solved. I have followed the instructions which did not work. I am trying to upload a Nifi XML template that I have previously exported, but am getting a 403.I am running Nifi version 1.17.0 in a docker container.The docker image is the latest, which should be 1.14.0It is run using docker compose.I am using Postman to send the REST requests. The steps I have taken are: Generate an access token using: POST https://localhost:8443/nifi-api/access/token with an x-www.form-urlencoded body containing the username and password set in the .env file. This token is then used as the bearer tokenGet the root process-group id using: GET https://localhost:8443/nifi-api/flow/process-group/root then taking the id from processGroupFlow:idFailing to import an XML template using: POST https://localhost:8443/nifi-api/process-groups/{id}/templates/import where the body is raw XML, containing the previo

Hi，I'm trying to connect the impala on cdp throught JDBC,but it didn't work.It just show: [Cloudera][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: Exec() rpc failed: Remote error: Service unavailable: ExecQueryFInstances request on impala.ControlService from 21.97.66.145:38012 dropped due to backpressure. The service queue contains 0 items out of a maximum of 2147483647; memory consumption is 0., Query: SELECT 1 as `expr_0`.java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: Exec() rpc failed: Remote error: Service unavailable: ExecQueryFInstances request on impala.ControlService from 21.97.66.145:38012 dropped due to backpressure. The service queue contains 0 items out of a maximum of 2147483647; memory consumption is 0., Query: SELECT 1 as `expr_0`.at com.cloudera.impala.hivecommon.api.HS2Client.pollForOperationCompletion(Unknown Source) ~[?:?]at com.cloudera.impala.hiveco

Hi All,Could you please help to resolve below concern as - We are executing below script and getting ERROR. I have enclosed the same for referene. hive -e "sethive.merge.tezfiles=true;set hive.merge.mapfiles=true;set hive.merge.mapredfiles=true;set tez.queue.name=BIAdv;set hive.execution.engine=tez;set hive.vectorized.execution.enabled =true;set hive.vectorized.execution.reduce.enabled ==true;set hive.exec.dynamic.partition=true;set hive.exec.dynamic.partition.mode=nonstrict;set hive.exec.max.dynamic.partitions.pernode=20000;set hive.exec.max.dynamic.partitions=100000;set hive.merge.size.per.task=134217724;set hive.merge.smallfiles.avgsize=134217724;set tez.grouping.split-count=1;INSERT overwrite TABLE <table-name1> partition (reported_date,last_usage_date)SELECT * from <table-name1> where reported_date='2022-11-16' and last_usage_date='2022-04-10';" ERROR - Log :----------------------------------------------------------------------------------------------ERR

Hi community,  I have found an issue on CDH 6 with hive log rotation. In this CDH version, hive log4j changes from rotation by size (configured by cloudera manager) to rotation by size and day (RFA to DRFA). ISSUEThis change results in:Cloudera can not remove old logs (and the FS can be filled), property "Maximum Log File Backups" does not work properly.The rotated log name changed from index to date, If we produced more logs than "Max Log Size", we only have one file per day, so we are overwriting this file with the new produced logs, losing the old logs on same day, and we will have X logs per day, but probably losing logs every day (huge problem when resolving issues). I'm completely sure that is a new issue, because hive log4j configuration changes on this version. SOLUTIONThe solution for me was changing the "appender.DRFA.rollingPolicy.FileNamePattern" property through Cloudera Safety Valve in Hive Server 2,  "Logging Advanced Configur

Whether to consider ambari supporting CentOS8?

I am trying to remove oozie from the cluster but when i am trying to do, it its showing dependency of hue. But when I removed the dependency of hue and trying to restart the cluster then I  am unable to restart the cluster with following error, Command failed to run because service Hue has an invalid configuration. Review and correct its configuration. First error: Missing required value: Oozie Service 

Hi, I have the below data: Record 6|Invalid values for Entry Date.Record 7|Processing failed due to error in record: 6Record 8|Processing failed due to error in record: 6Record 9|Processing failed due to error in record: 6Record 10|Processing failed due to error in record: 6Record 11|Duplicate of record Job id  : , data id : , first_name: , last_name: , Date: 2021, Entry Date: 2021.Record 12|Processing failed due to error in record: 11Record 13|Processing failed due to error in record: 11Record 14|Processing failed due to error in record: 11Record 15|Processing failed due to error in record: 11 I want to convert it like: [{"Record 6":"Invalid values for Entry Date."},{"Record 7":"Processing failed due to error in record: 6"},{"Record 8":"Processing failed due to error in record: 6"},{"Record 9":"Processing failed due to error in record: 6"},{"Record 10":"Processing failed due to error in record: 6"},{"Record 11":"Duplicate of record Job id  : , data id : ,

Hello,In my 3Masters cluster, one Kudu Master is starting and stopping all the time, this is the Log detail from Cloudera Manager: Time Log Level Source Log Message10:14:41.417 AM WARN cc:288Found duplicates in --master_addresses: the unique set of addresses is Master1:7051, Master2:7051, Master3:705110:15:11.823 AM WARN cc:254Call kudu.consensus.ConsensusService.RequestConsensusVote from 10.157.136.55:55402 (request call id 0) took 4542 ms (4.54 s). Client timeout 1775 ms (1.78 s)10:15:11.823 AM WARN cc:254Call kudu.consensus.ConsensusService.RequestConsensusVote from 10.157.136.37:59796 (request call id 0) took 30215 ms (30.2 s). Client timeout 9654 ms (9.65 s)10:15:11.823 AM WARN cc:260Trace:1112 10:15:07.281146 (+ 0us) service_pool.cc:169] Inserting onto call queue1112 10:15:07.281169 (+ 23us) service_pool.cc:228] Handling call1112 10:15:11.823245 (+4542076us) inbound_call.cc:171] Queueing success responseMetrics: {"spinlock_wait_cycles":384}10:15:11.823 AM WARN cc:260Trace:11

APAI am about to upgrade from cdh to cdp and I have some questions regarding new version of Hive.Until now I used to have hive as etl service because it is more stable but slower than impala. My tables that bi users see are in impala.My questions are:1) Is hive 3 fast enough to compete impala ?2) In case of bi use is it more appropriate to point hive or impala(I read that hive 3 uses cache and makes bi repeated requests faster)?3) In case of kafka flow, is it appropriate to create an acid table in hive 3 and store the fetched data live ?

Hello experts, I am using below api to create and update variable registry variables../nifi-api/process-groups/" + processGroupId + "/variable-registry/update-requests But I am unable to delete any already created variable from specific processor group variable registry using nifi api.Which api I am suppose to use to delete a variable registry variables? Thanks,Mahendra 

Where can I see a list of all the processors that will load NiFi's content into memory?

Hello there, Reached out to the main #, by email, and multiple folks on LinkedIn. I can't get a single Cloudera person to follow-up w/ me on a request to get pricing to support NiFi in a commercial environment. Can someone point me to the right person internally? Thank you.

Hey Everyone!I have encoutered this error while trying to create a kudu table using impala-shell.As you see i have created an impala table (yager) stored as textfile and i've tried to create another table (kuduyager) stored as kudu as select * from (yager) then it showed : ERROR: AnalysisException: Cannot analyze Kudu table 'kuduyager': Error determining if Kudu's integration with the Hive Metastore is enabled: cannot complete before timeout: KuduRpc(method=getHiveMetastoreConfig, tablet=null, attempt=95, TimeoutTracker(timeout=180000, elapsed=179150), Trace Summary(175448 ms): Sent(0), Received(0), Delayed(94), MasterRefresh(0), AuthRefresh(0), Truncated: falseDelayed: (UNKNOWN, [ getHiveMetastoreConfig, 94 ])) 

I'm looking at the Hbase Web UI and when I click on one of the region servers listed under base stats I see a list of region names.   Example:   img,0006943d-20150504220458043384375D00000002-00093,1527295748538.7b45a9f6f5584fc50b3152d41a5323a2.   My question is, what part of this is the actual region name.  My understanding is, img is the table name, 0006943d-20150504220458043384375D00000002-00093 is the beginning rowkey.  Is 1527295748538.7b45a9f6f5584fc50b3152d41a5323a2. the actual region name? 

I WAS TRYING TO INSTALL CDP THE OTHER DAY. I GOT A WIRED PROBLEM. CM AUTOMATICALLY  SKIPPED THE DATABASE SETTING AFTER I ASSIGNED ALL THE ROLES TO THE COMPONENTS. IS THERE ANY STEPS THAT I MISSED? CAN ANYONE HELP PLEASE?

Hi Team, We have 1 table which has more than 1 lakh partitions. Support team moved data from HDFS to other path and try to do MSCK but it is not working even drop table also. Getting below errors.. FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask While drooping partitions also getting same error after 10min can one help me on this.. 

Hive Fails to install kerberos authentication hive. But after uninstall kerberos , caused by an error occurs during the hive startup Unable to read HiveServer2 configs from Zookeeper

Hi, I have a flow that receives JSON arrays as input. I would like to validate each of these jsons' schema, however using the ValidateRecord processor doesn't quite seem to do the job. I need to validate things such as certain fields being Enum values, having a max/min length, and ensuring required fields are present (sometimes inside of optional nested jsons). It seems an avro schema does not allow some of these functionalities and as such the Record processors can't quite validate my data as I need it. I would love to hear if anyone has had a similar use case and what they did to solve it. I am considering using the ScriptedValidateRecord processor, however I would prefer to avoid that and might instead opt for using EvaluateJsonPath to extract all the fields I want to validate and then using RouteOnAttribute with the expression language to filter out bad records. If there is a more appropriate way to validate records like this then I'm all ears. Thanks I'm advance!

I have a NIFI workflow running, how do I start and stop the workflow from the CLI. Is it possible to only start/stop one workflow from multiple workflows?

when I try to read parquet file from Azure datalake container from databricks, I am getting spark exemption. Below is my query     import pyarrow.parquet as pqfrom pyspark.sql.functions import *from datetime import datetimedata = spark.read.parquet(f"/mnt/data/country/abb/countrydata.parquet")     org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 14.0 failed 4 times, most recent failure: Lost task 0.3 in stage 14.0 (TID 35) (10.135.39.71 executor 0): org.apache.spark.SparkException: Exception thrown in awaitResult:   what does this mean? What I need to do for this?

Hi Team, We have 3 DN's and Data Directories mount point size is around 1TB on each Node , total Data Directories size was 3TB. I deleted 500GB data from HDFS but space got release from only DN3 Data Directory not from other DN's. Please advise.. 

hello cloudera community, we are trying to use pyspark to interact with kudu We were able to successfully execute the command below: import kudu but when running the command below: from kudu.client import Partitioning returns the error ImportError: libkudu_client.so.0: cannot open shared object file: No such file or directory ps: we are using cdp, cdh 7.1.7 and cm 7.6.1

Hello everybody,I'm facing with a very weird behaviour with an UPSERT instruction: the operation doesn't have effect on columns that was previously stored with null value; I have 'tab1' defined as create table IF NOT EXISTS tab1(id VARCHAR not null, eventdate TIMESTAMP not null, category VARCHAR, history VARBINARY); UPSERT into tab1(id,eventdate,category,history) values ('id1',to_timestamp('2022-10-20 00:00:00.0','test',null); now I just try to execute the following statement:UPSERT into tab1(id,eventdate,category,history) values ('id1',to_timestamp('2022-10-20 00:00:00.0','edited test','ABC'); this is the resulting table:SELECT * FROM tab1+-----+-----------------------+-------------+---------+| id  |        eventdate      |  category   | history |+-----+-----------------------+-------------+---------+| id1 | 2022-10-20 00:00:00.0 | edited test |         |+-----+--------------------