Find answers, ask questions, and share your expertise
Recently active
Hi all, I'm having problems getting the JavaScript I wrote for ExecuteScript to work properly. What I'm trying to do:My flowfiles contain rows of data from a SELECT statement. Each flowfile is searched line by line for non-ASCII characters, and all non-ASCII characters that are found are replaced with empty strings before the data is loaded to a target database. I'd like to grab the value for the table's unique identifier column and write it to a new table, say "nifi_replaced_data" so that I can easily look in the source database and see the row with the non-ASCII character that NiFi replaced in the target database. For example, I might be processing a table with the columns:column_1, column_2, unique_id, column_4And the flowfile, which is tab delimited, might look like:somedata bad__character AA11BB22 someotherdataotherdata no_bad_character CC33DD44 someotherdata The process:Each flowfile is checked in it's entirety for no
Hi, There is a kafka producer converting decimals to bytes using this:import java.math.BigDecimal;import java.math.RoundingMode;import java.nio.ByteBuffer;import org.apache.avro.Conversion;import org.apache.avro.Conversions;import org.apache.avro.LogicalType;import org.apache.avro.LogicalTypes;import org.mapstruct.Mapper;@Mapperpublic class BigDecimalMapper { public ByteBuffer bigDecimalToByteBuffer(BigDecimal bigDecimal) { if (bigDecimal == null) return null; Conversion<BigDecimal> conversion = new Conversions.DecimalConversion(); LogicalType type = LogicalTypes.decimal(15, 4); BigDecimal scaledBigDecimal = bigDecimal.setScale(4, RoundingMode.HALF_UP); // Schema parameter is not used by Conversions so pass as null return conversion.toBytes(scaledBigDecimal, null, type); }We are using the confluent schema registry and this is the schema:{"type": "record","namespace": "event","name": "Event","fields": [{"name": "maxApprovedAmount","type": "bytes","logicalT
I need to create a flow that update rows in my database based on csv file.I have :1- GetFilestatusclient_idActive123Inactive456 2- UpdateAttribute to update the Baobab_file (the schema is already in the AvroSchemaRegistry (collection) 3- SplitRecord to get each line statusclient_idActive123 4- ExecuteSQL : update table_client set current_status = status where id = client_id How can I do this I tried ${status} and \status but no luck Thank you
Cloudera Manager (Trial) 7.4.4Cloudera Runtime 7.1.7 (Parcels) Cloudera SCM log: 2022-07-28 19:19:59,113 ERROR CommandPusher-1:com.cloudera.cmf.command.CommandHelpers: GenerateCredentials - Execution error:java.io.IOException: Encountered error with /opt/cloudera/cm/bin/gen_credentials.sh: Cannot access generated keytab file /var/run/cloudera-scm-server/cmf2117942825672975512.keytabat com.cloudera.cmf.security.GenerateCredentialsCommand.generateKeytab(GenerateCredentialsCommand.java:533)at com.cloudera.cmf.security.GenerateCredentialsCommand.createKeytab(GenerateCredentialsCommand.java:637)at com.cloudera.cmf.security.GenerateCredentialsCommand.generateMissingKeytabs(GenerateCredentialsCommand.java:350)at com.cloudera.cmf.security.GenerateCredentialsCommand.access$300(GenerateCredentialsCommand.java:70)at com.cloudera.cmf.security.GenerateCredentialsCommand$2$1.run(GenerateCredentialsCommand.java:745)at com.cloudera.cmf.security.GenerateCredentialsCommand$2$1.run(
Yarn's application priority can be found in yarn 8088 resource manager's website,How do the priority work in yarn? Now the version of CDH I use is 6.3.2.Hadoop 3.0.0 Hive 2.1.1 I use hive on spark.Can I use the config to manage the application priority? For some reason some hive sqls should have high application priority when those sqls are appending and run ahead of the other. Instead of running the same time and equal share the compute resources. When I set the following config in Hive, it seems do not work well in Yarn.MapReduce"-Dmapreduce.job.priority=xx"Flink"-yD yarn.applicaiton.priority=xx"Spark"spark.yarn.priority=xx" In hive sql."set spark.yarn.priority=10;"It does not work ...
Orginal data:{"data": [{"id": "1234569","Date": "2022-08-22"},{"id": "1234567","Date": "2022-08-22"},{"id": "34567","Date": "2022-08-22"}]}API Response:[ {"Record 2" : "Invalid values for emp info"}, {"Record 3" : "Invalid values for emp info"} ]Where Record 1,Record 2,Record 3 is array of json object index. Here we have three records in orginal data, Record 1 passed successfully and remaining two failed and due to that in response we are getting Record 2 and Record 3i.e. if index of orginal data = value of Record (for example original data's index 1 is passed successfully so it should not merge. In orginal Response Index 2 failed, so it should merge with Record 2 and so on.. ) then final output should look like below:{"data": [{"id": "1234567","Date": "2022-08-22","Record 2" : "Invalid values for emp info"},{"id": "34567","Date": "2022-08-22","Record 3" : "Invalid values for emp info"}]}How can I do that? Please help.
Hi Team, I have an requirement: Input:{"headers": {"comments": "3"},"data": [{"Id": "100000","Date": "2022-09-22"},{"Id": "100001","Date": "2022-09-02"}]} Jolt Spec:[{"operation": "shift","spec": {"data": {"*": {"Id": "[&1].Id","Date": "[&1].Date","$": "[&1].array_Ind"}}}}, {"operation": "modify-default-beta","spec": {"*": {"Record": "=concat('Record ',@(1,array_Ind))"}}}] I am getting below output:[ {"array_Ind" : "0","Id" : "100000","Date" : "2022-09-22","Record" : "Record 0"}, {"array_Ind" : "1","Id" : "100001","Date" : "2022-09-02","Record" : "Record 1"} ] Expected output: [ {"array_Ind" : "1","Id" : "100000","Date" : "2022-09-22","Record" : "Record 1"}, {"array_Ind" : "2","Id" : "100001","Date" : "2022-09-02","Record" : "Record 2"} ] Can you please help
I am getting info messages like the one below while reading and writing files in hdfs.can anyone explain the meaning of this message, will it cause any issues? I have 3 Namenode(open-source Hadoop) in HA. INFO retry.RetryInvocationHandler: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-errorat org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:108)at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2094)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1550)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3342)at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1208)at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(Clien
Hi,I have downloaded version 1.16 which is the latest. Im trying to secure nifi with TLS and LDAP. However I keep getting the followning message : Unknown user with identity 'CN=nifi_admin, OU=NIFI'. Contact the system administrator My Nifi,properties has the following set : nifi.security.user.authorizer=managed-authorizer nifi.security.user.login.identity.provider=ldap-provider If I have set as follows it works and it accepts the cert & authentication: nifi.security.user.authorizer=single-user-authorizer nifi.security.user.login.identity.provider=single-user-provider My Authorizer file has the identity set as follows: <userGroupProvider>...<property name="Initial User Identity 1">CN=nifi_admin, OU=NIFI</property></userGroupProvider> <accessPolicyProvider>...<property name="Initial Admin Identity">CN=nifi_admin, OU=NIFI</property><proper
I am trying to process a CSV file and convert it to a JSON in a specific format. My CSV file is as followsSTARTPI,0010002,25,king,address,phonePE,3.2,company1PE,1.9,company2STOPSTARTPI,0010003,25,prince,address,phonePE,3.2,company1PE,1.9,company2STOPI want help in extracting records from START till STOP. I was able to do the JSON conversion.
I have a problem very similar to this one which has been marked as solved. I have followed the instructions which did not work. I am trying to upload a Nifi XML template that I have previously exported, but am getting a 403.I am running Nifi version 1.17.0 in a docker container.The docker image is the latest, which should be 1.14.0It is run using docker compose.I am using Postman to send the REST requests. The steps I have taken are: Generate an access token using: POST https://localhost:8443/nifi-api/access/token with an x-www.form-urlencoded body containing the username and password set in the .env file. This token is then used as the bearer tokenGet the root process-group id using: GET https://localhost:8443/nifi-api/flow/process-group/root then taking the id from processGroupFlow:idFailing to import an XML template using: POST https://localhost:8443/nifi-api/process-groups/{id}/templates/import where the body is raw XML, containing the previo
Hi,I'm trying to connect the impala on cdp throught JDBC,but it didn't work.It just show: [Cloudera][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: Exec() rpc failed: Remote error: Service unavailable: ExecQueryFInstances request on impala.ControlService from 21.97.66.145:38012 dropped due to backpressure. The service queue contains 0 items out of a maximum of 2147483647; memory consumption is 0., Query: SELECT 1 as `expr_0`.java.sql.SQLException: [Cloudera][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: Exec() rpc failed: Remote error: Service unavailable: ExecQueryFInstances request on impala.ControlService from 21.97.66.145:38012 dropped due to backpressure. The service queue contains 0 items out of a maximum of 2147483647; memory consumption is 0., Query: SELECT 1 as `expr_0`.at com.cloudera.impala.hivecommon.api.HS2Client.pollForOperationCompletion(Unknown Source) ~[?:?]at com.cloudera.impala.hiveco
Hi All,Could you please help to resolve below concern as - We are executing below script and getting ERROR. I have enclosed the same for referene. hive -e "sethive.merge.tezfiles=true;set hive.merge.mapfiles=true;set hive.merge.mapredfiles=true;set tez.queue.name=BIAdv;set hive.execution.engine=tez;set hive.vectorized.execution.enabled =true;set hive.vectorized.execution.reduce.enabled ==true;set hive.exec.dynamic.partition=true;set hive.exec.dynamic.partition.mode=nonstrict;set hive.exec.max.dynamic.partitions.pernode=20000;set hive.exec.max.dynamic.partitions=100000;set hive.merge.size.per.task=134217724;set hive.merge.smallfiles.avgsize=134217724;set tez.grouping.split-count=1;INSERT overwrite TABLE <table-name1> partition (reported_date,last_usage_date)SELECT * from <table-name1> where reported_date='2022-11-16' and last_usage_date='2022-04-10';" ERROR - Log :----------------------------------------------------------------------------------------------ERR
Hi community, I have found an issue on CDH 6 with hive log rotation. In this CDH version, hive log4j changes from rotation by size (configured by cloudera manager) to rotation by size and day (RFA to DRFA). ISSUEThis change results in:Cloudera can not remove old logs (and the FS can be filled), property "Maximum Log File Backups" does not work properly.The rotated log name changed from index to date, If we produced more logs than "Max Log Size", we only have one file per day, so we are overwriting this file with the new produced logs, losing the old logs on same day, and we will have X logs per day, but probably losing logs every day (huge problem when resolving issues). I'm completely sure that is a new issue, because hive log4j configuration changes on this version. SOLUTIONThe solution for me was changing the "appender.DRFA.rollingPolicy.FileNamePattern" property through Cloudera Safety Valve in Hive Server 2, "Logging Advanced Configur
Whether to consider ambari supporting CentOS8?
I am trying to remove oozie from the cluster but when i am trying to do, it its showing dependency of hue. But when I removed the dependency of hue and trying to restart the cluster then I am unable to restart the cluster with following error, Command failed to run because service Hue has an invalid configuration. Review and correct its configuration. First error: Missing required value: Oozie Service
Hi, I have the below data: Record 6|Invalid values for Entry Date.Record 7|Processing failed due to error in record: 6Record 8|Processing failed due to error in record: 6Record 9|Processing failed due to error in record: 6Record 10|Processing failed due to error in record: 6Record 11|Duplicate of record Job id : , data id : , first_name: , last_name: , Date: 2021, Entry Date: 2021.Record 12|Processing failed due to error in record: 11Record 13|Processing failed due to error in record: 11Record 14|Processing failed due to error in record: 11Record 15|Processing failed due to error in record: 11 I want to convert it like: [{"Record 6":"Invalid values for Entry Date."},{"Record 7":"Processing failed due to error in record: 6"},{"Record 8":"Processing failed due to error in record: 6"},{"Record 9":"Processing failed due to error in record: 6"},{"Record 10":"Processing failed due to error in record: 6"},{"Record 11":"Duplicate of record Job id : , data id : ,
Hello,In my 3Masters cluster, one Kudu Master is starting and stopping all the time, this is the Log detail from Cloudera Manager: Time Log Level Source Log Message10:14:41.417 AM WARN cc:288Found duplicates in --master_addresses: the unique set of addresses is Master1:7051, Master2:7051, Master3:705110:15:11.823 AM WARN cc:254Call kudu.consensus.ConsensusService.RequestConsensusVote from 10.157.136.55:55402 (request call id 0) took 4542 ms (4.54 s). Client timeout 1775 ms (1.78 s)10:15:11.823 AM WARN cc:254Call kudu.consensus.ConsensusService.RequestConsensusVote from 10.157.136.37:59796 (request call id 0) took 30215 ms (30.2 s). Client timeout 9654 ms (9.65 s)10:15:11.823 AM WARN cc:260Trace:1112 10:15:07.281146 (+ 0us) service_pool.cc:169] Inserting onto call queue1112 10:15:07.281169 (+ 23us) service_pool.cc:228] Handling call1112 10:15:11.823245 (+4542076us) inbound_call.cc:171] Queueing success responseMetrics: {"spinlock_wait_cycles":384}10:15:11.823 AM WARN cc:260Trace:11
APAI am about to upgrade from cdh to cdp and I have some questions regarding new version of Hive.Until now I used to have hive as etl service because it is more stable but slower than impala. My tables that bi users see are in impala.My questions are:1) Is hive 3 fast enough to compete impala ?2) In case of bi use is it more appropriate to point hive or impala(I read that hive 3 uses cache and makes bi repeated requests faster)?3) In case of kafka flow, is it appropriate to create an acid table in hive 3 and store the fetched data live ?
Hello experts, I am using below api to create and update variable registry variables../nifi-api/process-groups/" + processGroupId + "/variable-registry/update-requests But I am unable to delete any already created variable from specific processor group variable registry using nifi api.Which api I am suppose to use to delete a variable registry variables? Thanks,Mahendra
Where can I see a list of all the processors that will load NiFi's content into memory?
Hello there, Reached out to the main #, by email, and multiple folks on LinkedIn. I can't get a single Cloudera person to follow-up w/ me on a request to get pricing to support NiFi in a commercial environment. Can someone point me to the right person internally? Thank you.
Hey Everyone!I have encoutered this error while trying to create a kudu table using impala-shell.As you see i have created an impala table (yager) stored as textfile and i've tried to create another table (kuduyager) stored as kudu as select * from (yager) then it showed : ERROR: AnalysisException: Cannot analyze Kudu table 'kuduyager': Error determining if Kudu's integration with the Hive Metastore is enabled: cannot complete before timeout: KuduRpc(method=getHiveMetastoreConfig, tablet=null, attempt=95, TimeoutTracker(timeout=180000, elapsed=179150), Trace Summary(175448 ms): Sent(0), Received(0), Delayed(94), MasterRefresh(0), AuthRefresh(0), Truncated: falseDelayed: (UNKNOWN, [ getHiveMetastoreConfig, 94 ]))
I'm looking at the Hbase Web UI and when I click on one of the region servers listed under base stats I see a list of region names. Example: img,0006943d-20150504220458043384375D00000002-00093,1527295748538.7b45a9f6f5584fc50b3152d41a5323a2. My question is, what part of this is the actual region name. My understanding is, img is the table name, 0006943d-20150504220458043384375D00000002-00093 is the beginning rowkey. Is 1527295748538.7b45a9f6f5584fc50b3152d41a5323a2. the actual region name?
I WAS TRYING TO INSTALL CDP THE OTHER DAY. I GOT A WIRED PROBLEM. CM AUTOMATICALLY SKIPPED THE DATABASE SETTING AFTER I ASSIGNED ALL THE ROLES TO THE COMPONENTS. IS THERE ANY STEPS THAT I MISSED? CAN ANYONE HELP PLEASE?
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.