Q&A

Just trying to use a simple Groovy Script for setting TTL on redis keys with a given pattern.  @Grab(group='org.springframework.data', module='spring-data-redis', version='2.6.2')import org.springframework.data.redis.connection.RedisConnectionimport org.springframework.data.redis.core.ScanOptionsdef pool = CTL.redis.getConnection()pool.withCloseable {conn-> def keyPattern = 'raw:extra:*' def scanOptions = ScanOptions.scanOptions().match(keyPattern).build() def cursor = conn.keyCommands().scan(scanOptions) while(cursor.hasNext()) { def key = cursor.next() // Check if the key has a TTL if(conn.keyCommands().pTtl(key) == -1) { // Set a 15 days TTL for the key conn.keyCommands().expire(key, 15 * 24 * 60 * 60) } }}  However this is giving this specific error. I tried using KeyScanOptions, without grabbing the spring-data-redis and a builder. Still getting the same error.   ExecuteGroovyScript[id=6f743677-05b0-1622-bb0b-cbb0e51

I converted non-ha hadoop cluster to ha hadoop cluster using below articlehttps://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.0/fault-tolerance/content/deploying_namenode_ha_cluster.html After conversion, getting "HiveServer2Error: ImpalaRuntimeException: Error making 'add_partitions' RPC to Hive Metastore" exception while trying to upload file What can cause this problem ?

Hi , I have a publish kafka processor where iam having a self loop and flowfile expiration time for that loop is 1 hours , if some file is there for more then 1 hour it will get removed. i want to print log message (e.g. dropping the flow file) after every flow file is removed from queue after getting expired. This is the flow -

Hi there,We've been using Nifi for awhile in our projects, mostly using Kafka and AMQP Client processors for receiving data; but recently a request came up for UDP, specifically DTLS (Datagram transport Layer Security).Looking at the existing UDP Client Processors, it appears that DTLS is not supported; or at least it's not clear for us how to do it. Is that correct?Are there any alternatives then? Maybe creating a custom ListenDTLS processor based on the existing ListenUDP? Could that work?Best regards

Hi there!I'm facing a connection refused problem in my apache nifi instance trying to reach an Elasticsearch instance using a queryElasticsearchHttp processor.The elasticsearch instance is reachable using CURL or in a web browser:But when I try to execute the processor, it fails due to a connection refused errorHere is the configuration I have in the processor: The server is not secured, but anyway I also tried to access with a valid user/password, facing the same error.Here my log:  2023-11-24 09:03:14 2023-11-24 08:03:14,682 ERROR [Timer-Driven Process Thread-4] o.a.n.p.e.QueryElasticsearchHttp QueryElasticsearchHttp[id=018b15d7-f6fe-11ed-b48b-3fd75e47c7d4] Failed to read from Elasticsearch due to Failed to connect to localhost/127.0.0.1:9200, this may indicate an error in configuration (hosts, username/password, etc.). Routing to retry2023-11-24 09:03:14 java.net.ConnectException: Failed to connect to localhost/127.0.0.1:92002023-11-24 09:03:14 at okhttp3.internal

I deployed NiFi and Kafka cluster of cdp public cloud  on the azure.And  I am trying to publish data to kafka(kafka-poc) user nifi(datahub-poc).I am facing "Topic XXX not present in metadata after 0000ms" when I set 'Use transaction=false'  And I am facing "Timeout expired after 0000milliseconds while awaiting  InitProducerId" after "Invaid transition attempted from state INITIALIZING to state ABORTING_TRANSACTION"  error when i set 'Use transaction=true'    Anything should I try to solve this problems?      

I have hdp cluster 2.6.5 (kerberized) on public cloud, and i need to access hdfs from outside (public access) through hdfs cli (not webhdfs)As the external hadoop-client host can't be enrolled through ambari i just downloaded hadoop 2.7.3 package and configured core-site.xml and hdfs-site.xml as bellowcore-silte.xml   <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://datalake-cstest9</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> </configuration>   hdfs-site.xml   <configuration> <property> <name>dfs.nameservices</name> <value>datalake-cstest9</value> </property> <property> <name>dfs.ha.namenodes.datalake-cstest9</name> <value>nn1,nn2</value> </prope

i want to send value of kafka.topic as header i.e (the topic from where i consumed message) to publish topic (another topic) by using publish processor.But iam getting below error in publish processor- 

Hi everybody. When I use https://github.com/cloudera/hue/blob/master/tools/ops/hue_sync_ldap_groups_cron.sh to sync ldap groups with hue on my first cluster it's works fine but when I use on a second cluster I have one error  "LD_LIBRARY_PATH can't be found, if you are using ORACLE for your Hue databasethen it must be set, if not, you can ignoreexport LD_LIBRARY_PATH=/path/to/instantclient". Cause this error,  we can't retreive de ldap_config used later in the scripts. Does someone know how to fix this issue. Thanks 😉

Hello everyone,my team using TEZ, in particular Hive, has noticed that during an insert with a very simple select a single parquet file of 1.5 gb per partition is generated in the output table.To try to remedy the problem, a number of settings were used at the session level but had no effect.Below are the sets used at the session level:SET hive.execution.engine=tez;SET hive.exec.dynamic.partition=true;SET hive.exec.dynamic.partition.mode=nonstrict;SET hive.optimise.sort.dynamic.partition.threshold=0;--SET tez.grouping.max-size=268435456;--SET hive.exec.reducers.bytes.per.reducer=536870912;--SET tez.grouping.split-count=18;SET hive.vectorized.execution.reduce.enabled = true;SET hive.vectorized.execution.reduce.groupby.enabled = true;--SET hive.tez.auto.reducer.parallelism=false;--SET mapred.reduce.tasks=12;--SET hive.tez.partition.size=104857600;--SET hive.tez.partition.num=10;SET hive.parquet.output.block.size=104857600;I would like to ask if there is a way to always have parquet type

CREATE TABLE IF NOT EXISTS emp ( eid int, name String,salary int). . . . . . . . . . . . . . . . . . . . . . .> COMMENT 'Emp Details'. . . . . . . . . . . . . . . . . . . . . . .> ROW FORMAT DELIMITED. . . . . . . . . . . . . . . . . . . . . . .> FIELDS TERMINATED BY '\t'. . . . . . . . . . . . . . . . . . . . . . .> LINES TERMINATED BY '\n'. . . . . . . . . . . . . . . . . . . . . . .> STORED AS TEXTFILE;INFO : Compiling command(queryId=hive_20190905115646_595a00a3-9a4c-4082-a81f-c426fb3e903a): CREATE TABLE IF NOT EXISTS emp ( eid int, name String,salary int)COMMENT 'Emp Details'ROW FORMAT DELIMITEDFIELDS TERMINATED BY '\t'LINES TERMINATED BY '\n'STORED AS TEXTFILEINFO : Semantic Analysis Completed (retrial = false)INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)INFO : Completed compiling command(queryId=hive_20190905115646_595a00a3-9a4c-4082-a81f-c426fb3e903a); Time taken: 0.094 secondsINFO : Executing command(queryId=hive_20190905115646_595a00a

If a have 2 original block and 1 Parity block in HDFS according to the EC policy, then If I lose 2 of this 3 blocks, How HDFS is going to Recover the missing data. Block A  xor  Block B = Parity BlockIf I lose any one of the block, I can see HDFS will be able to create the missing block. But if I lose any 2 block, I don't see how we will recover the missing 2 block  out of 1 block. If help me understand or redirect me to some helpful resources. 

Hi All,  I am upgrading cloudera manager from version 6.1.1 to 7.7.1 and when starting the cloudera-scm-server service I get the following error2023-11-04 15:53:32,946 ERROR main:com.cloudera.enterprise.dbutil.SqlFileRunner: Exception while executing ddl scripts.com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Duplicate column name 'DISPLAY_NAME'What should I do to handle the error?please your advice for me. Thanks

I receive a text/csv file with many lines through an InvokeHttp Processor. My requirement is that I don't want the first 7 lines. What should I do to remove the first 7 lines and keep the remaining the same text/csv format.

Using Nifi 1.23.2 I'm not able to resolve the YamlSlurper class in a ExecuteScript processor (Groovy 3.0.17) nor in the ExecuteGroovyScript processor (Groovy 3.0.18). With a minimal script body likeimport groovy.yaml.YamlSlurperreturnI get the following after clicking Verify Properties:Perform ValidationComponent is invalid: 'GroovyScript' is invalid because org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: Script53405bee.groovy: 1: unable to resolve class groovy.yaml.YamlSlurper @ line 1, column 392. che.nifi.logging.ComponentLog;import gro ^ org.codehaus.groovy.syntax.SyntaxException: unable to resolve class groovy.yaml.YamlSlurper @ line 1, column 392. at org.codehaus.groovy.ast.ClassCodeVisitorSupport.addError(ClassCodeVisitorSupport.java:262) at org.codehaus.groovy.control.ResolveVisitor.visitClass(ResolveVisitor.java:1424) at org.codehaus.groovy.control.ResolveVisitor.startResolving(ResolveVisitor.java:262) at org.codehaus.groovy.control.CompilationU

Hi,  I have CDP private Cloud Base version 7.4.4 running in EC2 instance. I have hive, hive onTez and Tez services running. When I try to execute query in hive I get below issue   Tez session hasn't been created yet. Opening sessionERROR : Failed to execute tez graph. org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configuration   Below is the complete error log INFO  : Compiling command(queryId=hive_20220915080651_6fe3fec1-a2ab-462b-ad3c-e7f4d53c2b46):  insert into customer values('sample1','1')INFO  : Semantic Analysis Completed (retrial = false)INFO  : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:col1, type:string, comment:null), FieldSchema(name:col2, type:string, comment:null)], properties:null)INFO  : Completed compiling command(queryId=hive_20220915080651_6fe3fec1-a2ab-462b-ad3c-e7f4d53c2b46); Time taken: 0.121 secondsI

I have enabled the repository Encryption with protocol version 1, after that  there is below error in Base64EncodeContent  for some of the flowfiles  and other flowfiles gets successfully processed.Can someone please help on this. 2023-08-08 19:26:07,368 ERROR [Timer-Driven Process Thread-5] o.a.n.p.standard.Base64EncodeContent Base64EncodeContent[id=7ba6e988-8bc9-392a-1ae8-f38733a69144] Processing halted: yielding [1 sec]org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find content for FlowFileat org.apache.nifi.controller.repository.StandardProcessSession.handleContentNotFound(StandardProcessSession.java:3701)at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3454)at org.apache.nifi.processors.standard.Base64EncodeContent.onTrigger(Base64EncodeContent.java:135)at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)at org.apache.nifi.controller.StandardProcessorNo

Hi everyone,I have a weird behavior with one of my DBCP Controllers.I'm trying to conncet to a Progress-Database on 10.254.249.24:31510. The controller is configured as shown: When I try to connect the Database (ie. with an "Execute SQL Processor" I get the following error message:Now the weird thing is: The shown Port number in the error mesage is not the Port I configured in the DBCP Connection.I thought it's a firewall thing, but when I check the IP via netcat, everything is fine:I also can succesfully connect to the database with the exact same parameters and *.jar-Files as shown in the first Screenshot when I use a SQL Client like "SQuirreL".Any Ideas?Best regardsMaik 

I want to create a flow In NIFI where i want to consumer data from 3 Kafka topics and produce that data into 3 different Kafka topics . each topic should produce data into unique topic . for example - kafka topics 1  -->  produce to topic Akafka topics 2 --> produce to topic Bkafka topics 3 --> produce to topic Ci want to use only one Consumer processor and one producer processor . Right now i am using 3 produce Kafka processor .Can anyone suggest better approach to do so and in more optimized way. Help me to reduce three publish processors to 1 , but it should be still able to consume from multiple topics and produce to multiple topics but dynamically , like topic1 produces data to topic A only and topic 2 to topic B and so on . i tried using expression language . kafka.topic is the attribute which contains consumer topics. so i added if else condition for different consumer topic but the output was not the producer topic but whole string me

I was trying to use python query to extract data from Hive table and create a data frame. It gave me the following error at cursor.execute line.pyhive.exc.OperationalError: TExecuteStatementResp(status=TStatus(statusCode=3, infoMessages=['*org.apache.hive.service.cli.HiveSQLException:Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask:17:16', 'org.apache.hive.service.cli.operation.Operation:toSQLException:Operation.java:348', 'org.apache.hive.service.cli.operation.SQLOperation:runQuery:SQLOperation.java:228', 'org.apache.hive.service.cli.operation.SQLOperation:runInternal:SQLOperation.java:265', 'org.apache.hive.service.cli.operation.Operation:run:Operation.java:260', 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatementInternal:HiveSessionImpl.java:575', 'org.apache.hive.service.cli.session.HiveSessionImpl:executeStatement:HiveSessionImpl.java:550', 'org.apache.hive.service.cli.CLIService:executeState

I am trying to transform a json using jolt where i am moving the array inside serviceCharacteristic to outside of service {}, eventually i plan to move to respective positions  { "serviceOrderItem": [ { "action": "modify", "id": "0965f63b-db26-4397-b222-190a891838f0", "service": { "href": "serviceInventory/v4/service/adaptiveNetworkSite/1b9a3fae-f50b-4310-887d-60988f8d5645", "id": "adaptiveNetworkSite", "serviceCharacteristic": [ { "name": "modifyPath", "value": [ { "op": "add", "path": "$.supportingService", "value": {} }, { "op": "sub", "path": "$.supportingService", "value": {} } ] } ] } }, { "action": "create", "id": "0965f63b-db26-4397-b222-190a891838f0", "service": { "href": "serviceInventory

Can someone please help me modify the below code?Based on the hours parameter, I am fetching folders from HDFS location, then from each folder, taking the latest csv file(based on getModificationTime) and loading it into the spark dataframe.The below code is working, but there are some issues since it's running in a loop, so after every iteration the data in the dataframe is overwritten by new data, and at the end I am only getting the last file data. Is there any way to collect all the latest files, i.e. in a collection, then execute the dataframe load step, so it can load all the latest files from every folder in the dataframe?Code:val static_path="/user/hdfs/test/partition_date="val hours=3//creating list of each folder. val paths = (0 until hours).map(h => currentTs.minusHours(h)) .map(ts => s"${static_path}${ts.toLocalDate}/hour=${ts.getHour}").toList// iterating every folder from the list for (eachfolder <- paths) { var New_Folder_Path: String = eachfolder.toString

I am trying to fetch a count of total columns for a list of individual tables/views from Impala from the same schema. however i wanted to scan through all the tables from that schema to capture the counts of columns of all the tables in a single query ? i have already performed a similar exercise from Oracle Exa data ,however since i a new to Impala is there a way to capture all the tables in a single query ?     Exadata query for reference   # Oracle Exadata query i used   select owner, table_name as view_name, count(*) as countsfrom dba_tab_cols /*DBA_TABLES_COLUMNS*/where (owner, table_name) in(select owner, view_namefrom dba_views /*DBA_VIEWS*/where 1=1and owner='DESIRED_SCHEMA_NAME')group by owner ,table_nameorder by counts desc;impala /hiveDESCRIBE schemaName.tableName; # Impala /Hive   how to find out ? if i need to run a single query check i was following the below    for multiple tables/view how to find out the total column counts ?  

I have a table and when I am trying to query it through imapala it throws me this error:File 'hdfs:/path/table/1_data.0.parq' has an incompatible Parquet schema for column 'db.table.parameter_11'. Column type: STRING, Parquet schema: optional int64 amount [i:10 d:1 r:0] But when I query it through hive it returns results correctly. How can I fix it?

Hbase service check is failing with below error:ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2829) at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:2085) at org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:706) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:392) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:359) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:339)For usage try 'help "create"'Took 9.2549 secondshbase:025:0> put 'ambarismoketest','row01','family:col01','id620aaa55_date300323'ERROR: Unknown table ambarismoketest!For usage try 'help "put"'Took 0.0754 secondshbase:026:0&gt