Q&A

We are planning to automate our hive schema changes as part of our CI/CD deployment. Currently we are using Gitlab to manage our repo and execute deployment pipeline. One of the tools we're considering is Liquibase. This blog detailed out the way to implement this https://docs.liquibase.com/install/tutorials/impala-hive.html. It explains that in order to use Liquibase with Hive, it needs an impala/hive extensions. This approach seems promising. However I'm concerned that the extension is outdated and may impose security risk in our system since the last update of it was 5 years ago (https://github.com/eselyavka/liquibase-impala).   I was wondering if anyone here have the same use case. What approach/options did you choose to implement the solution? I appreciate any comments or suggestion from this groups.   Thank you!

HiI am using many InvokeHTTP processors for sending APIs to a cloud service (complex flow in NIFI)I need to know how many requests NIFI is sending per minute (as I am paying by CallsPerMinutes). Besides the provenance window, where are the logs/files (on the Linux machine) showing those calls? I need to be able to read them and then I could analyze them. Thank you,Tamir 

I have a flow-file1. I need to generate an authentication token in the next step using a different flow-file2 through InvokeHttp processor which gives the response in another flow-file3. Now, using this token in this flow-file3, in the next step, I need to send flow-file1 to another HTTP endpoint using InvokeHttp. I am not able to figure out this flow. How to keep the flow-file1 and also handle other flow files which are completely for different purpose than the flow-file1? 

Cloudera Documentation - cdswctl and Using VS Code for Remote Editing describe how to create a ssh-endpoint using `cdswctl`. The command describes how to create a new session and associate a ssh-endpoint.   Is it possible to create a ssh-endpoint to an existing session? e.g. first list available sessions and then connect to one of them via its ID?

Hi, An external process creates an error.file on certain events and updates the process log file. The process that I'm creating starts when the error.file is created and need to read the last 10 - 15 lines of the log.file. My problem is that the log.file does not rollover and just grows continually and neither follows a log standard in order to use SysLog processors. How can I get that last 10 - 15 lines of the log file? Something like ( "tail -15 log.file" ). Thanks in advance.

Insufficient Permissions - Untrusted proxy CN=w-nifi-inf-wf02.dev.Company.com, OU=OCIO, O=Company -New 2-Node Cluster-SAML is working fine-Cluster is communicating fine-I have checked the user log and the DN is exactly the same as the authorizers below:<authorizers><userGroupProvider><identifier>file-user-group-provider</identifier><class>org.apache.nifi.authorization.FileUserGroupProvider</class><property name="Users File">./conf/users.xml</property><property name="Legacy Authorized Users File"></property><property name="Initial User Identity 1">user1</property><property name="Initial User Identity 2">CN=w-nifi-inf-wf02.dev.Company.com, OU=OCIO, O=Company</property><property name="Initial User Identity 3">CN=w-nifi-inf-wf03.dev.Company.com, OU=OCIO, O=Company</property></userGroupProvider><accessPolicyProvider><identifier>file-access-policy-provider</identifier>

What does it mean when a manually triggered job is in "Scheduling" job status? What happens behind the scenes during this Scheduling phase? Recently I have been noticing my job getting stuck in this phase for more than 10 minutes or so which is concerning.

I have a basic kafka 2.13_3.10 cluster with one broker, consumer and producer.I was testing if the "log.flush.interval.messages = 10"worked properly. It is supposed to write the kafka messages of the topics after 10 messages, but where are they saved in the disk by default??And, is there any way to specify the directory where you want data to be flushed overriding the default configuration?Thanks in advance.

I am using cloudera hdfs sink connector with parquetWriter.  "com.cloudera.dim.kafka.connect.hdfs.HdfsSinkConnector"   It seems to flush the data in kafka topic every 1 minute.  The number of partition of one of my kafka topics is 64.  64 (kafka partition) * 60 (minutes per hour) * 24 (hours per day) = 92160 files are sinked everyday in one directory. So, I created a job to delete files n days old. But this job is too slow because of the number of files of the directory where the parquet files are sinked by hdfs sink connector.   I have question about cloudera hdfs sink connector 1. Is it possible to sink the files in daily partition directory? ex) /blah/{topicName}/{yyyyMMdd} 2. Is there a way to change flush duration instead of every minutes?

I have a JSON document that I want to index into Hbase.   { "field1" : { "nestedfield" : "value1" } "field2" : "value2"}   In this case I want `nestedfield` to be the one used as a row identifier field, but I can't figure out how to specify this to the `PutHBaseJson` processor. I have tried multiple ways : - field1/nestedfiled - field1['nestedfiled'] - field1.nestedfield - $.filed1.nestedfiled None of them are working, the processor is saying that "the Row ID was null". Using `field2` for this purpose is working fwiw.

Hi Folks,After adding new columns in Kudu tables from impala, I had some issues computing stats for this table. Everything works fine until the Compute Stats command;The alter table command  ALTER TABLE myschema.mytable ADD COLUMNS (mycolumn TIMESTAMP DEFAULT now());  The issue when Compute Stats command. ImpalaRuntimeException: Error making 'updateTableColumnStatistics' RPC to Hive Metastore: CAUSED BY: MetaException: Column mycolumn doesn't exist in table mytable in database myschema  Someone can help me, please?

We are creating a PoC in a Cliente using CPD public on Azure.  Always when I try to create the Streams Messaging Light Duty.  The result I get is Failed: Orchestrator component went failed in 30 min(s), message: Status: 406 Not Acceptable Response: 406 Not Acceptable

Hello Team,   Spark job through Oozie is failing with the below exception in the Prod cluster. Note that the same job passes in the lower clusters (f.e. Dev)   22/03/24 05:05:22 INFO spark.SparkContext: Invoking stop() from shutdown hook22/03/24 05:05:22 ERROR scheduler.AsyncEventQueue: Listener EventLoggingListener threw an exceptionjava.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:477) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:627) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:583) at org.apache.hadoop.fs.FSDataOutputStream.hflush(FSDataOutputStream.java:134) at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:145) at org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:145) at scala.Option.foreach(Option.scala:257)

HI, Please find the below hbase-site.xml file in my hdp cluster.env - AWS logs message:2022-03-05 10:09:58.242 [dw-8144 - POST /blitz-reader/metric-reader/v3.0] WARN c.a.b.p.c.m.r.MetricDataHBaseReadProcessor - org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=3, exceptions:Sat Mar 05 10:09:50 UTC 2022, RpcRetryingCaller{globalStartTime=1646474990201, pause=1000, retries=3}, org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ip-10-145-250-154.us-west-2.compute.internal/10.145.250.154:60020Sat Mar 05 10:09:53 UTC 2022, RpcRetryingCaller{globalStartTime=1646474990201, pause=1000, retries=3}, org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: ip-10-145-250-154.us-west-2.compute.internal/10.145.250.154:60020Sat Mar 05 10:09:58 UTC 2022, RpcRetryingCaller{globalStartTime=1646474990201, pause=1000, retries=3}, java.net.ConnectException: Connection refusedjava.util.concurrent.Ex

Error Exception in thread "main" org.apache.spark.SparkException: No main class set in JAR; please specify one with --class at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657) at org.apache.spark.deploy.SparkSubmitArguments.validateSubmitArguments(SparkSubmitArguments.scala:266) at org.apache.spark.deploy.SparkSubmitArguments.validateArguments(SparkSubmitArguments.scala:251) at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:120) at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.<init>(SparkSubmit.scala:913) at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:913) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935) at org.apache.spark.deploy.Spark

Hi, is there a way to login to NiFi registry via openid (ej google)?I'm thinking of how I can secure the instance without a domain like in NiFiThanks!

Hi All,   I am trying all means to find out a latest Cloudera sandbox for practicing purpose on mac m1 , yes i know and experienced many issues due to M1 not supporting most of the softwares with out any tweaks.   I have found few options   Option 1 - Virtual box -> not compatible for M1 Option 2 - Parallels -> not open source  Option 3 - VM ware fusion  -> not open source  Option 4 - UTM - again some issues keyboard no sync  Option 5 - Docker desktop -> This options works with tweaks   I have picked up Docker desktop ,since there is a software available for M1 , i have then explored the options  to find a suitable cloudera/hortonwork sandbox images   1. Docker - Hortonworks sandbox 3.0.1 -> failed Hortonworks 3.0.1 -> I have tried with some tweaks but not able to configure in spite of spending a lot of time fixing the issues ,no luck    Issue/error - Failed to get D-Bus connection: No such file or direc

I'm a newbie, I want to calculate month on month percentage change in price change in my table. Im using hive and using query: select year(stock_date),month(stock_date),sum(stock_price) over (partition by year(stock_date),month(stock_date) ORDER BY month(stock_date)), PERCENT_RANK() OVER(ORDER BY month(stock_date) DESC) AS per from table4; it's generating, It's showing July 2019 at the top and goes descending thru the numbers. how do I get the months in correct order and the percentage change with it? can anyone help me understand what am I doing wrong? 

Is there any sort of formula or how did you came up with this value for users's processes? is it a random value? what can I check within my cluster in order to get a proper value for me? My current configuration is as of below. # Cloudera modifications* - nofile 32768* - nproc 32768

F0303 09:59:04.650674 32117 catalogd-main.cc:87] Couldn't open transport for hostname:11423 (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server not found in Kerberos database)) I have used a customized service name for impala, can we do that? if not how can we achieve it?principal - impala_<some text>@hostname@Domain

Hi there... I was wondering if there is a way, that can report the failure of read eventhub failure while trying to read the eventhub. Suggestions would be appreciated.Thanks in advance 🙂

Hello All, We are connecting to CM console via F5 Load Balancer (Reverse Proxy). We are trying to enable X-Forwarded-For (XFF) in HTTP header, to get actual source client IP address from the CM Audits log. How can I enable CM server to read XFF HTTP header for the source client IP address instead of reading source IP address from layer 3? Thank you in advance.

From the docs at https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/impala_string_functions.html#string_functions__regexp_extract "This example shows how a pattern string starting with .*? matches the shortest possible portion of the source string, returning the rightmost set of lowercase letters."    select regexp_extract('AbcdBCdefGHI','.*?([[:lower:]]+)',1); returns defEvery other Regex impl I've worked with would return bcd I can't make sense of the docs either - "shortest possible string.. returns the rightmost.." - the shortest possible string in a "search from the left" operation returns the leftmost, not the rightmost      

I have an issue with my JOLT transformation in NIFI:My INPUT:{"customerNumber": "40890036","customerOrderNumber": "TPACSAY4","orderType": "ES","orderEntryFlag": "createOrder","parts": [{"partNumberOrdered": "A0029864271","quantityOrdered": "2.00","customerInfo": "Cust information","chassisNo": "WDB45678901234567","additionalEntry": "ADDITIONAL ENTRY"},{"partNumberOrdered": "N000000001388","quantityOrdered": "10.00","customerInfo": "Cust information","chassisNo": "WDB45678901234567","additionalEntry": "ADDITIONAL ENTRY"}]} My current JOLT is below:[{"operation": "modify-default-beta","spec": {"CustNo1": "${CustNO}","Customer": "${CustNO}","ShippingSiteId": "0","SalesOrderOriginCode": "CFE","ShippingWarehouseId": "5559","LineDiscount": "${MCRHoldCode}","SALESORDERNUMBER": "","customerReqNo": "","OrderType": "${MCRHoldCode}","DELIVERYTERMSCODE": "${DELIVERYTERMSCODE}","ShippingDate_VO_Header": "=substring((${ShippingDate_VO}), 0, 20)","ShippingDate_EM_Header": "=substring((${Shipping

I'm trying to connect to dynamo db from nifi processor component "PutDynamoDB"I'm putting my secret key in the field Access Key IDsecret key in Secret Access Keyand the path of the credentials in the format: [default]aws_access_key_id=<key id>aws_secret_access_key=<access key> At the moment of make the validation i get this message: Perform Validation Component is invalid: '' validated against 'Access Key' is invalid because Cannot set both Credentials File and Secret Key/Access Key  Does someone have an idea about this? I haven't foun anything related