Q&A

I created a managed table called "questiuon".However, a path called "/warehouse/tablespace/external/hive/question.db" was created on the hdfs path.Conversely, creating an external table also creates a path called "/warehouse/tablespace/managed/hive/question.db".What is the reason?

Hello! I am creating a view `view_business` with IMPALA: since the source table `table_a` is a parquet formatted table with complex data types such as array and struct, I have to always run an query option statement "SET PARQUET_ARRAY_RESOLUTION=TWO_LEVEL_THEN_THREE_LEVEL" to load the arrays; otherwise those array columns are showing NULL value. DROP VIEW IF EXISTS discovery_database.view_business;SET PARQUET_ARRAY_RESOLUTION=TWO_LEVEL_THEN_THREE_LEVEL;CREATE VIEW discovery_database.view_business AS ( SELECT * FROM table_a)Once the view is created, if I want to query the view, I still need to run the query option statement first as follows: SET PARQUET_ARRAY_RESOLUTION=TWO_LEVEL_THEN_THREE_LEVEL; SELECT * FROM discovery_database.view_businessHowever, the query option statement above is not supported in ODBC connector when I tried to use the statements above to link the curated table `view_business` as a data source in a power BI report.  My question is: is it possi

Hi Team, We have 2 node managers of each having cpu 16 . we restrict CPU limit of YARN by set yarn.scheduler.maximum-allocation-vcores & yarn.nodemanager.resource.cpu-vcores to 12. But CPU utilization shows 100% utilization. May i know how to troubleshoot this issue. Though we limit cpu to 12 why it is taking 100% of cpu i.e..,16. Immediate suggestions much appreciated. Regards,Hanu 

Hi Team, When we are running Msck Repair Table command it is hanging , can anyone please help me on this issue?? Regards,Hanu

Good afternoon all, and thank you in advance.I have a Apache NIFI solution that is pulling from API a pair of dates, i've grabbed out the relevant bit from the json:"arrival" : "2023-07-24T14:33:00.263Z", "departure" : "2023-07-24T14:21:15.263Z"And i'm successfully getting those values down to attributes via extract attributes.However when i perform a toNumber() on them to convert to miliseconds since EPOCH to determine the time between the dates, i return an empty condition upon review of my flow in the queue.I've gone so far as to arbitarily grab "NOW()" and attempt to toNumber() it but with the same result as trying to modify the above attributes.  incidentally NOW() returns a format like: Mon Jul 24 15:39:21 UTC 2023 - its different that my UTC formatted items, but it also doesn't want to toNumber() - returning again an empty valueShows me attempting to convert the attributes:and this shows the flow in the queue after hitting that updateattrs.  The datetimestamps are

Hello experts, I am working on a project where I have two Gitlab instances available in two different networks i.e., A and B. I need to design a solution using NiFi to read data from certain branches of different Gitlab repos available in Network A and transfer the data in real time to the corresponding branches of the Gitlab instance in Network B. I will greatly appreciate if you could give me some high level idea to develop the solution using NiFi. If relevant, please mention some standard NiFi processor which can help read in this regard. Thank you

Need help if anyone is aware: Platform Ambari (HDP 3.1) with Kerberos Enabled, I am trying to push command after that I did KINIT -V -kt kafka.keytab kafka-principal@realm and got "Authenticated to Kerberos v5":   ./kafka-console-producer.sh --broker-list host.kafka:6667 --topic cleanCsv --producer-property security.protocol=SASL_PLAINTEXT < /tmp/clean_csv_full.csv     Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>&g

HelloI want to set up a local environment of Hive with Iceberg similar to my working environment - CDP public cloud 7.2.15 so I can test locally.The problem is that on my local environment insert into iceberg tables (via Hive) does not work with Tez execution engine (works only with MapReduce). And according to the Iceberg documentation it should not work with Tez for Hive 3. But the thing is that it does work on CDP - I am able to insert data into Iceberg tables with Tez on Hive 3.Is there a way to make it work locally? Thank you Here is my local config:Hadoop: 3.2.1Hive: 3.1.3 (on CDP 3.1.3000.2023.0.14.0-155 is used)Iceberg-hive-runtime: 0.13.1.7.2.15.0-147 (the same version is used on CDP)Tez: 0.9.1.7.2.15.0-147 (the same version is used on CDP)

Hello All,I'm trouble-shooting the following issue with our Cloudera Nutch cluster and would appreciate any help the community can offer:We have two NameNode roles and three JournalNode roles running, however both NameNode roles are failing to start and reporting the error below (IP addresses obfuscated).  This occurred following a restart of the underlying hosts.Any recommendations for a recovery path from this error would be greatly appreciated.  Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [x.x.x.95:8485, x.x.x.86:8485, x.x.x.130:8485], stream=null))org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 1 successful responses:x.x.x.130:8485: null [success]2 exceptions thrown:10.103.28.95:8485: tried to access method com.google.common.collect.Range.<init>(Lcom/google/common/collect/Cut;Lcom/google/common/collect/Cut;)V from class com.google.common.collect.Ranges at

Hi everyone,I'm trying to use NiFis Rest API to start and stop a Processor, but evertime I get the response The desired state is not setHere is my curl-Command which I use:curl -k --header "Authorization: Bearer $token" -i -H 'Content-Type: application/json' -XPUT -d'{ "revision": { "version": 18, "clientId": "01891026-da00-199f-9dc9-ae3a37dce067" }, "status": { "runStatus": "RUNNING" }, "component": { "id": "0189100e-1e29-115c-ec96-b857cd155f4a", "state": "RUNNING" }, "id": "0189100e-1e29-115c-ec96-b857cd155f4a" }' https://localhost:8443/nifi-api/processors/0189100e-1e29-115c-ec96-b857cd155f4a/run-statusHas anyone an idea what's the problem?Best regardsMaik

Hi ,          i've a flow  where there two flows one for xml file and other for corresponding Pdf file. Currently i'm merging those using Mergecontent and generating the zip file out of it.  by making sure minimum entries 2.if input files load are heavy the merge is not matching  corresponding file means (same invoice file should be merged) but sometimes other invoice file merging due to big load from input at a time. i already implemented 10 secs delay  for each flow and merging two files. is there a mechanism to merge files based on common attribute before mergecontent. Thanks in advance

Hello everyone ! for 24 hours i was trying to send data flow from apache nifi to minio s3. i have 2 processors getflowfile and puts3object. i'm putting everything correc like acces key  secret  key  etc... but it show me always the sdame error in nifi that they  don'tv recognize the acces key could you help please

I'm using the NIFI ExtractText Processor and I'm trying to come up with the regular expression to extract values from a JSON String that is in the flowfile-content coming from a response of an API (of course I cannot change how the API responds). JSON Data returned from the REST API is: {  "Result" : {    "SessionID" : null,    "ResultData" : {      "DATAENTITYNAME" : "List",      "CURRENTCURSORPOSITION" : 2,      "DATARECORD" : [ {        "DATAFIELD" : [ {          "FIELDNAME" : "name",          "FIELDLABEL" : "Name",          "DATATYPE" : "VARCHAR",          "FIELDVALUE" : "JSMITH",          "ISKEYFIELD" : true        }, {          "FIELDNAME" : "namedesc",          "FIELDLABEL" : "Name Desc.",

Hello. I am trying to run explorative Pyspark code within a Databricks ApacheSpark environs. I am pretty sure this syntax is correct, but the subject-referenced apache.spark error keeps throwing. Any insights, please? databaseName = "database"desiredColumn = "variable"database = spark.sql(f"show tables in {databaseName} ").collect()display(database) tablenames = []for row in database:     cols = spark.table(row.tableName).columns      listColumns= spark.table(row.tableName).columns      if desiredColumn in listColumns:           tablenames.append(row.tableName)

I need to convert the the request with Apache-nifi, can you help me how to do that? curl --location --request POST 'https://localhost:8080/token' \--header 'Content-Type: application/x-www-form-urlencoded' \--data-urlencode 'client_id=xxxxxxxx' \--data-urlencode 'client_secret=yyyyyyyy' \--data-urlencode 'type=EEEEEEEEEEE' \--data-urlencode 'scope=ZZZZZZ'  Thank for your help!!

Hi Team CM UI Down post OEL , But CM server is running in backend, getting below error in log...Could you please advise WARN avro-servlet-hb-processor-4:com.cloudera.server.cmf.AgentProtocolImpl: Received an optimized heartbeat for a host with ID 'bef21682-baaf-462d-92f8-e43464fc4152' that is not recognized Thankssam

Hello Experts, I need to query delta table from apache nifi.Since Nifi cant connect delta table directly, I am trying it through Spark Thrift Server. I have used 'Hive3ConnectionPool' controller service with 'SelectHive3QL' processor and used below jdbc url and it worked for me.jdbc:hive2://localhost:10002 But I dont know whether using jdbc:hive2//<host>:<port> is right approach or should I use jdbc:spark//<host>:<port> url?Will there be any performance difference between jdbc:hive2 & jdbc:spark jdbc url's with respect to performance or any other imp thing?I tried jdbc:spark//<host>:<port> url with 'DBCPConnectionPool 1.16.3' cntroller service but could not succeed with DatabricksJDBC jar.Any suggestion would be appreciated ThanksMahendra

Hello, I've been trying to use Spark jdbc to connect to a CDP cloud virtual warehouse using the Hive jar (hive-jdbc-3.1.0-SNAPSHOT-standalone.jar) but I have the following errorCaused by: java.sql.SQLFeatureNotSupportedException: Method not supportedat org.apache.hive.jdbc.HivePreparedStatement.addBatch(HivePreparedStatement.java:78)Looking into the jar sources I see that the addBatch method throws the exception by default. Spark needs to use this method to insert the data into hive. Is there any other driver that can be used to overcome this problem.If no, does cloudera will support spark to connect to CDP virtual warehouse. Thanks in advance Best regards,Diego

Hi everyone, We are unable to get the record count into the variable  after Kerberos Authentication implementation and Ranger Installation. we are using following command but it's showing count by printing. Following is we used commandI need to get the count into the variable. anyone please help me! count=$(beeline -u "jdbc:hive2://dev-lisa.realm.com:10000/default;principal=hive/dev-lisa.intranet.slt.com.lk@REALM.COM;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks" -n hadoopuser02 -e 'SELECT count(*) from table_name; ')

How to disable/remove auto TLS certificates and create self-signed certificate in Cloudera version 6.2. The Cloudera version I am using is 6.2 having self-signed certificates that expired a few days back. now the Cloudera manager is not restarting. I want to remove the existing ones and create a new self-signed certificate and apply it to the cluster. can anyone help?

Hello Community, I can assign an attribute's value using a reference to a parameter context value ( e.g. #{variablle_abc} ). But when I attempt to do this same thing within the "Advanced" window in an UpdateAttribute processor, it fails.  I've verified the Process Group Parameter Context is set.   Anyone know how to make this work in the Advanced window of an UpdateAttribute processor?

Hello all!Is there a way to use PGR and input/output ports in the same nifi instance?The aim of that is to reuse the same processor multiple times by only reference to one PGR.If not, are there any other suggestions to achieve the same idea؟

Hello everyone, can someone please take 15 minutes with me to solve some issues in apache nifi ? i just started learning apache nifi and there is some mistakes that i don't know how to solve. i'm reay to prepare a meet for that. if anyone is available it would be so grateful !! Thank you.

Problem Summary---------------------------------------------------Behavior of auth_user table in CDH 6.3.4Problem Description---------------------------------------------------SymptomUser's first name states only one letter in hue schema under user tableQuestion:Is this the default behavior of know issue in CDH 6.3.4?User's first name has only first letter in Hue auth_user table of Hue(configured LDAP)+-----+----------+------------------+-------------+--------------------------+| id | username | first_name | FirstLetter | last_name |+-----+----------+------------------+-------------+--------------------------+| 22 | mycp2cqj | T | T | Khoo || 90 | mycp2fep | W | W | Ng || 101 | cim26384 | M | M | Mohd Baharudin || 243 | mycp1sfs | J | J | Wong || 264 | mycp22yf | G | G | Lee || 272 | mycp2hg4 | W | W | Ab Rahman || 282 | mycp1vk8 | Hui Sue | H | Khoo | <<<<<<<<<<| 284 | phcc1yb9 | R | R | Rochelle Anne C de Matta |

Hello Experts, I am using ConsumeAzureEventHub processor to consume messages in Nifi 1.16.'Concurrent Tasks' field of this processor is disabled so I am not able to speed up the consume processor.What is the reason for this limitation and is there any work around for this ?(we usually run kafka consumer with higher concurrency matching partitions) Thanks,Mahendra