Q&A

Hi,I'm using the Cloudera CDP 7.1.7 Private Cloud Base version and would like to have a confirmation if there is a possibility to be able to set, in the capacity scheduler of Yarn, time rules as I did previously in the old CDH.It would be interesting to be able to use that functionality also in CDP, but in the Cloudera documentation I've searched on the web there's no mention of it.Could you give me some hints if this feature is still present in CDP? Thanks 

  Groovy Script Crashing: Unexpected Character '(' in groovy script in apache nifi?

I have this data and I need to split VisitList's content into different flowfile one by one.What should I write in JsonPath Expression so that data got split, I tried $.* but then it is considering this given data into one file only while I want to split on the basis of VisitList.Data[ { "employer": "98765", "loc_id": "312", "topId": "Management", "VisitList": [ { "S1": "HR", "S2": "Accountant" }, { "S1": "Manager", "S2": "Sr. Manager" } ] }] I want to split Data into files like this 1st flowfile[ { "employer": "98765", "loc_id": "312", "topId": "Management", "VisitList": [ { "S1": "HR", "S2": "Accountant" } ] }] 2nd flowfile[ { "employer": "98765", "loc_id": "312", "topId": "Management", "VisitList": [ { "S1": "Manager", "S2": "Sr. Manager" } ] }] I  tried putting $.* in JsonPath Expres

EnvironmentOS: Windowsmain batch Script: sample_Import_Load.bat@Former Member off && pushd "%~dp0" && setlocalcall "C:\Program Files\CARIS\BASE Editor\5.5\system\caris_env.bat"call "C:\sample_Testing\1_XYZ\import-points-loop.bat"set pythonpath=c:\Program Files\CARIS\BASE Editor\5.5\python\3.5python "C:\sample_Testing\2_Coverages_Upload\coverage_loader_sample.py" -L dba/sql@localhost -D sample_TEST -o surfac -t .csar -f "C:\sample_Testing\2_Coverages_Upload" -m "C:\sample_Testing\2_Coverages_Upload\Sample_Meta.csv" -vpopd && endlocalpauseNested batch script: Import-points-loop.bat@Former Member off && pushd "%~dp0" && setlocalcall "C:\Program Files\CARIS\BASE Editor\5.5\system\caris_env.bat"FOR %%f in (*.xyz) do (carisbatch --run ImportPoints --input-format ASCII --input-crs EPSG:3395 --output-crs EPSG:32659 --info-file "C:\sample_Testing\1_XYZ\Example.info" "%%~nf.xyz" "C:\sample_Testing\2_Coverages_Upload\%%~nf.csar")popd && endlocalREM

Hi,This is a reproducible, simple issue where the performance is surprisingly bad. It is a follow-up to the case under this link, where initially a stoackoverflow issue occurred. The script below ran for 26 hours in over 8 cores at full calculation as is seen in the hardware statistics.  Of course the object is "quite large"- but in similar operations with such an object it does not take so long. The generated size was 42.5GB in the 8 parquet files on HDFS.  Here is the code:from pyspark.sql import SparkSessionfrom pyspark.sql.functions import randss = SparkSession.builder.appName("test_replication") \.config("spark.kryoserializer.buffer.max.mb", "2047") \.config('spark.sql.execution.arrow.pyspark.enabled', "true") \.config("spark.driver.maxResultSize", "16G") \.config("spark.driver.memory", "4G") \.config("spark.executor.memory", "16G") \.config("spark.dynamicAllocation.maxExecutors","8") \.config("spark.executor.instances", "2") \.config("spark.executor.cores

Hello,I am running since approx 4 weeks into unsolvable OOM issues, using CDSW, yarn cluster, pyspark 2.4.7 and python 3.6. It seems that I am making generally something fundamentally wrong. The following is a reproducible code example to mimic one issue in its simplest form: A large spark dataframe is created and then it is supposed to be written in a parquet file on HDFS. The HDFS-disk has more than enough space. Below I took partitioning out. However, coalesce as also partitionBy lead to same issues. Generating only a small file works (to rule out other possible issues). The reproducible code is the following. from pyspark.sql import SparkSessionfrom pyspark.sql.functions import randss = SparkSession.builder.appName("test_replication") \.config("spark.kryoserializer.buffer.max.mb", "2047") \.config('spark.sql.execution.arrow.pyspark.enabled', "true") \.config("spark.driver.maxResultSize", "2G") \.config("spark.driver.memory", "3G") \.config("spark.executor.memory

Hi,I'm using a PutDatabaseRecord with UPSERT statement for a PostgreSQL database, but the ON CONFLICT keys are not being quoted event with the setting "Quote Column Identifiers" set to true.ERROR:Batch entry 0 INSERT INTO "eventstorage"."2062d999b1f26bcb32c3f39d22bd34"(..., "FDZ_UUID", ...) VALUES (...) ON CONFLICT (FDZUUID) DO UPDATE SET (...) = (..., EXCLUDED."FDZ_UUID", ...) was aborted: ERROR: column "fdzuuid" does not existI've seen an JIRA issue regarding this behavior and it is as Resolved: https://issues.apache.org/jira/browse/NIFI-8043.I'm using nifi 1.17.0@MattWhosorry for pinging you directly, but you were the assignee of the jira ticket and I've seen you commenting other similar posts. Thanks

Hi,I am using CDP PC trial version with AWS  for learning CDP PC and am trying to register an environment on CDP PC and repeatedly getting the below error. I am following https://www.cloudera.com/campaign/videos/cdp-setup.html (pdf booklet) Datalake creation failed. 1. Data Access Role (arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role) is not set up correctly. Please follow the official documentation on required policies for Data Access Role. Missing policies (chunked): arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn:aws:s3:::cdp-my-bucket/hive_replica_functions_dir/* arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn:aws:s3:::cdp-my-bucket/ranger/audit arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn:aws:s3:::cdp-my-bucket/hive_replica_functions_dir arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn

I want to fetch the files based on wildcard(*xyz*.txt) in nifi and if files are not available then it will keep on looping again check whether the files are exist or not. Can you please suggest the solution in NIFI. Files are present in server. 

now I am use Ambari2.7.5 and HDP3.1.5 to install my cluster, but I get some vulnerabilities. so where can I download the source code or jars of HDP3.1.5. please help me.

Hello, Recently we have noticed several ETL jobs receiving the following error: errorCode:10280, errorMessage:Error while compiling statement: FAILED: LockException [Error 10280]: Error communicating with the metastore), Query: create temporary external table `databasenamehere`.`tablenamehere_dmx_20230607074828418241275540` (TableAttributesHere TBLPROPERTIES ('transactional'='false').', vendor contact information: support.precisely.comConnect : (TBLCOUNT) 179 records were committed to target database "jdbc:hive2://servernamehere:10000;AuthMech=1;SSL=1;KrbHostFQDN=servernamehere;KrbRealm=domainhere;KrbServiceName=hive" table "tablenamehere", outof which 0 records were loaded successfully and 179 records were rejected We are on Hive 3.1.3000.7.1.7.1000-141 and the metastore is MySQL version mysql Ver 14.14 Distrib 5.7.37, for Linux (x86_64). We have also noticed recently that queries are taking much longer than normal to do basic tasks, and wonder if there is somethin

I am on a kerberized CM 7.7.1 private cloud and after watching the "Cloudera Search Training" tutorial, I'd like to create a collection with SolrCloud.I created a configuration and was able to upload it successfully to Zookeeper: kinit <my_username>solrctl instancedir --create testcollection <my_local_path> This looks alright to me: solrctl instancedir --list testcollectionlocalFSTemplatemanagedTemplatemanagedTemplateSecureschemalessTemplateschemalessTemplateSecure Calling: solrctl collection --create testcollection -s 1 Gives the error:Error: A call to SolrCloud WEB APIs failed:(there's nothing more) I suspect it is because no secure connection is being used. I therefore try it with: solrctl --solr https://<node with solr>:8995/solr collection --create testcollection -s 1 However, Solr cannot find the configuration:content-security-policy: default-src 'none'; base-uri 'none'; connect-src 'self'; form-action 'self'; fo

Unable to up the Nifi Cluster Mode with OIDC IntegrationHave Existing Cert-Manager v1.5.3 (Let's Encrypt)Currently using OIDC with Cluster ModeNifi Image version 1.19.1Pull nifi release v1.1.3Kong gateway/proxy version 3.2My value.yaml (Note: I only put necessary info value not a full value.yaml) replicaCount: 2properties: externalSecure: false isNode: true httpsPort: 8443 webProxyHost: nifi-cluster.domain.name:443 service: type: ClusterIP httpsPort: 8443 # nodePort: 30236 annotations: kubernetes.io/ingress.class: kong-nginx konghq.com/protocol: "https"ingress: enabled: true className: kong-nginx annotations: cert-manager.io/cluster-issuer: letsencrypt-xxx konghq.com/strip-path: "false" konghq.com/plugins: session konghq.com/session-affinity-cookie: my_session_cookie konghq.com/protocols: "http,https" # Enable both HTTP and HTTPS konghq.com/session-affinity: "true" # Enable session affinity tls: - secretName: nifi-certificate hosts

Hello,I'm trying to read messages from a IBM MQ. for that I have set up a JMSConnectionFactoryProvider Service, and configured a ConsumeJMS proccesor.The JMSConnectionFactoryProvider is connecting to the IBM Queue successfully, and even the ConsumeJMS outputs a file from the queue when I run it, but the file's content is corrupted. it is suppose to be readable, but it comes out as gibbirish.I noticed that the output flow file has 2 interesting header properties:JMS_IBM_Character_Set: IBM424JMS_IBM_Encoding: 785 Maybe it has something to do with the corrupted output? I tried to specify IBM424 in the 'Chracter Set' property on ConsumeJMS proccessor, but it had no effect.Thanks a lot!  

Hi,kindly need your support in below, I have Text file, I have to convert it to byteArray and send it through REST API, I tried many ways to convert the file to byteArray, last one is:- ExecuteScript with below code:import base64import codecswith codecs.open('/path/to/input/file.txt', 'r', encoding='utf-8') as file:file_content = file.read()file_bytes = file_content.encode('utf-8')encoded_bytes = base64.b64encode(file_bytes)encoded_string = encoded_bytes.decode('utf-8')flowFile = session.putAttribute(flowFile, 'fileContent', encoded_string)session.transfer(flowFile, REL_SUCCESS) but still get error: utf-8' codec can't decode byte 0xff in position 0: unexpected code byte in <script> at line number 6 REST API Config:HTTP Method: POSTAPI URL:.....Request Content-Type: application/octet-streamauthentication User: username authentication password: password we applied it in VB.net as below:Function CallApi(APIURL As String,filepath As String) As StringDim req A

Hello NiFi team, I'm new to NiFi, I'm trying to get data from Elasticsearch using QueryElasticsearchHttp however I have a self_signed certificate I'm not sure how to use that if you can give some examples or just some basic steps I can try to set that up on my end. Thank you in advance for the help. 

Hello, I am trying to run a mapreduce job and output it on a folder. Is there any way to change the default filename. My default filename is part-r-00000.snappy.parquet and I want to add a date to this. Is this possible. Is there a parameter to change this, thank you. 

Hi  I currently have to extract folder from other computer via NiFi SFTP. I been finding for solution and notice that the NiFi processor (GetFile, GetSFTP) only allow to extract files instead of folder. Is there any nifi processor that is able to extract the folder? Example: Computer 1directory: C:\User\localFolder Computer 2 Using GetSFTP processor (remote path: C:\User) to extract the folder Thanks 

Hello, How do we fetch the number of columns each table has in each DB from HMS DB? I have an urgent need to do a comparison for the number of columns each table has in each DB between 2 clusters. Doing that manually is practically impossible. Kindly help Thankssnm1523

Hi, How do we change the InputFormat and OutputFormat of an existing Hive table? I tried below, but got an error:alter table <table name> SET InputFormat "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"; Error:Cannot recognize input near 'SET' 'InputFormat' 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' Kindly suggest

Hello,Cluster has been Kerberized (LDAP / AD / Kerberos) and I have errors when I try to start the cluster. Zookeeper service start with following error: 2022-07-01 13:24:14,341 ERROR org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer: Failed to authenticate using SASLjavax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decryptAP-REQ - RC4 with HMAC)] at jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199) at org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer.authenticate(SaslQuorumAuthServer.java:99) at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:563) at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:487) at org.apache.zookeeper.server.quorum.

My manager forces me to find a way to install and use Spark 3 on CDH 6.x cluster. Is there any change?When I did some research, I found out that only CDP 7. supports Spark 3, and CDH 6.x only support Spark 2. But my manager said that you don't need to install Spark through Cloudera Manager, you can install Spark 3 separately (by downloading a tar from the internet or sth like that) and then find a way to make that Spark service connect with Cloudera service like Hive, HDFS,... (by copying the hive-site, hdfs-site,... to spark conf folder maybe?) So does anyone have any experience with this? My manager is insane!!!! 

Very new to NiFi and I am trying to create a daily email with throughput totals from the previous day.   I was getting nowhere figuring out how to capture and report on NiFi processor throughput so I figured I could start with a GetHDFSFileInfo process that could look at the various HDFS tables and report on the previous day's directory size.   Similar to "hdfs dfs -du -h /warehouse/tablespace/managed/hive/table1" and then grabbing the partition directory size from the previous day.   I could script it easy enough but I would like to keep everything in NiFi so I don't have to worry about scripts and cron jobs. When I try to use GetHDFSFileInfo and do not recurse directories, I can get a list of all the partition directories but the length value is always 0.   If I enable recurse, then I get every file (and they have the length value), Just curious if there was a way to have GetHDFSFileInfo provide partition level directory disk usage.&n

Hi All, I am running Nifi on a single Development server. (NAME="Debian GNU/Linux" VERSION_ID="10")I start the service using command line  -  bin/nifi.sh startDuring the initial phases as i experiment with RAM, CPU and disk i need to restart machine sometimes.When machine restarted I use above command to start Nifi and I go to UI and start the jobs. Is there a way that1. Nifi process starts automatically when ever system reboots or restarts ? 2. Nifi jobs also start automatically without me running from UI ? can I use startup scripts for this ? I want to get ready with production mindset as i soon will be launching.I want to understand how it is handled in realtime production servers..Please suggest.   

Hello, Is there a way get the hdfs locations of all tables in CDP 7.1.7 SP1? I tried running a for loop against a list of tables on below command, however, it will take ages to get the list as I am looking forward to 66k+ tables. hive -e “desc formatted <table>;” | grep Location Thought of to get from HMS DB directly, however, I can only see DB location URI, but nothing specific to tables. Kindly help as this is urgent. Thankssnm1523