Solved

Hello,I'm trying to edit Yarn Queue Manager queues and it seems i'm not able to edit them, there is not the option to edit them and there are no information in the doc about it...it's supposed to have this options panelBut this is not my case.the only information found it is to set the queue manager service in yarn, that's all...Any other ideas? 

I have a problem when sending JSON data to Elastic from Nifi - Nifi puts [ square brackets ] around a JSON array, and so Elastic doesn't recognise the data. The current fix is to use a SplitJSON processor, which removes the square brackets, but this will become unworkable when the number of flow files gets a bit big in number. Is there a way of creating an array of JSON that is recognised by Elastic that does not contain square brackets?I am also using an older version of Nifi - Ver 1.12.1

Hi,Would need some information on how to connect to Kubernetes pods of DW, DE DF ML on AWS. Do we need to make some configurations etc, what are the ways that we can connect? Any docs /guide would be helpful. Thanks Wert

I am setting up a stand-alone CDP cluster. However, getting errors when doing the agents installation steps. I have used this command but it still error:  

We have a cluster running OS opensuse 12sp5. I wonder if CDH 5.14.2 supports this OS version. I‘v seen the CDH release notes, but still want to make sure.

I've been stuck for so long, I don't know what to do. Help❤️ I have installed Hive using ambari,When I try to insert a piece of data into the table, I get a Tez error,Here's what I've done:hive services and Tez services are normal. The user name I use is root, and encryption-free login has been configured on all nodes.root has its own /user/root directory and /tmp/hive/root on hdfsroot is a hive administrator and has all permissions on related databases and tablesI use yarn application-list to verify that no tasks are runningI use yarn node-list to confirm that all nodes are RUNNINGI use hdfs dfsadmin-report to confirm that all nodes are Normal and in normal stateI use hadoop fsck confirmed /warehouse/tablespace/managed/hive/root.db/test is healthyMy question is detailed below:I have a list like this0: jdbc:hive2://hdp0:2181,hdp1:2181,hdp2:2181> desc test;INFO : Compiling command(queryId=hive_20230614090425_2f598338-0603-4ba5-b4a2-8b77796b6d1b): desc testINFO : Semantic Analy

Hi to all,hope someone can point me on the right direction.I have something like that:[ { "reference" : "PB", "date" : "20230613", "barcode" : "C337287V28490010", "shipment_number" : ""}, { "reference" : "PB", "date" : "20230613", "barcode" : "C337287V28490011", "shipment_number" : ""}, { "reference" : "PB", "date" : "20230613", "barcode" : "C337287V28490012", "shipment_number" : ""} ]I need to update only a specific record with a met condition.Speaking on a SQL language it should be something like that:UPDATE FLOWFILE SET shipment_number = '${Shipment_Number_Attribute} ' WHERE barcode = '${Barcode_Attribute}'Unfortunately this statement in QueryRecord doesn't work, maybe UPDATE not supported?Moreover is there any way to achieve the same with UpdateRecord? Many thanks for any help.Appreciate. 

I have a csv file which a new record comes and appends every 15 minutes. I need to extract only the latest record from the csv file and save as a JSON

I have a flow that fetches all files from a given directory—they could be gzip, zip or csv, with the gzip and zip files holding a single csv file. I then route on MIME type, decompress the gzip files, unpack the zip files, and then bring what are now ALL csv files back together. This is working. I then want to create a zip archive of all of these csv files, and MergeContent seemed like a good candidate (outside of using ExecuteStreamCommand to run zip from the OS). But no matter what I do, the results are inconsistent:The first time I ran it with default properties in MergeContent (and setting Merge Format to ZIP), it created a single zip file — I thought I was done!But on the second time, it created six zip files, so I realized I wasn't done, and started playing with propertiesChanged Maximum number of Bins to 1 - created five zip filesChanged Correlation Attribute Name to absolute.path, which is the same for all flow files - created four zip filesChanged Maximum Group Size to 50

Nifi UI is showing blank screen. Processors are not visible. This is happening for all users/Administrators

Hi,I'm using the Cloudera CDP 7.1.7 Private Cloud Base version and would like to have a confirmation if there is a possibility to be able to set, in the capacity scheduler of Yarn, time rules as I did previously in the old CDH.It would be interesting to be able to use that functionality also in CDP, but in the Cloudera documentation I've searched on the web there's no mention of it.Could you give me some hints if this feature is still present in CDP? Thanks 

Hi,This is a reproducible, simple issue where the performance is surprisingly bad. It is a follow-up to the case under this link, where initially a stoackoverflow issue occurred. The script below ran for 26 hours in over 8 cores at full calculation as is seen in the hardware statistics.  Of course the object is "quite large"- but in similar operations with such an object it does not take so long. The generated size was 42.5GB in the 8 parquet files on HDFS.  Here is the code:from pyspark.sql import SparkSessionfrom pyspark.sql.functions import randss = SparkSession.builder.appName("test_replication") \.config("spark.kryoserializer.buffer.max.mb", "2047") \.config('spark.sql.execution.arrow.pyspark.enabled', "true") \.config("spark.driver.maxResultSize", "16G") \.config("spark.driver.memory", "4G") \.config("spark.executor.memory", "16G") \.config("spark.dynamicAllocation.maxExecutors","8") \.config("spark.executor.instances", "2") \.config("spark.executor.cores

Hello,I am running since approx 4 weeks into unsolvable OOM issues, using CDSW, yarn cluster, pyspark 2.4.7 and python 3.6. It seems that I am making generally something fundamentally wrong. The following is a reproducible code example to mimic one issue in its simplest form: A large spark dataframe is created and then it is supposed to be written in a parquet file on HDFS. The HDFS-disk has more than enough space. Below I took partitioning out. However, coalesce as also partitionBy lead to same issues. Generating only a small file works (to rule out other possible issues). The reproducible code is the following. from pyspark.sql import SparkSessionfrom pyspark.sql.functions import randss = SparkSession.builder.appName("test_replication") \.config("spark.kryoserializer.buffer.max.mb", "2047") \.config('spark.sql.execution.arrow.pyspark.enabled', "true") \.config("spark.driver.maxResultSize", "2G") \.config("spark.driver.memory", "3G") \.config("spark.executor.memory

Hi,I am using CDP PC trial version with AWS  for learning CDP PC and am trying to register an environment on CDP PC and repeatedly getting the below error. I am following https://www.cloudera.com/campaign/videos/cdp-setup.html (pdf booklet) Datalake creation failed. 1. Data Access Role (arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role) is not set up correctly. Please follow the official documentation on required policies for Data Access Role. Missing policies (chunked): arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn:aws:s3:::cdp-my-bucket/hive_replica_functions_dir/* arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn:aws:s3:::cdp-my-bucket/ranger/audit arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn:aws:s3:::cdp-my-bucket/hive_replica_functions_dir arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn

now I am use Ambari2.7.5 and HDP3.1.5 to install my cluster, but I get some vulnerabilities. so where can I download the source code or jars of HDP3.1.5. please help me.

I am on a kerberized CM 7.7.1 private cloud and after watching the "Cloudera Search Training" tutorial, I'd like to create a collection with SolrCloud.I created a configuration and was able to upload it successfully to Zookeeper: kinit <my_username>solrctl instancedir --create testcollection <my_local_path> This looks alright to me: solrctl instancedir --list testcollectionlocalFSTemplatemanagedTemplatemanagedTemplateSecureschemalessTemplateschemalessTemplateSecure Calling: solrctl collection --create testcollection -s 1 Gives the error:Error: A call to SolrCloud WEB APIs failed:(there's nothing more) I suspect it is because no secure connection is being used. I therefore try it with: solrctl --solr https://<node with solr>:8995/solr collection --create testcollection -s 1 However, Solr cannot find the configuration:content-security-policy: default-src 'none'; base-uri 'none'; connect-src 'self'; form-action 'self'; fo

Hello NiFi team, I'm new to NiFi, I'm trying to get data from Elasticsearch using QueryElasticsearchHttp however I have a self_signed certificate I'm not sure how to use that if you can give some examples or just some basic steps I can try to set that up on my end. Thank you in advance for the help. 

Hi  I currently have to extract folder from other computer via NiFi SFTP. I been finding for solution and notice that the NiFi processor (GetFile, GetSFTP) only allow to extract files instead of folder. Is there any nifi processor that is able to extract the folder? Example: Computer 1directory: C:\User\localFolder Computer 2 Using GetSFTP processor (remote path: C:\User) to extract the folder Thanks 

Hello, How do we fetch the number of columns each table has in each DB from HMS DB? I have an urgent need to do a comparison for the number of columns each table has in each DB between 2 clusters. Doing that manually is practically impossible. Kindly help Thankssnm1523

Hi, How do we change the InputFormat and OutputFormat of an existing Hive table? I tried below, but got an error:alter table <table name> SET InputFormat "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"; Error:Cannot recognize input near 'SET' 'InputFormat' 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' Kindly suggest

Hello,Cluster has been Kerberized (LDAP / AD / Kerberos) and I have errors when I try to start the cluster. Zookeeper service start with following error: 2022-07-01 13:24:14,341 ERROR org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer: Failed to authenticate using SASLjavax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decryptAP-REQ - RC4 with HMAC)] at jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:199) at org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer.authenticate(SaslQuorumAuthServer.java:99) at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:563) at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:487) at org.apache.zookeeper.server.quorum.

My manager forces me to find a way to install and use Spark 3 on CDH 6.x cluster. Is there any change?When I did some research, I found out that only CDP 7. supports Spark 3, and CDH 6.x only support Spark 2. But my manager said that you don't need to install Spark through Cloudera Manager, you can install Spark 3 separately (by downloading a tar from the internet or sth like that) and then find a way to make that Spark service connect with Cloudera service like Hive, HDFS,... (by copying the hive-site, hdfs-site,... to spark conf folder maybe?) So does anyone have any experience with this? My manager is insane!!!! 

Very new to NiFi and I am trying to create a daily email with throughput totals from the previous day.   I was getting nowhere figuring out how to capture and report on NiFi processor throughput so I figured I could start with a GetHDFSFileInfo process that could look at the various HDFS tables and report on the previous day's directory size.   Similar to "hdfs dfs -du -h /warehouse/tablespace/managed/hive/table1" and then grabbing the partition directory size from the previous day.   I could script it easy enough but I would like to keep everything in NiFi so I don't have to worry about scripts and cron jobs. When I try to use GetHDFSFileInfo and do not recurse directories, I can get a list of all the partition directories but the length value is always 0.   If I enable recurse, then I get every file (and they have the length value), Just curious if there was a way to have GetHDFSFileInfo provide partition level directory disk usage.&n

Hi All, I am running Nifi on a single Development server. (NAME="Debian GNU/Linux" VERSION_ID="10")I start the service using command line  -  bin/nifi.sh startDuring the initial phases as i experiment with RAM, CPU and disk i need to restart machine sometimes.When machine restarted I use above command to start Nifi and I go to UI and start the jobs. Is there a way that1. Nifi process starts automatically when ever system reboots or restarts ? 2. Nifi jobs also start automatically without me running from UI ? can I use startup scripts for this ? I want to get ready with production mindset as i soon will be launching.I want to understand how it is handled in realtime production servers..Please suggest.   

Hello, Is there a way get the hdfs locations of all tables in CDP 7.1.7 SP1? I tried running a for loop against a list of tables on below command, however, it will take ages to get the list as I am looking forward to 66k+ tables. hive -e “desc formatted <table>;” | grep Location Thought of to get from HMS DB directly, however, I can only see DB location URI, but nothing specific to tables. Kindly help as this is urgent. Thankssnm1523