Q&A

Hi Team, I am trying to re-build my materialized view (ALTER MATERIALIZED VIEW <db>.<table> REBUILD;). However, I got an error message saying that there is another process re-building the MV.Error message:SQL Error [40000] [42000]: Error while compiling statement: FAILED: SemanticException org.apache.hadoop.hive.ql.parse.SemanticException: Another process is rebuilding the materialized viewI tried to check if there is any write lock on the MV or the table, however there is none. I also tried to re-build the MV, but then I got the same error again when I tried to rebuild.Any idea why this is happening or how to check the "another process"?

Hello, I am in process of setting up a CDP 7.1.7 cluster.At the moment, CM 7.6.1 is installed and integrated to AD on LDAPS protocol (Had to select authentication type as LDAP for the integration to work even though we have AD being used).Next step is I have added few basic services i.e. HDFS, YARN and Zookeeper and now I am enabling Kerberos.At the step of Generating credentials it fails with attached screenshot.However, in the same window I noticed that CM is trying to connect to AD on LDAP protocol on port 389. Ideally it should be connecting via LDAPS on 636 as we have TLS also configured and enabled. Not sure if this is even relevant. From where does CM gets the LDAP URL? I tried to understand gen_credentials_ad.sh script at /opt/cloudera/cm/bin, however, did not completely interpret. Please help as this is bit urgent. Thankssnm1523

I have a use case that requires malware analysis. The analysis and feature extraction is taking place on a separate designated server/network. A feature file is generated on that server/network. I will need to schedule the analysis to run on the most recent malware and generate feature files. The challenge is how do I send the feature file to Nifi from the server/network that has access to malware repository?

Hi,I have my project deployed in CDSW and am eager to know if there is an option to mask the code being printed/displayed on the output console log? or at least in the email that gets distributed on the job status after the run?

Hi all, I'm very new to NiFi and trying to find ways how to convert json array to be stored in SQL database. I currently have this input json  { "QueryHandle" : "#qb9ec959946924620a6f46013d745fe03", "Rows" : [ { "Values" : [ "A1", "Test Data A1" ] }, { "Values" : [ "A2", "Test Data A2" ] }, { "Values" : [ "A3", "Test Data A3" ] }, { "Values" : [ "A4", "Test Data A4" ] } ], "TotalRowCount" : 197, "DataTypes" : { "Values" : [ { "ColumnName" : "Value", "Type" : "String" }, { "ColumnName" : "Description", "Type" : "String" } ] }}  Here's the end result I would like to have before I can use PutSQL processor [ { "VALUES" : "A1", "DESCRIPTION" : "Test Data A1"}, { "VALUES" : "A2", "DESCRIPTION" : "Test Data A2"}, { "VALUES" : "A3", "DESCRIPTION" : "Test Data A3",}, { "VALUES" : "A4", "DESCRIPTION" : "Test Data A4",}  I have tried to use jolttransform, but couldn't be able to remove the array or t

hiI want to know if its possible to use the LOAD DATA INPATH to load parquet files into an external iceberg table? right now I am getting the following error: Error while compiling statement: FAILED: SemanticException [Error 10101]: A non-native table cannot be used as target for LOAD    does hive support LOAD DATA INPATH with Iceberg tables?

I have been trying to send an email via NiFi using PutEmail processor with Gmail/Outlook both and getting same exception in both of the configuration as   PutEmail[id=3611dcb8-0183-1000-8a1d-df8f12f92438] Failed to send email for StandardFlowFileRecord[uuid=017c80a8-fad0-493b-a427-fa72b9a530c7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1663081396353-45, container=default, section=45], offset=521106, length=538],offset=0,name=f37f23a1-e328-4eff-bb30-6f125369ebf7,size=538]: Could not convert socket to TLS; routing to failure: javax.mail.MessagingException: Could not convert socket to TLS; nested exception is: javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol is disabled or cipher suites are inappropriate)   The configuration is set as below :  Current NiFi version is  : 1.13.2Java Version : openjdk version "11.0.16" 2022-07-19OpenJDK Runtime Environment (build 11.0.16+8-post-Ubuntu-0ubuntu120.

We using PutHive3Streaming processor to send data to Hive from NiFi, I have an issue where we are getting LOTS of small delta files on our busier feeds which is causing issues with compaction etc. I have used a series of merges in NiFi to ensure each flowfile contains many thousand records but it still creates many delta files. I wondered if anyone had any advice on tuning 'Records Per Transaction' and 'Transactions per Batch' options on the PutHive3Streaming processor, I believe this could help with my issue but have had mixed/confusing results from testing. There isn't  a great deal of information on best practice that I have found. Has anyone else had similar issues/found adjustments helpful? 

The server of hive on tez shut down after it restarted.  We can find error message of hive's log below:2021-02-25 14:20:06,800 INFO org.apache.hadoop.hive.common.ZooKeeperHiveHelper: [shutdown-hook-0]: Server instance removed from ZooKeeper.2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopping/Disconnecting tez sessions.2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Stopped tez session pool manager.2021-02-25 14:20:06,800 INFO org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Shutting down HiveServer22021-02-25 14:20:06,800 ERROR org.apache.hive.service.server.HiveServer2: [shutdown-hook-0]: Error stopping schqjava.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalidat org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.4.0-203.jar:3.1.3000.7.1.4.0-2

Hi, I have a use case, where i have to see the content of each FlowFiles by triggering the API call. API i have explored so far :https://<URL>/nifi-api/flowfile-queues/<Queue-ID>/flowfiles/<FlowFile-UUID>/content I got the response as below : "The id of the node in the cluster is required." Note : My NiFi is setting up in three node cluster. Please suggest.  

Dear Team,My api is not retrieving any data using invokehttp. Im able to get expected json data from cli but not from Nifi processor.  https://resource-manager-serer:8090/ws/v1/cluster/apps?states=RUNNING -k --negotiate -u : I have https and Kerberos.   

Hi I am learning Apache Crunch. But I discovered that https://crunch.apache.org/ is retired.What is the replacement this project?

Can someone tell me if these is even possible.  We want to send a file through nifi to another nifi server on a different network, thus nifi is the only tunnel of communication between the two networks.  If we can send just the file data I'm ok with that. If we can maintain the attributes and send the actual flowfile then even better.

I have installed Apache nifi 1.19.1 and configured the ssl using below command  ./tls-toolkit.sh standalone -n 'nifi1,nifi2,nifi3' -B MyPassword -C 'CN=nifiadmin,OU=NIFI' -O -o /opt/nificert  and it was wotking fine after importing the certificate in my browser.Then configured LDAP like below:-loginIdentityProviders  <?xml version="1.0" encoding="UTF-8" standalone="yes"?><loginIdentityProviders> <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">cn=Myuser,ou=Myuser,dc=Mydomain,dc=com</property> <property name="Manager Password">Mypassword</property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property

Hello Experts, I am using nifi-api to update the variable registry of specific processor groups.It was working as expected, but in recent time its taking too long (some time infinite) time to complete the variable registry update.However if I restart nifi and try again this var registry update api it works quickly as expected. Is it because of increased number/size of variable registry or any resource crunch ?Could you please suggest a solution on this ?Any suggestion is much appreciated in advance. ThanksMahendra

Hello Experts, We have a nifi instance with 8 core machine, and recently nifi is looking quite slow with its performance (nifi-api response, UI etc).Recently we have added lot of processors but at any point of time the  processor which are processing data will be only 10% of total processors. Will nifi perf goes down with increase in number of processors though 90% total processors will be idle? ThanksMahendra

Hi All Good Day! Just want to share our problem after we upgrade our nifi version from 1.9.2 to 1.16.2. all processor running fine but after ingesting a data, We got an error on the nifi bulletin.   Upon checking on the logs this the error. Thank you 

I am using a ForkEnrichment - JoinEnrichment combo to enrich some data.   Within the JoinEnrichment processor, the "wrapper" and "sql" options for the "Join Strategy" both work fine.  However, the "insert enrichment fields" option doesn't.  I prefer this option because I want my enrichment data to be a nested dataset off one of the root fields.  The field exists in the original data, and is properly referenced in the "Insertion Record Path" field.  So... I'm scratching my head trying to figure out why this isn't working.   Anyone experience this or have any ideas?

Hi guys,I'm testing an upgrade from CDH 6.3.4 to CDP 7.1.8 , I successfully performed the upgrade but I'm trying to test the rollback process, I'm using  AUTO-TLS, but I'm having a one issue though,  it says the below in the HDFS rollback Cloudera : (Clusters with TLS enabled only) Edit the /etc/hadoop/conf.rollback.namenode/ssl-server.xml file on all NameNode hosts (located in the temporary rollback directory) and update the keystore passwords with the actual cleartext passwords. So the question is : How can I get the cleartext password ? 

I cannot figure out which local path is out of spaceI have a 3 node cluster. Each node has 1TB boot disk plus 1 TB X 2 attached 7200rpm disks. The directory structure on each of the two disks on each of the three nodes are /media/sanjay/hdd03/yarn/nm/media/sanjay/hdd03/dfs/dn/media/sanjay/hdd03/dfs/snn/media/sanjay/hdd03/dfs/snn/media/sanjay/hdd02/yarn/nm/media/sanjay/hdd02/dfs/dn/media/sanjay/hdd02/dfs/snndf -h /media/sanjay/hdd03Filesystem Size Used Avail Use% Mounted on/dev/sdc1 917G 228M 870G 1% /media/sanjay/hdd03df -h /media/sanjay/hdd02Filesystem Size Used Avail Use% Mounted on/dev/sdb1 917G 1.2G 869G 1% /media/sanjay/hdd02df -h /tmpFilesystem Size Used Avail Use% Mounted on/dev/mapper/ubuntu--vg-root 915G 41G 828G 5% / 2023-01-07 22:15:31,486 ERROR org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Local path for public localization is not found. May be disks failed.org.apache.hadoop.util.DiskChecker$DiskErrorException: No sp

I have a JSON dataset and need to create a unique key from it. No single field is unique, but the first two, together, are. What I want to convert from: [{"airport":"KDEN", "runway":"16R", "length":16000}, ...] and I want to concatenate the "airport" and "runway" values to create this: [{"key":"KDEN/16R", "airport":"KDEN", "runway":"16R", "length":16000}, ...] I've got it working with an ugly anti-pattern of SplitRecord, EvaluateJSONPath, UpdateAttribute, and AttributesToJSON  which changes each flow file to several thousand, but am looking for a better way to do this. 

I need help understanding the root cause of a job failure in Cloudera Data Science Workbench.What does it mean when a job is stuck in "Scheduling" phase and terminates as "Failed" after the timeout period? The log file, which usually displays the python code followed by the results from print statements and/or run time errors, states only a single line reading "Engine exited with status 34". Apparently, I was able to re-run the job successfully after a while. It could be a temporary glitch in the system but still trying to understand the root cause of this issue. Any insight on what happens during the "Job Scheduling" is much appreciated.

Hello everyone! Recently login ambari report 500 error box. I've been working on it for ages, but I can't solve it. The following are the error phenomena and related logs, hoping to get your help. Thanks!Error screenshot:500 status code received on GET method for API: /api/v1/clusters/bgybigdata/alerts?format=groupedSummary log:HTTP ERROR 500Problem accessing /api/v1/clusters/bgybigdata/alerts. Reason: Server Error Caused by:java.lang.NullPointerException at org.apache.ambari.server.controller.internal.AlertResourceProvider.toResource(AlertResourceProvider.java:260) at org.apache.ambari.server.controller.internal.AlertResourceProvider.getResources(AlertResourceProvider.java:240) at org.apache.ambari.server.controller.internal.AlertResourceProvider.queryForResources(AlertResourceProvider.java:166) at org.apache.ambari.server.controller.internal.ClusterControllerImpl$ExtendedResourceProviderWrapper.queryForResources(ClusterControllerImpl.java:966) at org.apache.ambari.s

Does anyone have experience with CDP Private Cloud on CentOS 9?Are there any plans to officially support it in the near future?

Hello,I'm brand new to Atlas and trying to see if this capability exists.  I understand that you can use a Hive Hook and whenever a Hive table is modified, Kafka is able to communicate to Atlas and have those changes appear in Atlas.  I'm trying to see if there is any reporting or auditing features that allows you to see what those changes were?For example: If a user modified a Hive table and added two columns and this gets replicated in Atlas, is there a way to see what change happened (columns added) and when it occurred?  I'm not sure if this is something Atlas can do natively or if there might be 3rd party tools I could use to see this?Thanks in advance!