Q&A

Hello,Any clue on how we configure Streams Messaging Manager server to send audit logs to HDFS and / or Solr (just like other services) and then they would get archived to /archive directory from where we can manually delete them?I am referring to logs that get stored under below location locally (not HDFS)/var/log/streams-messaging-manager/audit/<hdfs or solr>/spoolThankssnm1523

Hello All,On CDP PB, like we send ranger audit logs of services thats support Ranger plugin to HDFS, can we also send logs of Stream Messaging Manager service to HDFS? I tried to look for config to achieve this, however, I could find only 2 configs (listed below) that allow us to configure the local path. However, there was nothing found to send these logs to HDFS.Ranger Streams Messaging Manager Plugin Audit Hdfs Spool Directory PathRanger Streams Messaging Manager Plugin Audit Solr Spool Directory PathAny help / guidance will be highly appreciated.Thankssnm1523

We are facing the HttpClient error when we try to fetch data from the Hive using the  SelectHiveQL NiFi Processor.Please find the error details below and help us resolve this with your expertise. Controller Service - HiveConnectionPoolDatabase Connection URL: jdbc:hive2://localhost:10000/default;principal=hive@test@xyz.comHive Configuration Resources: /etc/hive/conf/hive-site.xml,/etc/hadoop/conf/core-site.xml NiFi Processor - SelectHiveQL Error Message :SelectHiveQL[id=23er-1234-9999-4444-17] org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$2132/2007351143@6ab4a383 failed to process due to java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient- Caused by: java.lang.ClassNotFoundException: org.apache.http.client.HttpClient; rolling back session: {}SelectHiveQL[id=sge33-1234-9999-21-33] Processing halted: yielding [1 sec]: java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient- Caused by: java.lang.ClassNotFoundException: org.apach

Good day,In SIT, I have manually create a hive table. Follow by manually create a partition on it. Then I copy the bucket file from other table and put inside the partition. At this moment, if I run select statement, I getting 0 result, I believe this is normal because data haven't write to meta store.Then I run the following commands:ANALYZE TABLE raw_zone_sit.jxtemp PARTITION (dl_created_yr='2023', dl_created_mth='12') COMPUTE STATISTICS;After I run this command, I able to select data from my table.After that I apply the same way in production, but I am still getting empty result.Currently I am trying to recover all the meta store in production, this is because 1 of the person accidently delete all the meta store in production. Thus now I am trying to recover on it.For your information, parquet tables working fine. But I have some ORC tables also, now pending ORC table to resolve.Please help.

Requirement:Building an open source Hadoop cluster. No Ambari (No support for RHEL 9) and No CM manager (My company is not ready). Server Information:I have 5 nodes in RHEL 9: 2 masters and 3 slaves. Apache Services to be installed as per my company requirement:- Hadoop- Hbase- ZookeeperOnline Resources:Overwhelmed to follow because Its tough for me to figure out how I will be able to interlinked everything together. Needed solution:Can someone please share the standard and most updated resources which i can follow to setup my cluster. And make those services talk to each other in a right direction. Like where to start ? As I know its hadoop but after that I got no clue. I did all the coding part in my previous project and now this admin part is a pain to me.Please someone help. Really appreciate. Thanks,Srikanth

Hello,Could you please help me what i am missing here.. i am trying to copy currency value from currency condition and adding it in calculatePrice array.Input { "tiers": [ { "rewards": [ { "rewardId": 8127, "changeType": 2, "changeAmount": 15, "changePercent": null, "quantityToDiscount": null, "applyInd": 2, "rewardsMerch": [] } ], "priceRuleAttributes": [ { "type": "CFA", "id": "PR_CURR_CONDITION", "values": [ "AED" ] } ] }, { "rewards": [ { "rewardId": 8128, "changeType": 2, "changeAmount": 18, "changePercent": null, "quantityToDiscount": null, "applyInd": 2, "rewardsMerch": [] } ], "priceRuleAttributes": [ { "type": "CFA", "id": "PR_CURR_CONDITION", "values": [ "USD"

When I use the GETMONGO processor to read source data from MongoDB with the output format set to Standard JSON, it converts the original date value to the local timestamp. However, when using the Extended JSON format, the original date value appears but in a different format.I need one of the following solutions:1. Original date value in Standard JSON format: The date should appear as "createdAt": "2012-08-27T22:32:56Z".2. Date format in Extended JSON: The date should appear as {"createdAt": "2012-08-27T22:32:56Z"}.3. Transformation: Convert the Extended JSON format from "createdAt": {"$date": "2012-08-27T22:32:56Z"} to {"createdAt": "2012-08-27T22:32:56Z"}.Here are examples of the outputs:**Standard JSON output:**[{"_id": "629f3a565f65e119263fa79e","id": "46dd1621-910f-4eb6-91a0-5ae609a77b10","firstName": "kelly","createdAt": "2012-08-28T10:32:56Z","modifiedAt": "2024-07-01T18:17:29Z","suspended": false,"linkedAccounts": [{"accountNumber": "100000000","accountType": "SAV","linkedAt":

After adding new datanodes to the existing cluster and updating topology file. I am not able to refresh the namenode to pick up rack of the new node. It puts under `/default-rack`.   Anyway to refresh the rack topology changes without restarting namenodes? 

We have a requirement to convert the CSV file content into a custom format. Please find the details of the input and expected output format below and help us with your expertise to achieve this using the NiFi Processors.Input File Content :username, first name, middle name, last nametest_user, test_FN, test_MN, test_LNtest_user2, test2_FN, test2_MN, test2_LNExpected Output Format :username = test_userfirst name = test_FNmiddle name = test_MNlast name = test_LNusername = test_user2first name = test2_FNmiddle name = test2_MNlast name = test2_LN

Hello,I am wondering if there is any way to secure HBase with Apache Knox so that if the Knox token is not provided HBase redirects to the Knox SSO page. I managed to do it for HDFS Yarn and Spark using filters but can' t seem to find any way to do that with HBase.```<property><name>hadoop.http.authentication.type</name><value>org.apache.hadoop.security.authentication.server.JWTRedirectAuthenticationHandler</value></property><property><name>hadoop.http.authentication.authentication.provider.url</name><value>https://<knox-host>:8443/gateway/knoxsso/api/v1/websso</value></property><property><name>hadoop.http.authentication.public.key.pem</name><value><Your public token from the .pem file></value></property>```Here is what I added in core-site.xml to protect HDFS and Yarn.Thank you for your time

Hai, everyoneMy principal KVNO is different from each other principals like thisroot@master:~# kvno hdfs-hadoop@EXAMPLE.COM kvno nn/master.hadoop.com@EXAMPLE.COMhdfs-hadoop@EXAMPLE.COM: kvno = 3kvno: Server kvno@EXAMPLE.COM not found in Kerberos database while getting credentials for kvno@EXAMPLE.COMnn/master.hadoop.com@EXAMPLE.COM: kvno = 2Can the kvno be different from each other's principals?

Hi All,My Apache Nifi is running properly but  unable to open my ec2 ubuntu IP-address in browser . I am getting This site can't be reached error. Please help me to resolve this issue .

Hi guys,I am currently evaluating if NiFi can replace one or two of our tools (Talend OS/VisualCron).Therefor I need to find out how NiFi can be used to transfer (mainly "move") files from one sftp source to another. We have at the moment around 500 jobs in Visual Cron doing this. There are 40-50 servers in use and the jobs differ by file pattern and path.The closest solution I have found so far would be a custom processor mentioned in this post. But it seems outdated and need to get some fixes before it can be successfully build again.Either I could store all server and job configurations in an external file (yaml/xml) to create flowfiles or as preferred to store this data directly in NiFi - but not sure how.The benefit I see against VisualCron to let NiFi do the job is his queue system. This will make our transfers a lot more stable which is not at the moment.Do you think NiFi would be the right choice for this and can help me to find a suitable solution without creating 500 flows in

Hello, I m sending a http request to an API and facing some difficulties to configure the multipart specificities.I have to send 2 files as part of request. Is there any way to send 2 files ? Since there can be only one flow file that is possible  @SAMSALApache NiFi processor 'InvokeHTTP' POST with MultipartFile formdata to remote API --This works fine when one file is being sent. Any help on sending 2 of them ? 

Hey everyone, after enabling Kerberos resource manager can't run, this log after try run resource manager. please AdviceFile "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/hdfs_resource.py", line 295, in _run_command raise WebHDFSCallException(err_msg, result_dict)resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X GET -d '' -H 'Content-Length: 0' --negotiate -u : 'http://master.hadoop.com:50070/webhdfs/v1/services/sync/yarn-ats?op=GETFILESTATUS'' returned status_code=403. <html><head><meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/><title>Error 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Invalid argument (400) - Cannot find key of appropriate type to decrypt AP REP - AES256 CTS mode with HMAC SHA1-96)</title></head><body><h2>HTTP ERROR 403 GSSException: Failure unspecified at GSS-API level (Mec

I'm running oozie HA 5.2.1 on EMR and I have an issue with this temporary directory. I have a workflow which has start node -> action node -> end node. The job start running -> runs for 10-15 minutes -> is initially marked as successful -> is marked as failed. The error is JA008 File or directory not found for the /user/oozie/oozie-oozi/"oozie_job_id"/"ActionName"--java directory. I'm running in parallel a script which looks at this directory as well as the state of the workflow and while running this directory contains action.xml and launcher.xml file -> before marking it as SUCCEDED it adds the action-data.seq file -> after it marks it as succeeded it deletes this directory -> after 1-2 minutes marks it as failed with the error above. It looks like oozie is checking again for that directory after it previously deleted it.Although this jobs is marked as failed from oozie perspective in reality this job is successful (checking in the RM UI the jobs status and th

How to set up TeraGen & TeraSort performance testing in a CDP private cloud cluster ? 

Hello, i am trying to implement a solution where i want to use Dynamic SOAP requests to a servicenow system but not using Basic Authentication mechanism and Oauth2.0 instead.I am not sure if this can be done without using invokeHTTP processor because i need to put some URL there as well in order to POST. Not sure how can i utilize this authentication mechanism seamlessly within PutServicenow. Need help.

HiI have an existing CDP 7.1.x cluster with Auto-tls enabled during the creation of the cluster. I followed the use case 2: https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/security-encrypting-data-in-transit/topics/cm-security-use-case-2.html, to use an existing Root CA. Recently the certificates expired and I'm trying to renew them. I've a couple of questions from the documentation.1. In the above page, it mentions "In this use case, rotation of the Auto-TLS certificate authority is not supported. Cloudera recommends creating an intermediate CA with a long lifetime. The host certificates can be rotated by using the generateHostCerts API." - Should I use this to generate the host certs. If so, can I get an example of the API call and it's usage.2. Or should I use this use case 3: https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/security-encrypting-data-in-transit/topics/cm-security-use-case-3.html. Generate the certificates myself and use the gener

We are on CDP 7.1.7 SP2 versionCan I ask what is the best strategy to backup the cluster? We are using 5 phyisical nodes (3 of which holds the actual HDFS data). What is a good strategy for backup?1. Create another cluster that will act as replica (lower CPU and RAM)2. In the same cluster, backup / to preserve the OS apps (CM etc.) and /data (HDFS)

Unable to execute the Sqoop command after configuring the existing environment into the new environment.we are using Sqoop import/export command to get the data from other DB to the HADOOP. But it's failed with following error. But we can do eval command and list command for same DB.Error: java.lang.RuntimeException: java.sql.SQLException: ORA-12541: Cannot connect. No listener at host prxd10-scan.intranet.slt.com.lk port 1521. (CONNECTION_ID=fnQPj6XGQbKYLTFy6NYkAA==) https://docs.oracle.com/error-help/db/ora-12541/but we are doing some Sqoop job with HADOOP and other DBs.please help me!

Hi! I am new in using Apache NiFi in Cloudera DataFlow. I am using NiFi 1.25.0.I made a flow in local that listens to port 80 for HTTP requests and the hostname was ofc localhost but when trying to replicate it in Cloudera I don't know what hostname I have to use. Some say it is one of the nodes that the cluster has but my cluster have 4 nodes (which one to use?).Is there a way to know what hostname I have to use to be able to reach the endpoint hosted in Cloudera by using Postman from my PC? Do I have to make some cluster/port forwarding/network configuration?Thank you guys!

Hi, On our Cloudera Cluster version 7.11.3, added new Nifi and nifi registry parcels and csds. Restarted cloudera-scm-server and cloudera-scm-agents and cloudera manager services, but still not seeting any of the nifi or nifi registries when I add service. Nifi version 1.26.0Checked scm-server and scm-agent logs no errors. What else am I missing ,Any idea why ?

Hi Team, I am using 3 - node Nifi cluster, version : 1.23.2. I observed that Nifi throws error for deleting any processor/parameter context if any node is down. Can you please share the reason of it? Is there any document from Apache that clearly list out the restrictions of task list if any one node is down?Thanks,Priyanka

I am experiencing an issue with FTP access from the host machine to a Docker container. FTP access works when I try directly from within the container. I am unsure where the problem lies.Context:I am using Apache NiFi 2.0-M2 deployed in a Docker container. I have configured a ListenFTP processor in NiFi to receive files via FTP on port 2221 and make them accessible from the host machine. docker-compose.ymlHere is my docker-compose.yml configuration:services:nifi:container_name: nifiimage: apache/nifi:2.0.0-M2ports: "8443:8443" "2221:2221" # ftpenvironment: NIFI_WEB_HTTPS_PORT=8443The port 2221 seems to be correctly mapped from the container to the host machine.Connection Tests from the Host Machine:Telnet:telnet localhost 2221Trying ::1... Connected to localhost. Escape character is '^]'. Connection closed by foreign host.Netcat:nc -zv localhost 2221Connection to localhost port 2221 [tcp/rockwell-csp1] succeeded!Curl:curl -4 -v -u nbo:******* -T test2.txt ``ftp://localhost:2221