Find answers, ask questions, and share your expertise
Recently active
Hi Team, I have configured the ldap connection for nifi login, when I try to access the nifi in web UI I am getting error like "Insufficient Permission Unknow user with identity 'xyz123' Contact the system administrator". Authentication is happing successfully, but authorization is not happening could you please advice.
Hi there,So, I need your experience with a flow I have been struggling with.My Flow is as follows: ConsumeKafka --> ConvertRecord --> ConvertJSONToSQL --> PutSQL.I am extracting some data out of some kafka brokers, which come in the following format: {"Column1": "1e39c17d25cb420e8e720aa4b1ae005e","TimeStamp": "2023-04-10T10:43:15.794241429+03:00","Column3": "some_string","Column4": false,"Column5": "some_string","Column6": "some_string","Column7": false,"Column8": "some_string","Column9": "","Column10": 0,"Column11": 0,"Column12": "","Column13": 0,"Column14": 137,"Column15": "","Column16": 0,"Column17": 0,"Column18": "","Column19": 138,"Column20": 1550,"Column21": "some_string","Column22": ""} I am using afterwards an ConverRecord, with an RecordReader = JsonTreeReader (with default configuration: Infer Schema and Root Node) and an RecordWriter = JsonRecordSetWriter (with default configuration: Inherit Record Schema and Pretty print json = false). T
I am trying to use NiFi to make a CURL call to SMM. Because SMM is using kerberos authentication I want to make a POST request to the SMM API using Kerberos. I am now at the last step where I need to sent a POST request to create a Kafka topic. I am having a problem where when I send the request using the ExecuteStreamCommand I think some formatting happens to the payload before the call gets sent to the SMM API. I tried escaping the quotes in the payload but still no luck. Here are my configurations for the ExecuteStreamCommand:Command Arguments Strategy: Command Arguments PropertyCommand Arguments: -X POST -H referer:${Referer} -H 'Content-Type: application/json' -d '{\"newTopics\": [{\"name\":\"testing123\",\"numPartitions\":3,\"replicationFactor\":3}], \"allTopicNames\":[\"testing123\"]}' --negotiate -u : -b /tmp/cookiejar.txt -c /tmp/cookiejar.txt http://SMM-HOSTNAME:8585/api/v1/admin/topicsCommand Path: curlIgnore STDIN: falseArgument Delimiter: " " (Sp
Hello, I'm using the InvokeHTTP processor to make a GET request from an API, however, before getting access to the API, there's a keycloak sign-in page that I need to bypass and the InvokeHTTP process returns that page as a csv response. Is the a way to sign-in or to use a token when making the GET request so I can make the right API call.
Caused by: java.lang.Exception: The specified login identity provider class 'org.apache.nifi.authentication.single.user.SingleUserLoginIdentityProvider' is not known to this nifi. I am trying to set up a nifi instance in a Ubuntu Server 22.04 machine.It's been hours I've been tinkering with the configurations and xml files and still I'm getting the error above.These are the configurations I am passing in the nifi.properties, authorizers.xml, login-identity-providers.xml and authorizations.xml files are in the screenshot below. I have tried specifying a user and password in the identity providers file but still nothing...
Im trying to make kafka ranger plugin communicate with the policy manager url over ssl, i have the truststore and keystore jks file but im getting PKIX path building failed, i know this error usually is visible when the files are not in the right place. I am using original ranger repository, please let me know where are we supposed to keep the files?
Hi,I am using distcp to copy data from hadoop hdfs to s3. below is the shorthand command of what i use hadoop distcp -pu -update -delete hdfs_path s3a://bucket recently got into an issue with the below case i have a file in hdfs -> temp_file with data 1234567890 with size 27kbfor the first time when i use distcp. it pushes the file to s3 bucket without any issue. second time i update the same file temp_file with different content abcdefghij but with same size 27kbnow when i run distcp. instead of checking the checksum of source and target distcp skips the file directly and doesnt copy the updated file from hdfs to s3 Am i missing any options in distcp command to make this scenario work?
Hi, We are looking to migrate our data from SQL server to CDP environment and would like some insights on getting this done. Would appreciate any help/guidance in the same. Thanks Amn
I have a pipeline running on a daily basis, I wish to receive an email alert whenever any of the processor fails. I tried testing put email processor, I am receiving success flow files after PutEmail but email is not being generated despite entering correct SMTP details. Kindly suggest if I am missing on anything. My nifi pipeline flow is as follows :getMongo > flatteJson > evaluateJsonPath > attributeToJson > queryRecord > updateAttribute > PutS3For eg : If I get an error in QueryRecord : column 'x' not found in list of columns. This failure alert should be sent on my email.PFB the properties I have used for configuration of PutEmail processor
Dears,I install NIFI 1.14 with a secure cluster. When open NIFI GUI, the user access policy and Users list is missing. I try single mode but the result is the same. First screen my new NIFI 1.14 version.Second screen old NIFI 1.11. How to add policy and users on NIFI1.14? NIFI 1.14 new versionNIFI 1.1 old version
I am consuming a messages from IBM MQ using processor (ConsumeJMS), then once you explore as a "List queue" you will see the following type and format: Hence, I added a following processor (ExtractText 1.16.1) in order to search for some texts using regular expression as following: (ACT.A.EMS..ST...AIR|ACT.X.EMS..ST...AIR) Finally, Sometimes the results of regular expression are valid and sometimes it satisfies a half of my condition because:1) if it does not match anything hence it match only the part of my condition. 2) or if the half of my condition is valid; hence it will match! Is there something wrong I am doing in my regular expression or something else? or I need to change type, format, or content type (application/octet-stream) to text so that I can search correctly instead of viewed as a hex? I hope my explanation was fine and clear for everyone.Thanks!
I am having a HDP hadoop cluster. I took snapshot of phoenix table from this cluster & I cloned that table to a CDP cluster. Hbase table was created from the snapshot & the hfiles are imported to the table's data directory . While selecting the table from phoenix shell, empty data it is showing. If I do a scan operation from hbase shell data is displaying (encoded). Is there any workaround for this issue. Thanks
I had a 3 node NiFi cluster and created a NiFi registry client with this a URL that goes through Knox and looks something like this https://xyz.com/gateway/registry/nifi-registry-app/nifi-registry-api. However, when I restart NiFi the registry client in NiFi strips the URL path so it comes back as https://xyz.com which isn't correct and can't connect to the registry. Has anyone experienced this before and/or solved this issue?
Hi there, any support would be much appreciated. First things first, here is our setup. We are running nifi on aws ec2 in a docker container. It's a two node cluster setup. We are planning to do an in place upgrade by upgrading the docker container and changing the nifi.properties parameters needed to get 1.19 running. changing the port number to 8443using https instead of httpsetting the sensitive properties, was empty in 1.13we are using the web.proxy.host with 3 entries, the hostname of the 2 nodes and the clusternamewe are running anonymously, without ldap. We'll set this up later.We made a backup of the flow.xml.gz file and deleted the original. We are planning to use the nifi-registry to import our flow on the clean canvas. So this works great! We have succesfully upgraded to 1.19. Now what we're trying to figure out is our fallback plan, just in case upgrading doesn't work in the production environment.When reverting all the changes above and deleting the flow.xml.gz,
Hi, I work with spatial data which is in netCDF format. To manipulate and analyze the data I use the NCO toolkit, which consists of a bunch of commands. I run these in the Windows CMD. Though this works fine, it is somewhat slow and cumbersome. Thus I would like to automate this with NiFi. I figured the ExecuteStreamCommand would do the job. The netCDF data comes in via InvokeHTTP (generates a flowfile with ‘Content Type:application/x-netcdf’), and is subsequently fed to the ExecuteStreamCommand. I have it configured as follows ‘ncks’ is the executable command from the NCO Toolkit. This is on my path, such that I am sure NiFi can reach it. When I run the flow as follows I receive errors on the ExecuteStreamCommand processor. These are as follows 2023-04-09 20:46:43,879 ERROR [Thread-187] o.a.n.p.standard.ExecuteStreamCommand ExecuteStreamCommand[id=65610950-0187-1000-7ef5-c5d0d5057e63] Failed to write FlowFile to Standard Input Streamjava.io
I have Installed CDP Private Cloud 7.1.7getting 500 error code while accessing 'http://hostname:50111/templeton/v1/status'
Hello, I have a scenario where I need to copy the file from HDFS to local file system on edge server. When i run the code in cluster mode with spark submit , I am getting below kind of error: User class threw exception: java.io.IOException: Mkdirs failed to create/some_specific/path/in_edge_server (exists=false, cwd=file:/some/path/yarn/nm/usercache/1234/appcache/application_123456744/container_t1234) It seems like it is trying to find the destination path where the execution takes place and since it is not finding the specific path it is trying to create a directory and failing? Below is the function being used:fs.copyToLocalFile(new Path(src_HDFSPath), new Path(dest_edgePath)) Do we have a solution to copy the file from HDFS to specific path of edge server when we run the program in cluster mode?
When I try to put files to HDFS with Talend, it creates the file but the file is empty I get this error :File /user/hdfs/data/file.xlsx could only be written to 0 of the 1 minReplication nodes. There are 1 datanode(s) running and 1 node(s) are excluded in this operation. I'm really struggling hope someone can help
Hello, Is there any provision to write the Kafka data in to cloud. This feature is available in Apache and Confluent Kafka. Thanks,Uday
Hello, i'm using the cetic helm chart to deploy nifi to GCP gke cluster. I have configured SSL and OIDC with azure AD, i was able to see the AD page and log in, after that i'm seeing this error Values: replicaCount: 1image: repository: apache/nifi tag: "1.16.3" pullPolicy: "IfNotPresent"securityContext: runAsUser: 1000 fsGroup: 1000sts: podManagementPolicy: Parallel AntiAffinity: soft useHostNetwork: null hostPort: null pod: annotations: security.alpha.kubernetes.io/sysctls: net.ipv4.ip_local_port_range=10000 65000 #prometheus.io/scrape: "true" serviceAccount: create: false #name: nifi annotations: {} hostAliases: [] startupProbe: enabled: false failureThreshold: 60 periodSeconds: 10properties: sensitiveKey: changeMechangeMe externalSecure: true isNode: false httpsPort: 8443 httpPort: 8080 webProxyHost: nifi-test.qa.cloud.paycomplete.com clusterPort: 6007 provenanceStorage: "8 GB" siteToSite: port: 100
Hello TeamI have a small development POC HDP Cluster installed on 4 machinesI was using the Public Repositories of Hortonworks/Cloudera that are no longer validhttp://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0/repodata/repomd.xmlhttp://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/repodata/repomd.xmlhttp://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/repodata/repomd.xmlhttp://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/repodata/repomd.xmlQuote"Starting with the HDP 3.1.5 release and Ambari 2.7.5 access to HDP, Ambari repositories requires authentication. Authentication credentials for new customers are sent sent from Cloudera to registered support contacts"To be a registered support contact I suppose one needs a subscription. How do I obtain the subscription for access to the repositories and how much does it cost annually and monthlyapproximately. Are they still available for legacy HDP and Ambari Platforms
Hello friends, I wrote a custom processor and now wanted to add a custom controller. The problem is, that the properties I wrote for the controller do not appear when I create the service. I followed the structure of this controller: https://github.com/Automna/NiFi/blob/master/nifi-rel-nifi-1.1.2/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java Here is what I did so far:public class MyNewControllerService extends AbstractControllerService implements MyControllerService { public static final PropertyDescriptor USERNAME = new PropertyDescriptor.Builder() .name("Username") .expressionLanguageSupported(ExpressionLanguageScope.NONE) .displayName("Set username") .required(true) .defaultValue("User") .description("Username to login to graph service") .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
Hello All, I have configured nifi flow with three node with external zookeepernode1.domain.comnode2.domain.comnode3.domain.com with three zookeeper nodeszk1.domain.comzk2.domain.comzk3.domain.com Everything is working fine but the requirement here is my flow has trigger points from external client systems to whom I have give a common URL likenode.domain.com to avoid any changes on the client teams if any of nodes goes down. Any idea how to configure this.
Hi all, I have HDFS service running on my CDP 7.1.8 private cloud base cluster with Kerberos enabled. Recently, I got two issues with my HDFS NameNode, here is the screen capture:The first one The second one:When looking into the role log, it shows Could anyone point out the root cause and the solution for this issue for me please? Thanks in advance. Please let me know if I need to provide more information.
Tried to add ambari metrics to a cluster and the metrics collector keeps failing on a node due to some issuehere are the logs:stderr:Traceback (most recent call last):File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 90, in <module>AmsCollector().execute()File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in executemethod(env)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/metrics_collector.py", line 51, in startams_service('collector', action = 'start')File "/usr/lib/ambari-agent/lib/ambari_commons/os_family_impl.py", line 89, in thunkreturn fn(*args, **kwargs)File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/AMBARI_METRICS/package/scripts/ams_service.py", line 84, in ams_serviceuser=params.ams_userFile "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__self.env.run()File "/usr/lib/ambari-agent
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.