About ashinde

ashinde · ‎01-19-2021

@Delio It would be bit complicated flow design as direct support for delete events is not available for GetHDFSEvents processor yet. But you can refer this solution, which can help you to move further. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here. -Akash

ashinde · ‎01-18-2021

@Law serveraddress is hostname of host where nifi service is installed. E.g you have installed it on samplehost.com then it would be "http://samplehost.com:8080/nifi" .If you enable SSL for nifi later then the URL will change to "https://samplehost.com:<secure_port>/nifi". Also you should check that host and port are reachable from your browser to access NiFi UI. If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here. -Akash

ashinde · ‎01-13-2021

@dzbeda You can use one of below approach. 1. Install minifi on windows machine and send logs from minifi to NiFi [Push] 2. You can use ConsumeWindowsEventLog to get event logs in NiFi [Pull]

ashinde · ‎01-13-2021

@Delio What do you mean by "But what about file delete?" Do you want to delete the data at source HDFS as soon as you fetch it in NiFi?

ashinde · ‎12-08-2020

@sarithe It is expected behaviour as Ranger has to pull resources from NiFi and as NiFi is secured it will look for authentication token from Ranger but as Ranger is un-secure it can not present the token and get 403 as response. You should enable SSL [or kerberise the cluster] for Ranger so Ranger can authenticate using SSL [or tgt]certificate. Meanwhile you can use https://community.cloudera.com/t5/Community-Articles/NiFi-Ranger-based-policy-descriptions/ta-p/246586 as reference to setup policies manually. -Akash

ashinde · ‎12-08-2020

Jaro, nipiapi Windows automated testing not attempted. Are there any particular things you want to achieve using nipiapi+Windows combination? -Akash

ashinde · ‎12-08-2020

@kumar79 I am not very sure with your complete use case, if you explain more would be better. Why you want to take date from shell/cron? you can use UpdateAttribute processor with custom property and use NiFi Regular Expression language to set date for you. E.g https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#dates The attribute later you can use in your flow to satisfy QL condition. Hope it helps. -Akash

ashinde · ‎12-08-2020

@DarkStar Is the join query works fine on PostgreSQL CLI? If yes, then what is the jdbc driver you are using? is that jdbc version supports joins with duplicate fields? It looks like you have T1.track_id and T3.track_id which are conflicting. May be can you try to change the field name, test it on psql CLI and then test again in NiFi ? -Akash

ashinde · ‎12-08-2020

@scoe74 If it is just token you have to take from SiteA and then using that token you can download data from SiteB then you can use 1st InvokeHTTP to take the token, add the token as flowfile attribute and then using that token you can use 2nd InvokeHTTP to download the data. Hope it helps you. -Akash

ashinde · ‎11-23-2020

@dzbeda You should extend the Content repo as it will store the content of your flowfiles. Please go through the best practices for setup https://community.cloudera.com/t5/Community-Articles/HDF-CFM-NIFI-Best-practices-for-setting-up-a-high/ta-p/244999 Also note that NiFi should not be used as backup for any failover mechanism as it is meant for processing and not for storage. I would recommend to store the data to may be some centralised location from where you can store and retrieve data as per use case and in case of failure as well you can have backup on that location. NiFi can be used to hold data for a day or 2 but if it is not processed then it will hold data at content repository and can fill the content repo storage.

ashinde · ‎11-20-2020

Do you see any Warns, Errors for "CN=admin, OU=ApacheNiFi" user in nifi-user.log file when you login to the ui?

ashinde · ‎11-17-2020

@Vapper Looks like there are issues with authorization, can you please check your authorizations.xml file and check if the user you have mentioned in INITIAL_ADMIN_IDENTITY is assigned to all expected policies?

ashinde · ‎11-17-2020

@fgarcia Cloudera NiFi Kafka processors should work with all Kafka distributions be it HDP, CDH, CDP, Confluent or Apache Kafka. We generally recommend to use *Kafka_0_10 with Broker version as 0.10 and likewise. Hope it helps 🙂

ashinde · ‎10-14-2020

@justenji The timestamp details are not exposed by NiFi on Flow but NiFi does maintain it in flow history. [Only if NiFi is secured with SSL] So you can try https://${hostname}:${NIFI_PORT}/ nifi-api/ flow/history/components/{componentId} to get the details. But it will provide all history of that component you have to parse it as per your need. Note that this info gets purged as per flow history parameters in nifi.properties so you will have these details not from beginning but beginning of the date [days/hours] mentioned in properties. If you are looking for flow timestamp when it is exported to NiFi-Registry then checkout for https://${hostname}:${NR_PORT}//nifi-registry-api/buckets/{bucketId}/flows/{flowId} Br, Akash

ashinde · ‎10-12-2020

@sunile_manjee I am not very familiar with AWS ELB but you can try to use HandleHttpRequest and HandleHttpResponse processors and check if it serves your use case

ashinde · ‎10-12-2020

@rstewart Counter is in-memory content for NiFi service. There is no direct way to get value of particular counter. But you can try below call to get a json response from NiFi about all counters. curl -v -X GET "https://$HOSTNAME:9091/nifi-api/counter" -H "Authorization: Bearer $TOKEN" -H "Connection: close" -k Once you get the Json response you can parse it as per your use case

ashinde · ‎10-12-2020

@justenji It is not possible yet to get all info in one NiFi API call as you desired but you can try below call to get the versioning info about particular process group curl -v -X GET "https://$HOSTNAME:9091/nifi-api/versions/process-groups/{id}" -H "Authorization: Bearer $TOKEN" -H "Connection: close" -k Br, Akash

ashinde · ‎10-12-2020

@varun_rathinam You can use MergeContent processor if it fits for your use case. It is better way to handle small file issue in NiFi. Please refer below link for more details. https://community.cloudera.com/t5/Support-Questions/Merge-Content-for-small-content-issue/td-p/167660 BR, Akash

ashinde · ‎08-24-2020

@Koffi Can you try to lower the initial memory and restart the service please?

ashinde · ‎08-03-2020

@abhinav_joshi NiFi supports " /access/kerberos" for kerberos/spnego negotiation and "/access/token" for username/password to provide token to use with " Authorization: Bearer" in api request. So you can try to use below calls 1. TOKEN=$( curl -k "https://$HOSTNAME:9091/nifi-api/access/token" -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' --data 'username=xxx@REALM&password=secret' --compressed ) 2. curl -v -X GET "https://$HOSTNAME:9091/nifi-api/flow/status" -H "Authorization: Bearer $TOKEN" -H "Connection: close" and so on.

ashinde · ‎08-03-2020

@bsivalingam83 This error generally occurs when you copied the flow.xml.gz from another NiFi cluster and tries to load it in new NiFi setup. This happens because the "nifi.sensitive.props.key" from first NiFi setup is different from "nifi.sensitive.props.key" value from second NiFi setup. If flow.xml.gz is not copied and CFM is 1.0.x version then you may need to check https://docs.cloudera.com/cfm/1.0.1/upgrading-cfm/content/upgrading-cfm.html to resolve the issue

ashinde · ‎05-30-2020

@samarsimha You can use configure MongoDBControllerService > SSL Context Service and provide certificate details.So NiFi can use these details to communicate with MongoDB database. [Note: Make sure that the certificate details like keystore and truststore path should be same on all NiFI nodes] SSL Context Service you can use StandardSSLContextService which will allow you to communicate with both legacy and modern systems Thanks, Akash

ashinde · ‎05-27-2020

@anand27krishn CDF packages are not supported on Windows OS but yes you can use single machine like you do for HDP 2.6.5 on virtualbox or vmware. Note that you have to install the VM and respective parcels manually there is no sandbox setup yet for CDF parcels. For more details for each service you can visit https://docs.cloudera.com/ and select for services like CFM, CSA,CSP and download the trail version parcels. Thanks, Akash

ashinde · ‎09-12-2018

Hi Inam, Please find below link for reference : https://community.hortonworks.com/articles/178747/mqtt-with-apache-nifi.html (I have tested it and worked fine for me) Also as you are using psk-identity please check for "use_identity_as_username" value in your mosquitto.conf file. I am not very much sure about psk-identity but you can use mosquitto_passwd to set password for particular user and redirect the password to a file which file can be used for authentication to publish MQTT messages.

Online	Offline
Last Visited	‎01-22-2021 07:14 AM

Date Registered	‎07-30-2018 12:32 PM
Last Visited	‎01-22-2021 07:14 AM
Total Messages Posted	25
Total Kudos Received	3

Re: NiFi - REST-API - searching for information ab...

Re: using AWS ELB with ListenHTTP on nifi

Re: nifi mongo x509 connection

Re: Is cloudera not free to use

Re: CDP Public Cloud : Data Replication

Re: HDF on HDI - NiFi

Re: How to collect windows event log using Nifi

Re: CDP Public Cloud : Data Replication

Re: Secured nifi with unsecured ranger issue

Re: nipyapi on windows 10

Re: NiFi: Passind date from command line

Re: Joining multiple tables in PostgreSQL using Ex...

Re: NiFi InvokeHTTP Problem

Re: Can NIFI storage be extended?

Re: NIFI docker ssl issue

Re: NIFI docker ssl issue

Re: Cloudera DataFlow and Confluent Kafka

Re: NiFi - REST-API - searching for information ab...

Re: using AWS ELB with ListenHTTP on nifi

Re: Retrieve Value of Signal Counter in Wait/Notif...

Re: NiFi - REST-API - searching for information ab...

Re: How to handle small file issue

Re: Nifi There is insufficient memory for the java...

Re: Starting process group from Linux for Secured ...

Re: Could not decrypt sensitive value - NiFi

Re: nifi mongo x509 connection

Re: Is cloudera not free to use

Re: Apache nifi : MQTT Bokerr consumer need help