Member since
01-04-2021
29
Posts
4
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
898 | 09-17-2023 03:09 AM |
05-28-2024
03:18 AM
Could someone please help me with this ? Fetch Provenance data using SiteToSiteProvenanceRe... - Cloudera Community - 388418 configuration site to site is not working in http when nifi is running on https
... View more
01-10-2024
08:29 PM
@MattWho @bbende Any thoughts on this?
... View more
12-01-2023
10:44 PM
I have been trying to build a custom plugin for a inhouse tool that we have built for apache ranger. I have been reading various articles and found pretty good ones including the one here. From what I understand the importance of specifying pollIntervalMs is to refresh the policies in Apache ranger admin periodically, hence without restarting Ranger admin. But when considering an example, as seen in the case of Nifi the polices are cached in a temporary directory in Nifi. Is this caching really required when developing the plugins? Is this caching done by this pollIntervalMs or is it something specific to Nifi? @MattWho @bbende Any thoughts on this?
... View more
Labels:
- Labels:
-
Apache Ranger
09-21-2023
12:35 AM
Looks like security zone is also working with nifi when configuring with ranger provided all the resource identifiers that needs to be added is also added in the security zone that we are creating. Probably this is a basic that I left out. Currently the resources identifiers in policies as shown below. But the security zone has only /flow as shown below The policies json corresponding to the above configuration is shown below. {"serviceName":"nifi","serviceId":1,"policyVersion":140,"policyUpdateTime":"20230921-12:51:41.126-+0530","policies":[],"serviceDef":{"name":"nifi","displayName":"nifi","implClass":"org.apache.ranger.services.nifi.RangerServiceNiFi","label":"NIFI","description":"NiFi","options":{"enableDenyAndExceptionsInPolicies":"false"},"configs":[{"itemId":400,"name":"nifi.url","type":"string","mandatory":true,"defaultValue":"http://localhost:8080/nifi-api/resources","uiHint":"{\"TextFieldWithIcon\":true, \"info\": \"The URL of the NiFi REST API that provides the available resources.\"}","label":"NiFi URL"},{"itemId":410,"name":"nifi.authentication","type":"enum","subType":"authType","mandatory":true,"defaultValue":"NONE","label":"Authentication Type"},{"itemId":411,"name":"nifi.ssl.use.default.context","type":"bool","subType":"YesTrue:NoFalse","mandatory":true,"defaultValue":"false","uiHint":"{\"TextFieldWithIcon\":true, \"info\": \"If true, then Ranger\u0027s keystore and truststore will be used to communicate with NiFi. If false, the keystore and truststore properties must be provided.\"}","label":"Use Ranger\u0027s Default SSL Context"},{"itemId":500,"name":"nifi.ssl.keystore","type":"string","mandatory":false,"label":"Keystore"},{"itemId":510,"name":"nifi.ssl.keystoreType","type":"string","mandatory":false,"label":"Keystore Type"},{"itemId":520,"name":"nifi.ssl.keystorePassword","type":"password","mandatory":false,"label":"Keystore Password"},{"itemId":530,"name":"nifi.ssl.truststore","type":"string","mandatory":false,"label":"Truststore"},{"itemId":540,"name":"nifi.ssl.truststoreType","type":"string","mandatory":false,"label":"Truststore Type"},{"itemId":550,"name":"nifi.ssl.truststorePassword","type":"password","mandatory":false,"label":"Truststore Password"},{"itemId":560,"name":"ranger.plugin.audit.filters","type":"string","mandatory":false,"defaultValue":"[]","label":"Ranger Default Audit Filters"}],"resources":[{"itemId":100,"name":"nifi-resource","type":"string","level":10,"mandatory":true,"lookupSupported":true,"recursiveSupported":false,"excludesSupported":false,"matcher":"org.apache.ranger.plugin.resourcematcher.RangerDefaultResourceMatcher","matcherOptions":{"wildCard":"true","ignoreCase":"true"},"label":"NiFi Resource Identifier","description":"NiFi Resource","accessTypeRestrictions":[],"isValidLeaf":true}],"accessTypes":[{"itemId":100,"name":"READ","label":"Read","impliedGrants":[]},{"itemId":200,"name":"WRITE","label":"Write","impliedGrants":[]}],"policyConditions":[{"itemId":1,"name":"_expression","evaluator":"org.apache.ranger.plugin.conditionevaluator.RangerScriptConditionEvaluator","evaluatorOptions":{"ui.isMultiline":"true"},"uiHint":"{ \"isMultiline\":true }","label":"Enter boolean expression","description":"Boolean expression"}],"contextEnrichers":[],"enums":[{"itemId":1,"name":"authType","elements":[{"itemId":1,"name":"NONE","label":"None"},{"itemId":2,"name":"SSL","label":"SSL"}],"defaultIndex":0}],"dataMaskDef":{"maskTypes":[],"accessTypes":[],"resources":[]},"rowFilterDef":{"accessTypes":[],"resources":[]},"id":10,"guid":"a23a32e1-4b74-4cbf-8b60-3e161d748954","isEnabled":true,"createTime":"20230912-17:43:16.455-+0530","updateTime":"20230912-17:43:16.798-+0530","version":1},"auditMode":"audit-default","securityZones":{"nifi":{"zoneName":"nifi","resources":[{"nifi-resource":["/flow"]}],"policies":[{"service":"nifi","name":"nifi-zone-policy","policyType":0,"policyPriority":0,"isAuditEnabled":true,"resources":{"nifi-resource":{"values":["/proxy"],"isExcludes":false,"isRecursive":false}},"additionalResources":[{"nifi-resource":{"values":["/flow"],"isExcludes":false,"isRecursive":false}},{"nifi-resource":{"values":["/process-groups/b2cf12da-018a-1000-9dd7-6e78a33341fb"],"isExcludes":false,"isRecursive":false}},{"nifi-resource":{"values":["/data/process-groups/b2cf12da-018a-1000-9dd7-6e78a33341fb"],"isExcludes":false,"isRecursive":false}}],"policyItems":[{"accesses":[{"type":"READ","isAllowed":true},{"type":"WRITE","isAllowed":true}],"users":["vishnu"],"groups":[],"roles":["ranger_users"],"conditions":[],"delegateAdmin":true}],"denyPolicyItems":[],"allowExceptions":[],"denyExceptions":[],"dataMaskPolicyItems":[],"rowFilterPolicyItems":[],"serviceType":"nifi","options":{},"validitySchedules":[],"policyLabels":[],"zoneName":"nifi","isDenyAllElse":false,"id":4,"guid":"b4577f50-f5fc-40ae-b745-7ec9380e2705","isEnabled":true,"version":13}],"containsAssociatedTagService":false}}} Since only flow was in the security zone the other configurations was not working. If I add other two resource specifiers in the nifi security zone, then then things are working properly. Also not sure if there is some other thing that you mentioned as security zone.
... View more
09-20-2023
10:36 AM
@pacman 1. There is noway to disable TLS. If you remove your TLS certificate from your browser or use and incognito window your client certificate will not be presented in the TLS exchange. NiFi requires TLS certificate authentication for NiFi to NiFi authenticated and authorized connections (for example in a multi-node NiFi cluster or utilizing NiFi Site-to-Site between different NiFi deployments). When NiFi is secured an NO other methods of authentication are configured, NiFi will "REQUIRE" a MutualTLS exchange. Once at least one additional method of authentication is configured, NiFi will "WANT" a client certificate and if one is not presented from the client, NiFi will move on to next authentication method. 2. I am not clear what you mean by "removed excessive permissions from authorizations.xml". If you are using Ranger, the authorizations.xml file is not being used. That file would have been created by the file-access-policy-provider. Ranger does not use this provider. There really is no concept of an "initial admin" when using Ranger. You'll need to add authorization for what you need manually in Ranger. The "Initial Admin" is used when NiFi authorization is handled by a local file provider so that a user can be setup on startup that has ability to access NiFi and setup additional authorizations from within the NiFi UI. I recommend starting a new community question so we don't make this thread overly complicated by solving many unrelated issues. There are some NiFi Resource Identifiers that would only apply to file based authorization, but all other do work when used correctly. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-18-2023
06:11 AM
Congratulations on resolving the issue and thanks for sharing the solution.
... View more
12-19-2022
03:51 AM
1 Kudo
I have tried several scenarios to generate cache miss in HBase with HDP 2.6.5. Different steps I followed include: 1) putting value in HBase using put command and fetching using get command. 2) putting value in HBase with put command , flushing and the trying to fetch the data. None of this creates cache misses. Infact the hits and hits caching keeps increasing by multiple counts during flushing. Misses and misses caching always remains zero. Why is this behaviour occurring? What is the difference between misses and misses caching and hits and hits caching? I will attach the screen shots of region server logs.
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache HBase
-
HDFS
12-19-2022
12:15 AM
Hi @pacman A Merge happens for each Read Operation i.e. BlockCache & MemStore. As such, Incorrect Values aren't observed. Having said that, If you observe any such scenario of Read/Write Inconsistency, Kindly share a Use-Case & any replication attempt to allow us to review accordingly. Regards, Smarak
... View more
12-08-2022
02:35 AM
In Hbase as per my reading the reads happens by first checking in blockcache, if missed then Memcache,if missed then use bloom filters to check for the record and finally use index on the HFile to read the data. But what if all the data is compressed? How can it find the index and read the data from a compressed Hfile? Even if read from where does the decompression occur? Is it from client?
... View more
Labels:
- Labels:
-
Apache HBase
-
HDFS
11-21-2022
02:24 AM
1 Kudo
Hello @sfdragonstorm & @pacman In the Region Name "img,0006943d-20150504220458043384375D00000002-00093,1527295748538.7b45a9f6f5584fc50b3152d41a5323a2.", the Table Name is "img", StartKey is "0006943d-20150504220458043384375D00000002-00093", Timestamp is "1527295748538" & "7b45a9f6f5584fc50b3152d41a5323a2" is the Region ID. Under HBase Data Directory, each Table Directory would have Region-Level Directories as identified by Region ID ("7b45a9f6f5584fc50b3152d41a5323a2" in Example). The Region ID is an MD5 encoded string for the Region Name & generated by HBase itself. Refer [1], if your Team wish to review the same. Regards, Smarak [1] https://hbase.apache.org/apidocs/src-html/org/apache/hadoop/hbase/client/RegionInfo.html#line.164
... View more