Member since
10-04-2017
43
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
4000 | 12-22-2017 12:34 PM |
12-27-2019
10:31 AM
I have 3 nodes NiFi and 1 node Registry instances running.
Registry - SSL enabled and AD-integrated
NIfi - SSL enabled and AD-integrated
how can I integrate NiFI with Registry, so I can start using Registry.
@MattWho please advice for any recommendations
... View more
Labels:
- Labels:
-
Apache NiFi
12-26-2019
12:55 PM
@MattWho this really helps, now i am getting login page once after the setting the value to false nifi.registry.security.needClientAuth=false Really Appreciate it. Can you help to configure file-based user authorizations as well in authorizers.xml i hope below part should be enough userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Initial User Identity 1">CN=sys_admin, OU=RegPoc</property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class> <property name="User Group Provider">file-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">CN=sys_admin, OU=RegPoc</property> <!--<property name="NiFi Identity 1"></property>--> </accessPolicyProvider> what should be the node identity here ?
... View more
12-25-2019
07:33 PM
do i need to configure authorization.xml to get the login page alone? please advice. Thanks!
... View more
12-25-2019
06:41 PM
@Matt yes, updated nifi-reg.properties file nifi.registry.security.identity.provider=ldap-provider but still no luck # security properties # nifi.registry.security.keystore=./conf/keystore.jks nifi.registry.security.keystoreType=jks nifi.registry.security.keystorePasswd=xxxxxxxx nifi.registry.security.keyPasswd=xxxxxx nifi.registry.security.truststore=./conf/truststore.jks nifi.registry.security.truststoreType=jks nifi.registry.security.truststorePasswd=xxxxxxxxxxx nifi.registry.security.user.authorizer=managed-authorizer #nifi.registry.security.user.login.identity.provider=ldap-identity-provider nifi.registry.security.needClientAuth= nifi.registry.security.authorizers.configuration.file=./conf/authorizers.xml nifi.registry.security.authorizer=managed-authorizer nifi.registry.security.identity.providers.configuration.file=./conf/identity-providers.xml nifi.registry.security.identity.provider=ldap-provider
... View more
12-22-2019
08:49 AM
Hi @MattWho thanks for the update. I have used listftp , since I don't own the source cant move or delete files . This is causing files to be repulled if nifi service is restarted. I will try list files , hope we don't have this pattern here . Really appreciate your help. Thanks
... View more
12-22-2019
02:58 AM
nifi registry security - able to login with certs but not getting login page when canceling certs
updated login-identity-provider.xml as below not sure what is missing
<identityProviders> <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">xxxxxxxx</property> <property name="Manager Password">>xxxxxxxx</property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://abc:389</property> <property name="User Search Base">OU=managed,DC=ams,DC=xx,DC=com</property> <property name="User Search Filter">sAMAccountName={0}</property> <property name="Identity Strategy">USE_USERNAME</property> <property name="Authentication Expiration">12 hours</property> <property name="Sync Interval">30 secs</property> </provider> </identityProviders>
please let me know what is missing here
... View more
Labels:
- Labels:
-
Apache NiFi
12-19-2019
03:56 PM
which property to use in RouteOnAttribute to filter files older a day after FetchFile from source?
Thanks!
... View more
Labels:
- Labels:
-
Apache NiFi
12-18-2019
10:51 PM
I know we have this issue in nifi 1.5 thought fixed in 1.9 . not sure in which version this is fixed?
Any thoughts? instead of a workaround , interested to know if this is handled in any newer versions?
... View more
Labels:
- Labels:
-
Apache NiFi
01-11-2019
01:59 PM
thanks for the quick response, is there a timeline when the document is available for public?
... View more
01-11-2019
11:34 AM
Hi, I am looking for any document to migrate CM Server from Postgres to Mysql , have anyone performed this? please share for any recommendations
... View more
Labels:
- Labels:
-
Cloudera Manager
11-16-2018
05:34 PM
is there a way to give Hive Temp table create access ONLY in Locked Down Env for a user?
... View more
- Tags:
- Data Processing
- Hive
Labels:
- Labels:
-
Apache Hive
11-16-2018
09:08 AM
We have Production env which is locked down and have read-only access to the user. How to give User to have a HIVE TEMP TABLE CREATE ACCESS ONLY in PROD
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Sentry
10-25-2018
10:20 AM
figured out there are locks on user temp tables, cleared locks and able to drop them. Thanks
... View more
10-17-2018
01:51 PM
Users will automatically be logged out after 'n' seconds of inactivity. any specific number for "n" ?
... View more
10-03-2018
01:50 PM
when I am trying to create a TEMP table getting below error drop table app_fwa.tmpsettime; select * from app_fwa.tmpsettime; create temporary table app_fwa.tmpsettime AS SELECT from_unixtime(unix_timestamp()) as vartime; Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
... View more
Labels:
- Labels:
-
Apache Hive
09-25-2018
10:13 AM
thanks that worked. I have an 18 data nodes each capacity of 150 TB, current data storage of each DN is around 14TB whereas 1 DN is 4 TB. I did disc balancing to equally distribute the data but it didn't help. I am thinking it depends on the math % of storage vs allocated. is there a way i can distribute the data equally now?
... View more
09-24-2018
03:10 PM
even I am getting the same error, I see no other error, trying to run the disc plan to run intra disc balancing error: hdfs diskbalancer -plan lrcdhdn009 18/09/24 17:01:46 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec 18/09/24 17:01:46 INFO block.BlockTokenSecretManager: Setting block keys 18/09/24 17:01:46 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec 18/09/24 17:01:46 ERROR tools.DiskBalancer: java.lang.IllegalArgumentException: Unable to find the specified node. lrcdhdn009
... View more
09-10-2018
10:40 PM
I have 10 files each of size 64GB, a hive table is created on top of these files. data read operations are slow, thinking 64GB file read is the culprit. file format is text can someone check this scenario
... View more
07-17-2018
10:21 AM
Hi, is there a way to hide or use encrypted passwords using curl command, basically i dont want to expose passwords in curl command to use api.. curl -u 'admin:admin' http://cm_host:7180/api/v1/clusters/dev01/services/hdfs2
... View more
07-12-2018
03:23 PM
1 Kudo
Here is the path to add hdfs acls through Ambari HDFS-> Configs -> Advance -> Custom HDFS-site -> Add
property -> add below and save changes dfs.namenode.acls.enabled Restsart all required services
... View more
07-12-2018
10:22 AM
Hi, i could see using API we can keep cluster to MaintenanceMode using below API's curl -u 'admin:admin' -X post 'http:// xxx .xxx.xxx .com:7180/api/v14/clusters/CDH_DEV/commands/enterMaintenanceMode' curl -u 'admin:admin' -X post 'http://xxx.xxx.xxx.com:7180/api/v14/clusters/CDH_DEV/commands/exitMaintenanceMode' Can we keep Cloudera Management Service to MaintenanceMode using APi's? please advice , Thanks!
... View more
06-19-2018
09:33 PM
Ambari doesn't have mechanism to install KDC and deploy kerberos set up like cloudera manager ? Appreciate if someone provide glimpse on this ?
... View more
01-16-2018
09:42 PM
i have two applications, do we need to have two separate variable.registry property files ?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache NiFi
12-26-2017
12:09 PM
Spark2 History server going down with Out of memory.
increased the heap size to 1gb and restarted again same issue.
Appreacite if someone can advice?
part of log:
'[' -n '-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark2_on_yarn_spar1957cecc-SPARK2_YARN_HISTORY_SERVER-e8531d5dc1f494cbe596792e_pid{{PID}}.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh' ']' ++ replace_pid -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/spark2_on_yarn_spar1957cecc-SPARK2_YARN_HISTORY_SERVER-e8531d5dc1f494cbe596792e_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh ++ echo -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/spark2_on_yarn_spar1957cecc-SPARK2_YARN_HISTORY_SERVER-e8531d5dc1f494cbe596792e_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh +
+ exec /opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/spark2/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/10402-spark2_on_yarn-SPARK2_YARN_HISTORY_SERVER/spark2-conf/spark-history-server.conf + grep -q OnOutOfMemoryError /proc/12752/cmdline
Attached full log.
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
-
HDFS
12-26-2017
12:03 PM
seems you have to create the directory as per the log, pls check /user/spark/spark2ApplicationHistory
... View more
12-22-2017
12:34 PM
restarted HIVE meta store and issue got fixed. Now queries are running fine from HUE verified for errors in hive logs but didn't find any.
... View more
12-22-2017
10:03 AM
I am trying to run the simple select query from HUE and beeline getting timed out an error. not sure what could be the reason? select * from table1 limit 10; operation timed out after a minute
... View more
Labels:
- Labels:
-
Apache Hive
12-22-2017
08:39 AM
Hi Folks, if we change umask mode to 002 from 022 will fix this? is there any version dependency? currently, we are in 5.9.0. My question is for the directory, seems we are forcing to give write permissions to group by changing to 002? directory: 022 - rwxr-xr-x 002 - rwxrwxr-w File: 022- rw-r--r-- 002 - rw-rw-r-- This is how much ACL on directory looks like, i have few groups that need only r-x, if we keep 002 will it get changed to rwx for group? hdfs dfs -getfacl /test/ky/val/claim/version=xx/ # file: /test/ky/val/claim/version=xx # owner: svc_ingest_t # group: supergroup user::rwx user:hive:rwx group::rwx group:g_ingt_ba:r-x group:g_ingt_bu:rwx group:g_ingt_du:r-x group:hive:rwx mask::rwx other::--- default:user::rwx default:user:hive:rwx default:group::rwx default:group:g_ingt_ba:r-x default:group:g_ingt_bu:rwx default:group:g_du:r-x default:group:hive:rwx default:mask::rwx default:other::---
... View more