Re: Query long running HUE

venkatesh_kanch · ‎12-27-2019

I have 3 nodes NiFi and 1 node Registry instances running. Registry - SSL enabled and AD-integrated NIfi - SSL enabled and AD-integrated how can I integrate NiFI with Registry, so I can start using Registry. @MattWho please advice for any recommendations

venkatesh_kanch · ‎12-26-2019

@MattWho this really helps, now i am getting login page once after the setting the value to false nifi.registry.security.needClientAuth=false Really Appreciate it. Can you help to configure file-based user authorizations as well in authorizers.xml i hope below part should be enough userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Initial User Identity 1">CN=sys_admin, OU=RegPoc</property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class> <property name="User Group Provider">file-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">CN=sys_admin, OU=RegPoc</property>  </accessPolicyProvider> what should be the node identity here ?

venkatesh_kanch · ‎12-25-2019

do i need to configure authorization.xml to get the login page alone? please advice. Thanks!

venkatesh_kanch · ‎12-25-2019

@Matt yes, updated nifi-reg.properties file nifi.registry.security.identity.provider=ldap-provider but still no luck # security properties # nifi.registry.security.keystore=./conf/keystore.jks nifi.registry.security.keystoreType=jks nifi.registry.security.keystorePasswd=xxxxxxxx nifi.registry.security.keyPasswd=xxxxxx nifi.registry.security.truststore=./conf/truststore.jks nifi.registry.security.truststoreType=jks nifi.registry.security.truststorePasswd=xxxxxxxxxxx nifi.registry.security.user.authorizer=managed-authorizer #nifi.registry.security.user.login.identity.provider=ldap-identity-provider nifi.registry.security.needClientAuth= nifi.registry.security.authorizers.configuration.file=./conf/authorizers.xml nifi.registry.security.authorizer=managed-authorizer nifi.registry.security.identity.providers.configuration.file=./conf/identity-providers.xml nifi.registry.security.identity.provider=ldap-provider

venkatesh_kanch · ‎12-22-2019

Hi @MattWho thanks for the update. I have used listftp , since I don't own the source cant move or delete files . This is causing files to be repulled if nifi service is restarted. I will try list files , hope we don't have this pattern here . Really appreciate your help. Thanks

venkatesh_kanch · ‎12-22-2019

nifi registry security - able to login with certs but not getting login page when canceling certs updated login-identity-provider.xml as below not sure what is missing <identityProviders> <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">xxxxxxxx</property> <property name="Manager Password">>xxxxxxxx</property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://abc:389</property> <property name="User Search Base">OU=managed,DC=ams,DC=xx,DC=com</property> <property name="User Search Filter">sAMAccountName={0}</property> <property name="Identity Strategy">USE_USERNAME</property> <property name="Authentication Expiration">12 hours</property> <property name="Sync Interval">30 secs</property> </provider> </identityProviders> please let me know what is missing here

venkatesh_kanch · ‎12-19-2019

which property to use in RouteOnAttribute to filter files older a day after FetchFile from source? Thanks!

venkatesh_kanch · ‎12-18-2019

I know we have this issue in nifi 1.5 thought fixed in 1.9 . not sure in which version this is fixed? Any thoughts? instead of a workaround , interested to know if this is handled in any newer versions?

venkatesh_kanch · ‎01-11-2019

thanks for the quick response, is there a timeline when the document is available for public?

venkatesh_kanch · ‎01-11-2019

Hi, I am looking for any document to migrate CM Server from Postgres to Mysql , have anyone performed this? please share for any recommendations

venkatesh_kanch · ‎11-16-2018

is there a way to give Hive Temp table create access ONLY in Locked Down Env for a user?

venkatesh_kanch · ‎11-16-2018

Please check DB name and make sure Table is not existed already

venkatesh_kanch · ‎11-16-2018

We have Production env which is locked down and have read-only access to the user. How to give User to have a HIVE TEMP TABLE CREATE ACCESS ONLY in PROD

venkatesh_kanch · ‎10-25-2018

figured out there are locks on user temp tables, cleared locks and able to drop them. Thanks

venkatesh_kanch · ‎10-17-2018

Users will automatically be logged out after 'n' seconds of inactivity. any specific number for "n" ?

venkatesh_kanch · ‎10-03-2018

when I am trying to create a TEMP table getting below error drop table app_fwa.tmpsettime; select * from app_fwa.tmpsettime; create temporary table app_fwa.tmpsettime AS SELECT from_unixtime(unix_timestamp()) as vartime; Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

venkatesh_kanch · ‎09-25-2018

thanks that worked. I have an 18 data nodes each capacity of 150 TB, current data storage of each DN is around 14TB whereas 1 DN is 4 TB. I did disc balancing to equally distribute the data but it didn't help. I am thinking it depends on the math % of storage vs allocated. is there a way i can distribute the data equally now?

venkatesh_kanch · ‎09-24-2018

even I am getting the same error, I see no other error, trying to run the disc plan to run intra disc balancing error: hdfs diskbalancer -plan lrcdhdn009 18/09/24 17:01:46 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec 18/09/24 17:01:46 INFO block.BlockTokenSecretManager: Setting block keys 18/09/24 17:01:46 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec 18/09/24 17:01:46 ERROR tools.DiskBalancer: java.lang.IllegalArgumentException: Unable to find the specified node. lrcdhdn009

venkatesh_kanch · ‎09-10-2018

I have 10 files each of size 64GB, a hive table is created on top of these files. data read operations are slow, thinking 64GB file read is the culprit. file format is text can someone check this scenario

venkatesh_kanch · ‎07-17-2018

Hi, is there a way to hide or use encrypted passwords using curl command, basically i dont want to expose passwords in curl command to use api.. curl -u 'admin:admin' http://cm_host:7180/api/v1/clusters/dev01/services/hdfs2

venkatesh_kanch · ‎07-12-2018

Here is the path to add hdfs acls through Ambari HDFS-> Configs -> Advance -> Custom HDFS-site -> Add property -> add below and save changes dfs.namenode.acls.enabled Restsart all required services

venkatesh_kanch · ‎07-12-2018

Thanks for the update .

venkatesh_kanch · ‎07-12-2018

Hi, i could see using API we can keep cluster to MaintenanceMode using below API's curl -u 'admin:admin' -X post 'http:// xxx .xxx.xxx .com:7180/api/v14/clusters/CDH_DEV/commands/enterMaintenanceMode' curl -u 'admin:admin' -X post 'http://xxx.xxx.xxx.com:7180/api/v14/clusters/CDH_DEV/commands/exitMaintenanceMode' Can we keep Cloudera Management Service to MaintenanceMode using APi's? please advice , Thanks!

venkatesh_kanch · ‎06-19-2018

Ambari doesn't have mechanism to install KDC and deploy kerberos set up like cloudera manager ? Appreciate if someone provide glimpse on this ?

venkatesh_kanch · ‎01-16-2018

i have two applications, do we need to have two separate variable.registry property files ?

venkatesh_kanch · ‎12-26-2017

Spark2 History server going down with Out of memory. increased the heap size to 1gb and restarted again same issue. Appreacite if someone can advice? part of log: '[' -n '-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark2_on_yarn_spar1957cecc-SPARK2_YARN_HISTORY_SERVER-e8531d5dc1f494cbe596792e_pid{{PID}}.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh' ']' ++ replace_pid -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/spark2_on_yarn_spar1957cecc-SPARK2_YARN_HISTORY_SERVER-e8531d5dc1f494cbe596792e_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh ++ echo -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/spark2_on_yarn_spar1957cecc-SPARK2_YARN_HISTORY_SERVER-e8531d5dc1f494cbe596792e_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh + + exec /opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/spark2/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/10402-spark2_on_yarn-SPARK2_YARN_HISTORY_SERVER/spark2-conf/spark-history-server.conf + grep -q OnOutOfMemoryError /proc/12752/cmdline Attached full log.

venkatesh_kanch · ‎12-26-2017

seems you have to create the directory as per the log, pls check /user/spark/spark2ApplicationHistory

venkatesh_kanch · ‎12-22-2017

restarted HIVE meta store and issue got fixed. Now queries are running fine from HUE verified for errors in hive logs but didn't find any.

venkatesh_kanch · ‎12-22-2017

I am trying to run the simple select query from HUE and beeline getting timed out an error. not sure what could be the reason? select * from table1 limit 10; operation timed out after a minute

venkatesh_kanch · ‎12-22-2017

Hi Folks, if we change umask mode to 002 from 022 will fix this? is there any version dependency? currently, we are in 5.9.0. My question is for the directory, seems we are forcing to give write permissions to group by changing to 002? directory: 022 - rwxr-xr-x 002 - rwxrwxr-w File: 022- rw-r--r-- 002 - rw-rw-r-- This is how much ACL on directory looks like, i have few groups that need only r-x, if we keep 002 will it get changed to rwx for group? hdfs dfs -getfacl /test/ky/val/claim/version=xx/ # file: /test/ky/val/claim/version=xx # owner: svc_ingest_t # group: supergroup user::rwx user:hive:rwx group::rwx group:g_ingt_ba:r-x group:g_ingt_bu:rwx group:g_ingt_du:r-x group:hive:rwx mask::rwx other::--- default:user::rwx default:user:hive:rwx default:group::rwx default:group:g_ingt_ba:r-x default:group:g_ingt_bu:rwx default:group:g_du:r-x default:group:hive:rwx default:mask::rwx default:other::---

Online	Offline
Last Visited	‎03-06-2020 07:19 PM

Date Registered	‎10-04-2017 08:13 AM
Last Visited	‎03-06-2020 07:19 PM
Total Messages Posted	43
Total Kudos Received	1

Re: Query long running HUE

NIfi and Nifi Registry Integration

Re: nifi registry security - able to login with ce...

Re: nifi registry security - able to login with ce...

Re: nifi registry security - able to login with ce...

Re: Nifi RouteOnAttribute - filter files older tha...

nifi registry security - able to login with certs ...

Nifi RouteOnAttribute - filter files older than a ...

HDF 3.4.1 Nifi 1.9 listftp and fetchftp - pulls f...

Re: Postgres to Mysql migration

Postgres to Mysql migration

Hive Temp table create access Only in Locked Down ...

Re: Error while running query on HIVE;

How to give User to have HIVE TEMP TABLE CREATE AC...

Re: unable to create TEMP TABLES in HIVE via HUE

Re: Hue Session Timeout

unable to create TEMP TABLES in HIVE via HUE

Re: Error trying to balance disks on node

Re: Error trying to balance disks on node

Re: Hive Multiple Small Files

Re: Maintenance Mode notifications

Re: Where to add dfs.namenode.acls.enabled for ACL...

Re: Maintenance Mode notifications

Re: Maintenance Mode notifications

Re: Kerberos Setup on HDP 2.6

Can we have multiple nifi.variable.registry.proper...

Spark2 History server going down with Out of memor...

Re: Spark2 History Server failed to start in CDH5....

Re: Query long running HUE

Query long running HUE timed out

Re: HDFS ACL Inheritance doesn't work