About venkatesh_kanch

venkatesh_kanch · ‎12-28-2019

Nifi build Requirments For Cloud (aws) does nifi build on the cloud should have to follow the same requirements as on-prem dedicated disc or single ebs volume is enough for flow_repo, content_repo, provenence_repo, database_repo n/w - min 10 gb ? I have one instance on the cloud with single ebs, noticed it's going out of sync (nifi 1.9) please advice. @MattWho

venkatesh_kanch · ‎12-27-2019

I have 3 nodes NiFi and 1 node Registry instances running. Registry - SSL enabled and AD-integrated NIfi - SSL enabled and AD-integrated how can I integrate NiFI with Registry, so I can start using Registry. @MattWho please advice for any recommendations

venkatesh_kanch · ‎12-26-2019

@MattWho this really helps, now i am getting login page once after the setting the value to false nifi.registry.security.needClientAuth=false Really Appreciate it. Can you help to configure file-based user authorizations as well in authorizers.xml i hope below part should be enough userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Initial User Identity 1">CN=sys_admin, OU=RegPoc</property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class> <property name="User Group Provider">file-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">CN=sys_admin, OU=RegPoc</property>  </accessPolicyProvider> what should be the node identity here ?

venkatesh_kanch · ‎12-25-2019

do i need to configure authorization.xml to get the login page alone? please advice. Thanks!

venkatesh_kanch · ‎12-25-2019

@Matt yes, updated nifi-reg.properties file nifi.registry.security.identity.provider=ldap-provider but still no luck # security properties # nifi.registry.security.keystore=./conf/keystore.jks nifi.registry.security.keystoreType=jks nifi.registry.security.keystorePasswd=xxxxxxxx nifi.registry.security.keyPasswd=xxxxxx nifi.registry.security.truststore=./conf/truststore.jks nifi.registry.security.truststoreType=jks nifi.registry.security.truststorePasswd=xxxxxxxxxxx nifi.registry.security.user.authorizer=managed-authorizer #nifi.registry.security.user.login.identity.provider=ldap-identity-provider nifi.registry.security.needClientAuth= nifi.registry.security.authorizers.configuration.file=./conf/authorizers.xml nifi.registry.security.authorizer=managed-authorizer nifi.registry.security.identity.providers.configuration.file=./conf/identity-providers.xml nifi.registry.security.identity.provider=ldap-provider

venkatesh_kanch · ‎12-22-2019

Hi @MattWho thanks for the update. I have used listftp , since I don't own the source cant move or delete files . This is causing files to be repulled if nifi service is restarted. I will try list files , hope we don't have this pattern here . Really appreciate your help. Thanks

venkatesh_kanch · ‎12-22-2019

nifi registry security - able to login with certs but not getting login page when canceling certs updated login-identity-provider.xml as below not sure what is missing <identityProviders> <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">xxxxxxxx</property> <property name="Manager Password">>xxxxxxxx</property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://abc:389</property> <property name="User Search Base">OU=managed,DC=ams,DC=xx,DC=com</property> <property name="User Search Filter">sAMAccountName={0}</property> <property name="Identity Strategy">USE_USERNAME</property> <property name="Authentication Expiration">12 hours</property> <property name="Sync Interval">30 secs</property> </provider> </identityProviders> please let me know what is missing here

venkatesh_kanch · ‎12-19-2019

which property to use in RouteOnAttribute to filter files older a day after FetchFile from source? Thanks!

venkatesh_kanch · ‎12-18-2019

I know we have this issue in nifi 1.5 thought fixed in 1.9 . not sure in which version this is fixed? Any thoughts? instead of a workaround , interested to know if this is handled in any newer versions?

venkatesh_kanch · ‎07-18-2019

How to achieve distributed copying of files from HDFS to Azure Blob Storage using NIFI Remote Processor group

venkatesh_kanch · ‎01-31-2019

Umask 002 hdfs folder created with Recursivlely 770 and underlying folders are inherting 770 and files permissions are set to rw-rw---- which is expected but when checking ACL on files its showing errors effectve can someone check below example Directory files and acls inherited correctly drwxrwx---+ - svc_ingest_t supergroup 0 2019-01-28 19:31 /test/abc/dlk/data/abc/abcd/tmp_umask_002 /test/abc/dlk/data/abc/abcd/tmp_umask_002 # owner: svc_ingest_t # group: supergroup user::rwx user:hive:rwx group::rwx group:g_app_cdht_ingt_ba:r-x group:g_app_cdht_ingt_bu:rwx group:g_app_cdht_ingt_du:r-x group:hive:rwx mask::rwx other::--- default:user::rwx default:user:hive:rwx default:group::rwx default:group:g_app_cdht_ingt_ba:r-x default:group:g_app_cdht_ingt_bu:rwx default:group:g_app_cdht_ingt_du:r-x default:group:hive:rwx default:mask::rwx default:other::--- File ACLs not inherited correctly -rw-rw----+ 3 svc_ingest_t supergroup 13 2019-01-28 19:31 /test/abc/dlk/data/abc/acas/tmp_umask_002/tmp_umask_002.txt#123456_654321 # owner: svc_ingest_t # group: supergroup user::rw- user:hive:rwx #effective:rw- group::rwx #effective:rw- group:g_app_cdht_ingt_ba:r-x #effective:r-- group:g_app_cdht_ingt_bu:rwx #effective:rw- group:g_app_cdht_ingt_du:r-x #effective:r-- group:hive:rwx #effective:rw- mask::rw- other::---

venkatesh_kanch · ‎01-11-2019

thanks for the quick response, is there a timeline when the document is available for public?

venkatesh_kanch · ‎01-11-2019

Hi, I am looking for any document to migrate CM Server from Postgres to Mysql , have anyone performed this? please share for any recommendations

venkatesh_kanch · ‎11-16-2018

is there a way to give Hive Temp table create access ONLY in Locked Down Env for a user?

venkatesh_kanch · ‎11-16-2018

Please check DB name and make sure Table is not existed already

venkatesh_kanch · ‎11-16-2018

We have Production env which is locked down and have read-only access to the user. How to give User to have a HIVE TEMP TABLE CREATE ACCESS ONLY in PROD

venkatesh_kanch · ‎10-25-2018

figured out there are locks on user temp tables, cleared locks and able to drop them. Thanks

venkatesh_kanch · ‎10-24-2018

Can someone share repo url for Ambari & HDF stack? OS version RHEL 6 Ambari - 2.6.5 - Repo url HDF - 3.2 -- Repo url i am not able to find repo urls for RHEL 6

venkatesh_kanch · ‎10-24-2018

NiFi - Version 1.2.0.3.0.2.0-76 we see puthdfs fails few times in the nifi cluster, but we have designed flows having retry module and processing the files. curious to know if there is any workaround for a fix . nifi-app_2018-10-23_11.0.log.gz:2018-10-23 11:10:13,045 ERROR [Timer-Driven Process Thread-67] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=6ad95325-fc20-3985-8f8d-df9464fcd032] Failed to write to HDFS due to org.apache.nifi.processor.exception.ProcessException: IOException thrown from PutHDFS[id=6ad95325-fc20-3985-8f8d-df9464fcd032]: org.apache.hadoop.ipc.RemoteException(java.io.IOException): BP-1112006255-10.30.144.36-1488657406341:blk_1084125753_10392458 does not exist or is not under Constructionnull nifi-app_2018-10-23_11.0.log.gz:org.apache.nifi.processor.exception.ProcessException: IOException thrown from PutHDFS[id=6ad95325-fc20-3985-8f8d-df9464fcd032]: org.apache.hadoop.ipc.RemoteException(java.io.IOException): BP-1112006255-10.30.144.36-1488657406341:blk_1084125753_10392458 does not exist or is not under Constructionnull

venkatesh_kanch · ‎10-17-2018

Users will automatically be logged out after 'n' seconds of inactivity. any specific number for "n" ?

venkatesh_kanch · ‎10-03-2018

when I am trying to create a TEMP table getting below error drop table app_fwa.tmpsettime; select * from app_fwa.tmpsettime; create temporary table app_fwa.tmpsettime AS SELECT from_unixtime(unix_timestamp()) as vartime; Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

venkatesh_kanch · ‎09-25-2018

thanks that worked. I have an 18 data nodes each capacity of 150 TB, current data storage of each DN is around 14TB whereas 1 DN is 4 TB. I did disc balancing to equally distribute the data but it didn't help. I am thinking it depends on the math % of storage vs allocated. is there a way i can distribute the data equally now?

venkatesh_kanch · ‎09-24-2018

even I am getting the same error, I see no other error, trying to run the disc plan to run intra disc balancing error: hdfs diskbalancer -plan lrcdhdn009 18/09/24 17:01:46 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec 18/09/24 17:01:46 INFO block.BlockTokenSecretManager: Setting block keys 18/09/24 17:01:46 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec 18/09/24 17:01:46 ERROR tools.DiskBalancer: java.lang.IllegalArgumentException: Unable to find the specified node. lrcdhdn009

venkatesh_kanch · ‎09-10-2018

I have 10 files each of size 64GB, a hive table is created on top of these files. data read operations are slow, thinking 64GB file read is the culprit. file format is text can someone check this scenario

venkatesh_kanch · ‎07-17-2018

Hi, is there a way to hide or use encrypted passwords using curl command, basically i dont want to expose passwords in curl command to use api.. curl -u 'admin:admin' http://cm_host:7180/api/v1/clusters/dev01/services/hdfs2

venkatesh_kanch · ‎07-12-2018

Here is the path to add hdfs acls through Ambari HDFS-> Configs -> Advance -> Custom HDFS-site -> Add property -> add below and save changes dfs.namenode.acls.enabled Restsart all required services

venkatesh_kanch · ‎07-12-2018

Thanks for the update .

venkatesh_kanch · ‎07-12-2018

Hi, i could see using API we can keep cluster to MaintenanceMode using below API's curl -u 'admin:admin' -X post 'http:// xxx .xxx.xxx .com:7180/api/v14/clusters/CDH_DEV/commands/enterMaintenanceMode' curl -u 'admin:admin' -X post 'http://xxx.xxx.xxx.com:7180/api/v14/clusters/CDH_DEV/commands/exitMaintenanceMode' Can we keep Cloudera Management Service to MaintenanceMode using APi's? please advice , Thanks!

venkatesh_kanch · ‎06-19-2018

Ambari doesn't have mechanism to install KDC and deploy kerberos set up like cloudera manager ? Appreciate if someone provide glimpse on this ?

venkatesh_kanch · ‎01-24-2018

Nifi Upgrade, register a version error from local repo Hello All, When i am trying to register a version not able to SAVE and proceed further Amabri -> Admin-> stacks&versions-> Manage versions-> Register version -> selected HDF3.0 , use local repo , rhel6 (given repo url) then not able to save and proceed further? @Matt Clarke

Online	Offline
Last Visited	‎03-06-2020 07:19 PM

Date Registered	‎10-04-2017 08:13 AM
Last Visited	‎03-06-2020 07:19 PM
Total Messages Posted	43
Total Kudos Received	1

Re: Query long running HUE

Nifi build Requirments For Cloud (aws)

NIfi and Nifi Registry Integration

Re: nifi registry security - able to login with ce...

Re: nifi registry security - able to login with ce...

Re: nifi registry security - able to login with ce...

Re: Nifi RouteOnAttribute - filter files older tha...

nifi registry security - able to login with certs ...

Nifi RouteOnAttribute - filter files older than a ...

HDF 3.4.1 Nifi 1.9 listftp and fetchftp - pulls f...

How to achieve distributed copying of files from H...

hdfs ACL inheritance

Re: Postgres to Mysql migration

Postgres to Mysql migration

Hive Temp table create access Only in Locked Down ...

Re: Error while running query on HIVE;

How to give User to have HIVE TEMP TABLE CREATE AC...

Re: unable to create TEMP TABLES in HIVE via HUE

Ambari, HDF upgrade

Nifi PutHdfs Auth issues

Re: Hue Session Timeout

unable to create TEMP TABLES in HIVE via HUE

Re: Error trying to balance disks on node

Re: Error trying to balance disks on node

Re: Hive Multiple Small Files

Re: Maintenance Mode notifications

Re: Where to add dfs.namenode.acls.enabled for ACL...

Re: Maintenance Mode notifications

Re: Maintenance Mode notifications

Re: Kerberos Setup on HDP 2.6

Nifi Upgrade, register a version error from local ...