About venkatesh_kanch

venkatesh_kanch · ‎07-18-2019

How to achieve distributed copying of files from HDFS to Azure Blob Storage using NIFI Remote Processor group

venkatesh_kanch · ‎01-31-2019

Umask 002 hdfs folder created with Recursivlely 770 and underlying folders are inherting 770 and files permissions are set to rw-rw---- which is expected but when checking ACL on files its showing errors effectve can someone check below example Directory files and acls inherited correctly drwxrwx---+ - svc_ingest_t supergroup 0 2019-01-28 19:31 /test/abc/dlk/data/abc/abcd/tmp_umask_002 /test/abc/dlk/data/abc/abcd/tmp_umask_002 # owner: svc_ingest_t # group: supergroup user::rwx user:hive:rwx group::rwx group:g_app_cdht_ingt_ba:r-x group:g_app_cdht_ingt_bu:rwx group:g_app_cdht_ingt_du:r-x group:hive:rwx mask::rwx other::--- default:user::rwx default:user:hive:rwx default:group::rwx default:group:g_app_cdht_ingt_ba:r-x default:group:g_app_cdht_ingt_bu:rwx default:group:g_app_cdht_ingt_du:r-x default:group:hive:rwx default:mask::rwx default:other::--- File ACLs not inherited correctly -rw-rw----+ 3 svc_ingest_t supergroup 13 2019-01-28 19:31 /test/abc/dlk/data/abc/acas/tmp_umask_002/tmp_umask_002.txt#123456_654321 # owner: svc_ingest_t # group: supergroup user::rw- user:hive:rwx #effective:rw- group::rwx #effective:rw- group:g_app_cdht_ingt_ba:r-x #effective:r-- group:g_app_cdht_ingt_bu:rwx #effective:rw- group:g_app_cdht_ingt_du:r-x #effective:r-- group:hive:rwx #effective:rw- mask::rw- other::---

venkatesh_kanch · ‎01-11-2019

thanks for the quick response, is there a timeline when the document is available for public?

venkatesh_kanch · ‎01-11-2019

Hi, I am looking for any document to migrate CM Server from Postgres to Mysql , have anyone performed this? please share for any recommendations

venkatesh_kanch · ‎11-16-2018

is there a way to give Hive Temp table create access ONLY in Locked Down Env for a user?

venkatesh_kanch · ‎11-16-2018

Please check DB name and make sure Table is not existed already

venkatesh_kanch · ‎11-16-2018

We have Production env which is locked down and have read-only access to the user. How to give User to have a HIVE TEMP TABLE CREATE ACCESS ONLY in PROD

venkatesh_kanch · ‎10-25-2018

figured out there are locks on user temp tables, cleared locks and able to drop them. Thanks

venkatesh_kanch · ‎10-24-2018

Can someone share repo url for Ambari & HDF stack? OS version RHEL 6 Ambari - 2.6.5 - Repo url HDF - 3.2 -- Repo url i am not able to find repo urls for RHEL 6

venkatesh_kanch · ‎10-24-2018

NiFi - Version 1.2.0.3.0.2.0-76 we see puthdfs fails few times in the nifi cluster, but we have designed flows having retry module and processing the files. curious to know if there is any workaround for a fix . nifi-app_2018-10-23_11.0.log.gz:2018-10-23 11:10:13,045 ERROR [Timer-Driven Process Thread-67] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=6ad95325-fc20-3985-8f8d-df9464fcd032] Failed to write to HDFS due to org.apache.nifi.processor.exception.ProcessException: IOException thrown from PutHDFS[id=6ad95325-fc20-3985-8f8d-df9464fcd032]: org.apache.hadoop.ipc.RemoteException(java.io.IOException): BP-1112006255-10.30.144.36-1488657406341:blk_1084125753_10392458 does not exist or is not under Constructionnull nifi-app_2018-10-23_11.0.log.gz:org.apache.nifi.processor.exception.ProcessException: IOException thrown from PutHDFS[id=6ad95325-fc20-3985-8f8d-df9464fcd032]: org.apache.hadoop.ipc.RemoteException(java.io.IOException): BP-1112006255-10.30.144.36-1488657406341:blk_1084125753_10392458 does not exist or is not under Constructionnull

venkatesh_kanch · ‎10-17-2018

Users will automatically be logged out after 'n' seconds of inactivity. any specific number for "n" ?

venkatesh_kanch · ‎10-03-2018

when I am trying to create a TEMP table getting below error drop table app_fwa.tmpsettime; select * from app_fwa.tmpsettime; create temporary table app_fwa.tmpsettime AS SELECT from_unixtime(unix_timestamp()) as vartime; Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

venkatesh_kanch · ‎09-25-2018

thanks that worked. I have an 18 data nodes each capacity of 150 TB, current data storage of each DN is around 14TB whereas 1 DN is 4 TB. I did disc balancing to equally distribute the data but it didn't help. I am thinking it depends on the math % of storage vs allocated. is there a way i can distribute the data equally now?

venkatesh_kanch · ‎09-24-2018

even I am getting the same error, I see no other error, trying to run the disc plan to run intra disc balancing error: hdfs diskbalancer -plan lrcdhdn009 18/09/24 17:01:46 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec 18/09/24 17:01:46 INFO block.BlockTokenSecretManager: Setting block keys 18/09/24 17:01:46 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec 18/09/24 17:01:46 ERROR tools.DiskBalancer: java.lang.IllegalArgumentException: Unable to find the specified node. lrcdhdn009

venkatesh_kanch · ‎09-10-2018

I have 10 files each of size 64GB, a hive table is created on top of these files. data read operations are slow, thinking 64GB file read is the culprit. file format is text can someone check this scenario

venkatesh_kanch · ‎07-17-2018

Hi, is there a way to hide or use encrypted passwords using curl command, basically i dont want to expose passwords in curl command to use api.. curl -u 'admin:admin' http://cm_host:7180/api/v1/clusters/dev01/services/hdfs2

venkatesh_kanch · ‎07-12-2018

Here is the path to add hdfs acls through Ambari HDFS-> Configs -> Advance -> Custom HDFS-site -> Add property -> add below and save changes dfs.namenode.acls.enabled Restsart all required services

venkatesh_kanch · ‎07-12-2018

Thanks for the update .

venkatesh_kanch · ‎07-12-2018

Hi, i could see using API we can keep cluster to MaintenanceMode using below API's curl -u 'admin:admin' -X post 'http:// xxx .xxx.xxx .com:7180/api/v14/clusters/CDH_DEV/commands/enterMaintenanceMode' curl -u 'admin:admin' -X post 'http://xxx.xxx.xxx.com:7180/api/v14/clusters/CDH_DEV/commands/exitMaintenanceMode' Can we keep Cloudera Management Service to MaintenanceMode using APi's? please advice , Thanks!

venkatesh_kanch · ‎06-19-2018

Ambari doesn't have mechanism to install KDC and deploy kerberos set up like cloudera manager ? Appreciate if someone provide glimpse on this ?

venkatesh_kanch · ‎01-24-2018

Nifi Upgrade, register a version error from local repo Hello All, When i am trying to register a version not able to SAVE and proceed further Amabri -> Admin-> stacks&versions-> Manage versions-> Register version -> selected HDF3.0 , use local repo , rhel6 (given repo url) then not able to save and proceed further? @Matt Clarke

venkatesh_kanch · ‎01-16-2018

i have two applications, do we need to have two separate variable.registry property files ?

venkatesh_kanch · ‎12-27-2017

HDFS ACL Inheritance failing when creating directory with NIFI HDFS is Cloudera with ACL's enabled, anyhow I am trying to create a directory with NIFI then load a file it is not inheriting the parent ACLs? hdfs dfs -getfacl /test/dlk/data/delete_later_umask002_venky1 # file: /test/dlk/data/delete_later_umask002_venky1 # owner: svc_ingest_t # group: supergroup user::rwx user:hive:rwx #effective:r-x group::rwx #effective:r-x group:g_app_cdht_ingt_ba:r-x group:g_app_cdht_ingt_bu:rwx #effective:r-x group:g_app_cdht_ingt_du:r-x group:hive:rwx #effective:r-x mask::r-x other::--- default:user::rwx default:user:hive:rwx default:group::rwx default:group:g_app_cdht_ingt_ba:r-x default:group:g_app_cdht_ingt_bu:rwx default:group:g_app_cdht_ingt_du:r-x default:group:hive:rwx default:mask::rwx default:other::--- any solution?

venkatesh_kanch · ‎12-26-2017

Spark2 History server going down with Out of memory. increased the heap size to 1gb and restarted again same issue. Appreacite if someone can advice? part of log: '[' -n '-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/spark2_on_yarn_spar1957cecc-SPARK2_YARN_HISTORY_SERVER-e8531d5dc1f494cbe596792e_pid{{PID}}.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh' ']' ++ replace_pid -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/spark2_on_yarn_spar1957cecc-SPARK2_YARN_HISTORY_SERVER-e8531d5dc1f494cbe596792e_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh ++ echo -XX:+HeapDumpOnOutOfMemoryError '-XX:HeapDumpPath=/tmp/spark2_on_yarn_spar1957cecc-SPARK2_YARN_HISTORY_SERVER-e8531d5dc1f494cbe596792e_pid{{PID}}.hprof' -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh + + exec /opt/cloudera/parcels/SPARK2-2.0.0.cloudera1-1.cdh5.7.0.p0.113931/lib/spark2/bin/spark-class org.apache.spark.deploy.history.HistoryServer --properties-file /var/run/cloudera-scm-agent/process/10402-spark2_on_yarn-SPARK2_YARN_HISTORY_SERVER/spark2-conf/spark-history-server.conf + grep -q OnOutOfMemoryError /proc/12752/cmdline Attached full log.

venkatesh_kanch · ‎12-26-2017

seems you have to create the directory as per the log, pls check /user/spark/spark2ApplicationHistory

venkatesh_kanch · ‎12-22-2017

restarted HIVE meta store and issue got fixed. Now queries are running fine from HUE verified for errors in hive logs but didn't find any.

venkatesh_kanch · ‎12-22-2017

I am trying to run the simple select query from HUE and beeline getting timed out an error. not sure what could be the reason? select * from table1 limit 10; operation timed out after a minute

venkatesh_kanch · ‎12-22-2017

Hi Folks, if we change umask mode to 002 from 022 will fix this? is there any version dependency? currently, we are in 5.9.0. My question is for the directory, seems we are forcing to give write permissions to group by changing to 002? directory: 022 - rwxr-xr-x 002 - rwxrwxr-w File: 022- rw-r--r-- 002 - rw-rw-r-- This is how much ACL on directory looks like, i have few groups that need only r-x, if we keep 002 will it get changed to rwx for group? hdfs dfs -getfacl /test/ky/val/claim/version=xx/ # file: /test/ky/val/claim/version=xx # owner: svc_ingest_t # group: supergroup user::rwx user:hive:rwx group::rwx group:g_ingt_ba:r-x group:g_ingt_bu:rwx group:g_ingt_du:r-x group:hive:rwx mask::rwx other::--- default:user::rwx default:user:hive:rwx default:group::rwx default:group:g_ingt_ba:r-x default:group:g_ingt_bu:rwx default:group:g_du:r-x default:group:hive:rwx default:mask::rwx default:other::---

venkatesh_kanch · ‎12-05-2017

Hi, i am trying to follow your steps. unfortunately getting error, please look once Step1: i am using to get token curl -X POST --insecure -H \ -H "Accept: */*" \ -H "Accept-Encoding: gzip" \ -H "Accept-Language: en-US,en;q=0.5" \ -H "Connection: keep-alive" \ -H "Content-Length: 38" \ -H "Content-Type: application/x-www-form-urlencoded; charset=UTF-8" \ -H "Host: https://nifitest.com/nifi" \ -H "Referer: https://nifitest.com/nifi/login" \ -H "X-Requested-With: XMLHttpRequest" \ -d 'username=XXXX&password=XXXX' \ https://nifitest.com/nifi-api/access/token | gunzip error: Curl: (6) Couldn't resolve host 'Accept: *' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 34 0 298 --:--:-- --:--:-- --:--:-- 0 gzip: stdin: unexpected end of file

venkatesh_kanch · ‎12-05-2017

Hello All, I am a newbie here, Can you help how to deploy NIFI Template or Process group from DEV to Test via API call? 1) validate token for auth 2) download template or Process groups to shared drive? 3) upload to TEST Appreciate if you can guide me @smanjee

Online	Offline
Last Visited	‎10-17-2019 02:34 PM

Date Registered	‎10-04-2017 08:13 AM
Last Visited	‎10-17-2019 02:34 PM
Total Messages Posted	33
Total Kudos Received	1

Re: Query long running HUE

How to achieve distributed copying of files from H...

hdfs ACL inheritance

Re: Postgres to Mysql migration

Postgres to Mysql migration

Hive Temp table create access Only in Locked Down ...

Re: Error while running query on HIVE;

How to give User to have HIVE TEMP TABLE CREATE AC...

Re: unable to create TEMP TABLES in HIVE via HUE

Ambari, HDF upgrade

Nifi PutHdfs Auth issues

Re: Hue Session Timeout

unable to create TEMP TABLES in HIVE via HUE

Re: Error trying to balance disks on node

Re: Error trying to balance disks on node

Re: Hive Multiple Small Files

Re: Maintenance Mode notifications

Re: Where to add dfs.namenode.acls.enabled for ACL...

Re: Maintenance Mode notifications

Re: Maintenance Mode notifications

Re: Kerberos Setup on HDP 2.6

Nifi Upgrade, register a version error from local ...

Can we have multiple nifi.variable.registry.proper...

HDFS ACL Inheritance failing when creating directo...

Spark2 History server going down with Out of memor...

Re: Spark2 History Server failed to start in CDH5....

Re: Query long running HUE

Query long running HUE timed out

Re: HDFS ACL Inheritance doesn't work

Re: Creating NiFi Template via Rest API

NIFI Deployment Via API's