Member since
09-25-2015
109
Posts
36
Kudos Received
8
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
771 | 04-03-2018 09:08 PM | |
1202 | 03-14-2018 04:01 PM | |
3322 | 03-14-2018 03:22 PM | |
809 | 10-30-2017 04:29 PM | |
464 | 10-17-2017 04:49 PM |
04-03-2018
09:12 PM
1 Kudo
hi @Aishwarya Dixit you can gracefully shutdown the region server, that will trigger hbase Mater to perform a bulk assignment of all regions hosted by that region server.
... View more
04-03-2018
09:10 PM
Hi @Anurag Mishra please accept the answer if it resolved your issue.
... View more
04-03-2018
09:08 PM
1 Kudo
Hi @Venkata Sudheer Kumar M CPU is capable of running multiple containers if the jobs are not cpu intensive. The stack advisor is only recommending to not go beyond "CPU(s) * 2". However, there is nothing stopping you from configuring higher. if you observe your container concurrency metrics and CPU utilization, you can identify your threshold of vcores to 1 CPU and set it accordingly. Note: yarn.nodemanager.resource.cpu-vcores would only be applicable if you enable CPU Scheduling.
... View more
03-19-2018
09:01 PM
Hi @Anurag Mishra You can use value of 'yarn.resourcemanager.cluster-id' in jobTracker. # grep -A1 'yarn.resourcemanager.cluster-id' /etc/hadoop/conf/*
jobTracker=yarn-cluster However, "Failing over to rm2" is just a "INFO" message , that indicates rm1 is Standby. Your issue with oozie spark2 action would be different.
... View more
03-19-2018
08:43 PM
Hi @Sergey Sheypak You may require hotfix on HDP with OOZIE-2606 OOZIE-2658 OOZIE-2787 OOZIE-2802. Please open a support case.
... View more
03-19-2018
08:41 PM
Hi @Bijay Deo You may require hotfix on HDP with OOZIE-2606 OOZIE-2658 OOZIE-2787 OOZIE-2802. Please open a support case.
... View more
03-19-2018
08:33 PM
Hi @Gagandeep Singh Chawla Did the above response help?
... View more
03-19-2018
06:36 PM
Hi @Aishwarya Dixit Did it work ? You can always Shutdown the Region Server process, and Hbase Master will reassign all Regions to a different Region Server host.
... View more
03-14-2018
04:08 PM
Please accept an Answer, so that we can mark this request close.
... View more
03-14-2018
04:01 PM
Do you have the YARN Resource Manager screenshot when you run 3 mapred job? http://<Active_RM_HOST>:8088/cluster/scheduler From what i read from screen shots, Maximum AM Resource 20% i.e. 20% of 391GB = 78GB. value of: yarn.app.mapreduce.am.resource.mb and tez.am.resource.memory.mb will determine how many AMs can fit in to run concurrently.
... View more
03-14-2018
03:42 PM
Hi @Alpesh Virani Can you also share the Resource Manager UI screenshot. This will tell what is the actual usage for your queue. http://<Active_RM_HOST>:8088/cluster/scheduler
... View more
03-14-2018
03:33 PM
Hi @Alpesh Virani There are several possibilities. When you have multiple hive sessions open with execution engine "mr", can you tell us: 1. How much resources you have used / available to run in "default" queue ? Check this on YARN RM UI > Scheduler 2. If you have 100% available in "default" queue, check the "am" container size "yarn.app.mapreduce.am.resource.mb" and check the "Maximum AM Resource" for "default" queue, see if queue has enough resources to run multiple "am" "mr" containers.
... View more
03-14-2018
03:22 PM
1 Kudo
Hi @Aishwarya Dixit You can use the command move in hbase shell. Examples:
hbase> move 'ENCODED_REGIONNAME'
hbase> move 'ENCODED_REGIONNAME', 'SERVER_NAME'
Here is some help for this command:
Move a region. Optionally specify target regionserver else we choose one at random.
NOTE: You pass the encoded region name, not the region name so this command is a little different to the others.
The encoded region name is the hash suffix on region names:
e.g. if the region name were
TestTable,0094429456,1289497600452.527db22f95c8a9e0116f0cc13c680396.
then the encoded region name portion is 527db22f95c8a9e0116f0cc13c680396
A server name is its host, port plus startcode.
For example: host187.example.com,60020,1289493121758
... View more
03-14-2018
03:11 PM
Hi @heta desai, Each service needs to be configured to use LDAP Server for login.(doc links below)
Once configured,
* For ambari-server, you need to run the following command periodically to sync up users and groups. /usr/sbin/ambari-server sync-ldap --ldap-sync-admin-name=admin --ldap-sync-admin-password=admin --all * For Ranger, user can login once setup for ldap authentication For both services, you need to grant respective privileges to user / groups, on what each can do upon login. Documentation Links: Ambari Server:
HORTONWORKS DOCS » APACHE AMBARI 2.6.1.5 » APACHE AMBARI ADMINISTRATION > Chapter 3. Managing Users and Groups > Local and LDAP User and Group Types: Configure Ambari to use LDAP Server https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-security/content/configure_ambari_to_use_ldap_server.html Synchronizing LDAP Users and Groups https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-security/content/synchronizing_ldap_users_and_groups.html Specific Set of Users and Groups https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-security/content/specific_set_of_users_and_groups.html Existing Users and Groups https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-security/content/existing_users_and_groups.html All Users and Groups https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.1.5/bk_ambari-security/content/all_users_and_groups.html Ranger:
HORTONWORKS DOCS » DATA PLATFORM 2.6.4 » SECURITY Chapter 2. Authentication
Configuring Ranger Authentication with UNIX, LDAP, or AD https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_security/content/_configuring_ambari_for_ldap_or_active_directory_authentication.html
... View more
03-13-2018
03:15 PM
Hi @Gagandeep Singh Chawla On Ambari Managed Cluster, Ambari > Oozie > configs > custom oozie-site > oozie.processing.timezone=GMT+0100 Reference: https://community.hortonworks.com/articles/73830/how-to-change-oozie-timezone-from-gmt-to-est.html
... View more
03-13-2018
03:08 PM
Hi @Swetha Nelwad Appears that you have not yet mapped existing hbase table to a phoenix table. Please follow the following guide to map hbase table in phoenix. https://phoenix.apache.org/faq.html#How_I_map_Phoenix_table_to_an_existing_HBase_table
... View more
03-13-2018
02:58 PM
Hi @Naveen Y Depending how you have provisioned your cluster, if it is ambari managed, could you please review the following article by @Kuldeep Kulkarni and see if this serves your purpose. https://community.hortonworks.com/articles/34747/ambari-admin-utility.html
... View more
03-13-2018
02:53 PM
1 Kudo
hi @Narasimha K Looks like rmhost2 is in standby. could you please check if the connection to rmhost2 is working without knox. i.e. does the url get auto redirected to Active rmhost2 when you connect against standby RM i.e. rmhost1.
Also, can you dump the output when rmhost1 is active resoucemanger. Need to check if curl -ivk -u narasik "http://knoxhost:8443/knox/topology1/resourcemanager/v1/cluster"
... View more
02-27-2018
07:04 PM
1 Kudo
Repo Description Background: To run Hadoop distcp command on a Cluster with NameNode High Availability (HA) enabled, the following is required: * Adding of nameservice information of both Source and destination cluster * Restarting of the services. The reason being that YARN ResourceManager renews delegation tokens for applications. Solution: To avoid server side configuration, the MapReduce jobs can send the configurations to RM at runtime and RM uses these configurations to renew tokens via mapreduce.job.send-token-conf We can leverage the same via Oozie Distcp Action. Git Repo contains Oozie distcp Action template that would allow basic oozie distcp action on a Kerberos environment and help parameterize on runtime. This way end users can run at their schedule.
job.properties workflow.xml Repo Info Github Repo URL https://github.com/saumilmayani/oozie-distcp_template.git Github account name saumilmayani Repo name oozie-distcp_template.git
... View more
- Find more articles tagged with:
- Distcp
- Oozie
- sample-aps
- solutions
Labels:
02-15-2018
02:50 PM
Hi @gsharma please reflect the title How to delete a Phoenix Table (created on Top of Existing Hbase table), Without Dropping HBase Table
... View more
11-14-2017
01:48 PM
Hi @Priyaranjan Swain Is this Kerberos enabled via Existing MIT KDC or Existing Active Directory? Please check if the principal is locked in your MIT KDC or Active Directory. Also as mention earlier, please check the /etc/krb5.conf for any issues in this file with respect to KDC information. You may try "Regenerating Keytabs" via Ambari-server and see if it helps.
... View more
11-14-2017
01:47 PM
Hi @Priyaranjan Swain Is this Kerberos enabled via Existing MIT KDC or Existing Active Directory? Please check if the principal is locked in your MIT KDC or Active Directory. Also as mention earlier, please check the /etc/krb5.conf for any issues in this file with respect to KDC information. You may try "Regenerating Keytabs" via Ambari-server and see if it helps.
... View more
11-13-2017
10:55 PM
Hi @sudi ts Create a "Config Group" under YARN Service. Note, just increasing RAM capacity might not be enough. You may need additional CPUs to handle more jobs / containers. yarn-site.xml
yarn.nodemanager.resource.memory-mb
yarn.nodemanager.resource.cpu-vcores
Add Node Manager Host to this config group Restart Respective Nodemanagers associated to this config group.
... View more
11-13-2017
10:40 PM
Hi @Priyaranjan Swain Try running following command on host where you have yarn resource Manager "RM" installed. /usr/bin/kinit -kt /etc/security/keytabs/rm.service.keytab rm/ip-192-168-0-50.eu-west-1.compute.internal@TECHNIPFMC.COM; If this does not return the following error, then try restarting service again via Ambari. returned 1. kinit: Preauthentication failed while getting initial credentials If returns above error, check the /etc/krb5.conf to check for any issues here. or in case of KDC Load balancer, see if the principals are replicated across all servers serving under this KDC. [realms]
TECHNIPFMC.COM = {
admin_server = <FQDN>
kdc = <FQDN>
}
... View more
10-30-2017
07:07 PM
Hi @deepak rathod yes, you are using HDP-2.3.2.0. You need to upgrade to HDP-2.6.2.0. Here is the doc: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.2/index.html https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-upgrade/content/ambari_upgrade_guide.html
... View more
10-30-2017
06:35 PM
1 Kudo
Hi @deepak rathod appears that the hdp version you are running, does not have this information on YARN Resource Manager UI. Sample screenshot I attached earlier was from HDP-2.6.2.0-205. You may need to upgrade HDP stack.
... View more
10-30-2017
06:24 PM
Hi @Pooja Kamle following error indicates that hbase user does not have correct permissions. Looks like you have ranger enabled. Can you please check if you have correct privileges for hbase user. ERROR [RS_OPEN_PRIORITY_REGION-hostname:16020-1] handler.OpenRegionHandler: Failed open of region=hbase:namespace,,1508913064554.16ee288e7e2f92b959283a91a2205c93., starting to roll back the global memstore size. org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'hbase/principal' (action=admin Also make sure following property is set correctly: Ambari Web UI > hbase > config > Advanced hbase-site >hbase.superuser=hbase
For Ranger Policy: Sample Screenshot
... View more
10-30-2017
06:14 PM
Hi @deepak rathod Could you please share the screen capture of the scheduler page. It should have for each application, Allocated Memory and vCores, allong with Running Containers. you can sort and see which application has the most Allocated Memory MB. Sample attached.
... View more
10-30-2017
04:29 PM
1 Kudo
Hi @deepak rathod Click on the Scheduler link on the Resource Manager page. It will tell you what is the resource utilization for each queue and and you can drill down to identify which jobs are consuming total of 279 containers ? Appears that you have 279 containers running with avg. of 12 GB , with total of 3.41 TB memory reserved.
... View more
10-17-2017
06:22 PM
Is this ORC data? For Application Master to run, you can set the following property and then run the query. This should reduce the memory pressure on AM, trying to figure out how many mappers , going through all hdfs files. set hive.exec.orc.split.strategy=BI; How many partitions would you have in total for 14TB data? Notes: hive.exec.orc.split.strategy
Default Value: HYBRID
Added In: Hive 1.2.0 with HIVE-10114
What strategy ORC should use to create splits for execution. The available options are "BI", "ETL" and "HYBRID".<br>
The HYBRID mode reads the footers for all files if there are fewer files than expected mapper count, switching over to generating 1 split per file if the average file sizes are smaller than the default HDFS blocksize. ETL strategy always reads the ORC footers before generating splits, while the BI strategy generates per-file splits fast without reading any data from HDFS. Other Article on how mappers are determined: https://community.hortonworks.com/questions/905/how-are-number-of-mappers-determined-for-a-query-w.html
... View more