Member since
04-25-2017
19
Posts
4
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3196 | 07-04-2019 04:16 PM | |
1379 | 06-08-2018 12:13 PM | |
5700 | 05-22-2017 10:24 PM |
11-01-2023
08:07 AM
On onStageCompleted you should have also the failures and the info on why it failed with StageInfo.failureReason() Check on the standard event log JSON if the logger stored that event, if it also does not have it, then it means your listener might have an issue. If you are not really seeing the StageCompleted event anywhere ( not even on the event log), it probably means the failure was more than just a regular error and it had a major error/crash that aborted the execution of that stage in a way that no StageCompleted was ever sent. Also bear in mind that there are specific listeners for SQL execution with fail/success events: QueryExecutionListener .
... View more
01-11-2022
08:11 AM
Many times, Support need to have a look at the Microsoft AD tree for AD internal configuration (e.g. CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration, <domain>). The problem with this part of the AD tree is - it is not visible by default in the AD directory structure.
This article will help the reader to explore the hidden AD tree. Any change in AD should be carried out by the customer’s AD team or admin.
Instructions
Here are the steps:
Log on to the AD server
Run the ADSIEdit.exe program from Start > Run menu.
Select the top-most entry of the left side, and then, select Action > Connect to... from the menu.
In the next dialog box, select Select a well known Naming Context radio button, and select Configuration from the drop-down menu.
Click OK and ADSI Edit will now show CN=Configuration and its sub-tree.
We can now traverse to CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration, <domain> part easily as seen above.
Example: For Microsoft AD vulnerability (CVE-2021-42282), if the user wants to check the value of dSHeuristics attribute under CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration, <domain>, they can follow the steps above, and then, right-click on CN=Directory Service and select Properties > Attribute Editor and find the value of dSHeuristics attribute.
... View more
Labels:
07-05-2019
02:55 PM
Great, @son trinh, let me know how it goes. Oh, and also, if you don't get the amount of disk space back you need, we can set TTLs to the other data. My recommendation above is for the metrics column families on the tables (like how much memory and CPU per container) which are the least important and also the ones that come with an expiration period by default, so that you don't lose job execution metadata (like where and what and when was executed, exit status, etc.), but if required and you are OK with not having that information after the retention period, we could also get the rest of the ATSv2 data to expire with: alter 'prod.timelineservice.application', {NAME=> 'c',TTL => 1296000}
alter 'prod.timelineservice.application', {NAME=> 'i',TTL => 1296000}
alter 'prod.timelineservice.app_flow', {NAME=> 'm',TTL => 1296000}
alter 'prod.timelineservice.entity', {NAME=> 'c',TTL => 1296000}
alter 'prod.timelineservice.entity', {NAME=> 'i',TTL => 1296000}
alter 'prod.timelineservice.flowrun', {NAME=> 'i',TTL => 1296000}
alter 'prod.timelineservice.flowactivity', {NAME=> 'i',TTL => 1296000}
alter 'prod.timelineservice.subapplication', {NAME=> 'c',TTL => 1296000}
alter 'prod.timelineservice.subapplication', {NAME=> 'i',TTL => 1296000} Regards, -- Tomas
... View more
07-04-2019
04:16 PM
1 Kudo
Hi @son trinh ! The default config for the ATSv2 tables is to keep data for 30 days, so you should decrease this config to get smaller footprint on ATSv2. You can change this by lowering the TTL on the tables, for example, setting expiration to 15 days (=1296000 seconds). Assuming you're running HBase in embedded mode for atsv2 (remember that ATSv2 HBase can also be run in Service mode) : Run this as yarn-ats user, with kerberos ticket if on a kerberized env.: hbase --config /etc/hadoop/conf/embedded-yarn-ats-hbase shell and inside the hbase shell, run these: alter 'prod.timelineservice.application', {NAME=> 'm',TTL => 1296000 }
alter 'prod.timelineservice.subapplication', {NAME=> 'm',TTL => 1296000 }
alter 'prod.timelineservice.entity', {NAME=> 'm',TTL => 1296000 } That should keep the ATSv2 db smaller. Regards -- Tomas
... View more
06-08-2018
12:13 PM
Hi @Vinay K It depends, you might have some underlying issue, first check who is holding up too many open files, by using the os command "lsof" If everything seems OK but and you just have this issue due to the workload, increase the number of open files in limit.conf, but beware that if you are using Ambari and using of of the hadoop users, its limit will be overridden by the value from the Ambari configuration.
... View more
06-06-2017
09:34 PM
1 Kudo
Hi @mel mendoza Yes, that was exactly what I was thinking that could be happening. You have networking issues on your data nodes. Take a look a the typical configuration issues that could cause this, like checking the network interfaces for dropped packets, that they (both the node's eth interfaces and the communication equipment between them) are set in the correct speed and duplex mode, and that they have proper MTU settings. Best regards, -- Tomas
... View more
06-05-2017
09:04 PM
@mel mendoza Hi Mel, It looks like the data transfer itself is taking the time. Could be network. Are these nodes virtual machines or are they physical nodes? Also, can you take a look in your datanode logs (/var/log/hadoop/hdfs/hadoop-hdfs-datanode* on each node), and look for some slowness message like "Slow BlockReceiver" ? Best Regards, -- Tomas
... View more
05-22-2017
10:24 PM
Hi Mel!, Are you using hdfs dfs commands in your script? If you are, try to add --loglevel TRACE to the hdfs commands, so that we can see at which point most of the time is being spent. Best regards, -- Tomas
... View more