About jorge-luis_hern

jorge-luis_hern · ‎08-07-2019

I see this suggestion on smartsense for my cluster and below steps on how to performe it https://community.hortonworks.com/questions/208928/increase-open-file-limit-of-the-user-to-scale-for.html?childToView=250058#answer-250058 But my concerning is, How to get a value to set the ulimit value for each user within my cluster, is there any sort of formula I am not aware of? or I need to check something within my cluster? I don't know, I am thinking on the amount of process ran by each user and based on that, to set a number. But even if I I do this, again ... what is the fundamental on setting a value x or y? Any hint? Regards!

jorge-luis_hern · ‎08-07-2019

Is there any sort of formula or how did you came up with this value for users's processes? is it a random value? what can I check within my cluster in order to get a proper value for me?

jorge-luis_hern · ‎03-29-2019

Any one who can help us to get an official site to get his jar to make it work with tibco? Regards!

jorge-luis_hern · ‎03-14-2019

I am configuring rack awareness within my cluster which is build with 5 data nodes on 3 racks as below: rack_1 = node1 rack_2 = node2 node3 rack_3 = node4 I want to understand that if without rack awareness configured, my 3 blocks from same replica could be allocated within one node? Could this happen if I do not set rack awareness? Regards!

jorge-luis_hern · ‎01-07-2019

In adittion, within my cluster, port was assigned to 9090, by getting the pid from above command I just ran netstat -tuplen | grep <pid> and I got the port where the UI is running

jorge-luis_hern · ‎10-11-2018

This is a known behavior from Hortonworks, check the discussion https://issues.apache.org/jira/browse/HDFS-8986 Regards!

jorge-luis_hern · ‎10-11-2018

Hello team. Current lecture directly to the involved directory is giving us: hadoop fs -du -h -s </path/dir> t 1.4 T </path/dir> of information but this is not accurate because when getting each and one size from the files and directories within above directory, performing a simple addition operation from excel we are getting 386574657917 which translated in GB is 360GB which is the expected size within this directory. Has someone face this before? Regards!

jorge-luis_hern · ‎10-10-2018

I am making a disctp between non secure clusters and I've made it works, the problem is when running below command: hadoop distcp -overwrite -pb <Snapshot_Dir/Snapshot_name/* <Target_DIR> since now I am playing with options. I am getting mentioned error (check complete message below): 18/10/10 11:12:42 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, overwrite=true, append=false, useDiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false, blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=100, sslConfigurationFile='null', copyStrategy='uniformsize', preserveStatus=[BLOCKSIZE], preserveRawXattrs=false, atomicWorkPath=null, logPath=null, sourceFileListing=null, sourcePaths=[hdfs://server/DIR/.snapshot/s20181010-085735.478/*], targetPath=hdfs://server/DIR, targetPathExists=true, filtersFile='null', verboseLog=false} 18/10/10 11:12:42 INFO client.AHSProxy: Connecting to Application History server at server 18/10/10 11:12:43 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 4478; dirCnt = 0 18/10/10 11:12:43 INFO tools.SimpleCopyListing: Build file listing completed. 18/10/10 11:12:43 ERROR tools.DistCp: Duplicate files in input path: org.apache.hadoop.tools.CopyListing$DuplicateFileException: File hdfs://source_server/DIR.snapshot/s20181010-085735.478/2018-07-23/000000_0 and hdfs://source_server/DIR/.snapshot/s20181010-085735.478/dt=2018-07-16/000000_0 would cause duplicates. Aborting at org.apache.hadoop.tools.CopyListing.validateFinalListing(CopyListing.java:165) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:93) at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90) at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:86) at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:398) at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:190) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:155) at org.apache.hadoop.tools.DistCp.run(DistCp.java:128) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.tools.DistCp.main(DistCp.java:462) I've also tried -update and same result. Does anyone who has fix this? Regards!

jorge-luis_hern · ‎09-13-2018

Also check https://community.hortonworks.com/articles/14309/demystify-tez-tuning-step-by-step.html https://community.hortonworks.com/questions/5780/hive-on-tez-query-map-output-outofmemoryerror-java.html https://community.hortonworks.com/questions/12067/what-is-the-workaround-when-getting-hive-outofmemo.html

jorge-luis_hern · ‎09-13-2018

What I understand from documentation (I am applying same changes) and from previous experience is that happens that some jobs has too many inputs and outputs to be handle by the system itself (mappers, containers, reducers etc...) this affects from one side your cluster performance since resources are not enough for such jobs to complete or simply they start to present a considerable delay to what is expected. I've seen that's the moment when OOM error shows up. Saying this, I guess explanation from the smartsense is clear since it tells us "Enabling auto-scaling scales down requests, which may otherwise be too large." Hope this helps 😉

jorge-luis_hern · ‎09-13-2018

Hello team. I've just applied the recommended smartsense setting to make my dfs.namenode.checkpoint.period more efficent and I want to take a measure on how fast is now my namenode recovering from an restart than it was before. Where can I get an old lecture from the nn last restart and how much time did it take? Regards!

jorge-luis_hern · ‎09-13-2018

Below the answer in case someone need it: "Each handler (worker) thread consume resources/memory on the NN. We can not set this value too high because it will consume unnecessary resources and cause extra burden on the NN."

jorge-luis_hern · ‎09-12-2018

I am following Smartsense recommendation to make my cluster operationally more efficient, however there is a lot of documentation on how to set mentioned feature within ambari an why handlers are important for RPC, no one is mentioning why having this feature set "too high" affects a cluster. Can anyone share some thought here? Regards!"

jorge-luis_hern · ‎09-03-2018

According to hwx team, this is a known bug https://issues.apache.org/jira/browse/AMBARI-23309https://hortonworks.jira.com/browse/BUG-92535 (Internal Link)

jorge-luis_hern · ‎09-03-2018

Hello all. I have installed ambari 2.6.0.9 using hdp 2.6.3.7 on my 3 env (dev, qa and pro) and none of the 3 are showing mentioned widget within yarn service - Check attached image. Any idea, I have not found information yet. Regards!!

jorge-luis_hern · ‎06-05-2018

It worked for me. Thanks!

jorge-luis_hern · ‎05-16-2018

Hey Geoffrey eventhough it worked. I kept monitoring it for a while and metrics went away again, but this time with a different message 2018-05-16 22:53:52,754 INFO TimelineMetricHostAggregatorHourly: End aggregation cycle @ Wed May 16 22:53:52 UTC 2018 2018-05-16 22:54:10,428 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:54:20,432 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:54:30,437 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:54:40,446 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:54:45,499 INFO TimelineClusterAggregatorSecond: Started Timeline aggregator thread @ Wed May 16 22:54:45 UTC 2018 2018-05-16 22:54:45,501 INFO TimelineClusterAggregatorSecond: Last Checkpoint read : Wed May 16 22:52:00 UTC 2018 2018-05-16 22:54:45,501 INFO TimelineClusterAggregatorSecond: Rounded off checkpoint : Wed May 16 22:52:00 UTC 2018 2018-05-16 22:54:45,501 INFO TimelineClusterAggregatorSecond: Last check point time: 1526511120000, lagBy: 165 seconds. 2018-05-16 22:54:45,501 INFO TimelineClusterAggregatorSecond: Start aggregation cycle @ Wed May 16 22:54:45 UTC 2018, startTime = Wed May 16 22:52:00 UTC 2018, endTime = Wed May 16 22:54:00 UTC 2018 2018-05-16 22:54:45,501 INFO TimelineClusterAggregatorSecond: Skipping aggregation for metric patterns : sdisk\_%,boottime 2018-05-16 22:54:50,453 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:55:00,460 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:55:10,462 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:55:20,463 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:55:30,473 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:55:40,476 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:55:50,487 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:56:00,490 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish 2018-05-16 22:56:10,494 INFO org.apache.hadoop.hbase.client.AsyncProcess: #1, waiting for 58080 actions to finish <br> and it's showing above all time. Any idea? I meant, the message is preatty clear and looks like my heap size is not enough for the amount of data the service is getting, this is what I have configure on my metrics collector heap size: metrics_collector_heapsize = 6144 If I have a cluster with 126 node and 106 of them has 899.50 GB as configure capaciy ant 20 of them with 399.75 GB what would be a fair amount of heap size to assign to this service does a formula exists for this? Regards!

jorge-luis_hern · ‎05-16-2018

Thank you Geoffrey it works and now logs are behaving as expected adn dashboard as well.

jorge-luis_hern · ‎05-16-2018

Problem: Amabri metrics dashboard within Ambari is not loading information (image attached) Actions: Ambari metrics service restart, Ambari collector restart. ambari-metrics-dashboard.png Also Ive founded this message on the logs. Therefore, I am trying to get mentioned server out from blacklist but I can't find information about it. 2018-05-16 20:32:17,258 [WARNING] emitter.py:146 - Error sending metrics to server. 2018-05-16 20:32:17,258 [WARNING] emitter.py:111 - Retrying after 5 ... 2018-05-16 20:35:42,259 [WARNING] emitter.py:146 - Error sending metrics to server. '' 2018-05-16 20:35:42,260 [WARNING] emitter.py:111 - Retrying after 5 ... 2018-05-16 20:35:47,260 [WARNING] emitter.py:120 - Metric collector host <server_name> was blacklisted. 2018-05-16 20:35:47,260 [INFO] emitter.py:96 - No valid collectors found... 2018-05-16 20:36:47,269 [INFO] emitter.py:96 - No valid collectors found... 2018-05-16 20:37:47,273 [INFO] emitter.py:96 - No valid collectors found... 2018-05-16 20:38:47,276 [INFO] emitter.py:96 - No valid collectors found... 2018-05-16 20:39:47,280 [INFO] emitter.py:96 - No valid collectors found... 2018-05-16 20:40:47,283 [INFO] emitter.py:154 - Calculated collector shard based on hostname : <server_name> 2018-05-16 20:44:07,284 [WARNING] emitter.py:146 - Error sending metrics to server. '' 2018-05-16 20:44:07,285 [WARNING] emitter.py:111 - Retrying after 5 ... 2018-05-16 20:44:12,285 [WARNING] emitter.py:146 - Error sending metrics to server. 2018-05-16 20:44:12,285 [WARNING] emitter.py:111 - Retrying after 5 ... 2018-05-16 20:47:37,286 [WARNING] emitter.py:146 - Error sending metrics to server. '' 2018-05-16 20:47:37,287 [WARNING] emitter.py:111 - Retrying after 5 ...

jorge-luis_hern · ‎05-16-2018

Did you figure it out? I am having same message.

jorge-luis_hern · ‎04-27-2018

Which node would be this file located? Or where in Ambari can be searched for?

jorge-luis_hern · ‎04-27-2018

This is not an issue any more somebody ran hive on safe mode therfore beeline.properties files was created. Check below link. It gave me the root cause, but in order to fix specifically my issue, I needed either to delete the beeline.properties file created, remove below property or comment it out within beeline.properties files located on /home/<user_account>/.beeline/beeline.properties beeline.hiveconfvariables={} https://issues.apache.org/jira/browse/HIVE-16116

jorge-luis_hern · ‎04-24-2018

I have 2 edge nodes where I can do this, one is working properly and the other is not. I want to know what should I need to check in order to clear posted message when accessing to hive through beeline? This is the one is not working: beeline -n srvc_ima_platform -u 'jdbc:hive2://server:2181,server:2181,server:2181/ea_fin;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;?tez.queue.name=srvc_platform' --showHeader=false --outputformat=tsv2 --hiveconf hive.fetch.task.conversion=none Exception in thread "main" java.lang.NullPointerException at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:677) at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:777) at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:491) at org.apache.hive.beeline.BeeLine.main(BeeLine.java:474) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) And this is he one working properly, both cli are exactly the same: beeline -n srvc_ima_platform -u 'jdbc:hive2://server:2181,server:2181,server:2181/ea_fin;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;?tez.queue.name=srvc_platform' --showHeader=false --outputformat=tsv2 --hiveconf hive.fetch.task.conversion=none Connecting to jdbc:hive2://server:2181,server:2181,server:2181/ea_fin;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;?tez.queue.name=srvc_platform Connected to: Apache Hive (version 1.2.1000.2.6.0.24-2) Driver: Hive JDBC (version 1.2.1000.2.6.0.3-8) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.2.1000.2.6.0.3-8 by Apache Hive 0: jdbc:hive2://server:2> Regards!!

jorge-luis_hern · ‎04-19-2018

We left the settings just as stated and cluster keeps working fine.

jorge-luis_hern · ‎03-24-2018

HADOOP_NFS3_OPTS="-Xmx{{nfsgateway_heapsize}}m -Dhadoop.security.logger=ERROR,DRFAS ${HADOOP_NFS3_OPTS}"

jorge-luis_hern · ‎03-23-2018

Hello team. I've founded we have some items already set within hdfs-site.xml with different names that the ones indicated on hwx documents. Therefore: 1.- I would like to know the difference between each item. 2.- To know if it will be better to add them within the hdfs-site.xml along the existing settings already there?. I meant duplicate the values but with different names setings. 3.- To remove current setting and set the new ones (the ones indicated on the document). https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_hdfs-nfs-gateway-user-guide/content/user-guide-hdfs-nfs-instructions.html For question 3 I would say this will depend on my organization requierements but I would like to read your ideas. Current = nfs.kerberos.principal Reccommended = dfs.nfsgateway.kerberos.principal Current = nfs.keytab.file Recommended = dfs.nfsgateway.keytab.file Regards!!

jorge-luis_hern · ‎02-13-2018

Actually, clearing cookies and rebooting my browser was enough for me since all previous steps were properly set.

jorge-luis_hern · ‎02-11-2018

I've seen it depends on the services that are failling. E.g. Below steps worked for me when HDFS or Node Manager giving mentioned message, but when I have all servicies stopped due to this message in a single node it does not work (I am figuring out how to). ambari-agent stop mv /var/lib/ambari-agent/data/structured-out-status.json /var/lib/ambari-agent/data/structured-out-status.json-old ambari-agent start Regards!

jorge-luis_hern · ‎01-26-2018

Any luck on this? I am also having error 401 and credentials are fine as well.

jorge-luis_hern · ‎12-11-2017

We are observing mentioned process is consuming high CPU resources for a particular server and going through the clutser looking for same process, I 've founded on other 2 servers, running under a different user and they are not even close to the consumpltion level like the one I am having the doubt with. What I want to know is if this is a normal behavior and if I should't worry abou it. server_1_[root@server_1 ~]# top -p 16193 top - 18:04:58 up 479 days, 21:34, 2 users, load average: 5.79, 3.53, 2.51 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie %Cpu(s): 20.4 us, 4.0 sy, 0.0 ni, 75.1 id, 0.0 wa, 0.0 hi, 0.5 si, 0.0 st KiB Mem : 26384640+total, 11807649+free, 26398576 used, 11937132+buff/cache KiB Swap: 4194556 total, 4194556 free, 0 used. 23283172+avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16193 alipiah 20 0 15.861g 2.012g 30100 S 0.7 0.8 14:42.70 java ********************************************************************************** server_2_[root@server_2 ~]# top -p 9864 top - 18:06:52 up 479 days, 21:36, 2 users, load average: 0.39, 0.47, 0.49 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.8 us, 0.3 sy, 0.0 ni, 98.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 26384640+total, 66057520 free, 26081160 used, 17170771+buff/cache KiB Swap: 4194556 total, 4194556 free, 0 used. 23298083+avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9864 alipiah 20 0 15.857g 438752 30048 S 0.0 0.2 3:12.83 java ********************************************************************************** server_3_[root@server_3 ~]# top -p 344095 top - 18:08:43 up 479 days, 21:38, 2 users, load average: 38.80, 35.37, 32.46 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie %Cpu(s): 81.3 us, 1.9 sy, 0.0 ni, 16.8 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem : 26384640+total, 19629884 free, 42436532 used, 20177996+buff/cache KiB Swap: 4194556 total, 4194556 free, 0 used. 21663824+avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 344095 root 20 0 29.954g 8.888g 23968 S 2129 3.5 82702:17 java

Online	Offline
Last Visited	‎08-07-2019 04:01 PM

Date Registered	‎01-14-2016 10:55 PM
Last Visited	‎08-07-2019 04:01 PM
Total Messages Posted	45
Total Kudos Received	3

Re: "hadoop fs -du" command giving incorrect value

Re: What's the problem on having dfs.namenode.hand...

Re: Getting "Exception in thread "main" java.lang....

Re: How to restart just spark client on Ambari

Increase open file limit of the user to scale for ...

Re: Increase open file limit of the user to scale ...

Where can I download hadoop-core-1.2.1.jar?

Could blocks from same replica be allocated within...

Re: Jenkins is not working in HDP2.6

Re: "hadoop fs -du" command giving incorrect value

"hadoop fs -du" command giving incorrect value

Distcp ErrorMessage "Duplicate files in input pat...

Re: Need to understand impact for setting tez.task...

Re: Need to understand impact for setting tez.task...

How to check how long did Namenode take to restart...

Re: What's the problem on having dfs.namenode.hand...

What's the problem on having dfs.namenode.handler....

Re: Yarn - Widget - Cluster Memory not available

Yarn - Widget - Cluster Memory not available

Re: How to reset root password for sandbox hdp 2.4

Re: Ambari 2.6.* - Amabri Metrics UI is not loadin...

Re: Ambari 2.6.* - Amabri Metrics UI is not loadin...

Ambari 2.6.* - Amabri Metrics UI is not loading in...

Re: Error sending metrics to server. 'NoneType' ob...

Re: SmartSense Agent refusing to connect after upg...

Re: Getting "Exception in thread "main" java.lang....

Getting "Exception in thread "main" java.lang.Null...

Re: Differences between ambari settings and hwx do...

How can I know specified JVM heap space (HADOOP_NF...

Differences between ambari settings and hwx docume...

Re: Issue - Oozie web console is disabled.To enabl...

Re: Error: Unable to run the custom hook script

Re: Hive JDBC Connection string with KNOX and Kerb...

"ActivityAnalyzerFacade" consuming high ammount of...