I support a Hortonworks Hadoop cluster that is on an isolated network.  Currently running HDP-2.5.3.0 and RedHat 6.6 for the operating system.   I need to rebuild a couple of my data nodes and no longer have access to the RedHat repo.  I do have access the a centos repo.   Question is, can I rebuild the data nodes using Centos 7 when the majority of my cluster is running RH6?  Ultimately I would like to move everything to Centos 7 but the cluster is not running HA and I am not sure what the process for migrating everything would be ... View more

I installed patches and bounced a node on my cluster and now the datanode won't start 2018-12-20 14:26:07,962 ERROR datanode.DataNode (BPServiceActor.java:run(772)) - Initialization failed for Block pool <registering> (Datanode Uuid fa50d7aa-c305-47bd-9935-6d8f947e0d27) service to m02.pnl.gov/192.168.41.52:8020. Exiting. java.io.IOException: All specified directories are failed to load. Redhat 6.10 Ambari 2.5.0.3 Stack 2.7.3.2.6 ... View more

We have a data retention requirement that is pushing storage capacity on my cluster. It is not in the budget to increase storage or add more data nodes. I do have some older servers that I am thinking of using to archive older data that is not accessed very often (if at all). Trying to find some guidance as to best practices for doing this. One though is to dump HDFS partition directories into a very similar file structure on the archive server. Seems there would be a huge IO/network hit with this option and I would have to make room to pull data back into the cluster if I ever needed to access it. Anyone have experience with archiving data from hadoop? ... View more

I lost a drive on one of my data nodes that apparently stored some ambari-metrics collection stuff. Besides the hadoop directory structure that was created when the datanode was restarted I see a var/lib/ambari-metrics-collector that was created as well replaced the drive and everything is back up except the metrics collector. Getting these errors 22:22:12,598 WARN [main-SendThread(datanode03.foo:61181)] ClientCnxn:1146 - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused 22:22:04,011 WARN [main] DefaultPhoenixDataSource:84 - Unable to connect to HBase store using Phoenix. java.sql.SQLException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations ... View more

Rebooted my cluster last week and I am seeing unusually high CPU loads on 3 datanodes. Nothing changed, except the reboot. Load average being 120-150 with the others having a load of less than 5. Looking at top, it seems to be yarn java processes that are running. All nodes are running Java 1.7.0 Running HDP 2.5.3 ambari 2.4.2 37 datanodes all configured the same Dell R515 servers 2 8-core AMD CPU's RHEL 6 64-bit 256GB memory same hard drives The 3 nodes in question only differ by CPU. They are running AMD Opteron 4386 vs AMD Opteron 4284 on all the other datanodes. I have been running this cluster for over year with no problems. Any ideas as to why this is happening? I have looked at bios setting to see if I had some configured different but they are all the same ... View more

My cluster nodes have two versions on python installed. 2.6.6 in /usr/bin 2.7.12 in /usr/local/bin. I installed some python modules for the 2.7.12 version associated with geolocation. When I run a job locally on one of the nodes it runs fine. When it is submitted through yarn I get the following “ImportError: No module named ipaddress” ipaddress is one of the modules I installed. I suspect yarn is using the 2.6.6 version of python. How can I determine if this is the case and if it is how can I define yarn to use the python in /usr/local/bin? Thanks ... View more

NameNode Heap Usage (weekly) alert - I get these periodically Critical is set to 60%, warning at 50% minimum heap set to 1000M. The variance for this alert is 504MB which is 66% of the 7560MB average (4536MB is the limit). Is this a concern? If so, what should I be looking at? ... View more

Running Ambari 2.4.2 and HDP 2.5.3.0 It looks like my cluster is defined to use mysql for the hive metastore. The log file is being written to /var/run/mysqld and is 239GB and growing and is close to filling up the /var file system Can I delete or rotate out the log and do I need to shutdown hive to do this? Is there a way to reduce the level of logging or to automatically rotate this log? ... View more

Planning on some housekeeping If I delete an account in ambari, what happens to any data? Is it deleted as well? ... View more

running a job in hive, writes cache to /var/hadoop/yarn/local/usercache This is causing the /var file system to fill up resulting in node manage failures on some nodes and the job to hang. Is it possible to direct usercache to a different location? ... View more

Running this in the ambari hive 2.0 view. Trying to join 2 tables into a new table. Query seems to run then stalls at 100% and returns with an error but no specific error message is provided in the hive view. Where can I find the error message or at least logs associated with the job ID provided by the hive view? ... View more

adding new nodes to my cluster, I have one that I cannot get the ambari-agent to start RHEL 6.9, HDP-2.6.0.3, ambari 2.5.0.3, ambari-agent-2.5.0.3-7.x86_64 I get the following error in the ambari-agent log ERROR 2017-09-25 14:59:26,625 Controller.py:502 - Controller thread failed with exception: Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 489, in run self.register = Register(self.config) File "/usr/lib/python2.6/site-packages/ambari_agent/Register.py", line 35, in __init__ self.hardware = Hardware(self.config) File "/usr/lib/python2.6/site-packages/ambari_agent/Hardware.py", line 52, in __init__ self.hardware.update(Facter(self.config).facterInfo()) File "/usr/lib/python2.6/site-packages/ambari_agent/Facter.py", line 571, in facterInfo facterInfo = super(FacterLinux, self).facterInfo() File "/usr/lib/python2.6/site-packages/ambari_agent/Facter.py", line 248, in facterInfo 'ipaddress': self.getIpAddress(), File "/usr/lib/python2.6/site-packages/ambari_agent/Facter.py", line 81, in getIpAddress return socket.gethostbyname(self.getFqdn().lower()) gaierror: [Errno -2] Name or service not known ... View more

A datanode service on one of my cluster nodes is down because I lost a hard drive/file system. I have a drive on order but it may be a couple of days until I have it in hand. What is the impact of continuing to run in this state? ... View more

running HDP 2.5.3 The root file system filled up on the node that runs hive metastore and hiveserver2. Both processes were stopped. I cleared up disk space but could not get either process to start hiveserver2 error yarn.exceptions.ApplicationNotFoundException: Application with id "application_1496410945618_25770" doesn't exist in RM hivemetastore error transport exception could not create ServerSocket I suspect it was a job that didn't finish or die cleanly because of the disk filling up. There were no jobs running in the yarn RM I tried stopping all the hive processes and restarting yarn and then start hive. Still had the same problem. I resorted to stopping all the services on the node where the disk filled up and bouncing the node. This fixed the problem as hiveserver2 and hive metastore now run. Any ideas of a better way to troubleshoot and resolve something like this? ... View more

Trying to get serde working with zeppelin and/or the spark shell. Running the following in either val links = sqlContext.sql("SELECT * FROM test_links LIMIT 10") links.show() Produces an error ERROR hive.log: error in initSerDe: java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found Tried added this to /etc/spark/conf/hive-site.xml <property> <name>hive.aux.jars.path</name> <value>file:///usr/hdp/2.5.3.0-37/hive2/lib/hive-hcatalog-core.jar</value> </property> But am still getting the error. ... View more

Running spark or pyspark from the zeppelin notebook hangs. Looking in the zeppelin-interpreter-spark log I am seeing the following ERROR [2017-05-18 11:51:39,130] ({BoneCP-pool-watch-thread} PoolWatchThread.java[fillConnections]:118) - Error in trying to obtain a connection. Retrying in 7000ms java.sql.SQLException: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.setReadOnly(Unknown Source) at com.jolbox.bonecp.ConnectionHandle.setReadOnly(ConnectionHandle.java:1324) at com.jolbox.bonecp.ConnectionHandle.<init>(ConnectionHandle.java:262) at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115) at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.sql.SQLException: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) I have oozie installed and have verified the ownership of the oozie-db oozie:hadoop ... View more

HDP 2.5.3 Ambari 2.4.2 18 data nodes 190TB HDFS disk usage is at about 92% ~15TB free with critical alarms or warnngs on most all the data nodes Percent DataNodes With Available Space is alarming as well Are the best practice recommendations for setting these thresholds, for managing the percent of HDFS disk usage? Are there concerns for running HDFS disk usage above a certain percentage? ... View more

HDP 2.5.3 37 datanodes I had 34 data nodes that are identical with regards to CPU,. memory, storage. I recently added 3 data nodes with the same CPU and memory but less storage (9TB as opposed to 11TB on the older nodes) I ran hdfs balancer, which chugged for a while and moved data to the 3 new nodes. Problem is, that 3 file systems on each of the 3 new nodes do not seem to be getting data. I have run the balancer from the CLI and the output states the cluster is balanced.The older nodes (with more storage) are at about 50% disk utilization, the 3 new nodes are at about 30% disk utilization ... View more

Running HDP 2.5.3 and ambari 2.4.2 I am having trouble getting some of the metrics and graphs to display. Dashboard page displays graphs OK Service page displays graphs OK except for Ambari Metrics - graphs just show spinning circles Main hosts pages do not show Disk Usage or Load Avg. Specific host page sometimes (rarely) shows graphs, usually shows spinners. But if I go back to the main hosts page, disk usage and load avg will display briefly for that host, then disappear. I have seen posts regarding the restart of ntp servers - time on all hosts are synced together. I have seen a post regarding setting up a proxy host but the cluster is on an isolated network with a local repository - no internet access. Any ideas? ... View more