@Timothy Spann - Ambari on our system does not provide any facility to install lzo. You keep referring to this, but it isn't there. If you believe it should be, please tell me where I might find the dialog? - I followed ALL the steps you outlined above, except for Hive. I DO NOT WANT LZO COMPRESSION ON HIVE. If that's not optional, then it should be documented as such. - I did have things stopped when I installed the RPMs and updated configuration. We're in a real mess here and currently trying to find someone to help us recover. I wish your company provided per-incident support, but that doesn't seem to be the case. ... View more

@Timothy Spann The cluster itself was installed through Ambari and has been running about a year. One of my users needed LZO compression enabled several days ago. You web site told me that Ambari does not install or configure LZO, so I followed the instructions as you entered them above. I added two changes to core-site.xml that were similarly documented in the HDP 2.3.2 web pages. After fixing an initial typo, we had working LZO and could explicitly invoke LzoIndexer on files in HDFS. Shortly after that I started receiving reports about Hive being broken. Originally it was complaining that it could not find the LzoCodec. I never told it to use the LzoCodec. I did not change Hive configuration. After removing the entries in core-site.xml, the Hive problems continued but it now tells me it cannot find "com" - a nonsense class name. I did restart everything that needed to be restarted - several times, in fact. The only thing amiss in the Hive logs is the same traceback the user gets on a failed query: 2016-11-02 12:57:18,278 WARN [HiveServer2-Handler-Pool: Thread-2740]: thrift.ThriftCLIService (ThriftCLIService.java:FetchResults(681)) - Error fetching results: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException: Error in configuring object at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:352) at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:221) at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:685) at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:454) at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:672) at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553) at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672) at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:347) ... 13 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hive.common.util.ReflectionUtil.setJobConf(ReflectionUtil.java:115) at org.apache.hive.common.util.ReflectionUtil.setConf(ReflectionUtil.java:103) at org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:87) at org.apache.hadoop.hive.ql.exec.FetchOperator.getInputFormatFromCache(FetchOperator.java:207) at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:361) at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:295) at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446) ... 17 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.common.util.ReflectionUtil.setJobConf(ReflectionUtil.java:112) ... 23 more Caused by: java.lang.IllegalArgumentException: Compression codec com not found. at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:139) at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:179) at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45) ... 27 more Caused by: java.lang.ClassNotFoundException: Class com not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132) ... 29 more 2016-11-02 12:57:18,281 INFO [HiveServer2-Handler-Pool: Thread-2740]: exec.ListSinkOperator (Operator.java:close(613)) - 10800 finished. closing... We are running HDP-2.3.2 on Centos 6.7. I do not know where to start troubleshooting this, particularly since it's not deterministic. Only some queries are blowing up with no obvious common denominator across them. Again, we made no changes to Hive and my users have made no changes in the way they are querying it. ... View more

@Timothy Spann And that is precisely what I had done - to the letter. If there was an option for installation from Ambari, it is not evident. Where exactly is this "wizard" you refer to? Your web documentation states clearly that Ambari neither installs nor configures LZO. The proximate issue is that Hive is totally broken now - even after removal of the two changes made to core-site.xml. Why is Hive even TRYING to use LZO? I did not configure that - I did not so much as touch Hive. ... View more

@ripunjay godhani I want to be sure I understand your post. Are you saying that modifying a single Ambari property will relocate logs for all components on a restart? If so, can share what the name of that property is? The page you linked to does not have a single mention of log location. In a perfect world, I would have left plenty of room under /var for logging, but we have a heavily used cluster with a lot of data and constant crashes from full /var on many of the machines. I need to move everything to a new location. ... View more

@Benjamin Leonhardi Thanks for the explanation. We're having significant scaling issues with our 32-host Hortonworks HDP-2.3.2.0-2950 installation. How do I determine which version of ATS is installed? I do not see it listed in the 'Stacks and Versions' page in Ambari. Assuming we're running one of the troublesome versions, what's the most expedient way to disable reporting to ATS from Hive? Since it is useful for debugging, I'm hoping there's a session parameter we can set at query time to suppress reporting when performance is an issue. ... View more

I'm curious what the advantage of suppressing Hive ATS reporting might be. From an esthetic standpoint we really don't want the UI filled up with myriads of successful, short-running queries, but it would be nice to switch it on in a case by case basis for debugging purposes. Beyond that, would turning it off improve query latency? ... View more

I wanted to post a quick followup on this thread. We recently found ourselves in a situation where we needed to deploy the hbase client code on an arbitrary number of machines and did not want the overhead of using Ambari. It was very straightforward to setup the Hortonworks repository reference and pull down hbase, however even after adding Phoenix the hbase shell would fail at startup with the dreaded (and spectacularly uninformative) exception: NativeException: java.io.IOException: java.lang.reflect.InvocationTargetException initialize at /usr/hdp/2.3.2.0-2950/hbase/lib/ruby/hbase/hbase.rb:42 (root) at /usr/hdp/2.3.2.0-2950/hbase/bin/hirb.rb:131 After almost 1/2 day of hair-pulling, I ran strace against the shell startup on a working node and compared it to the trace from the failing one. It turns out that the shell absolutely requires this directory path to exist (it can be empty): /hadoop/hbase/local/jars Once I created that hierarchy the shell was able to start successfully: $ mkdir /hadoop $ chmod 1777 /hadoop $ mkdir -p /hadoop/hbase/local/jars $ chmod -R 755 /hadoop/hbase Hopefully this will save someone else the time and aggravation. ... View more

@stevel Fantastic! That's a great example of useful and practical documentation. I'll let you know what I turn up from making the REST calls. ... View more

@stevel I made those two changes and restarted Spark. A job submitted with '--master yarn-client' still behaves as before, with the history server not correctly tracking the job. A job submitted with '--master yarn-cluster' does get picked up as a completed job in history, but when I drill in there is absolutely no information available relative to the job. The environment tab is populated, but not with anything obviously job-specific. The 'executors' tab has the following: executors.png which is suspiciously devoid of any actual activity. 'Stages', 'Storage' and 'Jobs' are completely blank. I understand in the abstract what you're asking for in terms of querying the ATS server, but it's going to take me some time to determine the required web-service calls and put that together. It's something I probably need to know about, but won't have the time to dig in for a day or so. Thanks for your help to this point! I'll try to get the rest of the information later this week. ... View more

@stevel I moved the yarn timeline directory to /hadoop/yarn and restarted. I'm no longer seeing the 500 error from the Spark History UI, but it continues to list completed Spark jobs in 'incomplete' telling me that there are hundreds of tasks remaining to be run. The YARN history UI does correctly report that the job is complete. incomplete.png The developer who owns the application tells me that it appears to be returning proper results. ... View more

@stevel I am seeing that same 500 error when working directory from the browser. Moving the timeline storage path is not a problem. I've read some suggestions about moving it to HDFS, but I'm not sure what other ramifications that may have so I'll stick with machine-local storage for now. Not sure if you saw one of my earlier posts where I mentioned that the Spark daemon log is filling with errors whose meaning is not clear (see beginning of thread above). Perhaps that will go away when I relocate the log directory. ... View more

@stevel Hi. We are using HDP-2.3.2.0-2950, with all nodes running Centos 6.7. Not sure I know how to answer the question about logs. For starters, it's not easy to understand where these would be. If I assume to history server to be the machine that I connect to for the Spark History UI and I assume that job-related logs would be under /tmp, then there's nothing relevant on that box. If I look on the namenode I can see /tmp/hadoop/yarn/timeline with populated subdirectories. Are those what you are referring to? I restarted the history server and now things are utterly non-functional. The Spark History UI shows nothing under either complete or incomplete and displays an error: Last Operation Failure: java.io.IOException: Bad GET request: status code 500 against http://bigfoot6.watson.ibm.com:8188/ws/v1/timeline/spark_event_v01?fields=PRIMARYFILTERS,OTHERINFO; {"exception":"WebApplicationException","message":"java.io.IOException: org.iq80.leveldb.DBException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /tmp/hadoop/yarn/timeline/leveldb-timeline-store.ldb/005567.sst: No such file or directo Indeed, there is no file by that particular name, but there are dozens of other .sst files present. What is causing it to look for that specific file and, further, why is it giving up completely after not finding it? We are using the YARN history service as the backend. FYI: After restarting the history server, I'm getting this is in the daemon logs on the history server host: spark-spark-orgapachesparkdeployhistoryhistoryserv.txt It looks very unhappy. All of this had been working fine as recently as late January, and I have not (knowingly) made any changes whatsoever to the Spark history configuration. Please let me know if you need any further information. I've looked through the Hortonworks courses on Hadoop management, but haven't seen any syllabus that claims to cover troubleshooting at a sufficiently low level. If that's not the case, can you advise which of them would provide enough background to be able to help in a case such as this? ... View more

Thanks, but we do not have a support agreement. We'll just have to live with it. I've provided all the information I have. ... View more

I'm starting to get concerned about this issue. We have run about 50 jobs in Spark that return results without any exceptional conditions, and which the YARN UI reports as complete. All of them are languishing in the Spark UI incomplete job listing with over 150 steps (it claims) left to go. The offending operation is either 'toPandas' or 'treeAggregate at GradientDescent.scala:189'. I do not see any sign that these processes are actually alive. Why are they not being reported as done? ... View more

The Tez view appears to be working correctly. That endless cascade of exceptions from the history server must be pointing to something specific, but I unfortunately do not know how to interpret it. One of our users mentioned to me that the lingering jobs in the Spark UI are all using a Python method called 'toPandas', while the few that do get properly noted as complete do not. Is that a useful clue? The Spark "incomplete" history continues to pile up dozens of jobs that are reported on the console (and by YARN) as being finished. ... View more

More information: Around the time the history server stopped working correctly, a cascade of exceptions appeared in the spark logs: 2016-02-04 14:55:09,035 WARN timeline.TimelineDataManager (TimelineDataManager.java:doPostEntities(366)) - Skip the timeline entity: { id: tez_con tainer_e07_1453990729709_0165_01_000043, type: TEZ_CONTAINER_ID } org.apache.hadoop.yarn.exceptions.YarnException: The domain of the timeline entity { id: tez_container_e07_1453990729709_0165_01_000043, type: TEZ_ CONTAINER_ID } is not allowed to be changed from Tez_ATS_application_1453990729709_0165 to Tez_ATS_application_1453990729709_0165_wanghai_201602041 45330_86a58f3a-0891-4c24-bf0f-0375575077da:1 Does that shed any light on the underlying problem? The log contains > 50 MB of such messages. ... View more

The behavior I'm looking for is something like this: Export all deltas for all configuration files beyond settings that are built-in defaults for Hortonworks installation. I believe this would be defined as version numbers > 1 (is that correct?) Import these into a newly-built cluster and be prompted for manual intervention when a delta includes a machine name, IP address or port (the latter can probably be determined by a regex match on the property name). As an audit tool, the presentation of environment (shell) scripts should be in the form of unified diffs rather than a dump of the entire file at each revision. Just a few ideas off the top of my head. There's no way this process can be totally automated, but I think it's possible to get very close. ... View more

Thanks, Jonas. As I mentioned in my last post, I'm in search of a way to generate a set of deltas that can propagate my tuning and tweaks to a new cluster. I try to keep notes, but there have been many occasions where I was in the midst of troubleshooting and failed to write down what I did. Exporting configuration in bulk isn't really what's needed to propagate diffs to a new cluster, since it will have machine names (and perhaps port addresses) that won't apply to the new target. Your script looks promising for a starting point, but I'd have to flesh it out with a framework that traverses all configuration files at a minimum. From a quick look it also appears that the change tracking on shell environment files will require a lot of manual work, since there's no attempt at differencing - the entire file is dumped. ... View more

Ah - thanks. I am far from an expert on Python and did not realize this was a non-Hadoop package. A quick 'yum install python-requests' did the trick. FYI: I'm not sure if you're implying that 'audit.py' should be present on the system, but it certainly isn't here. I grabbed it from GitHub. This is a very valuable utilitity, and would be even more so if it had the ability to crawl all configuration files and generate a master listing. My proximate problem is how to effectively clone a running configuration to a new cluster. I'm surprised to find so little discussion of this task. Perhaps I'm missing something obvious, but exporting and copying configurations does not correct the myriad of machine names and IP addresses that will certainly be inappropriate in a new setting. Ambari badly needs the ability to generate bulk delta files (differences from defaults at install time) that can be imported "smartly" into a new cluster - e.g. spotting anything that looks like a URL or port address and prompting for manual intervention. ... View more

Looks useful, but exactly which python binary is this intended to work with? It fails immediately with a complaint about not being able to find the 'request' package. I grepped the /usr/hdp tree, but cannot find a 'request.py' module. ... View more