@Timothy Spann - Ambari on our system does not provide any facility to install lzo. You keep referring to this, but it isn't there. If you believe it should be, please tell me where I might find the dialog? - I followed ALL the steps you outlined above, except for Hive. I DO NOT WANT LZO COMPRESSION ON HIVE. If that's not optional, then it should be documented as such. - I did have things stopped when I installed the RPMs and updated configuration. We're in a real mess here and currently trying to find someone to help us recover. I wish your company provided per-incident support, but that doesn't seem to be the case. ... View more

@Timothy Spann The cluster itself was installed through Ambari and has been running about a year. One of my users needed LZO compression enabled several days ago. You web site told me that Ambari does not install or configure LZO, so I followed the instructions as you entered them above. I added two changes to core-site.xml that were similarly documented in the HDP 2.3.2 web pages. After fixing an initial typo, we had working LZO and could explicitly invoke LzoIndexer on files in HDFS. Shortly after that I started receiving reports about Hive being broken. Originally it was complaining that it could not find the LzoCodec. I never told it to use the LzoCodec. I did not change Hive configuration. After removing the entries in core-site.xml, the Hive problems continued but it now tells me it cannot find "com" - a nonsense class name. I did restart everything that needed to be restarted - several times, in fact. The only thing amiss in the Hive logs is the same traceback the user gets on a failed query: 2016-11-02 12:57:18,278 WARN [HiveServer2-Handler-Pool: Thread-2740]: thrift.ThriftCLIService (ThriftCLIService.java:FetchResults(681)) - Error fetching results: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException: Error in configuring object at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:352) at org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:221) at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:685) at org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:454) at org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:672) at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1553) at org.apache.hive.service.cli.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1538) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1672) at org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:347) ... 13 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hive.common.util.ReflectionUtil.setJobConf(ReflectionUtil.java:115) at org.apache.hive.common.util.ReflectionUtil.setConf(ReflectionUtil.java:103) at org.apache.hive.common.util.ReflectionUtil.newInstance(ReflectionUtil.java:87) at org.apache.hadoop.hive.ql.exec.FetchOperator.getInputFormatFromCache(FetchOperator.java:207) at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:361) at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:295) at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446) ... 17 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.common.util.ReflectionUtil.setJobConf(ReflectionUtil.java:112) ... 23 more Caused by: java.lang.IllegalArgumentException: Compression codec com not found. at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:139) at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:179) at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45) ... 27 more Caused by: java.lang.ClassNotFoundException: Class com not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:132) ... 29 more 2016-11-02 12:57:18,281 INFO [HiveServer2-Handler-Pool: Thread-2740]: exec.ListSinkOperator (Operator.java:close(613)) - 10800 finished. closing... We are running HDP-2.3.2 on Centos 6.7. I do not know where to start troubleshooting this, particularly since it's not deterministic. Only some queries are blowing up with no obvious common denominator across them. Again, we made no changes to Hive and my users have made no changes in the way they are querying it. ... View more

@Timothy Spann And that is precisely what I had done - to the letter. If there was an option for installation from Ambari, it is not evident. Where exactly is this "wizard" you refer to? Your web documentation states clearly that Ambari neither installs nor configures LZO. The proximate issue is that Hive is totally broken now - even after removal of the two changes made to core-site.xml. Why is Hive even TRYING to use LZO? I did not configure that - I did not so much as touch Hive. ... View more

@ripunjay godhani I want to be sure I understand your post. Are you saying that modifying a single Ambari property will relocate logs for all components on a restart? If so, can share what the name of that property is? The page you linked to does not have a single mention of log location. In a perfect world, I would have left plenty of room under /var for logging, but we have a heavily used cluster with a lot of data and constant crashes from full /var on many of the machines. I need to move everything to a new location. ... View more

@Benjamin Leonhardi Thanks for the explanation. We're having significant scaling issues with our 32-host Hortonworks HDP-2.3.2.0-2950 installation. How do I determine which version of ATS is installed? I do not see it listed in the 'Stacks and Versions' page in Ambari. Assuming we're running one of the troublesome versions, what's the most expedient way to disable reporting to ATS from Hive? Since it is useful for debugging, I'm hoping there's a session parameter we can set at query time to suppress reporting when performance is an issue. ... View more

I'm curious what the advantage of suppressing Hive ATS reporting might be. From an esthetic standpoint we really don't want the UI filled up with myriads of successful, short-running queries, but it would be nice to switch it on in a case by case basis for debugging purposes. Beyond that, would turning it off improve query latency? ... View more