Re: Hive to S3 Error - timeout?

Jonathan_M_Maes · ‎10-16-2018

We have data that is gzip compressed with no file extension. Currently hive just outputs garbage as it assumes the data is raw text files. Is there a table/hive property we can use to flag this data as gzip compression? This is an interesting solution, but hoping there may be an easier way by now: http://daynebatten.com/2015/11/override-hadoop-compression-codec-file-extension/ HDP: 2.6.4 Hive: 1.2.1000

Jonathan_M_Maes · ‎08-31-2017

We either need to figure out how to get MirrorMaker 0.9.0 to use the new client api OR get MirrorMaker 0.10.1 to use the 0.9.0 compatible message format.

Jonathan_M_Maes · ‎08-31-2017

Hi, We need to use MirrorMaker to replicate data from 2 Kafka clusters running different versions. Source ------------------------------> Target Kafka 0.10.1 -> MirrorMaker -> Kafka 0.9.0 We are able to get this to work running MirrorMaker 0.9.0 using the "old zookeeper" consumer type, but would like to know if this is possible with MirrorMaker 0.10.1? We would like to limit the exposure of the source ZK cluster, hence the need to use the new MirrorMaker 0.10.1 consumer. Are there any consumer/producer configs we can use in MirrorMaker to allow us to use version 0.10.1? # Exception in MirrorMaker when running version 0.10.1 [2017-08-31 16:43:15,799] ERROR Uncaught error in kafka producer I/O thread: (org.apache.kafka.clients.producer.internals.Sender) org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'brokers': Error reading field 'host': Error reading string of length 26992, only 2176 bytes Thanks, Jon

Jonathan_M_Maes · ‎06-27-2017

We receive around 40 messages per sec.

Jonathan_M_Maes · ‎06-27-2017

We receive around 40 messages per sec in this topology.

Jonathan_M_Maes · ‎06-27-2017

Hi @Scott Shaw, The XMLs are 50-75kb (700 elements).

Jonathan_M_Maes · ‎06-27-2017

We need to parse a real-time feed of XML documents. Using Storm, what is the best approach to process a real-time XML feed in an input parser bolt? JAXB? Best practices. Both good and bad. Pros and cons. Kinesis > Input XML Parser Bolt > Other Bolts

Jonathan_M_Maes · ‎06-20-2017

Here are the final hive configs that seem to have fixed this issue. Seems to be related to timeouts. set hive.execution.engine=mr; set hive.default.fileformat=Orc; set hive.exec.orc.default.compress=SNAPPY; set hive.exec.copyfile.maxsize=1099511627776; set hive.warehouse.subdir.inherit.perms=false; set hive.metastore.pre.event.listeners=; set hive.stats.fetch.partition.stats=false; set hive.exec.dynamic.partition.mode=nonstrict; set hive.exec.dynamic.partition=true; set fs.trash.interval=0; set fs.s3.buffer.dir=/tmp/s3a; set fs.s3a.attempts.maximum=50; set fs.s3a.connection.establish.timeout=120000; set fs.s3a.connection.timeout=120000; set fs.s3a.fast.upload=true; set fs.s3a.fast.upload.buffer=disk; set fs.s3a.multiobjectdelete.enable=true; set fs.s3a.max.total.tasks=2000; set fs.s3a.threads.core=30; set fs.s3a.threads.max=512; set fs.s3a.connection.maximum=30; set fs.s3a.fast.upload.active.blocks=12; set fs.s3a.threads.keepalivetime=120;

Jonathan_M_Maes · ‎06-13-2017

This seems to be random. Sometimes we see this error; if we run it again and it succeeds. Not sure why we're seeing it though. Here are the hive properties we're using: set hive.execution.engine=mr; set hive.default.fileformat=Orc; set hive.exec.orc.default.compress=SNAPPY; set fs.s3a.attempts.maximum=50; set fs.s3a.connection.establish.timeout=30000; set fs.s3a.connection.timeout=30000; set fs.s3a.fast.upload=true; set fs.s3a.fast.upload.buffer=disk; set fs.s3n.multipart.uploads.enabled=true; set fs.s3a.threads.keepalivetime=60; set hive.exec.dynamic.partition.mode=nonstrict; set hive.exec.dynamic.partition=true; We're running HDP 2.4.2 (HDP-2.4.2.0-258).

Jonathan_M_Maes · ‎06-13-2017

We are using Hive to load data to S3 (using s3a). We've started seeing the following error: 2017-06-13 08:51:49,042 ERROR [main]: exec.Task (SessionState.java:printError(962)) - Failed with exception Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CopyObjectResultHandler). Response Code: 200, Response Text: OK com.amazonaws.AmazonClientException: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CopyObjectResultHandler). Response Code: 200, Response Text: OK at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:738) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:399) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.copyObject(AmazonS3Client.java:1507) at com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInOneChunk(CopyCallable.java:143) at com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:131) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.copy(CopyMonitor.java:189) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:134) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CopyObjectResultHandler at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:150) at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseCopyObjectResponse(XmlResponsesSaxParser.java:417) at com.amazonaws.services.s3.model.transform.Unmarshallers$CopyObjectUnmarshaller.unmarshall(Unmarshallers.java:192) at com.amazonaws.services.s3.model.transform.Unmarshallers$CopyObjectUnmarshaller.unmarshall(Unmarshallers.java:189) at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62) at com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44) at com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30) at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:712) ... 13 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:251) at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:209) at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:171) at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:138) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.read1(BufferedReader.java:212) at java.io.BufferedReader.read(BufferedReader.java:286) at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source) at org.apache.xerces.impl.XMLEntityScanner.skipSpaces(Unknown Source) at org.apache.xerces.impl.XMLDocumentScannerImpl$PrologDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:141) ... 20 more Anyone else seen this before? Is it a data size/length issue? Loading too much data at once? Timeout?

Jonathan_M_Maes · ‎02-21-2017

@Tony Bolt After you do the downconfig, do your configs look correct? If you're not upconfig'ing them to the correct location in ZK, solr won't see the correct version of your configs. Also, check in the ZK CLI to make sure you're using the right znode. If you're znode isn't /solr, then you'll need to adjust the above commands appropriately. And make sure solr is looking in the right znode. I believe my znode was /solr and my configs were in /solr/configs.

Jonathan_M_Maes · ‎01-13-2017

Storm Version: 0.10.0.2.4 Using a Kafka Spout. How does storm handle failed tuples? How many times will storm retry a failed tuple? What frequency will storm retry the failed tuple? What is the max tuple count a topology can handle between all spouts and bolts?

Jonathan_M_Maes · ‎12-01-2016

The user's saved queries weren't in this table. Which explains why they aren't seeing them. I opened one of our nightly pg dumps and pulled the user's query file location from the ds_savedquery_* table and cat'd them from hdfs and sent the output to the user. cat hdfs_files.out /user/xxxxxxx/hive/jobs/hive-job-813-2016-07-28_11-46/query.hql /user/xxxxxxx/hive/jobs/hive-job-1952-2016-10-18_09-31/query.hql ... for f in `cat hdfs_files.out`;do > hdfs dfs -cat $f >> saved_queries.hql > echo >> saved_queries.hql > echo >> saved_queries.hql > done Thanks @jss for your help with this.

Jonathan_M_Maes · ‎12-01-2016

Jonathan_M_Maes · ‎12-01-2016

Also, is there a difference in location between, ambari-only users vs ambari/linux users? (Still referring to the hive view) linux user = the user has an account on the linux box ambari user = the user has an ambari account We have a user that only had an ambari account. They seem to have lost their queries after we created a linux account for them. Both ambari and linux account names are the same.

Jonathan_M_Maes · ‎12-01-2016

In the Ambari Hive view, there is a "Saved Queries" tab, where are these queries saved? Are they in the Ambari DB? A local file on the Ambari node? HDFS?

Jonathan_M_Maes · ‎08-26-2016

Hi @Simran Kaur, You can run the following command to execute a HQL file in hive: hive -f filename.hql Be sure each of your queries in your HQL are terminated with a semi-colon ; Here's the help output of hive too: $ hive -H WARNING: Use "yarn jar" to launch YARN applications. hivusage: hive -d,--define <key=value> Variable subsitution to apply to hive commands. e.g. -d A=B or --define A=B --database <databasename> Specify the database to use -e <quoted-query-string> SQL from command line -f <filename> SQL from files -H,--help Print help information --hiveconf <property=value> Use value for given property --hivevar <key=value> Variable subsitution to apply to hive commands. e.g. --hivevar A=B -i <filename> Initialization SQL file -S,--silent Silent mode in interactive shell -v,--verbose Verbose mode (echo executed SQL to the console) And here is great documentation for all hive cli options: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli

Jonathan_M_Maes · ‎08-15-2016

After setting this property to false, we are no longer seeing extremely high ulimits for the hive user account. hive.server2.logging.operation.enabled=false Any idea what this is doing to consume so many open files?

Jonathan_M_Maes · ‎08-04-2016

Thanks @Hajime for the suggestion, netstat looks OK, not too many connections opened.

Jonathan_M_Maes · ‎08-02-2016

Thanks @Benjamin Leonhardi. That's what I am wondering as well. Something doesn't seem right. Can you point me to the ATS issue you are referring to?

Jonathan_M_Maes · ‎08-02-2016

Thanks @Scott Shaw We have found that the user's managed version of the limits conf (/etc/security/limits.d/hive.conf) is being used instead of the limits.conf file. Changing the hive.conf file manually gets overwritten once ambari sees that it has changed. But the real question is, should hive really have 64k+ files open?

Jonathan_M_Maes · ‎07-30-2016

By default, hive's ullimit (managed through ambari) is 32000. We reached that limit last week on our hiveserver2 server and decided to increase this value in ambari to 64000. We just hit the 64k nofile ulimit. This leads me to believe that hiveserver2 is not cleaning up connections like it should and files are not being released. Has anyone else experienced this issue? Any suggestions on what to check? What do you have your hive user's nofile limit set too? # lsof -u hive | wc -l 64450 We are on HDP 2.4.2, Ambari 2.2.2 Should hive really have that many files open? Update: We're approaching the 64k nofile ulimit setting again for the hive user. # lsof -u hive | wc -l 57090 After digging through the output of lsof, I see a lot of temporary operation_logs. /tmp/hive/operation_logs/658c3930-8975-47db-ad7f-7cbef6279b11/acc2043a-d3bb-4a8c-9a7d-d0b743b9ce5d Here is the total number of open operation_logs files open right now. # lsof -u hive | grep operation_logs | wc -l 56102 These files are 3 to 4 days old.

Jonathan_M_Maes · ‎07-03-2016

Hi @Sunile Manjee, Thank you for your response. This is the documentation I followed to setup this environment: https://doc.lucidworks.com/lucidworks-hdpsearch/2.3/Guide-Install.html I will be testing performance against HDFS indexing with NRT setup. I have local SSD disks setup as a fallback if this isn't fast enough or too unreliable. Thanks, Jon

Jonathan_M_Maes · ‎07-03-2016

After more digging, I discovered the solrconfig.xml in ZK was not the correct version. I did a series of downconfig and upconfig to load the correct configs and verify everything is OK. After loading the correct solrconfig.xml and restarting each solr node, the create collection command succeeded. /opt/lucidworks-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -cmd downconfig -d collection -z $zk_quorum:2181/solr -n collection /opt/lucidworks-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -d $path_to_configs -z $zk_quorum:2181/solr -n collection

Jonathan_M_Maes · ‎07-02-2016

Hello, I am trying to setup and configure HDPSearch. I have 4 solr boxes running 6 instances of solr. I have setup HDFS with NN HA. All 4 boxes can successfully reach HDFS using the NN HA name. However, I am receiving the below error when trying to create a collection in solr. What is solr missing that it can't connect to HDFS? 126330 ERROR (qtp59559151-22) [c:collection s:shard23 r:core_node86 x:collection_shard23_replica3] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'collection_shard23_replica3': Unable to create core [collection_shard23_replica3] Caused by: NN_HA_Name. .. 31 more Caused by: java.net.UnknownHostException: NN_HA_Name ... 45 more Here is the command to start solr cloud: solr -c -p 8983 -z $zk_quorum:2181/solr -Dsolr.directoryFactory=HdfsDirectoryFactory -Dsolr.lock.type=hdfs-Dsolr.hdfs.home=hdfs://NN_HA_Name/apps/solr Here is the command to create the collection: solr create -c collection -d collection -n collection -s 48 -rf 3 Here are my solrconfig.xml DirectoryFactory Settings: <directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory"> <str name="solr.hdfs.home">hdfs://NN_HA_Name/apps/solr</str> <str name="solr.hdfs.confdir">/etc/hadoop/conf</str> <bool name="solr.hdfs.blockcache.enabled">true</bool> <int name="solr.hdfs.blockcache.slab.count">1</int> <bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool> <int name="solr.hdfs.blockcache.blocksperbank">16384</int> <bool name="solr.hdfs.blockcache.read.enabled">true</bool> <bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool> <int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int> <int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int> </directoryFactory> I have installed the hdfs clients on the solr nodes and can successfully hdfs dfs -ls hdfs://NN_HA_Name/apps/solr I also see core-site.xml and hdfs-site.xml (with the correct NN configurations) in the /etc/hadoop/conf directory. Thanks, Jon

Jonathan_M_Maes · ‎05-05-2016

@Venkat ramanann On your Postgres server, you will need to update your pg_hba.conf file to allow access for the ambari user on the ambari database coming from 127.0.0.1. Here is the location of our pg_hba.conf file: /data/pghadoop/pg_hba.conf If its not there, run: find / -name pg_hba.conf Here is Postgres documentation for configuring this file: http://www.postgresql.org/docs/9.5/static/auth-pg-hba-conf.html This is what our pg_hba.conf file has for the ambari user: local all ambari trust host all ambari 0.0.0.0/0 trust host all ambari ::/0 trust Once you have made these changes, you will need to restart the Postgres server: /etc/init.d/postgresql restart Let me know if you have any other questions.

Jonathan_M_Maes · ‎05-04-2016

@Predrag Minovic, can you explain why Kafka needs its own Zk quorum? Why can't it utilize an existing Zk quorum? We are migrating to Kafka in production and I would like to get your take on this.

Jonathan_M_Maes · ‎05-04-2016

@Shankar B On your ambari server, edit the nagios config file in this location: /var/lib/ambari-server/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/ This is the location where ambari stores the configuration files for nagios. Once you have edited the file, restart the ambari-server process. This will push your new nagios changes to your nagios server. Then in ambari, restart the nagios service. You should now see your changes in the Nagios Admin UI. We had a similiar case where we needed more than one contact in nagios. Updating the contacts.cfg.j2 on the ambari server pushed the new contacts into the nagios config and everything worked like a charm.

Jonathan_M_Maes · ‎05-04-2016

@Michel Brown, here is the HDP documentation for using non-default databases in Ambari. These are for Ambari 2.2.1.0: Using Non-Default Databases - Ambari <- For Ambari Server Using Non-Default Databases <- For other HDP services We currently use Postgres as our backend DB for all of our services.

Jonathan_M_Maes · ‎05-04-2016

@Sami Ahmad, The permissions on your /user/hadoop HDFS directory are incorrect. You need to run this command to change the ownership of the HDFS directory: hdfs dfs -chown hadoop:hdfs /user/hadoop The error message shows you the current permissions on the /user/hadoop HDFS directory along with the type of access the process is trying to perform. Changing the /user/hadoop HDFS directory's owner using the above command should get you going again. ERROR : Failed to execute tez graph. org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop, access=WRITE, inode="/user/hadoop":hdfs:hdfs:drwxr-xr-x

Online	Offline
Last Visited	‎10-29-2018 12:52 PM

Date Registered	‎11-17-2015 09:15 PM
Last Visited	‎10-29-2018 12:52 PM
Total Messages Posted	33
Total Kudos Received	9

Re: Hive to S3 Error - timeout?

Re: How to run all of hive queries in a file in on...

Re: HDPSearch - failed to create collection - Unkn...

Re: PSQLException: FATAL: no pg_hba.conf entry for...

Re: Ambari is overiding the config change

Use gzip input codec on files without .gz extensio...

Re: Kafka MirrorMaker - Mirroring different Kafka ...

Kafka MirrorMaker - Mirroring different Kafka vers...

Re: Storm XML parser

Re: Storm XML parser

Re: Storm XML parser

Storm XML parser

Re: Hive to S3 Error - timeout?

Re: Hive to S3 Error - timeout?

Hive to S3 Error - timeout?

Re: HDPSearch - failed to create collection - Unkn...

How does storm handle failed tuples?

Re: Ambari Hive View - User's Saved Queries Locati...

Re: Ambari Hive View - User's Saved Queries Locati...

Re: Ambari Hive View - User's Saved Queries Locati...

Ambari Hive View - User's Saved Queries Location

Re: How to run all of hive queries in a file in on...

Re: HiveServer2 Hive user's nofile ulimit above 64...

Re: HiveServer2 Hive user's nofile ulimit above 64...

Re: HiveServer2 Hive user's nofile ulimit above 64...

Re: HiveServer2 Hive user's nofile ulimit above 64...

HiveServer2 Hive user's nofile ulimit above 64000?

Re: HDPSearch - failed to create collection - Unkn...

Re: HDPSearch - failed to create collection - Unkn...

HDPSearch - failed to create collection - UnknownH...

Re: PSQLException: FATAL: no pg_hba.conf entry for...

Re: Kafka cluster design

Re: Ambari is overiding the config change

Re: Ambari install without derby

Re: insert in HIVE failing