Member since
11-17-2015
33
Posts
12
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2538 | 06-20-2017 02:10 PM | |
64041 | 08-26-2016 01:14 PM | |
1140 | 07-03-2016 06:10 AM | |
30357 | 05-05-2016 02:58 PM | |
1500 | 05-04-2016 08:00 PM |
08-31-2017
06:30 PM
We either need to figure out how to get MirrorMaker 0.9.0 to use the new client api OR get MirrorMaker 0.10.1 to use the 0.9.0 compatible message format.
... View more
08-31-2017
05:11 PM
Hi, We need to use MirrorMaker to replicate data from 2 Kafka clusters running different versions. Source ------------------------------> Target Kafka 0.10.1 -> MirrorMaker -> Kafka 0.9.0 We are able to get this to work running MirrorMaker 0.9.0 using the "old zookeeper" consumer type, but would like to know if this is possible with MirrorMaker 0.10.1? We would like to limit the exposure of the source ZK cluster, hence the need to use the new MirrorMaker 0.10.1 consumer. Are there any consumer/producer configs we can use in MirrorMaker to allow us to use version 0.10.1?
# Exception in MirrorMaker when running version 0.10.1
[2017-08-31 16:43:15,799] ERROR Uncaught error in kafka producer I/O thread: (org.apache.kafka.clients.producer.internals.Sender) org.apache.kafka.common.protocol.types.SchemaException:
Error reading field 'brokers': Error reading field 'host': Error reading string
of length 26992, only 2176 bytes
Thanks,
Jon
... View more
Labels:
- Labels:
-
Apache Kafka
06-27-2017
04:30 PM
1 Kudo
We need to parse a real-time feed of XML documents. Using Storm, what is the best approach to process a real-time XML feed in an input parser bolt? JAXB? Best practices. Both good and bad. Pros and cons. Kinesis > Input XML Parser Bolt > Other Bolts
... View more
Labels:
- Labels:
-
Apache Storm
06-20-2017
02:10 PM
Here are the final hive configs that seem to have fixed this issue. Seems to be related to timeouts. set hive.execution.engine=mr;
set hive.default.fileformat=Orc;
set hive.exec.orc.default.compress=SNAPPY;
set hive.exec.copyfile.maxsize=1099511627776;
set hive.warehouse.subdir.inherit.perms=false;
set hive.metastore.pre.event.listeners=;
set hive.stats.fetch.partition.stats=false;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.dynamic.partition=true;
set fs.trash.interval=0;
set fs.s3.buffer.dir=/tmp/s3a;
set fs.s3a.attempts.maximum=50;
set fs.s3a.connection.establish.timeout=120000;
set fs.s3a.connection.timeout=120000;
set fs.s3a.fast.upload=true;
set fs.s3a.fast.upload.buffer=disk;
set fs.s3a.multiobjectdelete.enable=true;
set fs.s3a.max.total.tasks=2000;
set fs.s3a.threads.core=30;
set fs.s3a.threads.max=512;
set fs.s3a.connection.maximum=30;
set fs.s3a.fast.upload.active.blocks=12;
set fs.s3a.threads.keepalivetime=120;
... View more
06-13-2017
03:35 PM
This seems to be random. Sometimes we see this error; if we run it again and it succeeds. Not sure why we're seeing it though. Here are the hive properties we're using: set hive.execution.engine=mr;
set hive.default.fileformat=Orc;
set hive.exec.orc.default.compress=SNAPPY;
set fs.s3a.attempts.maximum=50;
set fs.s3a.connection.establish.timeout=30000;
set fs.s3a.connection.timeout=30000;
set fs.s3a.fast.upload=true;
set fs.s3a.fast.upload.buffer=disk;
set fs.s3n.multipart.uploads.enabled=true;
set fs.s3a.threads.keepalivetime=60;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.dynamic.partition=true;
We're running HDP 2.4.2 (HDP-2.4.2.0-258).
... View more
06-13-2017
02:38 PM
We are using Hive to load data to S3 (using s3a). We've started seeing the following error: 2017-06-13 08:51:49,042 ERROR [main]: exec.Task (SessionState.java:printError(962)) - Failed with exception Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CopyObjectResultHandler). Response Code: 200, Response Text: OK com.amazonaws.AmazonClientException: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CopyObjectResultHandler). Response Code: 200, Response Text: OK at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:738) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:399) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at com.amazonaws.services.s3.AmazonS3Client.copyObject(AmazonS3Client.java:1507) at com.amazonaws.services.s3.transfer.internal.CopyCallable.copyInOneChunk(CopyCallable.java:143) at com.amazonaws.services.s3.transfer.internal.CopyCallable.call(CopyCallable.java:131) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.copy(CopyMonitor.java:189) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:134) at com.amazonaws.services.s3.transfer.internal.CopyMonitor.call(CopyMonitor.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$CopyObjectResultHandler at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:150) at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseCopyObjectResponse(XmlResponsesSaxParser.java:417) at com.amazonaws.services.s3.model.transform.Unmarshallers$CopyObjectUnmarshaller.unmarshall(Unmarshallers.java:192) at com.amazonaws.services.s3.model.transform.Unmarshallers$CopyObjectUnmarshaller.unmarshall(Unmarshallers.java:189) at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62) at com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:44) at com.amazonaws.services.s3.internal.ResponseHeaderHandlerChain.handle(ResponseHeaderHandlerChain.java:30) at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:712) ... 13 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:251) at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:209) at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:171) at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:138) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.read1(BufferedReader.java:212) at java.io.BufferedReader.read(BufferedReader.java:286) at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source) at org.apache.xerces.impl.XMLEntityScanner.skipSpaces(Unknown Source) at org.apache.xerces.impl.XMLDocumentScannerImpl$PrologDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:141) ... 20 more Anyone else seen this before? Is it a data size/length issue? Loading too much data at once? Timeout?
... View more
Labels:
- Labels:
-
Apache Hive
02-21-2017
03:39 PM
@Tony Bolt After you do the downconfig, do your configs look correct? If you're not upconfig'ing them to the correct location in ZK, solr won't see the correct version of your configs. Also, check in the ZK CLI to make sure you're using the right znode. If you're znode isn't /solr, then you'll need to adjust the above commands appropriately. And make sure solr is looking in the right znode. I believe my znode was /solr and my configs were in /solr/configs.
... View more
01-13-2017
11:58 PM
Storm Version: 0.10.0.2.4 Using a Kafka Spout. How does storm handle failed tuples? How many times will storm retry a failed tuple? What frequency will storm retry the failed tuple? What is the max tuple count a topology can handle between all spouts and bolts?
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Storm
12-01-2016
05:03 PM
The user's saved queries weren't in this table. Which explains why they aren't seeing them. I opened one of our nightly pg dumps and pulled the user's query file location from the ds_savedquery_* table and cat'd them from hdfs and sent the output to the user.
cat hdfs_files.out
/user/xxxxxxx/hive/jobs/hive-job-813-2016-07-28_11-46/query.hql
/user/xxxxxxx/hive/jobs/hive-job-1952-2016-10-18_09-31/query.hql
...
for f in `cat hdfs_files.out`;do
> hdfs dfs -cat $f >> saved_queries.hql
> echo >> saved_queries.hql
> echo >> saved_queries.hql
> done
Thanks @jss for your help with this.
... View more
12-01-2016
03:41 PM
Thanks @jss! Which column in ds_jobimpl_* references that the query is a saved one? The user doesn't have 2100+ saved queries. This looks more like job history.
select count(*) from ds_jobimpl_6 where ds_owner = 'xxxxxxx';
2182
\d ds_jobimpl_6
ds_id | character varying(255) | not null
ds_applicationid | character varying(3000) |
ds_conffile | character varying(3000) |
ds_dagid | character varying(3000) |
ds_dagname | character varying(3000) |
ds_database | character varying(3000) |
ds_datesubmitted | bigint |
ds_duration | bigint |
ds_forcedcontent | character varying(3000) |
ds_globalsettings | character varying(3000) |
ds_logfile | character varying(3000) |
ds_owner | character varying(3000) |
ds_queryfile | character varying(3000) |
ds_queryid | character varying(3000) |
ds_referrer | character varying(3000) |
ds_sessiontag | character varying(3000) |
ds_sqlstate | character varying(3000) |
ds_status | character varying(3000) |
ds_statusdir | character varying(3000) |
ds_statusmessage | character varying(3000) |
ds_title | character varying(3000) |
hdfs dfs -find /user/xxxxxxx -name *.hql | wc -l
2546
... View more
12-01-2016
03:08 PM
Also, is there a difference in location between, ambari-only users vs ambari/linux users? (Still referring to the hive view) linux user = the user has an account on the linux box ambari user = the user has an ambari account We have a user that only had an ambari account. They seem to have lost their queries after we created a linux account for them. Both ambari and linux account names are the same.
... View more
12-01-2016
03:00 PM
1 Kudo
In the Ambari Hive view, there is a "Saved Queries" tab, where are these queries saved? Are they in the Ambari DB? A local file on the Ambari node? HDFS?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
08-26-2016
01:14 PM
1 Kudo
Hi @Simran Kaur, You can run the following command to execute a HQL file in hive: hive -f filename.hql Be sure each of your queries in your HQL are terminated with a semi-colon ; Here's the help output of hive too: $ hive -H
WARNING: Use "yarn jar" to launch YARN applications.
hivusage: hive
-d,--define <key=value> Variable subsitution to apply to hive
commands. e.g. -d A=B or --define A=B
--database <databasename> Specify the database to use
-e <quoted-query-string> SQL from command line
-f <filename> SQL from files
-H,--help Print help information
--hiveconf <property=value> Use value for given property
--hivevar <key=value> Variable subsitution to apply to hive
commands. e.g. --hivevar A=B
-i <filename> Initialization SQL file
-S,--silent Silent mode in interactive shell
-v,--verbose Verbose mode (echo executed SQL to the
console)
And here is great documentation for all hive cli options: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Cli
... View more
08-15-2016
09:06 PM
1 Kudo
After setting this property to false, we are no longer seeing extremely high ulimits for the hive user account. hive.server2.logging.operation.enabled=false Any idea what this is doing to consume so many open files?
... View more
08-04-2016
01:06 PM
Thanks @Hajime for the suggestion, netstat looks OK, not too many connections opened.
... View more
08-02-2016
06:48 PM
Thanks @Benjamin Leonhardi. That's what I am wondering as well. Something doesn't seem right. Can you point me to the ATS issue you are referring to?
... View more
08-02-2016
06:47 PM
Thanks @Scott Shaw We have found that the user's managed version of the limits conf (/etc/security/limits.d/hive.conf) is being used instead of the limits.conf file. Changing the hive.conf file manually gets overwritten once ambari sees that it has changed. But the real question is, should hive really have 64k+ files open?
... View more
07-30-2016
06:56 PM
1 Kudo
By default, hive's ullimit (managed through ambari) is 32000. We reached that limit last week on our hiveserver2 server and decided to increase this value in ambari to 64000. We just hit the 64k nofile ulimit. This leads me to believe that hiveserver2 is not cleaning up connections like it should and files are not being released. Has anyone else experienced this issue? Any suggestions on what to check? What do you have your hive user's nofile limit set too? # lsof -u hive | wc -l 64450 We are on HDP 2.4.2, Ambari 2.2.2 Should hive really have that many files open? Update: We're approaching the 64k nofile ulimit setting again for the hive user. # lsof -u hive | wc -l 57090 After digging through the output of lsof, I see a lot of temporary operation_logs. /tmp/hive/operation_logs/658c3930-8975-47db-ad7f-7cbef6279b11/acc2043a-d3bb-4a8c-9a7d-d0b743b9ce5d Here is the total number of open operation_logs files open right now. # lsof -u hive | grep operation_logs | wc -l 56102 These files are 3 to 4 days old.
... View more
Labels:
- Labels:
-
Apache Hive
07-03-2016
06:17 AM
Hi @Sunile Manjee, Thank you for your response. This is the documentation I followed to setup this environment: https://doc.lucidworks.com/lucidworks-hdpsearch/2.3/Guide-Install.html I will be testing performance against HDFS indexing with NRT setup. I have local SSD disks setup as a fallback if this isn't fast enough or too unreliable. Thanks, Jon
... View more
07-03-2016
06:10 AM
After more digging, I discovered the solrconfig.xml in ZK was not the correct version. I did a series of downconfig and upconfig to load the correct configs and verify everything is OK. After loading the correct solrconfig.xml and restarting each solr node, the create collection command succeeded. /opt/lucidworks-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -cmd downconfig -d collection -z $zk_quorum:2181/solr -n collection
/opt/lucidworks-hdpsearch/solr/server/scripts/cloud-scripts/zkcli.sh -cmd upconfig -d $path_to_configs -z $zk_quorum:2181/solr -n collection
... View more
07-02-2016
06:14 AM
Hello, I am trying to setup and configure HDPSearch. I have 4 solr boxes running 6 instances of solr. I have setup HDFS with NN HA. All 4 boxes can successfully reach HDFS using the NN HA name. However, I am receiving the below error when trying to create a collection in solr. What is solr missing that it can't connect to HDFS?
126330 ERROR (qtp59559151-22) [c:collection s:shard23 r:core_node86 x:collection_shard23_replica3] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Error CREATEing SolrCore 'collection_shard23_replica3': Unable to create core [collection_shard23_replica3] Caused by: NN_HA_Name.
.. 31 more
Caused by: java.net.UnknownHostException: NN_HA_Name
... 45 more
Here is the command to start solr cloud:
solr -c -p 8983 -z $zk_quorum:2181/solr -Dsolr.directoryFactory=HdfsDirectoryFactory -Dsolr.lock.type=hdfs-Dsolr.hdfs.home=hdfs://NN_HA_Name/apps/solr
Here is the command to create the collection:
solr create -c collection -d collection -n collection -s 48 -rf 3
Here are my solrconfig.xml DirectoryFactory Settings:
<directoryFactory name="DirectoryFactory" class="solr.HdfsDirectoryFactory">
<str name="solr.hdfs.home">hdfs://NN_HA_Name/apps/solr</str>
<str name="solr.hdfs.confdir">/etc/hadoop/conf</str>
<bool name="solr.hdfs.blockcache.enabled">true</bool>
<int name="solr.hdfs.blockcache.slab.count">1</int>
<bool name="solr.hdfs.blockcache.direct.memory.allocation">true</bool>
<int name="solr.hdfs.blockcache.blocksperbank">16384</int>
<bool name="solr.hdfs.blockcache.read.enabled">true</bool>
<bool name="solr.hdfs.nrtcachingdirectory.enable">true</bool>
<int name="solr.hdfs.nrtcachingdirectory.maxmergesizemb">16</int>
<int name="solr.hdfs.nrtcachingdirectory.maxcachedmb">192</int>
</directoryFactory>
I have installed the hdfs clients on the solr nodes and can successfully hdfs dfs -ls hdfs://NN_HA_Name/apps/solr I also see core-site.xml and hdfs-site.xml (with the correct NN configurations) in the /etc/hadoop/conf directory. Thanks,
Jon
... View more
Labels:
- Labels:
-
Apache Solr
05-05-2016
02:58 PM
2 Kudos
@Venkat ramanann On your Postgres server, you will need to update your pg_hba.conf file to allow access for the ambari user on the ambari database coming from 127.0.0.1. Here is the location of our pg_hba.conf file: /data/pghadoop/pg_hba.conf If its not there, run: find / -name pg_hba.conf Here is Postgres documentation for configuring this file: http://www.postgresql.org/docs/9.5/static/auth-pg-hba-conf.html This is what our pg_hba.conf file has for the ambari user: local all ambari trust
host all ambari 0.0.0.0/0 trust
host all ambari ::/0 trust Once you have made these changes, you will need to restart the Postgres server: /etc/init.d/postgresql restart Let me know if you have any other questions.
... View more
05-04-2016
08:21 PM
1 Kudo
@Predrag Minovic, can you explain why Kafka needs its own Zk quorum? Why can't it utilize an existing Zk quorum? We are migrating to Kafka in production and I would like to get your take on this.
... View more
05-04-2016
08:00 PM
@Shankar B On your ambari server, edit the nagios config file in this location:
/var/lib/ambari-server/resources/stacks/HDP/2.0.6/services/NAGIOS/package/templates/ This is the location where ambari stores the configuration files for nagios. Once you have edited the file, restart the ambari-server process. This will push your new nagios changes to your nagios server. Then in ambari, restart the nagios service. You should now see your changes in the Nagios Admin UI. We had a similiar case where we needed more than one contact in nagios. Updating the contacts.cfg.j2 on the ambari server pushed the new contacts into the nagios config and everything worked like a charm.
... View more
05-04-2016
07:42 PM
1 Kudo
@Michel Brown, here is the HDP documentation for using non-default databases in Ambari. These are for Ambari 2.2.1.0:
Using Non-Default Databases - Ambari <- For Ambari Server
Using Non-Default Databases <- For other HDP services We currently use Postgres as our backend DB for all of our services.
... View more
05-04-2016
07:19 PM
@Sami Ahmad, The permissions on your /user/hadoop HDFS directory are incorrect. You need to run this command to change the ownership of the HDFS directory:
hdfs dfs -chown hadoop:hdfs /user/hadoop The error message shows you the current permissions on the /user/hadoop HDFS directory along with the type of access the process is trying to perform. Changing the /user/hadoop HDFS directory's owner using the above command should get you going again. ERROR : Failed to execute tez graph. org.apache.hadoop.security.AccessControlException: Permission denied: user=hadoop, access=WRITE, inode="/user/hadoop":hdfs:hdfs:drwxr-xr-x
... View more