Member since
02-22-2017
33
Posts
6
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1210 | 10-28-2016 09:38 AM |
04-18-2018
07:21 AM
We have an application that reads messages from specific kafka topics, and process it, and when it reads message from topic it puts offset to the HBase table. after some amount of working application fails (time varries from 30 minutes to 15 hours ), in the driver stderr we see following log entries: 18/04/17 17:31:15 WARN client.AsyncProcess: #3121, the task was rejected by the pool. This is unexpected. Server is ***hostname masked***,60020,1523949367813 java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3f377224 rejected from java.util.concurrent.ThreadPoolExecutor@639d4dae[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1013) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$000(AsyncProcess.java:600) at org.apache.hadoop.hbase.client.AsyncProcess.submitMultiActions(AsyncProcess.java:449) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:429) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:238) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1495) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1098) And after some amount of time this ERRORS: 18/04/17 17:31:15 ERROR client.AsyncProcess: Cannot get replica 0 location for {"totalColumns":1,"row":"predictor_passport_ru_number_gold","families":{"cf":[{"qualifier":"\x00\x00\x00\x00","vlen":8,"tag":[],"timestamp":9223372036854775807}]}} 18/04/17 17:31:15 ERROR spark.Utils: Error saving offsets [OffsetRange(topic: 'predictor_passport_ru_number_gold', partition: 0, range: [2536631 -> 2536718])] org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1 time, at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:247) at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:227) at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1766) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1495) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1098) In the HBase logs I see an gap in messages on that period of time you can see this on attached screenshot - memstoreflush.png In addition full log of driver in index.zip. Please help to investigate and solve the issue.
... View more
Labels:
- Labels:
-
Apache HBase
-
Apache Spark
12-19-2017
03:34 AM
Hi, was You able to fix the issue? We have the same problem.
... View more
04-04-2017
03:53 PM
This event occurs only if we are using NiFi Hive Streaming. ls -R will later.
... View more
04-04-2017
01:58 PM
Hi All, Periodically, in some ORC tables in Hive we get duplicate partition "base" directory inside /table_name/partition_date=/base/ meaning: all contents of /table_name/partition_date=/base/* are in /table_name/partition_date=/base/base/*. After that partition become bad and from this bad partition we can’t do select count(*) or any other selects because of error occurring. But when we dropping duplicate “base” directory problem goes away. Why we got this duplicate folder in our buckets?
... View more
Labels:
- Labels:
-
Apache Hive
03-31-2017
02:21 PM
Our NiFi is co-located with other Hadoop components.
This is physical servers. 24 Cores per machine.templates.zip Zookeper is separate but on this machines. Errors on DetectDuplicate processor are symptoms of this issue. Socket Timeouts too. We have 3 Process Group on Our NiFi Cluster, their templates are in attachment.
... View more
03-31-2017
12:49 PM
Our NiFi have 8 Gb of heap, NiFi version is 1.1.0.2.
... View more
03-29-2017
10:17 AM
nifi-app.zipAfter working one weak our NiFi cluster become very unstable. Nodes are disconnect and reconnect every 5 - 30 minutes, processors don't work fine too. Restarting all 3 nodes solve the issue. Restarting NiFi weakly is not a good solution but we can work only with this approach. Example of log file from one of the node in attachment.
... View more
Labels:
- Labels:
-
Apache NiFi
03-16-2017
02:50 AM
Hi, explain statement is very huge so You can download it from our share: https://drive.croc.ru/display/data/list?dataId=c43e16e0-e0af-40f1-935e-1c44e4b01f91 login: 024741 password: E804F9487956
... View more
03-15-2017
07:03 AM
Hi, Tried to increase this property 10x but no results. Regards, Ramil.
... View more
03-13-2017
02:09 AM
Attachments mentioned above can be found on https://drive.croc.ru/display/data/list?dataId=02745bf5-e54d-47a9-8797-15f108fc057e login: 024556 password: 0B908ECFE563
... View more
- Tags:
- Hive
03-13-2017
02:09 AM
Attachments mentioned above can be found on https://drive.croc.ru/display/data/list?dataId=02745bf5-e54d-47a9-8797-15f108fc057e login: 024556 password: 0B908ECFE563
... View more
03-13-2017
02:03 AM
Hi Community, We have a script - collmx_consents_snp.hql (in attachement). In this script we use join of following tables: consent_service_consent_hst consent_service_consent_subject_hst consent_service_client_hst DDL of those table are in attachment too. All of this tables are partitioned by date. The root cause of problem is that join of two tables in production don’t working(there is consistent data in tables but query don’t get nothing): select * from ( SELECT consent_uid,CASE WHEN for_contract = true THEN evid_srv ELSE NULL END evid_srv,entity_type,to_date(modif_time) apply_date,id_client FROM consent_service_consent_snp ) csc join consent_service_consent_subject_snp cscs on (csc.consent_uid = cscs.consent_uid) In test environment all fine. When we add some filter on table consent_service_consent_snp by partition, than query giva us results: When we are running select count(*) on this tables, we didn’t get any errors. In our test environment we have less data than in production. And when we add constraint on date in select clause all working fine, so we think that problem may depend on number of rows in the table. Logs of HiveServer2 and HiveMetastore in attachement. When query is fails we see following in hiveserver2 log: 2017-03-09 20:00:30,406 INFO org.apache.hadoop.hive.ql.plan.ConditionalResolverCommonJoin: [HiveServer2-Background-Pool: Thread-7269]: Failed to resolve driver alias (threshold : 25000000, length mapping : {cscs:consent_service_consent_subject_hst=571829172, csc:consent_service_consent_snp:consent_service_consent_hst=434475747})
... View more
Labels:
- Labels:
-
Hive
10-28-2016
09:38 AM
Issue was solved by my self. The solutin was: 1) under folder in which workflow.xml is create folder lib and put there all hive jar files from sharedlibDir(/user/oozie/share/lib/lib_20160928171540)/hive; 2) Create hive-site.xml with contents: <configuration>
<property>
<name>ambari.hive.db.schema.name</name>
<value>hive</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://xxxxx:9083</value>
</property>
<property>
<name>hive.zookeeper.quorum</name>
<value>xxxx:2181,yyyyy:2181,zzzzzz:2181</value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/smartdata/hive/</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>org.postgresql.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:postgresql://xxxxx:5432/hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
</configuration>
and put it on hdfs for example in /tmp/hive-site.xml 3) Add following line in workflow.xml: <file>/tmp/hive-site.xml</file> This solved my issue.
... View more
10-27-2016
09:25 AM
Hello thanks for advice but all fine with shared libraries: $ oozie admin -oozie http://localhost:11000/oozie -shareliblist
[Available ShareLib]
hive
distcp
mapreduce-streaming
spark
oozie
hcatalog
hive2
sqoop
pig
spark_orig $ oozie admin -oozie http://localhost:11000/oozie -sharelibupdate
[ShareLib update status]
sharelibDirOld = hdfs://os-2471.homecredit.ru:8020/user/oozie/share/lib/lib_20160928171540
host = http://localhost:11000/oozie
sharelibDirNew = hdfs://os-2471.homecredit.ru:8020/user/oozie/share/lib/lib_20160928171540
status = Successful $ oozie admin -oozie http://localhost:11000/oozie -shareliblist
[Available ShareLib]
hive
distcp
mapreduce-streaming
spark
oozie
hcatalog
hive2
sqoop
pig
spark_orig On Resource manager UI all fine, see attached logs.
... View more
10-26-2016
12:47 PM
resource-manager-ui.txtHello, Our HDP version 2.5 When we trying to run sqoop action(to load data from oracle to hive) from oozie we get folowing error in /var/log/oozie/oozie-error.log: JOB[0000004-161024200820785-oozie-oozi-W] ACTION[0000004-161024200820785-oozie-oozi-W@sqoop] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] And there is nothing more usefull for diagnostic. Job.properties file listed below: # properties nameNode = hdfs://xxxxx:8020 resourceManager = xxxx:8050 queueName=default oozie.use.system.libpath=true oozie.wf.application.path = hdfs://xxxxxx:8020/smartdata/oozie/hive_test.xml mapreduce.framework.name = yarn When we running this job from command line with "sqoop ..... " as command all working fine. Please some one tell me how to solve or troubleshoot this.
... View more
Labels:
- Labels:
-
Apache Oozie
-
Apache Sqoop
10-17-2016
08:05 AM
Is the any workaround of this? or some hot fix?
... View more
10-14-2016
06:58 PM
1 Kudo
After we enabled HDFS HA PutHiveStreaming processor in our NiFi stopped working and generate following errors: 2016-10-14 21:50:53,840 WARN [Timer-Driven Process Thread-6] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=01571000-c4de-1bfd-0f09-5c439230e84e] Processor Administratively Yielded for 1 sec due to processing failure 2016-10-14 21:50:53,840 WARN [Timer-Driven Process Thread-6] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding PutHiveStreaming[id=01571000-c4de-1bfd-0f09-5c439230e84e] due to uncaught Exception: java.lang.IllegalArgumentException: java.net.UnknownHostException: hdpCROC 2016-10-14 21:50:53,847 WARN [Timer-Driven Process Thread-6] o.a.n.c.t.ContinuallyRunProcessorTask java.lang.IllegalArgumentException: java.net.UnknownHostException: hdpCROC at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:411) ~[na:na] at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311) ~[na:na] at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) ~[na:na] at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:688) ~[na:na] at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:629) ~[na:na] at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:159) ~[na:na] at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2761) ~[na:na] at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) ~[na:na] at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2795) ~[na:na] at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2777) ~[na:na] at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:386) ~[na:na] at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) ~[na:na] at org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.<init>(OrcRecordUpdater.java:234) ~[na:na] at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat.getRecordUpdater(OrcOutputFormat.java:289) ~[na:na] at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.createRecordUpdater(AbstractRecordWriter.java:253) ~[na:na] at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.createRecordUpdaters(AbstractRecordWriter.java:245) ~[na:na] at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.newBatch(AbstractRecordWriter.java:189) ~[na:na] at org.apache.hive.hcatalog.streaming.StrictJsonWriter.newBatch(StrictJsonWriter.java:41) ~[na:na] at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:607) ~[na:na] at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:555) ~[na:na] at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.fetchTransactionBatchImpl(HiveEndPoint.java:441) ~[na:na] at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.fetchTransactionBatch(HiveEndPoint.java:421) ~[na:na] at org.apache.nifi.util.hive.HiveWriter.lambda$nextTxnBatch$7(HiveWriter.java:250) ~[na:na] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: java.net.UnknownHostException: hdpCROC hdpCROC - our HDP cluster and dfs.servicenames property. All files such as hive-site.xml, hdfs-site.xml, hdfs-core.xml are actual. What can cause this issue?
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
-
Apache NiFi
10-07-2016
04:11 PM
Thank You for quick reply, can You tell me please where can I get consumeKafka_o_10 nifi processor?
... View more
10-07-2016
02:49 PM
1 Kudo
When we trying to use getkafka we see following error: 2016-10-07 17:37:39,469 INFO [pool-24-thread-1-EventThread] org.I0Itec.zkclient.ZkClient zookeeper state changed (Expired) 2016-10-07 17:37:39,470 INFO [ZkClient-EventThread-465-hdp-name1.lab.croc.ru:2181] k.consumer.ZookeeperConsumerConnector [95446e62-0157-1000-7951-fd4244e9aec2_###############-1475841346967-f0d261ce], exception during rebalance kafka.common.KafkaException: Failed to parse the broker info from zookeeper: {"jmx_port":-1,"timestamp":"1475501559373","endpoints":["PLAINTEXT://############:6667"],"host":"#############","version":3,"port":6667} next we see: Caused by: kafka.common.KafkaException: Unknown version of broker registration. Only versions 1 and 2 are supported.{"jmx_port":-1,"timestamp":"1475501559373","endpoints":["PLAINTEXT://#########:6667"],"host":"##########","version":3,"port":6667} Our hdp version is 2.5 and hdf version is 2.0.
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
10-05-2016
07:05 AM
Thank You very much, suggestion You've provided solved my issue.
... View more
10-04-2016
12:45 PM
Hello, thanks for quick answer! Is any way to solve this issue between HDP v 2.5 and HDF v 2.0? Or we have only one way is to downgrade our HDP to version 2.4? Is any documented way to do this downgrade?
... View more
10-04-2016
11:59 AM
Hello, We have the same error, have You solved this? Our HDP is version 2.5.
... View more