Re: ERROR running sqoop action from oozie

RAkhmadeev · ‎04-18-2018

We have an application that reads messages from specific kafka topics, and process it, and when it reads message from topic it puts offset to the HBase table. after some amount of working application fails (time varries from 30 minutes to 15 hours ), in the driver stderr we see following log entries: 18/04/17 17:31:15 WARN client.AsyncProcess: #3121, the task was rejected by the pool. This is unexpected. Server is ***hostname masked***,60020,1523949367813 java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3f377224 rejected from java.util.concurrent.ThreadPoolExecutor@639d4dae[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1013) at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$000(AsyncProcess.java:600) at org.apache.hadoop.hbase.client.AsyncProcess.submitMultiActions(AsyncProcess.java:449) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:429) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:238) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1495) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1098) And after some amount of time this ERRORS: 18/04/17 17:31:15 ERROR client.AsyncProcess: Cannot get replica 0 location for {"totalColumns":1,"row":"predictor_passport_ru_number_gold","families":{"cf":[{"qualifier":"\x00\x00\x00\x00","vlen":8,"tag":[],"timestamp":9223372036854775807}]}} 18/04/17 17:31:15 ERROR spark.Utils: Error saving offsets [OffsetRange(topic: 'predictor_passport_ru_number_gold', partition: 0, range: [2536631 -> 2536718])] org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1 time, at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:247) at org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1800(AsyncProcess.java:227) at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1766) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:240) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1495) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1098) In the HBase logs I see an gap in messages on that period of time you can see this on attached screenshot - memstoreflush.png In addition full log of driver in index.zip. Please help to investigate and solve the issue.

RAkhmadeev · ‎12-19-2017

Hi, was You able to fix the issue? We have the same problem.

RAkhmadeev · ‎04-04-2017

This event occurs only if we are using NiFi Hive Streaming. ls -R will later.

RAkhmadeev · ‎04-04-2017

Hi All, Periodically, in some ORC tables in Hive we get duplicate partition "base" directory inside /table_name/partition_date=/base/ meaning: all contents of /table_name/partition_date=/base/* are in /table_name/partition_date=/base/base/*. After that partition become bad and from this bad partition we can’t do select count(*) or any other selects because of error occurring. But when we dropping duplicate “base” directory problem goes away. Why we got this duplicate folder in our buckets?

RAkhmadeev · ‎03-31-2017

Our NiFi is co-located with other Hadoop components. This is physical servers. 24 Cores per machine.templates.zip Zookeper is separate but on this machines. Errors on DetectDuplicate processor are symptoms of this issue. Socket Timeouts too. We have 3 Process Group on Our NiFi Cluster, their templates are in attachment.

RAkhmadeev · ‎03-31-2017

Our NiFi have 8 Gb of heap, NiFi version is 1.1.0.2.

RAkhmadeev · ‎03-29-2017

nifi-app.zipAfter working one weak our NiFi cluster become very unstable. Nodes are disconnect and reconnect every 5 - 30 minutes, processors don't work fine too. Restarting all 3 nodes solve the issue. Restarting NiFi weakly is not a good solution but we can work only with this approach. Example of log file from one of the node in attachment.

RAkhmadeev · ‎03-16-2017

Hi, explain statement is very huge so You can download it from our share: https://drive.croc.ru/display/data/list?dataId=c43e16e0-e0af-40f1-935e-1c44e4b01f91 login: 024741 password: E804F9487956

RAkhmadeev · ‎03-15-2017

Hi, Tried to increase this property 10x but no results. Regards, Ramil.

RAkhmadeev · ‎03-13-2017

Attachments mentioned above can be found on https://drive.croc.ru/display/data/list?dataId=02745bf5-e54d-47a9-8797-15f108fc057e login: 024556 password: 0B908ECFE563

RAkhmadeev · ‎03-13-2017

Attachments mentioned above can be found on https://drive.croc.ru/display/data/list?dataId=02745bf5-e54d-47a9-8797-15f108fc057e login: 024556 password: 0B908ECFE563

RAkhmadeev · ‎03-13-2017

Hi Community, We have a script - collmx_consents_snp.hql (in attachement). In this script we use join of following tables: consent_service_consent_hst consent_service_consent_subject_hst consent_service_client_hst DDL of those table are in attachment too. All of this tables are partitioned by date. The root cause of problem is that join of two tables in production don’t working(there is consistent data in tables but query don’t get nothing): select * from ( SELECT consent_uid,CASE WHEN for_contract = true THEN evid_srv ELSE NULL END evid_srv,entity_type,to_date(modif_time) apply_date,id_client FROM consent_service_consent_snp ) csc join consent_service_consent_subject_snp cscs on (csc.consent_uid = cscs.consent_uid) In test environment all fine. When we add some filter on table consent_service_consent_snp by partition, than query giva us results: When we are running select count(*) on this tables, we didn’t get any errors. In our test environment we have less data than in production. And when we add constraint on date in select clause all working fine, so we think that problem may depend on number of rows in the table. Logs of HiveServer2 and HiveMetastore in attachement. When query is fails we see following in hiveserver2 log: 2017-03-09 20:00:30,406 INFO org.apache.hadoop.hive.ql.plan.ConditionalResolverCommonJoin: [HiveServer2-Background-Pool: Thread-7269]: Failed to resolve driver alias (threshold : 25000000, length mapping : {cscs:consent_service_consent_subject_hst=571829172, csc:consent_service_consent_snp:consent_service_consent_hst=434475747})

RAkhmadeev · ‎10-28-2016

Issue was solved by my self. The solutin was: 1) under folder in which workflow.xml is create folder lib and put there all hive jar files from sharedlibDir(/user/oozie/share/lib/lib_20160928171540)/hive; 2) Create hive-site.xml with contents: <configuration> <property> <name>ambari.hive.db.schema.name</name> <value>hive</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://xxxxx:9083</value> </property> <property> <name>hive.zookeeper.quorum</name> <value>xxxx:2181,yyyyy:2181,zzzzzz:2181</value> </property> <property> <name>hive.metastore.warehouse.dir</name> <value>/smartdata/hive/</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.postgresql.Driver</value> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:postgresql://xxxxx:5432/hive</value> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>hive</value> </property> </configuration> and put it on hdfs for example in /tmp/hive-site.xml 3) Add following line in workflow.xml: <file>/tmp/hive-site.xml</file> This solved my issue.

RAkhmadeev · ‎10-27-2016

Hello thanks for advice but all fine with shared libraries: $ oozie admin -oozie http://localhost:11000/oozie -shareliblist [Available ShareLib] hive distcp mapreduce-streaming spark oozie hcatalog hive2 sqoop pig spark_orig $ oozie admin -oozie http://localhost:11000/oozie -sharelibupdate [ShareLib update status] sharelibDirOld = hdfs://os-2471.homecredit.ru:8020/user/oozie/share/lib/lib_20160928171540 host = http://localhost:11000/oozie sharelibDirNew = hdfs://os-2471.homecredit.ru:8020/user/oozie/share/lib/lib_20160928171540 status = Successful $ oozie admin -oozie http://localhost:11000/oozie -shareliblist [Available ShareLib] hive distcp mapreduce-streaming spark oozie hcatalog hive2 sqoop pig spark_orig On Resource manager UI all fine, see attached logs.

RAkhmadeev · ‎10-26-2016

resource-manager-ui.txtHello, Our HDP version 2.5 When we trying to run sqoop action(to load data from oracle to hive) from oozie we get folowing error in /var/log/oozie/oozie-error.log: JOB[0000004-161024200820785-oozie-oozi-W] ACTION[0000004-161024200820785-oozie-oozi-W@sqoop] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] And there is nothing more usefull for diagnostic. Job.properties file listed below: # properties nameNode = hdfs://xxxxx:8020 resourceManager = xxxx:8050 queueName=default oozie.use.system.libpath=true oozie.wf.application.path = hdfs://xxxxxx:8020/smartdata/oozie/hive_test.xml mapreduce.framework.name = yarn When we running this job from command line with "sqoop ..... " as command all working fine. Please some one tell me how to solve or troubleshoot this.

RAkhmadeev · ‎10-17-2016

Is the any workaround of this? or some hot fix?

RAkhmadeev · ‎10-14-2016

After we enabled HDFS HA PutHiveStreaming processor in our NiFi stopped working and generate following errors: 2016-10-14 21:50:53,840 WARN [Timer-Driven Process Thread-6] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=01571000-c4de-1bfd-0f09-5c439230e84e] Processor Administratively Yielded for 1 sec due to processing failure 2016-10-14 21:50:53,840 WARN [Timer-Driven Process Thread-6] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding PutHiveStreaming[id=01571000-c4de-1bfd-0f09-5c439230e84e] due to uncaught Exception: java.lang.IllegalArgumentException: java.net.UnknownHostException: hdpCROC 2016-10-14 21:50:53,847 WARN [Timer-Driven Process Thread-6] o.a.n.c.t.ContinuallyRunProcessorTask java.lang.IllegalArgumentException: java.net.UnknownHostException: hdpCROC at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:411) ~[na:na] at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:311) ~[na:na] at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) ~[na:na] at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:688) ~[na:na] at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:629) ~[na:na] at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:159) ~[na:na] at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2761) ~[na:na] at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) ~[na:na] at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2795) ~[na:na] at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2777) ~[na:na] at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:386) ~[na:na] at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) ~[na:na] at org.apache.hadoop.hive.ql.io.orc.OrcRecordUpdater.<init>(OrcRecordUpdater.java:234) ~[na:na] at org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat.getRecordUpdater(OrcOutputFormat.java:289) ~[na:na] at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.createRecordUpdater(AbstractRecordWriter.java:253) ~[na:na] at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.createRecordUpdaters(AbstractRecordWriter.java:245) ~[na:na] at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.newBatch(AbstractRecordWriter.java:189) ~[na:na] at org.apache.hive.hcatalog.streaming.StrictJsonWriter.newBatch(StrictJsonWriter.java:41) ~[na:na] at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:607) ~[na:na] at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.<init>(HiveEndPoint.java:555) ~[na:na] at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.fetchTransactionBatchImpl(HiveEndPoint.java:441) ~[na:na] at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.fetchTransactionBatch(HiveEndPoint.java:421) ~[na:na] at org.apache.nifi.util.hive.HiveWriter.lambda$nextTxnBatch$7(HiveWriter.java:250) ~[na:na] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] Caused by: java.net.UnknownHostException: hdpCROC hdpCROC - our HDP cluster and dfs.servicenames property. All files such as hive-site.xml, hdfs-site.xml, hdfs-core.xml are actual. What can cause this issue?

RAkhmadeev · ‎10-07-2016

I've found, thank You very much for solution!

RAkhmadeev · ‎10-07-2016

Thank You for quick reply, can You tell me please where can I get consumeKafka_o_10 nifi processor?

RAkhmadeev · ‎10-07-2016

When we trying to use getkafka we see following error: 2016-10-07 17:37:39,469 INFO [pool-24-thread-1-EventThread] org.I0Itec.zkclient.ZkClient zookeeper state changed (Expired) 2016-10-07 17:37:39,470 INFO [ZkClient-EventThread-465-hdp-name1.lab.croc.ru:2181] k.consumer.ZookeeperConsumerConnector [95446e62-0157-1000-7951-fd4244e9aec2_###############-1475841346967-f0d261ce], exception during rebalance kafka.common.KafkaException: Failed to parse the broker info from zookeeper: {"jmx_port":-1,"timestamp":"1475501559373","endpoints":["PLAINTEXT://############:6667"],"host":"#############","version":3,"port":6667} next we see: Caused by: kafka.common.KafkaException: Unknown version of broker registration. Only versions 1 and 2 are supported.{"jmx_port":-1,"timestamp":"1475501559373","endpoints":["PLAINTEXT://#########:6667"],"host":"##########","version":3,"port":6667} Our hdp version is 2.5 and hdf version is 2.0.

RAkhmadeev · ‎10-05-2016

Thank You very much, suggestion You've provided solved my issue.

RAkhmadeev · ‎10-04-2016

Hello, thanks for quick answer! Is any way to solve this issue between HDP v 2.5 and HDF v 2.0? Or we have only one way is to downgrade our HDP to version 2.4? Is any documented way to do this downgrade?

RAkhmadeev · ‎10-04-2016

Hello, We have the same error, have You solved this? Our HDP is version 2.5.

Online	Offline
Last Visited	‎05-07-2018 09:09 AM

Date Registered	‎02-22-2017 02:28 AM
Last Visited	‎05-07-2018 09:09 AM
Total Messages Posted	33
Total Kudos Received	6

Re: ERROR running sqoop action from oozie

Problem with writing from spark streaming to hbase

Re: :8020 failed on connection exception: java.net...

Re: In some orc tables in Hive we get duplicate pa...

In some orc tables in Hive we get duplicate partit...

Re: after weak of working nifi cluster become unst...

Re: after weak of working nifi cluster become unst...

after weak of working nifi cluster become unstable

Re: Hive select from tables with join don't give r...

Re: Hive select from tables with join don't give r...

Re: Hive select from tables with join don't give r...

Re: Hive select from tables with join don't give r...

Hive select from tables with join don't give resul...

Re: ERROR running sqoop action from oozie

Re: ERROR running sqoop action from oozie

ERROR running sqoop action from oozie

Re: after enabling HDFS HA nifi PutHiveStreaming p...

after enabling HDFS HA nifi PutHiveStreaming proce...

Re: getkafka don't working in nifi

Re: getkafka don't working in nifi

getkafka don't working in nifi

Re: PutHiveStreaming Nifi processor; various error...

Re: PutHiveStreaming Nifi processor; various error...

Re: PutHiveStreaming Nifi processor; various error...