Hi, I want to process data from the HDFS file using the Spark Java code. While processing files, I am performing simple transformation such as replace a new line with space and find patterns using regex from the file. I used the wholeTextFiles method to read data from HDFS files but it took 2 hours to process only 4 MB files. I tried to increase spark executor memory to 15g with 4 executor instances still it took 2 hours. I have 1 master with 56GiB memory,8 cores, and 3 workers with 28 GiB memory,8 cores. How to improve the performance of the spark job using the above nodes configurations.   Thanks, ... View more

I have created Oozie workflow for Hive and Pig action on Azure hortonworks cloudera But while running Oozie workflow Hive action run successfully but Pig action throws below error :   19/12/10 14:11:59 INFO client.RMProxy: Connecting to ResourceManager at bigdata-m0.xxxxxxx.cloudapp.net/10.0.0.7:8050 19/12/10 14:12:00 INFO client.AHSProxy: Connecting to Application History server at bigdata-m0.xxxx.cloudapp.net/10.0.0.7:10200 File /app-logs/hdfs/logs/application_15770947956_0159 does not exist. Can not find any log file matching the pattern: [ALL] for the application: application_15770947956_0159 Can not find the logs for the application: application_15770947956_0159 with the appOwner: hdfs     When I run the pig script using ' pig -x tez -useHCatalog -f script.pig' then it run successfully but while running from Oozie workflow it throws above error so I added shell action before Pig action to give permission to the directory [/app-logs/hdfs/logs/] still it throws above error. workflow.xml :   <workflow-app xmlns="uri:oozie:workflow:0.5" name="PigHiveworkflow"> <start to="hive-node"/> <action name="hive-node"> <hive2 xmlns="uri:oozie:hive2-action:0.2"> <job-tracker>${resourceManager}</job-tracker> <name-node>${nameNode}</name-node> <jdbc-url>jdbc:hive2://xxxxxxxxxxx:2181/default;httpPath=cliservice;serviceDiscoveryMode=zooKeeper;transportMode=http;user=xxxx;zooKeeperNamespace=hiveserver2</jdbc-url> <password>xxxx</password> <script>${appPath}/hive.hql</script> </hive2> <ok to="shell-node"/> <error to="kill"/> </action> <action name="shell-node"> <shell xmlns="uri:oozie:shell-action:0.3"> <job-tracker>${resourceManager}</job-tracker> <name-node>${nameNode}</name-node> <exec>script.sh</exec> <file>${appPath}/script.sh</file> <capture-output/> </shell> <ok to="pig-node"/> <error to="kill"/> </action> <action name="pig-node"> <pig> <job-tracker>${resourceManager}</job-tracker> <name-node>${nameNode}</name-node> <job-xml>${appPath}/hive-site.xml</job-xml> <configuration> <property> <name>oozie.action.sharelib.for.pig</name> <value>pig,hive,hcatalog</value> </property> </configuration> <script>${appPath}/demo.pig</script> </pig> <ok to="end"/> <error to="kill"/> </action> <kill name="kill"> <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name="end"/> </workflow-app>   I tried to give permission to directory [/app-logs/hdfs/logs] using fs command in Pig script: fs -chmod -R 777 /app-logs/root; Still Pig action in Oozie workflow throws error. I am running Oozie workflow as a hdfs user.I am not getting which file permission is still missing.   Thanks,                     ... View more

Hi, I am running Pig action from Oozie workflow But Pig action ended with below error,So I tried changing yarn.timeline-service.version from 2.0f to 1.5f still it throws below error,     2019-12-06 13:54:21,074 [main] WARN org.apache.pig.PigServer - Error posting to ATS: org.apache.hadoop.service.ServiceStateException: java.io.IOException: Timeline V1 client is not properly configured. Either timeline service is not enabled or version is not set to 1.x at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:173) at org.apache.pig.backend.hadoop.PigATSClient.<init>(PigATSClient.java:68) at org.apache.pig.backend.hadoop.PigATSClient.getInstance(PigATSClient.java:57) at org.apache.pig.PigServer.<init>(PigServer.java:260) at org.apache.pig.PigServer.<init>(PigServer.java:219) at org.apache.pig.tools.grunt.Grunt.<init>(Grunt.java:46) at org.apache.pig.Main.run(Main.java:495) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:273) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:216) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:78) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:67) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:231) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by: java.io.IOException: Timeline V1 client is not properly configured. Either timeline service is not enabled or version is not set to 1.x at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceInit(TimelineClientImpl.java:100) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) ... 24 more     Then I set yarn.timeline-service.enabled=false after setting this property above error gets resolve but Pig action stuck in running state. Below is the pig job log:     2019-12-06 14:15:42,253 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete 2019-12-06 14:15:42,253 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1575641509651_0005] Heart beat Heart beat Heart beat Heart beat 2019-12-06 14:17:43,613 [HiveClientCache-cleaner-0] INFO org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Closed a connection to metastore, current connections: 1 2019-12-06 14:17:43,613 [HiveClientCache-cleaner-0] INFO org.apache.hadoop.hive.metastore.HiveMetaStoreClient - Closed a connection to metastore, current connections: 0 Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat Heart beat End of LogType:stdout.This log file belongs to a running container (container_e24_1575641509651_0004_01_000002) and so may not be complete.     It seems like Pig job lost connection to the HiveMetaStoreClient but Hive services are up and running still it stuck in running state.   Thanks, ... View more