I am finding many libraries part of HDP base install are added to class path like jackson-core-2.2.3.jar. However these libraries do not come with the vanilla (non hdp) install. Does anyone know why and how these libraries are used? If new version of the library exist, HDP may force to specify classpath for each application running with different jar. Is there a possible work around where HDP separate core hadoop libraries classpth from non core libraries (like jackson..*)? ... View more

Which file stores ambari configurations such as ambari classpath? ... View more

I am running a csv file with approx 300,000 records through routetext processor. I am getting the following error to many files open: NiFi App Log: 2016-01-05 23:53:57,540 WARN [Timer-Driven Process Thread-10] o.a.n.c.t.ContinuallyRunProcessorTask org.apache.nifi.processor.exception.FlowFileAccessException: Exception in callback: java.io.FileNotFoundException: /opt/nifi-1.1.0.0-10/content_repository/100/1452038037470-66660 (Too many open files) at org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2048) ~[nifi-framework-core-1.1.0.0-10.jar:1.1.0.0-10] at org.apache.nifi.processors.standard.RouteText.appendLine(RouteText.java:499) ~[na:na] at org.apache.nifi.processors.standard.RouteText.access$100(RouteText.java:79) ~[na:na] at org.apache.nifi.processors.standard.RouteText$1.process(RouteText.java:433) ~[na:na] at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1806) ~[nifi-framework-core-1.1.0.0-10.jar:1.1.0.0-10] at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:1777) ~[nifi-framework-core-1.1.0.0-10.jar:1.1.0.0-10] at org.apache.nifi.processors.standard.RouteText.onTrigger(RouteText.java:360) ~[na:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.1.0.0-10.jar:1.1.0.0-10] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1146) ~[nifi-framework-core-1.1.0.0-10.jar:1.1.0.0-10] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:139) [nifi-framework-core-1.1.0.0-10.jar:1.1.0.0-10] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49) [nifi-framework-core-1.1.0.0-10.jar:1.1.0.0-10] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119) [nifi-framework-core-1.1.0.0-10.jar:1.1.0.0-10] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_91] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) [na:1.7.0_91] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_91] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_91] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_91] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_91] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_91] Caused by: java.io.FileNotFoundException: /opt/nifi-1.1.0.0-10/content_repository/100/1452038037470-66660 (Too many open files) at java.io.FileOutputStream.open(Native Method) ~[na:1.7.0_91] at java.io.FileOutputStream.<init>(FileOutputStream.java:221) ~[na:1.7.0_91] at org.apache.nifi.controller.repository.FileSystemRepository.write(FileSystemRepository.java:862) ~[nifi-framework-core-1.1.0.0-10.jar:1.1.0.0-10] at org.apache.nifi.controller.repository.FileSystemRepository.write(FileSystemRepository.java:831) ~[nifi-framework-core-1.1.0.0-10.jar:1.1.0.0-10] at org.apache.nifi.controller.repository.StandardProcessSession.append(StandardProcessSession.java:2008) ~[nifi-framework-core-1.1.0.0-10.jar:1.1.0.0-10] ... 18 common frames omitted I have run the following hadoop dfsadmin -report and all is fine I have checked ulimit -Sn ulimit -Hn which both have 10000 limit. ... View more

@Tim Hall Any best practices when Migrating from Fair Scheduler to Capacity Scheduler? ... View more

@Tim Hall Is there any feature or functionality with capacity scheduler where it would provide resources to higher priority job which kicked off during lower priority large job which is comsuming most of the resources. Assuming same org and que. ... View more

Following the tutorial http://hortonworks.com/hadoop-tutorial/processing-data-pipeline-with-apache-falcon/ during rawEmailIngestProcess shell-wf the shellnode process fails. shellnode is basically calling ingest.sh: curl -sS http://bailando.sims.berkeley.edu/enron/enron_wit... | tar xz && hadoop fs -mkdir -p $1 && hadoop fs -put enron_with_categories/*/*.txt $1 I changed ingest.sh and perform a simple hadoop fs -ls and the shellnode process succeeds. Here is the oozie log: 2015-12-31 05:50:49,213 WARN ShellActionExecutor:523 - SERVER[sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000420-151222214138596-oozie-oozi-W] ACTION[0000420-151222214138596-oozie-oozi-W@shell-node] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1] 2015-12-31 05:50:49,239 INFO ActionEndXCommand:520 - SERVER[sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000420-151222214138596-oozie-oozi-W] ACTION[0000420-151222214138596-oozie-oozi-W@shell-node] ERROR is considered as FAILED for SLA 2015-12-31 05:50:49,261 INFO ActionStartXCommand:520 - SERVER[sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000420-151222214138596-oozie-oozi-W] ACTION[0000420-151222214138596-oozie-oozi-W@fail] Start action [0000420-151222214138596-oozie-oozi-W@fail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] Any help would be appreciated. @Anderw Ahn ... View more