Member since
12-06-2016
136
Posts
12
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2683 | 01-18-2018 12:56 PM |
10-03-2018
02:34 PM
Hi! Have same problem, but Ranger not installed [root@serv12 ~]# grep -i Ranger /var/lib/ambari-agent/data/output-869.txt
2018-10-03 17:49:28,216 - Ranger admin not installed
... View more
09-25-2018
11:32 AM
How to respond to messages related to GC Failure notifications ? : Logs for container_e152_1537807360391_1057_01_000081
ResourceManager RM Home NodeManager Tools
2018-09-25 12:22:15 Starting to run new task attempt: attempt_1537807360391_1057_2_04_000182_0
1.148: [GC (Allocation Failure) [PSYoungGen: 134807K->24906K(204288K)] 134807K->24994K(2068480K), 0.0255463 secs] [Times: user=0.21 sys=0.02, real=0.03 secs]
1.411: [GC (Metadata GC Threshold) [PSYoungGen: 74769K->22331K(379904K)] 74857K->22427K(2244096K), 0.0233332 secs] [Times: user=0.12 sys=0.03, real=0.02 secs]
1.435: [Full GC (Metadata GC Threshold) [PSYoungGen: 22331K->0K(379904K)] [ParOldGen: 96K->21509K(264192K)] 22427K->21509K(644096K), [Metaspace: 20929K->20929K(1069056K)], 0.0289288 secs] [Times: user=0.32 sys=0.03, real=0.03 secs]
6.755: [GC (Allocation Failure) [PSYoungGen: 237172K->28660K(379904K)] 258681K->95006K(644096K), 0.0204362 secs] [Times: user=0.08 sys=0.24, real=0.02 secs]
15.663: [GC (Allocation Failure) [PSYoungGen: 213057K->28659K(547328K)] 279403K->193109K(811520K), 0.0390040 secs] [Times: user=0.14 sys=0.50, real=0.04 secs]
15.702: [Full GC (Ergonomics) [PSYoungGen: 28659K->0K(547328K)] [ParOldGen: 164449K->172137K(616960K)] 193109K->172137K(1164288K), [Metaspace: 32575K->32575K(1079296K)], 0.0628115 secs] [Times: user=0.96 sys=0.16, real=0.06 secs]
33.575: [GC (Allocation Failure) [PSYoungGen: 280808K->28661K(569344K)] 5743954K->5559068K(6478848K), 0.0278063 secs] [Times: user=0.21 sys=0.32, real=0.03 secs]
34.210: [GC (Allocation Failure) [PSYoungGen: 490957K->82416K(791552K)] 6021364K->5612831K(6701056K), 0.0320689 secs] [Times: user=0.57 sys=0.16, real=0.03 secs]
34.243: [Full GC (Ergonomics) [PSYoungGen: 82416K->0K(791552K)] [ParOldGen: 5530414K->5603149K(6443008K)] 5612831K->5603149K(7234560K), [Metaspace: 33968K->33968K(1079296K)], 0.9965374 secs] [Times: user=26.00 sys=0.29, real=0.99 secs]
36.154: [GC (Allocation Failure) [PSYoungGen: 459509K->36576K(683520K)] 6062659K->5639733K(7126528K), 0.0232837 secs] [Times: user=0.69 sys=0.02, real=0.02 secs]
36.746: [GC (Allocation Failure) [PSYoungGen: 466467K->13984K(783360K)] 6069625K->5617149K(7226368K), 0.0202414 secs] [Times: user=0.54 sys=0.01, real=0.02 secs]
37.123: [GC (Allocation Failure) [PSYoungGen: 321932K->14048K(784896K)] 5925097K->5617213K(7227904K), 0.0258958 secs] [Times: user=0.69 sys=0.01, real=0.02 secs]
37.723: [GC (Allocation Failure) [PSYoungGen: 566423K->15086K(787968K)] 6169588K->5618252K(7230976K), 0.0238336 secs] [Times: user=0.58 sys=0.00, real=0.02 secs]
38.036: [GC (Allocation Failure) [PSYoungGen: 259899K->14478K(790016K)] 5863064K->5617644K(7233024K), 0.0251177 secs] [Times: user=0.55 sys=0.01, real=0.03 secs]
38.293: [GC (Allocation Failure) [PSYoungGen: 222330K->1806K(792576K)] 5825496K->5617860K(7235584K), 0.0262013 secs] [Times: user=0.57 sys=0.02, real=0.02 secs]
38.624: [GC (Allocation Failure) [PSYoungGen: 270228K->1248K(793088K)] 5886282K->5617856K(7236096K), 0.0277400 secs] [Times: user=0.56 sys=0.00, real=0.03 secs]
... View more
Labels:
- Labels:
-
Apache Hive
07-31-2018
10:39 AM
Hi! Try One of this changes:
Zeppelin
NIFI
ExecuteSparkInteractive
preds =
spark.sql(‘select * from sandbox.CHURN_PRP_D_PREDS_S ’)
preds =
spark.sql("select
* from andbox.CHURN_PRP_D_PREDS_S")
dir = ‘
/user/CHURN_PRP_D’
dir =
"hdfs:/user/CHURN_PRP_D"
date_df =
spark.sql("select to_date('{}') as score_date".format(date_calc))
ttt =
spark.createDataFrame([(date_calc,)], ["t"])
date_df =
ttt.select(to_date(ttt.t))
sqlContext.registerDataFrameAsTable(train,
"train")
train.registerTempTable("train")
... View more
06-11-2018
01:45 PM
Hi! one more question, how control count "livy-session-"" ?
... View more
06-11-2018
11:46 AM
Problem was in "sqlContext.registerDataFrameAsTable" What investigate similar (HTTP) errors ?
... View more
06-06-2018
03:48 PM
Do you have example run Zeppelin REST API ?
... View more
06-06-2018
02:00 PM
Labels:
- Labels:
-
Apache NiFi
-
Apache Zeppelin
06-06-2018
12:38 PM
CODE : executesparkinteractive-code.txt
ERROR :
2018-06-06 15:32:13,566 ERROR [Timer-Driven Process Thread-5] o.a.n.p.livy.ExecuteSparkInteractive ExecuteSparkInteractive[id=aeb74038-5333-13d2-0000-00001ea7e32e] ExecuteSparkInteractive[i
d=aeb74038-5333-13d2-0000-00001ea7e32e] failed to process session due to java.lang.RuntimeException: Failed : HTTP error code : 400 : Bad Request: {}
java.lang.RuntimeException: Failed : HTTP error code : 400 : Bad Request
at org.apache.nifi.processors.livy.ExecuteSparkInteractive.readJSONObjectFromUrlPOST(ExecuteSparkInteractive.java:282)
at org.apache.nifi.processors.livy.ExecuteSparkInteractive.submitAndHandleJob(ExecuteSparkInteractive.java:234)
at org.apache.nifi.processors.livy.ExecuteSparkInteractive.onTrigger(ExecuteSparkInteractive.java:197)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Spark
05-31-2018
08:03 PM
1 Kudo
Thank you! But what to do with processor , for example FetchFTP , after import change (version control) from DEV to PROD property: Password "No value set" Can add pasword to variables (on processor group), but this is not secured.
... View more