Member since
04-19-2019
4
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2637 | 04-23-2019 03:18 AM |
07-01-2019
07:04 AM
@Shu Thanks for your reply, and I'm sorry I didn't describe it clear, it's my fault. Actually I want to count is all the duplicated columns in a dataframe. Not specified column and not the duplicated rows. From your suggestion, maybe I can find the answer by comparing each columns and count the (duplicated rows == row count).
... View more
07-01-2019
02:28 AM
Hi all, I want to count the duplicated columns in a spark dataframe, for example: id col1 col2 col3 col4 1 3 999 4 999 2 2 888 5 888 3 1 777 6 777 In this case, the col2 and col4's values(font style is bold) are the same, which is my interest, so let the count +1. BTW, my dataset contain 20000K+ rows. Is there any function could solve this? Any idea will be appreciate, thank you.
... View more
Labels:
- Labels:
-
Apache Spark
04-23-2019
03:18 AM
Hi all, I had resolved this problem by add missing jar files to "/usr/hdp/3.1.0.0-78/storm/lib/" . Thanks.
... View more
04-22-2019
01:54 PM
Hi all, I had built up a HDF 3.3.1.0 with ambari, but I got NoClassDefFoundError when I'm building a data processing kafka to druid by Streaming Analytics Manager, how to fix it? My environment: Storm:1.2.1
SAM: 0.6.0
Kafka: 1.0.0.3.1
Druid: 0.12.1 Below is the logs in storm worker. java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonProcessingException
at org.apache.storm.kafka.spout.KafkaSpout.open(KafkaSpout.java:142) ~[dep-org.apache.storm-storm-kafka-client-jar-1.2.1.3.3.1.0-10.jar.1555487788000:1.2.1.3.3.1.0-10]
at org.apache.storm.daemon.executor$fn__10149$fn__10164.invoke(executor.clj:615) ~[storm-core-1.2.1.3.1.0.0-78.jar:1.2.1.3.1.0.0-78]
at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:482) [storm-core-1.2.1.3.1.0.0-78.jar:1.2.1.3.1.0.0-78]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonProcessingException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_112]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_112]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[?:1.8.0_112]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_112]
... 5 more
Thank you very much for your help. Best regards, DADA
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Storm