I am trying to save data to Elasticsearch using spark, I am working with xml files, as you know Elasticsearch accept only Json files, so I need to convert the xml file into Json file using Scala. Could you help me? ... View more

I am trying to save a json file to Elasticsearch, I am getting this error (Exception in thread "main" java.lang.AbstractMethodError) I am using "spark 2.3.0" and "scala 2.11.6" please find the code Below: import org.apache.spark.sql.SQLContext import org.elasticsearch.spark.sql._ import org.apache.spark.{SparkConf, SparkContext} object OrangetoES { def main(args: Array[String]) { val conf = new SparkConf().setAppName("OrangetoES").setMaster("local[*]") val sc = new SparkContext(conf) conf.set("es.index.auto.create", "true") val sqlc = new SQLContext(sc) val df = sqlc.read.json("../Orange.json") df.saveToEs("orangetoes/people") } } ... View more

I am trying to save a json file in Elasticsearch using spark scala, but the code not working. here is the code: import org.apache.spark import org.apache.spark.{SparkConf, SparkContext} import org.elasticsearch.spark._ import org.apache.spark.sql._ object JsontoES { def main(args: Array[String]): Unit = { val conf = new SparkConf().setAppName("HelloWorld").setMaster("local") val sc = new SparkContext(conf) conf.set("es.index.auto.create", "true") val spark = SparkSession .builder() .appName("Spark SQL basic example") .config("spark.some.config.option", "some-value") .getOrCreate() val jsonpath = "/home/aymenstien/Bureau/mydoc/Orange.json" //val ds= spark.read.json(jsonpath) sc.esJsonRDD(jsonpath) val RDD = sc.esRDD("radio/artists") //rddds.saveToEs("jsontoes/json-trips") //sc.makeRDD(Seq(ds)).saveToEs("jsontoEs/docs") } } <br> ... View more

0favorite I am trying to execute this command: aymenstien@aymenstien-VPCEH2Q1E:/usr/share/logstash$ ./bin/logstash -f /home/aymenstien/Bureau/fb.conf here is the config file: input { file { path => "/home/aymenstien/Bureau/mydoc/*" start_position => beginning codec => json sincedb_path => "/home/aymenstien/Bureau/mydoc/postj1.sincedb" } } output { stdout { codec => rubydebug } elasticsearch { hosts => "http://localhost:9200" index => "fbpost" document_type => "post" timeout => 30 workers => 1 } } and I am getting the result of execution below: aymenstien@aymenstien-VPCEH2Q1E:/usr/share/logstash$ ./bin/logstash -f /home/aymenstien/Bureau/fb.conf WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [FATAL] 2018-07-05 12:47:56.496 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:448:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:230:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:141:in `block in validate_all'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:140:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:279:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:238:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `<main>'"]} [ERROR] 2018-07-05 12:47:56.554 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit Can anyone help me? ... View more

I am working with sqoop, I can import and export data from mysql to hive or hbase with the command: sqoop import \ --connect "jdbc:mysql://localhost:3306/retail_db" \ --username=retail_dba \ --password=password \ --table departments \ --as-sequencefile \ --target-dir=/user/departments It work fine for me. Now I am trying to import and export from diffrent machine (my friend's machine)what I have to do. Thanks ... View more

I am lokking for the current stable release of hadoop. ... View more

I am trying to execute this command line $sqoop import --connect jdbc:mysql://hadoop:3306/Testdb --table widgets -m 4 --username root --password mypwd --driver com.mysql.jdbc.Driver but i am getting this error: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.0.3-8/accumulo/lib/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 18/03/09 01:15:55 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.6.0.3-8 18/03/09 01:15:55 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 18/03/09 01:15:55 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time. 18/03/09 01:15:55 INFO manager.SqlManager: Using default fetchSize of 1000 18/03/09 01:15:55 INFO tool.CodeGenTool: Beginning code generation 18/03/09 01:15:55 ERROR manager.SqlManager: Error executing statement: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure ... View more

Hi everyone, I hope that all is fine with you'll, I am running HDP 2.6 docker version on ubuntu. I was trying to start the HDP "./start_sandbox-hdp.sh" but it look its not going to start, alot of problems and a lot of errors. I am a beginner in Big Data, who can help me to resolve these problems. please find screen shot below. Thanks 🙂 ... View more