I am applying correlation on a csv file using apache spark, when loading data i am obliged to skipe the first row as a header which are columns in the dataset otherwise i can't load the data. i get the correlation computed but when i got the correlation matrix, i can't add the columns name as a header in the new matrix please would you help me get the matrix with its header ,thanks ,this what i did import org.apache.spark.mllib.linalg.{Vector,Vectors} import org.apache.spark.mllib.stat.Statistics import org.apache.spark.mllib.linalg.Matrix import org.apache.spark.rdd.RDD val data = sc.textFile(strfilePath).mapPartitionsWithIndex {case(index, iterator)=>if(index ==0) iterator.drop(1)else iterator } val inputMatrix = data.map { line =>val values = line.split(",").map(_.toDouble)Vectors.dense(values)} val correlationMatrix=Statistics.corr(inputMatrix,"pearson") ... View more

hi guys am using dataset with column name containing "." , here is the code am using and after that the issue i got , i think there is a problem with the "." character ,what should i do ? Thanks. val spark = SparkSession.builder.master("local").appName("my-spark-app").getOrCreate() val df = spark.read.option("inferSchema", "true").option("header", "true").csv("D:\ProcessDataSet\anis_data\Set _1Mud Pumps_Merged.csv") val aggs = df.columns.map(c => stddev(c).as(c)) val stddevs = df.select(aggs: _*) stddevs.show(5,false) ... View more

i have 4 csv files , i want to join and merge these files into one files based on a column timestamps to get one file.using spark or hadoop Please any help would be appreciated ... View more

Hi guys i am trying to save a dataframe to a csv file , that contains a timestamp. The problem that this column changes of format one written in the csv file .when showing via df.show i got a correct format when i check the csv file i got this format i also tried some think like this ,and still got the same problem finalresult.coalesce(1).write.option("header",true).option("inferSchema","true").option("dateFormat","yyyy-MM-dd HH:mm:ss").csv("C:/mydata.csv") val spark =SparkSession.builder.master("local").appName("my-spark-app").getOrCreate()val df = spark.read.option("header",true).option("inferSchema","true").csv("C:/Users/mhattabi/Desktop/dataTest2.csv")//val df = spark.read.option("header",true).option("inferSchema", "true").csv("C:\dataSet.csv\datasetTest.csv")//convert all column to numeric value in order to apply aggregation function df.columns.map { c =>df.withColumn(c, col(c).cast("int"))}//add a new column inluding the new timestamp columnval result2=df.withColumn("new_time",((unix_timestamp(col("time"))/300).cast("long")*300).cast("timestamp")).drop("time")val finalresult=result2.groupBy("new_time").agg(result2.drop("new_time").columns.map((_ ->"mean")).toMap).sort("new_time")//agg(avg(all columns..) finalresult.coalesce(1).write.option("header",true).option("inferSchema","true").csv("C:/mydata.csv") ... View more

I have dobe this code ,my question is for the function cast data type ,how can i cast all columns'datatype included in dataset at the same time except the column timestamp, and the other question is how to apply function avg on all column except also column timestamp,thanks a lot . val df = spark.read.option("header",true).option("inferSchema", "true").csv("C:/Users/mhattabi/Desktop/dataTest.csv") val result=df.withColumn("new_time",((unix_timestamp(col("time")) /300).cast("long") * 300).cast("timestamp")) result("value").cast("float")//here the first question val finalresult=result.groupBy("new_time").agg(avg("value")).sort("new_time")//here the second question about avg finalresult.coalesce(1).write.format("com.databricks.spark.csv").option("header", "true").save("C:/mydata.csv") ... View more

Hi guys i am working with scala and spark , i would like what is the best api or the best way to aggregate data based on interval of timestamps ,knowing that my data includes column timestamp that are taken every second , i would like to get a new dataframe based on aggregation and interval .Example my timestamp column is taken every second , i would like to know how would be my data if the timestamp was taken every minute ? val spark = SparkSession.builder.master("local").appName("my-spark-app").getOrCreate() val df = spark.read.option("header",true).csv("C:/Users/mhattabi/Desktop/Clean _data/Mud_Pumps _Cleaned/Set_1_Mud_Pumps_Merged.csv") df("DateTime").cast("timestamp") Thanks any help would Any help would be appreciated ... View more

Hi guys i am trying this source code val spark = SparkSession.builder.master("local").appName("my-spark-app").getOrCreate() val df = spark.read.option("header",true).csv("C:/Users/mhattabi/Desktop/Clean _data/Mud_Pumps _Cleaned/Set_1_Mud_Pumps_Merged.csv") df("DateTime").cast("timestamp") df("ADCH_Mud Pumps.db.MudPump.2.On.value").cast("integer") val result=df.select( col("*"), date_format(df("DateTime"),"yyyy-MM-dd hh:mm").alias("DateTime")).groupBy(df("DateTime")) .agg(avg(df("ADCH_Mud Pumps.db.MudPump.2.On.value"))) result.show(5) It says what sould i do for the attribute it is really existing in my data set,Thanks ... View more

hi guys i have a code written in pyspark any help to run it under scala thanks it is urgent please thanks from functools import reduce files =["/tmp/test_1.csv","/tmp/test_2.csv","/tmp/test_3.csv"] df = reduce(lambda x,y: x.unionAll(y), [sqlContext.read.format('com.databricks.spark.csv') .load(f, header="true", inferSchema="true") for f in files]) df.show() ... View more

hi guys Am new to spark ans scala,i have csv files that i want tomerge in the same csv file or dataframe i want just to handle them as if they are only one file Any help thanks ... View more

Hi I was able to load data remotely through the webhdfs rest api ,but it doesn't allow to load a big volume of data remotely.Is there any possibilty to load huge data remotely ,i ask if there is a hadoop api to do that ? Thank you ... View more

I am using HDP , i would like to use the webhdfs api of hadoop through .Net hadoop webhdfs api, i would like to access the hadoop in a remote machine through the webhdfs , i would like to know the URI that should be done to aceess hdp remotely using http.It should be something like http://host:port/webhdfs, what is the port to use Thank you ... View more

Hello guys I am running hadoop cluster using HDP , when i run hadoop there are two CLI of the datanode and the namenode , please is there is any possibilty to run them in background ? Thank you a lot ! ... View more

Hi @Sagar Shimpi I would like to know about the HDP Price just in case ,we want to use your product (HortonWorks Data Platform) to develop our personal software. Thank you ... View more

Hello sorry , but i want to know what is the difference between sandbox and HDP as two different tools Thanks ... View more

Hi guys i have downloaded sandbox under vmware machine , i started it i got this screen i opened a web browser in my windows host machine and wrote the address and there is nothing happened ? what should i do thanks ... View more

Hello i downloaded the hortonworks sandbox i couldn't log into the platform ,what should i insert as login and password before using the web Ui of the sandbox Thanks ... View more