I want spark api which can write data into excel file not in CSV file. Best solution to write data into excel file directly. ... View more

We have created spark application for client reporting . Client wants report in CSV format. We have coded it that way and it is generating desired output with requested format. When we see result data in log it shows correct format and correct data(I.e date format requested is 2018-07-26 11:19:04.0 and it is correct format shows in log but when we see same data in CSV file format is getting changed. It shows 6/7/2018 12:27 format. Why this issue with csv file when we see log file it shows correct results and same we have written to csv file through flie write command, it shows fomat changed. How to resolve this? Sample Code: val selectedData = dataFrame3.select(concat(col("ticket_number"),lit("-"),date_format(col("as_of_date"),"yyMMdd")).as("transref"),col("newmCanc").as("newmCanc"), when(col("trade_action") === "CXL",concat(col("master_ticket_num"),lit("-"),date_format(col("as_of_date"),"yyMMdd"))).otherwise("").as("relTransref"), col("trader_name").as("portfolioIdAm"), col("portfolioIdKvg"), col("name").as("portfolioName"), when(col("buy_sell_desc") === "Buy", "BUY").when(col("buy_sell_desc") === "Sell", "SELL").otherwise("OTHER").as("buyisell"), col("trade_feed_trade_amount").as("quantity"), col("secIdType"), col("id_isin").as("secId"), when(col("instrument_name").isNotNull,col("instrument_name")).otherwise(col("security_name")).as("secName"), format_number(col("trade_price").cast("Double"),2).as("price"), col("currency").as("tradeCCY"), format_number(col("settlement_costs_in_settlement_currency").cast("Double"),2).as("tradeComm"), format_number(col("Transaction_Cost_2_Amount").cast("Double"),2).as("fees"), format_number(col("Transaction_Cost_3_Amount").cast("Double"),2).as("tax"), format_number(col("Transaction_Cost_5_Amount").cast("Double"),2).as("others"), format_number(col("Accrued_Interest"),2).as("interest"), format_number(col("settlement_total_in_settlement_currency").cast("Double"),2).as("settlAmount"), when(col("Number_of_days_accrued_interest").isNull, "0").otherwise(col("Number_of_days_accrued_interest")).as("interestDays"), date_format(col("as_of_date"),"yyyy-MM-dd").cast("String").as("tradeDate"), date_format(col("receiveddate"),"yyyy-MM-dd HH:mm:ss").cast("String").as("executionTimestamp"), date_format(col("settlement_date"), "yyyy-MM-dd").cast("String").as("settlementDate") ... View more

We have a project where currently Shell script, Hive, Execution engine: TEZ is being used. For POC purpose we tried replacing shell scripts with spark and we executed HQLs through spark . One of the client cam back with a question that why would we need spark application as we can set spark as an execution engine and we can run our regular shell scripts and oozie workflow. What is the better option to choose just choose set hive.execution.engine=spark; OR make spark application and execute HQLs with spark APIs. If performance is same for both of them then why do we need to write code in Spark? What is the advantage of writing spark application using SPARK SQL? ... View more

I need to process CSV file through spark , I need load CSV file into hive tables through spark however my files itself has comma in data not as a separator but as a content at several places in this case there are three questions 1) How will spark identify that this is not a separator and consider this comma as a content of data 2) How can we process such data and load into hive including comma which is content and not a separator Please share some techniques to achieve above points. ... View more

I have spark job and while submitting I am giving X number of executors and Y memory however somebody else is also using same cluster and they also want to run several jobs during that time only with X number of executors and Y memory and both of them do not know about each other. In this case how number of executors/memory should be calculated and given to our spark job? ... View more

I see people ask that what are the optimization techniques you use for your spark job , what are these optimization techniques we can use for spark jobs? Like while writing spark job code or for submitting or to run job with optimal resources. ... View more

Can somebody guide me with Spark+ Scala complete examples? Is there any documents or online link or any book where I can see some complete examples of Spark + Scala ... View more

I need to schedule jobs which imports data from external tables to HDFS through SQOOP using incremental mode however it should not load duplicate values or if it is duplicate it should overwrite the existing data available in the file ... View more

I have two relations Relation A(1 w 2 x 3 y) Relation B( 1 z 2 x 3 y 4 K 5 L) Want to merge these relations in third relation with no duplicates using pig script Relation C 1 w 2 x 3 y 4 K 5 L ... View more

I have installed HDP 2.5 and I am able to login into sandbox in cent using root uname and hadoop password but how would I start working with hadoop components like pig,hive and all, Do I need to install all hadoop compnents and configure it or it is configured and installed by default? How would I get a prompt of pig, hive spark, scala and all? also I am unable to login into Ambari using admin/admin uname password? ... View more