Do you want to say like this? import org.apache.scala.sql.* ... View more

Great. Remember to set the best answer. By ... View more

Hi @Alex Ye, I looked on my NiFi logback.xml and my lines is equal to the folow: <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> ... View more

Hi @Alex Ye, If I understend correctly, your problem is that on the log is showing two lines with the same information. That is it? About this, let me know how many instances of NiFi do you have on your cluster? ... View more

Aways that you perform a save opperation the files will be created acording the number of partition of you DF, and this process generate files names same "part-xxxxx", so, this is the complete file name. The file name never will be different this. The variable is how many files will be generated. So sorry if I understand you desire. ... View more

On the shell, before execute your script. export SPARK_MAJOR_VERSION=1; ./start_ipython_notebook.sh ... View more

Hi @Paul Byrum, Reading the tutorial I noted that on Step 7 the "--notebook-dir='/usr/hdp/2.3.2.0-2950/spark/'" is setted to Spark 1 and not Spark 2. I suggest that you change your spark version "SPARK_MAJOR_VERSION=1" and try to start your script again. Regards ... View more

Hi @Prudhvi Rao Shedimbi, What exactly is your need? I ask this becouse if you simple want to read saved file is only necessary that you set the folder and all content will be read. sc.textFile("foldername/*") So, if what you want is write one unique file, from a previous processing of a DataFrame then you can do this using the "df.repartition(1).saveAsTextFile('HDFSFolder/FileName')" and so, only one file "part-00000" will be generated. If you are using a library like DataBricks you can do so: df.write.format("csv").save("/HDFSFolder/FileName.csv") That's it? ... View more

Hi @Vinod Kumar, you're right. If this type of question is part of your proof the answer will be as you quoted, via spark-submit informing the parameters required for submission of the job. ... View more

Go with faith. ... View more

During the exam, if you have a question about this, you will be asked to submit a spark application, and in the context of the question, you will be informed the JAR name, the JAR main class and how the application should be processed (local, etc.). I recently completed certification and it was so. Hope to help you. ... View more

Hi @Sidi Mahmoud RHIL, I searched for information about and found this: Current Apache hive release doesn't support this feature. In the shared apache Hive Jira HIVE-4847, you can see this is a new feature and the patch is available but it is yet to be merged and tagged to a particular release. We can expect this feature in any of the later releases. ... View more

I don't know Pyton, but if you don't need specifically to use Pyton, you can use NIFI. The NIFI has many processors to this purpose. You can get files from FTP, FS, HDFS, and to ingest to HDFS. I hope to helped. ... View more