Created on 05-03-2023 12:46 PM - edited 05-03-2023 12:54 PM
I'm new in the area, so sorry if this is a stupid question, but i have a sqoop command in one of my projects that i suspect is writing a .jar file inside /tmp. It should not write into /tmp.
this is the command:
time sqoop import \
-D mapreduce.job.classloader="${<env_var>}" \
-D mapreduce.job.queuename="${<env_var>}" \
-D mapreduce.job.name="${<env_var>}" \
-D tez.queue.name="${<env_var>}" \
-D org.apache.sqoop.splitter.allow_text_splitter=true \
--driver ${<env_var>} \
--connect ${<env_var>} \
--username ${<env_var>} \
--password-file ${<env_var>} \
--query ${<env_var>}
--split-by ${<env_var>} \
--num-mappers ${<env_var>} \
--target-dir ${<env_var>} \
${DELETE_TARGET_DIR} \
--fields-terminated-by ${<env_var>}
I also found in the log the following message: orm.CompilationManager: Wiriting jar file: <file_path>
What is this orm.CompilationManager?
Created 05-03-2023 12:56 PM
@arnxld Welcome to the Cloudera Community!
To help you get the best possible solution, I have tagged our Sqoop experts @JoseManuel and @jphelps who may be able to assist you further.
Please keep us updated on your post, and we hope you find a satisfactory solution to your query.
Regards,
Diana Torres,Created 05-03-2023 01:14 PM
Hello!
SQOOP takes advantage of YARN to create a distributed IMPORT/EXPORT job
To do so, it needs to compile your query for the mappers to actually execute the job
Check out the following links for further information:
https://sqoop.apache.org/docs/1.4.7/SqoopUserGuide.html - Section 8.2 Table 18
https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/orm/CompilationManager.java#L50
Hope this helps,
Sincerely,
José M Martínez Poblete
Cloudera Support Hive SME US/East/Shift5
my.cloudera.com/faq/support.html
USA: 1-866-809-4659
INTL: 1-650-425-9421