Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

[#1]

I am trying to import oracle table to HDFS directory, but getting the error "Generating splits for a textual index column allowed only in case of "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" property passed as a parameter"

I fixed the import issue by giving "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" in sqoop import. But why do we need to set this property? I imported other tables without setting this property. When should we set this property?

[#2 · Accepted solution]

@Gayathri Reddy G

The property "Dorg.apache.sqoop.splitter.allow_text_splitter=true" is required when you are using --split-by is used on a column which is of text type. There is difference in the TextSplitter class of Sqoop jars in HDP 2.4 and HDP 2.5 because of sqoop command fails without the required argument in HDP 2.5.

[#3]

This JIRA has some info about why you need to set the property: https://issues.apache.org/jira/browse/SQOOP-2910

[#4]

@Gayathri Reddy G or @Sindhu

can you tell me exactly how you added the -Dorg.apache.sqoop.splitter.allow_text_splitter=true to your import statement?

I'm trying to create a sqoop job with this import statement but it keeps failing:

16/10/27 17:53:25 ERROR tool.BaseSqoopTool: Error parsing arguments for import:
16/10/27 17:53:25 ERROR tool.BaseSqoopTool: Unrecognized argument: -Dorg.apache.sqoop.splitter.allow_text_splitter=true

my shell script to create the sqoop job looks like this:

#!/bin/bash

sqoop job \
  --create <tablename> \
  -- \
  import "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" \
  --connect "jdbc:<URL>"

If I remove the -Dorg line all goes file but offcourse the execution of the job failes thats why I need to pass this allow_text_splitter as a parameter.

I tried with double quotes, without, but no luck unfortunately

[#5]

Try giving it just after sqoop job. Eg: Sqoop job "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" -- import ...

[#6]

worked! many thanks, you saved my day.

[#7]

Can you include that in a line through code?

sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true \
--connect jdbc:mysql://localhost/sqoop \
--username root \
-passqord 0range \
--split-by id \
--columns id,name \
--table customer \
--target-dir /user/cloudera/ingest/raw/customers \
--fields-terminated-by "," \
--hive-import \
--create-hive-table \
--hive-table sqoop_workspace.customers

[#8]

@sri_man 

Since this thread was marked 'Solved' back in 2016, you would have a better chance of receiving a relevant response by posting a new question. This will also provide the opportunity to provide details specific to your environment that could aid other members in providing a more tailored answer to your issue.