question Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" in Support Questions

Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

reddy_gayathri — Mon, 17 Oct 2016 17:20:17 GMT

I am trying to import oracle table to HDFS directory, but getting the error "Generating splits for a textual index column allowed only in case of "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" property passed as a parameter"

I fixed the import issue by giving "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" in sqoop import. But why do we need to set this property? I imported other tables without setting this property. When should we set this property?

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

ssubhas — Mon, 17 Oct 2016 18:34:04 GMT

@Gayathri Reddy G

The property "Dorg.apache.sqoop.splitter.allow_text_splitter=true" is required when you are using --split-by is used on a column which is of text type. There is difference in the TextSplitter class of Sqoop jars in HDP 2.4 and HDP 2.5 because of sqoop command fails without the required argument in HDP 2.5.

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

clukasik — Mon, 17 Oct 2016 19:40:37 GMT

This JIRA has some info about why you need to set the property: https://issues.apache.org/jira/browse/SQOOP-2910

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

rene_sluiter — Thu, 27 Oct 2016 23:02:18 GMT

@Gayathri Reddy G or @Sindhu

can you tell me exactly how you added the -Dorg.apache.sqoop.splitter.allow_text_splitter=true to your import statement?

I'm trying to create a sqoop job with this import statement but it keeps failing:

16/10/27 17:53:25 ERROR tool.BaseSqoopTool: Error parsing arguments for import:
16/10/27 17:53:25 ERROR tool.BaseSqoopTool: Unrecognized argument: -Dorg.apache.sqoop.splitter.allow_text_splitter=true

my shell script to create the sqoop job looks like this:

#!/bin/bash

sqoop job \
  --create <tablename> \
  -- \
  import "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" \
  --connect "jdbc:<URL>"

If I remove the -Dorg line all goes file but offcourse the execution of the job failes thats why I need to pass this allow_text_splitter as a parameter.

I tried with double quotes, without, but no luck unfortunately

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

reddy_gayathri — Thu, 27 Oct 2016 23:35:37 GMT

Try giving it just after sqoop job. Eg: Sqoop job "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" -- import ...

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

rene_sluiter — Thu, 27 Oct 2016 23:52:24 GMT

worked! many thanks, you saved my day.

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

sri_man — Fri, 06 Mar 2020 21:28:17 GMT

Can you include that in a line through code?

sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true \
--connect jdbc:mysql://localhost/sqoop \
--username root \
-passqord 0range \
--split-by id \
--columns id,name \
--table customer \
--target-dir /user/cloudera/ingest/raw/customers \
--fields-terminated-by "," \
--hive-import \
--create-hive-table \
--hive-table sqoop_workspace.customers

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

ask_bill_brooks — Sat, 07 Mar 2020 02:06:25 GMT

@sri_man

Since this thread was marked 'Solved' back in 2016, you would have a better chance of receiving a relevant response by posting a new question. This will also provide the opportunity to provide details specific to your environment that could aid other members in providing a more tailored answer to your issue.