Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

Solved Go to solution
Highlighted

Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

I am trying to import oracle table to HDFS directory, but getting the error "Generating splits for a textual index column allowed only in case of "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" property passed as a parameter"

I fixed the import issue by giving "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" in sqoop import. But why do we need to set this property? I imported other tables without setting this property. When should we set this property?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

@Gayathri Reddy G

The property "Dorg.apache.sqoop.splitter.allow_text_splitter=true" is required when you are using --split-by is used on a column which is of text type. There is difference in the TextSplitter class of Sqoop jars in HDP 2.4 and HDP 2.5 because of sqoop command fails without the required argument in HDP 2.5.

View solution in original post

7 REPLIES 7
Highlighted

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

@Gayathri Reddy G

The property "Dorg.apache.sqoop.splitter.allow_text_splitter=true" is required when you are using --split-by is used on a column which is of text type. There is difference in the TextSplitter class of Sqoop jars in HDP 2.4 and HDP 2.5 because of sqoop command fails without the required argument in HDP 2.5.

View solution in original post

Highlighted

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

Expert Contributor

This JIRA has some info about why you need to set the property: https://issues.apache.org/jira/browse/SQOOP-2910

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

Contributor

@Gayathri Reddy G or @Sindhu

can you tell me exactly how you added the -Dorg.apache.sqoop.splitter.allow_text_splitter=true to your import statement?

I'm trying to create a sqoop job with this import statement but it keeps failing:

16/10/27 17:53:25 ERROR tool.BaseSqoopTool: Error parsing arguments for import:
16/10/27 17:53:25 ERROR tool.BaseSqoopTool: Unrecognized argument: -Dorg.apache.sqoop.splitter.allow_text_splitter=true

my shell script to create the sqoop job looks like this:

#!/bin/bash

sqoop job \
  --create <tablename> \
  -- \
  import "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" \
  --connect "jdbc:<URL>"

If I remove the -Dorg line all goes file but offcourse the execution of the job failes thats why I need to pass this allow_text_splitter as a parameter.

I tried with double quotes, without, but no luck unfortunately

Highlighted

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

Try giving it just after sqoop job. Eg: Sqoop job "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" -- import ...

Highlighted

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

Contributor

worked! many thanks, you saved my day.

Highlighted

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

New Contributor

Can you include that in a line through code?

sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true \
--connect jdbc:mysql://localhost/sqoop \
--username root \
-passqord 0range \
--split-by id \
--columns id,name \
--table customer \
--target-dir /user/cloudera/ingest/raw/customers \
--fields-terminated-by "," \
--hive-import \
--create-hive-table \
--hive-table sqoop_workspace.customers

 

Highlighted

Re: Sqoop Import: "-Dorg.apache.sqoop.splitter.allow_text_splitter=true"

Community Manager

@sri_man 

Since this thread was marked 'Solved' back in 2016, you would have a better chance of receiving a relevant response by posting a new question. This will also provide the opportunity to provide details specific to your environment that could aid other members in providing a more tailored answer to your issue. 

 

 

Bill Brooks, Community Manager
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Don't have an account?
Coming from Hortonworks? Activate your account here