I am trying to export a directory using sqoop into SQL Server. I am having trouble with this one particular directory and getting the error found below. I have searched all over and tried many different iterations of my sqoop parameters to try to get it to work, but still cannot get it to work. If anyone has any suggestions they would be greatly appreaciated.
ERROR from logs:
2018-05-18 08:41:20,378 WARN [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:rjkeen (auth:SIMPLE) cause:java.io.IOException: Can't export data, please check failed map task logs
2018-05-18 08:41:20,378 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:122)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.RuntimeException: Can't parse input data: 'http://tableau.abc.edu/views/SmartAlerts-Commercial/SmartAlerts?iframeSizedToWindow=true&:embed=y&:s...'
at smart_alert.__loadFromFields(smart_alert.java:708)
at smart_alert.parse(smart_alert.java:606)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:89)
... 10 more
Caused by: java.util.NoSuchElementException
at java.util.ArrayList$Itr.next(ArrayList.java:834)
at smart_alert.__loadFromFields(smart_alert.java:703)
... 12 more
Sqoop Export looks like:
export
--connect jdbc:sqlserver://ABC123;databaseName=test;integratedSecurity=false;applicationName=SQOOP; --username ********* --password **************
-m
1
--table
smart_alert
--columns
"customer_account_id,alert_date,alert_level,fromdate,todate,expected_usage,actual_usage,percentdifference,alert_type,dashboard,dt"
--input-fields-terminated-by
','
--input-lines-terminated-by
'\n'
--input-null-string
"\\N"
--input-null-non-string
"\\N"
--
--schema
dbo
Hive Table structure:
| CREATE EXTERNAL TABLE dev_smart_alert.commercial_usage_alert_test ( customer_account_id INT, alert_date STRING, alert_level INT, fromdate STRING, todate STRING, expected_usage DOUBLE, actual_usage DOUBLE, percentdifference DOUBLE, alert_type STRING, dashboard STRING ) PARTITIONED BY ( dt STRING ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' WITH SERDEPROPERTIES ('field.delim'=',', 'line.delim'='\n', 'serialization.format'=',') STORED AS TEXTFILE LOCATION 'hdfs://nameservice1/development/warehouse/smart_alert/commercial_usage_alert_test' |
SQL Table Structure:
[dbo].[smart_alert](
[customer_account_id] [int] NOT NULL,
[alert_date] [date] NOT NULL,
[alert_level] [smallint] NOT NULL,
[fromdate] [date] NOT NULL,
[todate] [date] NOT NULL,
[expected_usage] [float] NOT NULL,
[actual_usage] [float] NOT NULL,
[percentdifference] [float] NOT NULL,
[alert_type] [varchar](100) NOT NULL,
[dashboard] [varchar](8000) NOT NULL,
[dt] [date] NOT NULL
)
