The problem went away after I changed my script adding "order by field1". select row_number() over (order by Field1) as Key1 from mytable; ... View more

I got the same problem. I have a table with over 3k records. I ran the following query in Hive and see the Key1 column recycling the number from 1 to 1024. select row_number() over () as Key1 from mytable; ... View more

By the way, on the Job Tracker UI page for this job, I saw a link to "Log", but that log page does not look like a mapper log, it contains sections for different log types, such as directory info, launch_container.sh, … , stderr, stdout, and syslog. In addition, on the Ambari - YARN - Configs - Advance page, the "Enable Log Aggregation" is set to be enabled. ... View more

@Josh Elser Thanks for the reply. Where can I find those mapper log files? I tried the job tracker UI (port 8088) via the Ambari link "Yarn - ResourceManagerUI", but could not find any log record. I did locate the map reduce jo/application id. On the "logs" tab for this job, I picked the only one attempt "appattempt_1557438882545_0078_00001", then I got the message "No container data available!" ... View more

I was able to bypass the error using the "--ignore-errors" option and loaded the file with a smaller record count. So some records were not loaded into the table. Where can I find those missing records? I read somewhere that Phoenix or HBase is heavily logged. I wonder whether those errors were logged or some Phoenix configuration I need to do to enable the logging. ... View more

where can I find a full documentation regarding this tool? org.apache.phoenix.mapreduce.CsvBulkLoadTool ... View more

I just found out that the flat file behind the table has extra spaces in that AFFILCODE column. So rtrim has to be used all the time when querying against that column. Sorry for the confusion. ... View more

Here is an update. This issue only occurs when Beeline is connected to HiveServer2 via embedded mode. When Beeline is connected to HiveServer2 via Zookeeper Discovery Mode, there is no warning and therefore no frozen Beeline. It looks like Zookeeper works betters with Kafka than HiveSever2 does. ... View more

My sqoop import job still doesn't complete and create the hive tables. Currently, my workaround is to create a dbo view on top of my warehouse view/table and then run the import job. ... View more

@Geoffrey Shelton Okot My sqoop import job still doesn't complete and create the hive tables. Currently, my workaround is to create a dbo view on top of my warehouse view/table and then run the import job. ... View more

@Geoffrey Shelton Okot Target-dir is not a scratch directory. https://stackoverflow.com/questions/37415989/difference-between-warehouse-dir-and-target-dir-commands-in-sqoop But anyway, I did see the scratch directory created under the hdfs://users/hive path every time and there were files inside. I expect the scratch directory to be deleted automatically if the import command runs through the end. Actually, I had to manually delete this scratch folder every time before I ran my test because my test import never got completed and always stopped half-way. Thanks again for your help. ... View more

Those warnings appeared from time to time after a successful hive query. The warning may or may not freeze the Beeline shell. Sometimes if I wait long enough the "0: jdbc:hive2://> " prompt appears. ... View more

I am interested in the same issue. I kept getting this error when I ran an import command the second time, although the command eventually completed successfully. 19/05/21 11:41:12 ERROR orm.CompilationManager: Could not rename /tmp/sqoop-hive/compile/71c6a7a1adae560199851f6fbb29b992/vw_Dim_IPA.java to /home/hive/./vw_Dim_IPA.java. Error: Destination '/home/hive/./vw_Dim_IPA.java' already exists vw_Dim_IPA is the source SQL table name. My questions are: Does this error matter? Can I reuse those java files? I guess that there is a reason that those java files are not deleted automatically. ... View more

@Geoffrey Shelton Okot I tested again with "target-dir" specified and got the same result. There is no error but the processed stopped at the following output. 19/05/22 12:02:24 INFO mapreduce.ImportJobBase: Transferred # bytes in # seconds 19/05/22 12:02:24 INFO mapreduce.ImportJobBase: Retrieved # records. I checked the path specified as target-dir and it is empty. ... View more

Additional test showed that, the source sql server database schema name is the root cause. After I re-create the Dim_Member (a sql view) in dbo schema, the import command below works. sqoop import --connect 'jdbc:sqlserver://xyz\\sql1:57830;database=XYZ' --username xyz --password xyz  --table 'Dim_Member' --split-by Member_SKey --hive-import --hive-table Testdb.dim_member I can not find any documentation regarding specifying database schema in Sqoop tool. My only reference is this link. https://community.hortonworks.com/questions/50557/sqoop-import-sql-server-non-default-schema.html I think the syntax of specifying the database schema is correct because at least the mapreduce job was created successfully and completed halfway. ... View more

@Geoffrey Shelton Okot Thanks for the reply. I tried including “create-hive-table” parameter and still got the same result. I did not apply the “target-dir” parameter because I want Hive to control this. This is the import command ran successfully. I am able to re-run this one several times successfully. Certainly, I dropped the TestHive.DimDepartmentGroup table before the rerun. sqoop import --connect "jdbc:sqlserver://abc:1433;database=ABC" --username abc --password abc --table "DimDepartmentGroup" --hive-import --hive-table TestHive.DimDepartmentGroup This is the import command that did not create the table. sqoop import --connect 'jdbc:sqlserver://xyz\\sql1:57830;database=XYZ' --username xyz --password xyz  --table 'Dim_Member' --split-by Member_SKey -- --schema mydw --hive-import --hive-table Testdb.dim_member --create-hive-table You can see there are three key differences between the two imports but I do not think any of them matters. Source SQL Server instance – default vs named Source SQL db schema name – default (dbo) vs user-defined (mydw) Whether to use “split-by” ... View more