Member since
05-07-2018
331
Posts
45
Kudos Received
35
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5386 | 09-12-2018 10:09 PM | |
2081 | 09-10-2018 02:07 PM | |
7778 | 09-08-2018 05:47 AM | |
2382 | 09-08-2018 12:05 AM | |
3352 | 08-15-2018 10:44 PM |
07-09-2018
03:49 AM
Got it @Krish E! If you change your split-by for boundary-queries, do you still have the same issue? --boundary-query "SELECT MIN(cast(order_number as UNSIGNED)), MAX(cast(order_number as UNSIGNED)) FROM archive_orders" Not sure if your issue isn't related to the single-quote. INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(` <- cast(order_number as UNSIGNED) ->`), MAX(`<-cast(order_number as UNSIGNED)->`) FROM `archive_orders` As you using the split-by alongside with the sqoop import-Dorg.apache.sqoop.splitter.allow_text_splitter=true, guess sqoop is taking the whole cast function as a column name, but again, it's just a guess 🙂 Hope this helps!
... View more
07-09-2018
03:24 AM
Hi @Krish E! Sorry, I miss that 😞 And have you tried to run without the cast? This parameter should allow you to run split-by with a varchar column. Another thing that's intriguing me, I assume that this Sqoop Job worked in the past right? If so, could check on the DB side if it's possible to run the following query? (Perhaps find an uncast exception) select cast(order_number as UNSIGNED) from archive_orders; I'm not sure if the error is in the sqoop side, cause I took a look at the sqoop github and didn't found any exception related to Unknown Column, so my guess would be that probably the JDBC got the error from MySQL and throw to Sqoop. Hope this helps!
... View more
07-07-2018
09:04 AM
Hi @Erkan ŞİRİN! Hm gotcha! Just asking, but do you also have the property atlas.authentication.method.kerberos=False on atlas-application.properties? Do you have anything else on the logs (sqoop,kafka/atlas)? And when sqoop works do you still see the same ERROR from InMemoryJAASConfiguration? PS: I was taking a look at the code, and you're hitting this msg https://github.com/apache/atlas/blob/master/intg/src/main/java/org/apache/atlas/security/InMemoryJAASConfiguration.java#L294 There's a lot of DEBUG trace there, if we don't get any progress with the other logs, you can try to raise the level of the logs, to get a better trace of what's happening 🙂 Hope this helps!
... View more
07-07-2018
08:21 AM
1 Kudo
Hi @Simon Jespersen! Oh man, sorry I forgot to escape the double quotes inside the XML 😞 hive -e "select xpath_string('<tns:root xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:tns=\"http://test.com\" xmlns=\"http://xmlns.oracle.com/pcbpel/adapter/noname\"> <tns:second><tns:third>10379</tns:third><tns:four>stats</tns:four><tns:five>1</tns:five><tns:six><tns:DokumentFilIndhold>K</tns:DokumentFilIndhold></tns:six><tns:seven>2018-06-28T12:57:36</tns:seven><tns:eight>2018-06-28T13:02:28</tns:eight></tns:second></tns:root>','root/second/four'); " It should work, and now that you mentioned about the hive version. I'm wondering if isn't some set value for Hive. What you can try is to check your hive client properties (you can use the example below). hive -e "set;" > hive.properties I am attaching mine for you to compare. hive.txt Enjoy your PTO! Hope this helps when you get back 🙂
... View more
07-06-2018
11:13 PM
Hi @Krish E Have you tried to set this property on your sqoop command? -Dorg.apache.sqoop.splitter.allow_text_splitter=true Then, in this case, you won't need to cast the PK. Ps: Not sure if this property works alongside with the others parameters passed, guess it's worth to test it first 🙂 Hope this helps!
... View more
07-06-2018
05:36 PM
Hi @Erkan ŞİRİN! I'm not used to Atlas, but does your keytab has an expiry date? Check if you can use the keytabs for Atlas/Kafka. Also, check this link http://www.hadoopadmin.co.in/tag/error-security-inmemoryjaasconfiguration-unable-to-add-jaas-configuration/ Hope this helps!
... View more
07-06-2018
03:56 PM
Good to know! 🙂
... View more
07-06-2018
06:10 AM
1 Kudo
Hey @Yun Ding ! Could you check the following outputs? hdfs dfs -du -h / hdfs dfsadmin -report lsblk
df -h
And also check the value for this parameter on Ambari: dfs.datanode.du.reserved PS: Just in case, check the permission for the dfs.datanode.data.dir directory, should it be owned by hdfs:hadoop. Hope this helps!
... View more
07-06-2018
05:41 AM
Hi @Simon Jespersen! Hmmm that's quite strange 😞 I made the same test as you using beeline this time and still having the stat value: [root@node3 ~]# beeline -u 'jdbc:hive2://node3:10000/default' -n hive
Connecting to jdbc:hive2://node3:10000/default
Connected to: Apache Hive (version 1.2.1000.2.6.5.0-292)
Driver: Hive JDBC (version 1.2.1000.2.6.5.0-292)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1000.2.6.5.0-292 by Apache Hive
0: jdbc:hive2://node3:10000/default> select xpath_string('<tns:root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://test.com" xmlns="http://xmlns.oracle.com/pcbpel/adapter/noname"> <tns:second><tns:third>10379</tns:third><tns:four>stats</tns:four><tns:five>1</tns:five><tns:six><tns:DokumentFilIndhold>K</tns:DokumentFilIndhold></tns:six><tns:seven>2018-06-28T12:57:36</tns:seven><tns:eight>2018-06-28T13:02:28</tns:eight></tns:second></tns:root>','root/second/four');
+--------+--+
| _c0 |
+--------+--+
| stats |
+--------+--+
1 row selected (3.282 seconds)
0: jdbc:hive2://node3:10000/default> select xpath_string('<tns:root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://test.com" xmlns="http://xmlns.oracle.com/pcbpel/adapter/noname"> <tns:second><tns:third>10379</tns:third><tns:four>stats</tns:four><tns:five>1</tns:five><tns:six><tns:DokumentFilIndhold>K</tns:DokumentFilIndhold></tns:six><tns:seven>2018-06-28T12:57:36</tns:seven><tns:eight>2018-06-28T13:02:28</tns:eight></tns:second></tns:root>','root/second/four');
+--------+--+
| _c0 |
+--------+--+
| stats |
+--------+--+
1 row selected (0.54 seconds)
Do you mind to test it using hiveCLI? Would be something like this: hive -e "select xpath_string('<tns:root xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:tns="http://test.com" xmlns="http://xmlns.oracle.com/pcbpel/adapter/noname"> <tns:second><tns:third>10379</tns:third><tns:four>stats</tns:four><tns:five>1</tns:five><tns:six><tns:DokumentFilIndhold>K</tns:DokumentFilIndhold></tns:six><tns:seven>2018-06-28T12:57:36</tns:seven><tns:eight>2018-06-28T13:02:28</tns:eight></tns:second></tns:root>','root/second/four');" The only thing that set us apart is the minor number on the hive version: Mine version 1.2.1000.2.6.5.0-292 Your version 1.2.1000.2.5.0.0-1245 This shouldn't be a problem, anyway I'm gonna check if I find smtg useful between these two versions. Hope this helps!
... View more
07-05-2018
06:05 AM
Hey @Simon Jespersen! I'm so sorry for the long delay 😞 So regarding your issue, try to take off the first "/" on the /root/second/four (2nd param for the xpath_string). Instead of '/root/second/four')would be 'root/second/four') Hope this helps!
... View more