Re: How to use sqoop password-alias together with...

lucy_zhang_us · ‎10-04-2017

You can find the GC time in LLAP monitor link. for example: http://XXX-hddb-01-b:15002/

lucy_zhang_us · ‎08-18-2017

Thank you @Sonu Sahi for the response. I checked your colleague's blog. He is using password file but not the password alias (credential provider) which is saving password in keystore instead of file. Any way I finally figure out how to make it work. In case some one else need this in the future, I paste my solution here: The option file: import -connect "jdbc:sqlserver://host:1433;database=DB1" Script for Sqoop job defination: sqoop job -Dhadoop.security.credential.provider.path=jceks://hdfs/user/root/mysql.password.jceks --create sqoop_test -- options-file db.props --username sqoop_ro --password-alias sqoop_ro.passwd --target-dir /XXX --query 'select ....' This works for me. Thanks

lucy_zhang_us · ‎08-17-2017

I'm able to use credential provider to encrypt password. Also be able to use it with sqoop command. But I can't work it out if I want to put connection string in option files. Anybody can give me some clue ? Thanks

lucy_zhang_us · ‎05-23-2017

Same issue with script init_nifi_in_src_data.sh. Command not found. It's a pain. You don't have this issue ? @Sonu Sahi

lucy_zhang_us · ‎05-23-2017

I'm able to open Nifi UI by http://localhost:19090/nifi/ . Maybe I can ignore the script error ?

lucy_zhang_us · ‎05-23-2017

Yes, I'm able to log on http://127.0.0.1:9080 and http://127.0.0.1:18888. Can't open http://sandbox-hdf.hortonworks.com:19090/nifi Have issue with the tutorial script . It says virtualBox command not found. @Sonu Sahi

lucy_zhang_us · ‎05-23-2017

Hi @Sonu Sahi . Any idea with below error? I tried this script in both virtualBox and Vmware. Neither works. In VirtualBox, it complains vboxmanage not found. Thanks [root@sandbox-hdf ~]# ./auto_scripts/launch_ambari_ui.sh HDF Sandbox Virtual Environment Selection Menu 'Docker' - is my environment 'VirtualBox' - is my environment Enter the 'virtual environment' for HDF Sandbox: VirtualBox You entered HDF Sandbox runs on: VirtualBox Start VirtualBox VM's Docker HDF Sandbox... ./auto_scripts/virtualbox-scripts/virtualbox_sandbox_hdf.sh: line 3: vboxmanage: command not found Starting Ambari Services, may take 2.5 minutes...

lucy_zhang_us · ‎05-19-2017

Finally I figured out the reason. I downloaded the HDF sandbox vmware image. When I click the file to open, it pop up and automatically opened by virtualbox which can't work correctly for the network part. Realized this, I turned off and opened in VMWare. Issue gone. Thanks for your help.

lucy_zhang_us · ‎05-19-2017

Thank you @Sonu Sahi for your response. I'm actually also using virtual box. Attached is my screenshot: NOt sure what's wrong. My network configuration also NAT.

lucy_zhang_us · ‎05-19-2017

Yes, the VM setup is using NAT, After I started the VM. Before press ALT+F5, The VMware says "please open a browser's address field: http://10.0.2.15:18888/. However I got "connection time out " error. I press ALT+F5 to log in the Vmware as root user. Then tried "ssh -p 2222 root@127.0.0.1" it doesn't work. @Sonu Sahi Could you help check if you did the same but got different result ? Thanks

lucy_zhang_us · ‎05-18-2017

I have been able to connect to HDP sanbox by ssh -p 2222. But with the new HDF sandbox, I was stuck again with the "connection refused" error. After I brought up the sandbox, I tried the URL "http://10.0.2.15:18888/" but just got timeout error. Any idea ? @mqureshi

lucy_zhang_us · ‎05-18-2017

I removed the original VM from the virtualBOX and restarted the HDF VM. Then the error seems gone ? Not sure why removing another VMware from virtualBOX can hellp though. Thanks @mqureshi

lucy_zhang_us · ‎05-18-2017

I have downloaded the right HDF sandbox and confirmed my downloading has the same MD5 checksum. After I started the virtual machine, I keep getting below error, Any idea ? rejecting I/O to offline devices @mqureshi

lucy_zhang_us · ‎05-18-2017

Issue with HDF sandbox: After starting the virtual machine, I got lines of errors saying "sd 2:0:0:0: rejecting I/O to offline device"

lucy_zhang_us · ‎05-18-2017

You are right, it should be HDF sandbox. Seems I downloaded wrong VMware. Will redownload and try again. Thanks.

lucy_zhang_us · ‎05-18-2017

@mqureshi ; The tutorial can be found at https://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi/#section_3. First I logged in the Vmware, then I used ssh into container by port 2222.

lucy_zhang_us · ‎05-16-2017

Not sure if hortornworks support team really run these command when they prepare the Nifi tutorial. it doesn't work for me. vboxmanage is not found in the sandbox. Open command doesn't work either. Same issue with the other script launch_nifi_ui.sh [root@sandbox ~]# ./auto_scripts/launch_ambari_ui.sh HDF Sandbox Virtual Environment Selection Menu 'Docker' - is my environment 'VirtualBox' - is my environment Please enter 'Docker' or 'VirtualBox'. HDF Sandbox Virtual Environment Selection Menu 'Docker' - is my environment You entered HDF Sandbox runs on: VirtualBox Start VirtualBox VM's Docker HDF Sandbox... ./auto_scripts/virtualbox-scripts/virtualbox_sandbox_hdf.sh: line 3: vboxmanage: command not found Starting Ambari Services, may take 2.5 minutes.. . Launch Ambari UI... ./auto_scripts/launch_ambari_ui.sh: line 34: open: command not found

lucy_zhang_us · ‎04-14-2017

@thussain, It does logged some java error in hivemetastore.log. Please ese attachment hivemetastore.txt

lucy_zhang_us · ‎04-14-2017

I'm running a hive POC. We have a combined big table which has > 2300 columns and 44million rows. I submitted the command "analyze table TABXXX COMPUTE STATISTICS for COLUMNS;" It failed with below messages in the end: Map 1: 79(+1)/80 Reducer 2: 0(+1)/1 Map 1: 79(+1)/80 Reducer 2: 0(+1)/1 Map 1: 79(+1)/80 Reducer 2: 0(+1)/1 Map 1: 79(+1)/80 Reducer 2: 0(+1)/1 Map 1: 80/80 Reducer 2: 0(+1)/1 Map 1: 80/80 Reducer 2: 1/1 FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exe Anybody can give me some hint how to make it work ?

lucy_zhang_us · ‎03-31-2017

Thanks. The upper case of first letter for java type works. --map-column-java isactive=Integer,etl_isactive=Integer. Execution successfully: INFO mapreduce.ImportJobBase: Retrieved 88281 records.

lucy_zhang_us · ‎03-31-2017

I'm importing data from our SQL server database to HDFS with Sqoop. Some tables have bit data type in SQL Server. By default, sqoop will convert the bit column like 1->true 0 -> false. We want the data be converted to int type 1 or 0. So I tried the option --map-column-java. For example sqoop ... --map-column-java isactive=int; and even --map-column-java isactive=string, Neither works. I always got error : ERROR orm.ClassWriter: No ResultSet method for Java type integer ERROR tool.ImportTool: Imported Failed: No ResultSet method for Java type integer. Seems in Hortonworks platform, sqoopt doesn't support this option?

Online	Offline
Last Visited	‎10-04-2017 10:59 PM

Date Registered	‎03-31-2017 04:56 PM
Last Visited	‎10-04-2017 10:59 PM
Total Messages Posted	21
Total Kudos Received	3

Re: How to use sqoop password-alias together with...

Re: Connection refused for HDF sandbox

Re: Where to view the gc time for LLAP queries in ...

Re: How to use sqoop password-alias together with...

How to use sqoop password-alias together with sqo...

Re: Connection refused for HDF sandbox

Re: Connection refused for HDF sandbox

Re: Connection refused for HDF sandbox

Re: Connection refused for HDF sandbox

Re: Connection refused for HDF sandbox

Re: Connection refused for HDF sandbox

Re: Connection refused for HDF sandbox

Connection refused for HDF sandbox

Re: Issue with HDF sandbox

Re: Issue with Nifi Tutorial from the new Sandbox ...

Issue with HDF sandbox

Re: Issue with Nifi Tutorial from the new Sandbox ...

Re: Issue with Nifi Tutorial from the new Sandbox ...

Issue with Nifi Tutorial from the new Sandbox HDP ...

Re: ColumnStatsTask failed on one big and wide tab...

ColumnStatsTask failed on one big and wide table

Re: Issue with --map-column-java in sqoop command ...

Issue with --map-column-java in sqoop command wit...