Member since
10-10-2017
13
Posts
4
Kudos Received
0
Solutions
10-29-2018
09:18 AM
1 Kudo
Greetings Doug, I just ran into this same issue and resolved it. I wanted to give you some context first. The Telemetry Publushing Service is not a necessary component of a Cloudera cluster deploy - it is to supplement an add-on service called "Workload Experience Manager or Workload XM". You can safely delete the service and it will not affect cluster operations If Workload XM is a functionality you wish to leverage then you will need to take a couple steps to enable the Telemetry Publisher to send telemetry to Cloudera. Here is the Guide I Followed: https://www.cloudera.com/documentation/wxm/latest/topics/wxm_setup.html Step 1. Get Altus Credentials Step 2. Add Altus Credentials to Cloudera Manager Step 3. Add the Telemetry Publisher Service Role Basically you need to log into your Altus account and generate a key pair for WXM to leverage - you will then need to add this account in Cloudera Manager as an external account under the altus tab. That can be done here. https://altus.cloudera.com/wxm/home.html#/ Next in Cloudera Manager: Administration -> External Accounts -> click "Altus Credentials" tab and add the account. You may need to configure the account as your Altus account in Administration -> Settings -> "Telemetry Altus Account" The docs don't say this but I had to restart Cloudera Manager to get this setting to take effect - I could not start the Telemetry Publisher until after I restarted Cloudera Manager To access Workload XM, perform the following steps: Log in to the Workload XM console: wxm.cloudera.com/
... View more
10-10-2017
09:03 AM
1 Kudo
I am able to do this by running (or adding this to your ~/.bashrc) export PATH=/opt/cloudera/parcels/Anaconda/bin:$PATH After this conda, pip, and python commands all defaulted to the Anaconda version.
... View more
03-14-2017
03:56 AM
1 Kudo
I have found a solution to this provided by another user here: https://community.hortonworks.com/questions/20719/sqoop-to-sql-server-with-integrated-security.html Basically if you switch to the jtds driver which you can download here: http://jtds.sourceforge.net/ Per Rajendra Manjunath " Sqoop SQL Server data import to HDFS worked with manual parametric the authentication(using windows credential) with added parameter on the SQL Server JDBC driver, as integrated security is not supported by the SQL driver as of now due to the Kerberos authentication(Delegated tokens distributed over cluster while running MR job). So we need to pass the windows authentication with password and with the integrated security disabled mode to import the data to the system. As normal SQL server driver does not support, so I had used the jtds.jar and the different driver class to pull the data to the Hadoop Lake. Sample Command I tried on the server as follows, sqoop import --table Table1 --connect "jdbc:jtds:sqlserver://<Hostname>:<Port>;useNTLMv2=true;domain=<WindowsDomainName>;databaseName=XXXXXXXXXXXXX" \ --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username XXXXX --password 'XXXXXXX' \ --verbose --target-dir /tmp/33 -m 1 -- --schema dbo " Here are some examples that worked for me: # List databases sqoop list-databases --connect "jdbc:jtds:myactivedirectorydomain.com" --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username XXXXX -P # List tables sqoop list-tables --connect "jdbc:jtds:myactivedirectorydomain.com;databaseName=DATABASENAMEHERE" --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username jmiller.admin -P # Pull data example sqoop import --table TABLENAMEHERE --connect "jdbc:jtds:myactivedirectorydomain.com;databaseName=DATABASENAMEHERE" --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username XXXXX -P --fields-terminated-by '\001' --target-dir /user/XXXXX/20170313 -m 1 -- --schema dbo Note* In the above example you need to change the username to your username and database name in the list-tables or pull to the one you need (note the AD account you use will require access to the data).
... View more
03-14-2017
03:55 AM
I have found a solution to this provided by another user here: https://community.hortonworks.com/questions/20719/sqoop-to-sql-server-with-integrated-security.html Basically if you switch to the jtds driver which you can download here: http://jtds.sourceforge.net/ Per Rajendra Manjunath " Sqoop SQL Server data import to HDFS worked with manual parametric the authentication(using windows credential) with added parameter on the SQL Server JDBC driver, as integrated security is not supported by the SQL driver as of now due to the Kerberos authentication(Delegated tokens distributed over cluster while running MR job). So we need to pass the windows authentication with password and with the integrated security disabled mode to import the data to the system. As normal SQL server driver does not support, so I had used the jtds.jar and the different driver class to pull the data to the Hadoop Lake. Sample Command I tried on the server as follows, sqoop import --table Table1 --connect "jdbc:jtds:sqlserver://<Hostname>:<Port>;useNTLMv2=true;domain=<WindowsDomainName>;databaseName=XXXXXXXXXXXXX" \ --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username XXXXX --password 'XXXXXXX' \ --verbose --target-dir /tmp/33 -m 1 -- --schema dbo " Here are some examples that worked for me: # List databases sqoop list-databases --connect "jdbc:jtds:myactivedirectorydomain.com" --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username XXXXX -P # List tables sqoop list-tables --connect "jdbc:jtds:myactivedirectorydomain.com;databaseName=DATABASENAMEHERE" --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username jmiller.admin -P # Pull data example sqoop import --table TABLENAMEHERE --connect "jdbc:jtds:myactivedirectorydomain.com;databaseName=DATABASENAMEHERE" --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username XXXXX -P --fields-terminated-by '\001' --target-dir /user/XXXXX/20170313 -m 1 -- --schema dbo Note* In the above example you need to change the username to your username and database name in the list-tables or pull to the one you need (note the AD account you use will require access to the data).
... View more
03-14-2017
03:55 AM
I have found a solution to this provided by another user here: https://community.hortonworks.com/questions/20719/sqoop-to-sql-server-with-integrated-security.html Basically if you switch to the jtds driver which you can download here: http://jtds.sourceforge.net/ Per Rajendra Manjunath " Sqoop SQL Server data import to HDFS worked with manual parametric the authentication(using windows credential) with added parameter on the SQL Server JDBC driver, as integrated security is not supported by the SQL driver as of now due to the Kerberos authentication(Delegated tokens distributed over cluster while running MR job). So we need to pass the windows authentication with password and with the integrated security disabled mode to import the data to the system. As normal SQL server driver does not support, so I had used the jtds.jar and the different driver class to pull the data to the Hadoop Lake. Sample Command I tried on the server as follows, sqoop import --table Table1 --connect "jdbc:jtds:sqlserver://<Hostname>:<Port>;useNTLMv2=true;domain=<WindowsDomainName>;databaseName=XXXXXXXXXXXXX" \ --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username XXXXX --password 'XXXXXXX' \ --verbose --target-dir /tmp/33 -m 1 -- --schema dbo " Here are some examples that worked for me: # List databases sqoop list-databases --connect "jdbc:jtds:myactivedirectorydomain.com" --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username XXXXX -P # List tables sqoop list-tables --connect "jdbc:jtds:myactivedirectorydomain.com;databaseName=DATABASENAMEHERE" --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username jmiller.admin -P # Pull data example sqoop import --table TABLENAMEHERE --connect "jdbc:jtds:myactivedirectorydomain.com;databaseName=DATABASENAMEHERE" --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username XXXXX -P --fields-terminated-by '\001' --target-dir /user/XXXXX/20170313 -m 1 -- --schema dbo Note* In the above example you need to change the username to your username and database name in the list-tables or pull to the one you need (note the AD account you use will require access to the data).
... View more
03-14-2017
03:54 AM
I have found a solution to this provided by another user here: https://community.hortonworks.com/questions/20719/sqoop-to-sql-server-with-integrated-security.html Basically if you switch to the jtds driver which you can download here: http://jtds.sourceforge.net/ Per Rajendra Manjunath " Sqoop SQL Server data import to HDFS worked with manual parametric the authentication(using windows credential) with added parameter on the SQL Server JDBC driver, as integrated security is not supported by the SQL driver as of now due to the Kerberos authentication(Delegated tokens distributed over cluster while running MR job). So we need to pass the windows authentication with password and with the integrated security disabled mode to import the data to the system. As normal SQL server driver does not support, so I had used the jtds.jar and the different driver class to pull the data to the Hadoop Lake. Sample Command I tried on the server as follows, sqoop import --table Table1 --connect "jdbc:jtds:sqlserver://<Hostname>:<Port>;useNTLMv2=true;domain=<WindowsDomainName>;databaseName=XXXXXXXXXXXXX" \ --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username XXXXX --password 'XXXXXXX' \ --verbose --target-dir /tmp/33 -m 1 -- --schema dbo " Here are some examples that worked for me: # List databases sqoop list-databases --connect "jdbc:jtds:myactivedirectorydomain.com" --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username XXXXX -P # List tables sqoop list-tables --connect "jdbc:jtds:myactivedirectorydomain.com;databaseName=DATABASENAMEHERE" --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username jmiller.admin -P # Pull data example sqoop import --table TABLENAMEHERE --connect "jdbc:jtds:myactivedirectorydomain.com;databaseName=DATABASENAMEHERE" --connection-manager org.apache.sqoop.manager.SQLServerManager --driver net.sourceforge.jtds.jdbc.Driver --username XXXXX -P --fields-terminated-by '\001' --target-dir /user/XXXXX/20170313 -m 1 -- --schema dbo Note* In the above example you need to change the username to your username and database name in the list-tables or pull to the one you need (note the AD account you use will require access to the data).
... View more
03-13-2017
08:49 PM
1 Kudo
I have found a solution to this provided by another user here: https://community.hortonworks.com/questions/20719/sqoop-to-sql-server-with-integrated-security.html Basically if you switch to the jtds driver which you can download here: http://jtds.sourceforge.net/ Per Rajendra Manjunath " Sqoop SQL Server data import to HDFS worked with manual
parametric the authentication(using windows credential) with added parameter on
the SQL Server JDBC driver, as integrated security is not supported by the SQL
driver as of now due to the Kerberos authentication(Delegated tokens
distributed over cluster while running MR job). So we need to pass the windows authentication with password and
with the integrated security disabled mode to import the data to the system. As
normal SQL server driver does not support, so I had used the jtds.jar and the
different driver class to pull the data to the Hadoop Lake. Sample Command I tried on the server as follows, sqoop import --table Table1 --connect
"jdbc:jtds:sqlserver://<Hostname>:<Port>;useNTLMv2=true;domain=<WindowsDomainName>;databaseName=XXXXXXXXXXXXX"
\ --connection-manager org.apache.sqoop.manager.SQLServerManager
--driver net.sourceforge.jtds.jdbc.Driver --username XXXXX --password 'XXXXXXX'
\ --verbose --target-dir /tmp/33 -m 1 -- --schema dbo " Here are some examples that worked for me: # List
databases sqoop
list-databases --connect
"jdbc:jtds:sqlserver://databasehostname.yourdomain.com:1433;useNTLMv2=true;domain=myactivedirectorydomain.com"
--connection-manager org.apache.sqoop.manager.SQLServerManager --driver
net.sourceforge.jtds.jdbc.Driver --username XXXXX -P # List
tables sqoop
list-tables --connect "jdbc:jtds:sqlserver://databasehostname.yourdomain.com:1433;useNTLMv2=true;domain=myactivedirectorydomain.com;databaseName=DATABASENAMEHERE"
--connection-manager org.apache.sqoop.manager.SQLServerManager --driver
net.sourceforge.jtds.jdbc.Driver --username jmiller.admin -P # Pull
data example sqoop import --table
TABLENAMEHERE --connect
"jdbc:jtds:sqlserver://databasehostname.yourdomain.com:1433;useNTLMv2=true;domain=myactivedirectorydomain.com;databaseName=DATABASENAMEHERE"
--connection-manager org.apache.sqoop.manager.SQLServerManager --driver
net.sourceforge.jtds.jdbc.Driver --username XXXXX -P --fields-terminated-by '\001'
--target-dir /user/XXXXX/20170313 -m 1 -- --schema dbo Note* In the above
example you need to change the username to your username and database name in
the list-tables or pull to the one you need (note the AD account you use will
require access to the data).
... View more
03-13-2017
07:04 PM
Greetings Rajendra, You have saved my life, your solution works!!! We have spent two years on this thinking there was no way to get Sqoop working, I want to buy you a drink!!
... View more
02-09-2017
06:39 PM
Definately has not been resolved. It's something Sqoop can't currently do.
... View more
02-09-2017
06:19 PM
1 Kudo
I spent months on this and was told by Cloudera that it is just not possible. The kerberos tokens are lost when the mappers spawn (as the YARN transitions the job to its internal security subsystem). I think this is something that would have to be added to Sqoop itself, much like HBase mapreduce jobs have to pass job config to mappers in the code Sqoop able to do the same. I spent weeks trying different things, doing network traces (all my servers are tied to AD with Centrify, ext), watched the token drop, and finally found a paper written by Yahoo about YARN that explains the internal token subsystem it uses (after using Kerberos to externally verify someone it moves to different token based subsystem for performance). My company would be willing to pay some $$ to get this working. All our data is being held hostage in SQL Server (from a company we are merging with).
... View more
02-09-2017
06:17 PM
I spent months on this and was told by Cloudera that it is just not possible. The kerberos tokens are lost when the mappers spawn (as the YARN transitions the job to its internal security subsystem). I think this is something that would have to be added to Sqoop itself, much like HBase mapreduce jobs have to pass job config to mappers in the code Sqoop needs to do the same.
... View more
02-01-2017
06:19 PM
This project is actually broken, spent quite a bit of effort trying to get it to work a year ago. It's a shame because it would be great. Looks like it was built during a hackathon then abandoned.
... View more