Member since
10-25-2016
12
Posts
2
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 4452 | 11-04-2016 10:23 AM | |
| 3433 | 10-27-2016 09:01 AM |
02-15-2017
08:12 AM
thanks for your feedback cdguna. The file already exists: [root@bda1node03 managers.d]# pwd
/etc/sqoop/conf/managers.d
[root@bda1node03 managers.d]# ls -lrt
total 4
-rwxrwxrwx 1 root root 105 Feb 8 17:24 cldra_td_connector.txt
[root@bda1node03 managers.d]# cat cldra_td_connector.txt
com.cloudera.connector.teradata.TeradataManagerFactory=/var/lib/sqoop/sqoop-connector-teradata-1.2c5.jar
[root@bda1node03 managers.d]# but the errors (Teradata and Oracle) persists. The cldra_td_connector.txt should also solve the oracle error? Thanks
... View more
02-09-2017
02:05 AM
Hello I have a 6 nodes cluster with cloudera CDH 5.7.1 and Sqoop version: 1.4.6 I've installed the connector in the proper location /var/lib/sqoop bash-4.1$ ls -lrt total 13928 -rwxr-xr-x 1 root root 10489904 Oct 12 10:10 sqoop-connector-teradata-1.2c5.jar -rwxr-xr-x 1 root root 2455 Oct 12 10:10 tdgssconfig.jar -rwxr-xr-x 1 root root 991265 Oct 12 10:10 terajdbc4.jar -rwxrwxrwx 1 root root 2739670 Feb 8 17:57 ojdbc6.jar -rw-r--r-- 1 root root 11166 Feb 8 18:13 QueryResult.java when I try any to ingest data... for example sqoop import --connect jdbc:oracle:thin:@dbquad0:3999:QUAD0 --username user--password pwd--query "select cod_ateco,id_datc from qzt_dim_ateco" --hive-import --target-dir '/tmp/sqoopimport' --hive-table dece_sqoop_quad --split-by object_type I allways the following error Warning: /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/02/09 10:55:43 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.1
17/02/09 10:55:43 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/02/09 10:55:43 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
17/02/09 10:55:43 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
17/02/09 10:55:43 ERROR sqoop.ConnFactory: Could not load ManagerFactory com.cloudera.sqoop.manager.TeradataManagerFactory (not found)
17/02/09 10:55:43 ERROR tool.BaseSqoopTool: Got error creating database manager: java.io.IOException: No manager for connect string: jdbc:oracle:thin:@dbquad0:3999:QUAD0
at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:191)
at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:258)
at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:89)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:603)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236) if i use option --driver oracle.jdbc.driver.OracleDriver for listtable work fine can yuo help me?
... View more
Labels:
- Labels:
-
Apache Sqoop
11-04-2016
10:23 AM
1 Kudo
I finaly solved the problem! probably was a problem of creating tables on mysql. to solve this error: Error: Index column size too large. The maximum column size is 767 bytes It's necessary to create the sentry tables using ROW_FORMAT = DYNAMIC. In the cloudera version CDH 5.7.1.1 in the scritps /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/sentry/scripts/sentrystore/upgrade/sentry-mysql-1.5.0.sql the ROW_FORMAT option is omitted, the default is COMPACT. I must use the scripts of version CDH 5.7.0.1 /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p1464.1349/lib/sentry/scripts/sentrystore/upgrade/sentry-mysql-1.5.0.sql where the option ROW_FORMAT = DYNAMIC is explicit. With this script i created correctly the sentry table. After I restart the service i have created the role on HUE and beeline without any problem. My only doubt is, why cdh 5.7.1.1 doesn't contain the option ROW_FORMAT?
... View more
11-04-2016
05:50 AM
I try to recreate Sentry Database Table from CM( https://cloudera-portal.force.com/articles/KB_Article/Recreating-Database-Tables-Sentry-Service?popup=false&navBack=H4sIAAAAAAAAAFWLuwrCMBRA_yXgFEkHN0FERBCEOtRNRPK4JrFtbsijWsR_NxUcHM_hnPOLRLIkB4ePDpSGPfZA5qQtznMNJ5u6iXPh6ppt1f7CqsDftY7AgzSrLc8RFBXjbLGhGDTjnksDLIJLYWQ-4GAVBKYEa76qxiZLcxR3kGn3lOCTRTfNNdKeJ2ms03SAEIumN8xOkfflA4t1TeC4AAAA ) , i received this error: + exec /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/bin/hadoop jar /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/sentry/lib/sentry-core-common-1.5.1-cdh5.7.1.jar
org.apache.sentry.SentryMain --command schema-tool --verbose --conffile /var/run/cloudera-scm-agent/process/3000-sentry-create-dbtables/sentry-site.xml --dbType mysql --initSchema
Connecting to jdbc:mysql://bda1node03.sede.corp.sanpaoloimi.com:3306/sentry?useUnicode=true&characterEncoding=UTF-8
Connected to: MySQL (version 5.6.30-enterprise-commercial-advanced-log)
Driver: MySQL Connector Java (version mysql-connector-java-commercial-5.1.34 ( Revision: jess.balint@oracle.com-20141014163213-wqbwpf1ok2kvo1om ))
Transaction isolation: TRANSACTION_REPEATABLE_READ
Error: Index column size too large. The maximum column size is 767 bytes. (state=HY000,code=1709)
org.apache.sentry.SentryUserException: Schema initialization FAILED! Metastore state would be inconsistent !!
org.apache.sentry.SentryUserException: Schema initialization FAILED! Metastore state would be inconsistent !!
at org.apache.sentry.provider.db.tools.SentrySchemaTool.doInit(SentrySchemaTool.java:322)
at org.apache.sentry.provider.db.tools.SentrySchemaTool.doInit(SentrySchemaTool.java:295)
at org.apache.sentry.provider.db.tools.SentrySchemaTool$CommandImpl.run(SentrySchemaTool.java:569)
at org.apache.sentry.SentryMain.main(SentryMain.java:114) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.io.IOException: Schema script failed, errorcode 2
at org.apache.sentry.provider.db.tools.SentrySchemaTool.runBeeLine(SentrySchemaTool.java:423)
at org.apache.sentry.provider.db.tools.SentrySchemaTool.runBeeLine(SentrySchemaTool.java:396)
at org.apache.sentry.provider.db.tools.SentrySchemaTool.doInit(SentrySchemaTool.java:318)
... 9 more *** Sentry schemaTool failed *** The InnoDB storage engine is supported
... View more
11-04-2016
01:59 AM
Hi all, - CDH 5.7.1 - Hive 1.1.0 - Impala 2.5.0 - Sentry service Enable - NO LDAP - NO Kerberos I have installed and configured Sentry Service from CM. In HUE i have created user "hive"and relative group ("hive") with all privilege (hue privilege) When i try to create a new role i receveid thi error: hive ERROR could not create role
Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/apps/security/src/security/api/hive.py", line 176, in create_role
api.create_sentry_role(role['name'])
File "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/desktop/libs/libsentry/src/libsentry/api.py", line 49, in decorator
raise e
SentryException: Unknown error for request: TCreateSentryRoleRequest(protocol_version:2, requestorUserName:hive, roleName:prova), message: Error(s) were found while auto-creating/validating the datastore for classes. The errors are printed in the log, and are attached to this exception. I try to create role from beeline and receveid the same error command: 0: jdbc:hive2://bda1node04:10000> create role role1; error:
2016-11-03 14:43:32,563 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-55]: </PERFLOG method=Driver.execute start=1478180612438 end=1478180612563 duration=125 from=org.apache.hadoop.hive.ql.Driver>
2016-11-03 14:43:32,563 INFO org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-55]: Completed executing command(queryId=hive_20161103144343_efb717ff-b63c-408c-90f6-a93944eede95); Time taken: 0.125 seconds
2016-11-03 14:43:32,563 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-55]: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
2016-11-03 14:43:32,563 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-55]: </PERFLOG method=releaseLocks start=1478180612563 end=1478180612563 duration=0 from=org.apache.hadoop.hive.ql.Driver>
2016-11-03 14:43:32,564 ERROR org.apache.hive.service.cli.operation.Operation: [HiveServer2-Background-Pool: Thread-55]: Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for request: TCreateSentryRoleRequest(protocol_version:2, requestorUserName:hive, roleName:role1), message: Error(s) were found while auto-creating/validating the datastore for classes. The errors are printed in the log, and are attached to this exception.. Server Stacktrace: javax.jdo.JDODataStoreException: Error(s) were found while auto-creating/validating the datastore for classes. The errors are printed in the log, and are attached to this exception.
at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
at org.apache.sentry.provider.db.service.persistent.SentryStore.createSentryRole(SentryStore.java:316)
at org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.create_sentry_role(SentryPolicyStoreProcessor.java:230)
at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$create_sentry_role.getResult(SentryPolicyService.java:893)
at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$create_sentry_role.getResult(SentryPolicyService.java:878)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
NestedThrowablesStackTrace:
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 3072 bytes
at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
at com.mysql.jdbc.Util.getInstance(Util.java:360)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:978)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2526)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:848)
at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:742)
at com.jolbox.bonecp.StatementHandle.execute(StatementHandle.java:300)
at org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatement(AbstractTable.java:760)
at org.datanucleus.store.rdbms.table.TableImpl.createIndices(TableImpl.java:648)
at org.datanucleus.store.rdbms.table.TableImpl.createConstraints(TableImpl.java:422)
at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.performTablesValidation(RDBMSStoreManager.java:3487)
at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.addClassTablesAndValidate(RDBMSStoreManager.java:3218)
at org.datanucleus.store.rdbms.RDBMSStoreManager$ClassAdder.run(RDBMSStoreManager.java:2869)
at org.datanucleus.store.rdbms.AbstractSchemaTransaction.execute(AbstractSchemaTransaction.java:122)
at org.datanucleus.store.rdbms.RDBMSStoreManager.addClasses(RDBMSStoreManager.java:1606)
at org.datanucleus.store.AbstractStoreManager.addClass(AbstractStoreManager.java:954)
at org.datanucleus.store.rdbms.RDBMSStoreManager.getDatastoreClass(RDBMSStoreManager.java:680) Have you any suggestions? thanks
... View more
10-27-2016
09:01 AM
1 Kudo
I have solved adding this value into sentry-site.xml: <property><name>sentry.verify.schema.version</name><value> false</value><description> value: true, false true Sentry store will verify the schema version in backed DB with expected version in jar. The service won't start if there's a mismatch </description></property>
... View more
10-27-2016
02:40 AM
Hi all, on cloudera manager 5.7.1 i have use this guide ( http://www.cloudera.com/documentation/enterprise/latest/topics/sg_sentry_service_install.html#concept_zsk_pr5_1q__section_pwm_fsx_kq ) to install Sentry Service. Before to install service i have create an external MySql database with this commands: mysql> create database sentry DEFAULT CHARACTER SET utf8; Query OK, 1 row affected (0.00 sec) mysql> grant all on sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry_password'; Query OK, 0 rows affected (0.00 sec) When i try to start sentry service i received this error: exec /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/sentry/bin/sentry --command service --log4jConf /var/run/cloudera-scm-agent/process/2713-sentry-SENTRY_SERVER/sentry-log4j.properties -conffile /var/run/cloudera-scm-agent/process/2713-sentry-SENTRY_SERVER/sentry-site.xml
Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.Exception: Error starting server
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.sentry.service.thrift.SentryService.waitOnFuture(SentryService.java:312)
at org.apache.sentry.service.thrift.SentryService$CommandImpl.run(SentryService.java:398)
at org.apache.sentry.SentryMain.main(SentryMain.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.Exception: Error starting server
at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:172)
at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalStateException: Could not create org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessorFactory
at org.apache.sentry.service.thrift.SentryService.runServer(SentryService.java:207)
at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:168)
... 5 more
Caused by: org.apache.sentry.provider.db.SentryNoSuchObjectException: No matching version found
at org.apache.sentry.provider.db.service.persistent.SentryStore.getMSentryVersion(SentryStore.java:1453)
at org.apache.sentry.provider.db.service.persistent.SentryStore.getSentryVersion(SentryStore.java:1405)
at org.apache.sentry.provider.db.service.persistent.SentryStore.verifySentryStoreSchema(SentryStore.java:196)
at org.apache.sentry.provider.db.service.persistent.SentryStore.<init>(SentryStore.java:176)
at org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.<init>(SentryPolicyStoreProcessor.java:103)
at org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessorFactory.register(SentryPolicyStoreProcessorFactory.java:31)
at org.apache.sentry.service.thrift.SentryService.runServer(SentryService.java:201)
... 6 more Do yuo have any suggestion? Thanks, Simone
... View more
Labels:
