Member since
11-24-2017
76
Posts
8
Kudos Received
5
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 3426 | 05-14-2018 10:28 AM | |
| 6314 | 03-28-2018 12:19 AM | |
| 3234 | 02-07-2018 02:54 AM | |
| 3560 | 01-26-2018 03:41 AM | |
| 4859 | 01-05-2018 02:06 AM |
01-30-2018
03:29 AM
Hello everyone I am trying to correctly configure smtp to send notification email through Oozie email action. In Cloudera Manager I have set the following properties: df When I launch the action in Hue scheduler it fails with the following error JOB[0000000-180130111414032-oozie-oozi-W] ACTION[0000000-180130111414032-oozie-oozi-W@email-e637] Error starting action [email-e637]. ErrorType [ERROR], ErrorCode [EM007], Message [EM007: Encountered an error while sending the email message over SMTP.]
org.apache.oozie.action.ActionExecutorException: EM007: Encountered an error while sending the email message over SMTP.
at org.apache.oozie.action.email.EmailActionExecutor.email(EmailActionExecutor.java:297)
at org.apache.oozie.action.email.EmailActionExecutor.validateAndMail(EmailActionExecutor.java:171)
at org.apache.oozie.action.email.EmailActionExecutor.start(EmailActionExecutor.java:110)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:332)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:261)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.mail.MessagingException: Could not connect to SMTP host: smtp.live.com, port: 25;
nested exception is:
java.net.SocketTimeoutException: connect timed out
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1282)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:370)
at javax.mail.Service.connect(Service.java:297)
at javax.mail.Service.connect(Service.java:156)
at javax.mail.Service.connect(Service.java:105)
at javax.mail.Transport.send0(Transport.java:168)
at javax.mail.Transport.send(Transport.java:98)
at org.apache.oozie.action.email.EmailActionExecutor.email(EmailActionExecutor.java:293)
... 12 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:230)
at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:189)
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1250)
... 19 more I've tried both with Outlook mails (on port 25 and 465) and Gmail (ports 465 and 587) but all fail. My cluster is on Azure, thus I've open in the network the ports 25, 465, 587. Does anyone know how to properly configure smtp for Oozie? Do I need SSL/TLS enabled (actually is disabled)? In that case, can I use a self-signed certificate or do I need a certificate from a CA? Thanks for any info!
... View more
Labels:
- Labels:
-
Apache Oozie
-
Cloudera Manager
01-26-2018
03:41 AM
Finally solved by adding the connection manager parameter and the Teradata jdbc drivers in the path: sqoop import \
--connection-manager com.teradata.jdbc.TeraDriver \
--connect jdbc:teradata://host/DATABASE=db \
[...]
The following files should be available to Oozie (I've put them in the application lib folder): sqoop-connector-teradata-1.6c5.jar tdgssconfig.jar terajdbc4.jar I guess the documentation should be updated, since with the informations provided the import from Teradata works from Sqoop standalone but fails always inside Oozie.
... View more
01-25-2018
12:51 AM
Thank you for the answer The JAVA_HOME is not set, and I've installed the jdk version from the CDH repository. Then I guess the Cloudera Manager also installed the jdk 6 during the isntallation/configuration phase. So you recommend to uninstall jdk 6 and 7 and install the latest supported jdk on each host?
... View more
01-23-2018
04:22 AM
1 Kudo
Hi everyone, which version of java is used in the cluster? I found: /usr/java/jdk1.7.0_67-cloudera/bin/javac /usr/java/jdk1.6.0_31/bin/javac /usr/java/jdk1.6.0_31/sample/javac Is the one with "cloudera" the used one? Because java -version gives: java version "1.6.0_31"
Java(TM) SE Runtime Environment (build 1.6.0_31-b04)
Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01, mixed mode) Thus I am not sure if Cloudera Manager and other tools are configured to use jdk 6 or 7. Thanks for any info
... View more
Labels:
- Labels:
-
Cloudera Manager
01-22-2018
07:58 AM
Hi bgooley, yhank you for the answer! I've set the Sqoop 1 Client Gateway and deployed the configuration. The Teradata Connector parcel was already activated during the cluster configuration phase. What puzzles me is that in the Cloudera documentation, for the manual installation, says that the following property must be added to the sqoop-site.xml file in order to use the Teradata connector with Oozie: <configuration>
<property>
<name>sqoop.connection.factories</name>
<value>com.cloudera.connector.teradata.TeradataManagerFactory</value>
</property>
<configuration> I've followed the teradata connector installation path trough Cloaudera Manager, thus I was expecting that the property had been injected automatically by the CM, but it seems missing in the sqoop-site.xml files: /run/cloudera-scm-agent/process/ccdeploy_sqoop-conf_etcsqoopconf.cloudera.sqoop_client_1328941158299821742/sqoop-conf/sqoop-site.xml <?xml version="1.0" encoding="UTF-8"?> <!--Autogenerated by Cloudera Manager--> <configuration> <property> <name>sqoop.connection.factories</name> <value></value> </property> <property> <name>sqoop.tool.plugins</name> <value></value> </property> </configuration> /etc/sqoop/conf.cloudera.sqoop_client/sqoop-site.xml <?xml version="1.0" encoding="UTF-8"?>
<!--Autogenerated by Cloudera Manager-->
<configuration>
<property>
<name>sqoop.connection.factories</name>
<value></value>
</property>
<property>
<name>sqoop.tool.plugins</name>
<value></value>
</property>
</configuration> In this case should I inject it manually from the configuration in Cloudera Manager web console as explained from @saranvisa? Thanks for the help
... View more
01-20-2018
01:54 PM
Thank you, I've solved creating a new user in MySQL which had also write permissions on HDFS.
... View more
01-19-2018
12:01 PM
Thank you very much for the for informations. Thus if I understand correctly I should use Cloudera Manager for everything... During the installation procedure with CM I've included the Teradata Connector parcel (which I need to conenct to Teradata database through Oozie/Sqoop). In CM web console I can see that the Teradata Connector parcel has been deployed and activated by the CM, but a Sqoop job using Teradata connector will fail (it does not find the connector). According to https://www.cloudera.com/documentation/other/connectors/teradata/1-x/topics/cctd_topic_3.html#concept_kv1_cj2_4n the connector should be copied in the Sqoop library inside the parcel, but the file is not present. Also, the documentations says that in order to use a Sqoop action inside Oozie that performs a Teradata import the following should be present in the oozie-site.xml file: <configuration>
<property>
<name>sqoop.connection.factories</name>
<value>com.cloudera.connector.teradata.TeradataManagerFactory</value>
</property>
<configuration> but in none of the oozie-site.xml files inside /run/cloudera-scm-agent/process/* folders this property is set, neither it seems to be present in the Cloudera Manager configuration for Oozie in the web console. In this situation what should I do?
... View more
01-19-2018
03:28 AM
Hello everyone! I've MySQL server running on the master node (10.0.0.4) with user root which I gave all privileges. MySQL connection through sqoop works correctly on the master with the following command: sqoop list-tables \
--driver com.mysql.jdbc.Driver \
--connect jdbc:mysql://10.0.0.4/employees \
--username root --password $pwd When I launch sqoop job to import a table from MySQL, I got the following error: java.sql.SQLException: Access denied for user 'root'@'worker01' (using password: YES) It's like the worker try to access to the mysql db with its hostname, instead of the hostname of the master where I started the job. The sqoop command is the following: sqoop import \
--driver com.mysql.jdbc.Driver \
--connect jdbc:mysql://10.0.0.4/employees \
--username root --password $pwd \
--table titles \
--split-by emp_no \
--warehouse-dir /user/cloudera/sqoop Does anyone know how to solve this issue?
... View more
Labels:
- Labels:
-
Apache Sqoop
01-19-2018
01:46 AM
Hi everyone!
I have a cluster installed by Cloudera Manager Path B with Parcel CDH 5.13.1 activated. I have some doubts about how configuration files work in this environment (I was used to edit manually them in the Quickstart virtual machine provided by Cloudera).
For example: I need to add/modify a property configuration in Oozie, thus i searched in the node where Oozie server is installed for the file "oozie-site.xml". I got the following results:
/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/etc/oozie/conf.dist/oozie-site.xml
/run/cloudera-scm-agent/process/49-oozie-OOZIE_SERVER/oozie-site.xml
/run/cloudera-scm-agent/process/48-oozie-OOZIE-SERVER-upload-sharelib/oozie-site.xml
/run/cloudera-scm-agent/process/43-oozie-OOZIE-SERVER-createdbtables/oozie-site.xml
The content is very different between the one in /opt and the others in /run (that seem to have the same content):
/opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/etc/oozie/conf.dist/oozie-site.xml
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<!--
Refer to the oozie-default.xml file for the complete list of
Oozie configuration properties and their default values.
-->
<!-- Proxyuser Configuration -->
<!--
<property>
<name>oozie.service.ProxyUserService.proxyuser.#USER#.hosts</name>
<value>*</value>
<description>
List of hosts the '#USER#' user is allowed to perform 'doAs'
operations.
The '#USER#' must be replaced with the username o the user who is
allowed to perform 'doAs' operations.
The value can be the '*' wildcard or a list of hostnames.
For multiple users copy this property and replace the user name
in the property name.
</description>
</property>
<property>
<name>oozie.service.ProxyUserService.proxyuser.#USER#.groups</name>
<value>*</value>
<description>
List of groups the '#USER#' user is allowed to impersonate users
from to perform 'doAs' operations.
The '#USER#' must be replaced with the username o the user who is
allowed to perform 'doAs' operations.
The value can be the '*' wildcard or a list of groups.
For multiple users copy this property and replace the user name
in the property name.
</description>
</property>
-->
<!-- Default proxyuser configuration for Hue -->
<property>
<name>oozie.service.ProxyUserService.proxyuser.hue.hosts</name>
<value>*</value>
</property>
<property>
<name>oozie.service.ProxyUserService.proxyuser.hue.groups</name>
<value>*</value>
</property>
</configuration>
/run/cloudera-scm-agent/process/49-oozie-OOZIE_SERVER/oozie-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<!--Autogenerated by Cloudera Manager-->
<configuration>
<property>
<name>oozie.service.JPAService.jdbc.driver</name>
<value>org.postgresql.Driver</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.username</name>
<value>oozie_oozie_server</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.password</name>
<value>********</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.url</name>
<value>jdbc:postgresql://master:7432/oozie_oozie_server</value>
</property>
<property>
<name>oozie.services.ext</name>
<value>org.apache.oozie.service.PartitionDependencyManagerService,org.apache.oozie.service.HCatAccessorService,org.apache.oozie.service.MetricsInstrumentationService</value>
</property>
<property>
<name>oozie.service.EventHandlerService.event.listeners</name>
<value></value>
</property>
<property>
<name>oozie.service.URIHandlerService.uri.handlers</name>
<value>org.apache.oozie.dependency.FSURIHandler,org.apache.oozie.dependency.HCatURIHandler</value>
</property>
<property>
<name>oozie.service.HCatAccessorService.hcat.configuration</name>
<value>hive-conf/hive-site.xml</value>
</property>
<property>
<name>oozie.service.ActionService.executor.ext.classes</name>
<value></value>
</property>
<property>
<name>oozie.service.SchemaService.wf.ext.schemas</name>
<value></value>
</property>
<property>
<name>oozie.email.smtp.host</name>
<value>localhost</value>
</property>
<property>
<name>oozie.email.smtp.port</name>
<value>25</value>
</property>
<property>
<name>oozie.email.from.address</name>
<value>oozie@localhost</value>
</property>
<property>
<name>oozie.email.smtp.auth</name>
<value>false</value>
</property>
<property>
<name>oozie.service.CoordMaterializeTriggerService.lookup.interval</name>
<value>300</value>
</property>
<property>
<name>oozie.service.coord.normal.default.timeout</name>
<value>120</value>
</property>
<property>
<name>oozie.service.WorkflowAppService.system.libpath</name>
<value>/user/oozie/share/lib</value>
</property>
<property>
<name>oozie.service.CallableQueueService.callable.concurrency</name>
<value>10</value>
</property>
<property>
<name>oozie.service.CallableQueueService.queue.size</name>
<value>10000</value>
</property>
<property>
<name>oozie.service.CallableQueueService.threads</name>
<value>50</value>
</property>
<property>
<name>oozie.service.PurgeService.older.than</name>
<value>30</value>
</property>
<property>
<name>oozie.service.PurgeService.coord.older.than</name>
<value>7</value>
</property>
<property>
<name>oozie.service.PurgeService.bundle.older.than</name>
<value>7</value>
</property>
<property>
<name>oozie.service.PurgeService.purge.old.coord.action</name>
<value>true</value>
</property>
<property>
<name>oozie.service.DBLiteWorkflowStoreService.status.metrics.collection.interval</name>
<value>1</value>
</property>
<property>
<name>oozie.service.ProxyUserService.proxyuser.hue.groups</name>
<value>*</value>
</property>
<property>
<name>oozie.service.ProxyUserService.proxyuser.hue.hosts</name>
<value>*</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.nameNode.whitelist</name>
<value>master:8020</value>
</property>
<property>
<name>oozie.actions.default.name-node</name>
<value>hdfs://master:8020</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.jobTracker.whitelist</name>
<value>master:8032</value>
</property>
<property>
<name>oozie.actions.default.job-tracker</name>
<value>master:8032</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=yarn-conf</value>
</property>
<property>
<name>oozie.service.HadoopAccessorService.action.configurations</name>
<value>*=yarn-conf</value>
</property>
<property>
<name>oozie.base.url</name>
<value>http://master:11000/oozie</value>
</property>
<property>
<name>oozie.service.GroupsService.hadoop.security.group.mapping</name>
<value>org.apache.hadoop.security.ShellBasedUnixGroupsMapping</value>
</property>
<property>
<name>oozie.service.CallbackService.base.url</name>
<value>http://master:11000/oozie/callback</value>
</property>
<property>
<name>oozie.service.SparkConfigurationService.spark.configurations</name>
<value>*=/etc/spark/conf</value>
</property>
<property>
<name>hadoop.security.credential.provider.path</name>
<value>localjceks://file//run/cloudera-scm-agent/process/49-oozie-OOZIE_SERVER/creds.localjceks</value>
</property>
</configuration>
Are the files in /run the ones used by the cluster? If I need to make an edit to configuration, can I do it manually (in this case which file in /run should I modify?) or I have to do it through the Cloudera Manager web console?
Thanks for any information!
... View more
Labels:
01-15-2018
01:34 AM
Hi, I've replaced all my mysql-connector*.jar files in the system to the last version: mysql-connector-java-5.1.45-bin.jar, but I got always the same warning. Also I found that if I not specify the driver --driver com.mysql.jdbc.Driver \ I got an error: 18/01/15 01:32:26 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
18/01/15 01:32:26 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
18/01/15 01:32:26 INFO teradata.TeradataManagerFactory: Loaded connector factory for 'Cloudera Connector Powered by Teradata' on version 1.6c5
18/01/15 01:32:26 ERROR tool.BaseSqoopTool: Got error creating database manager: java.io.IOException: No manager for connect string: jdbc:mysql://localhost/loudacre
at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:191)
at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:258)
at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:89)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:593)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236) Thus I was wondering if I need to specify other settings to proper configure the mysql connector...According to here https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cdh_ig_jdbc_driver_install.html I just need to put the .jar connector in the lib folder of Sqoop, but maybe there is something more to do? Thanks for the help
... View more
- « Previous
- Next »