Member since
05-31-2016
12
Posts
4
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3713 | 03-21-2018 04:56 PM | |
3350 | 10-02-2017 07:37 PM | |
5768 | 02-03-2017 02:40 PM |
05-07-2020
06:35 AM
Hi, I did restart nifi , but the problem still persists. The next release of nifi will add a new DBCP connection Pool: Hadoop DBCP connection pool: https://issues.apache.org/jira/browse/NIFI-7257 It will, hopefully, solve the issue. For my part, I implemented a specific connector which modifies the classpath: https://github.com/dams666/nifi-dbcp-connectionpool In short: public static final PropertyDescriptor DB_DRIVER_LOCATION = new PropertyDescriptor.Builder() .name("database-driver-locations") .displayName("Database Driver Location(s)") .description("Comma-separated list of files/folders and/or URLs containing the driver JAR and its dependencies (if any). For example '/var/tmp/mariadb-java-client-1.1.7.jar'") .defaultValue(null) .required(false) .addValidator(StandardValidators.createListValidator(true, true, StandardValidators.createURLorFileValidator())) .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY) .dynamicallyModifiesClasspath(true) .build(); ... dataSource = new BasicDataSource(); dataSource.setDriverClassName(drv); dataSource.setDriverClassLoader(this.getClass().getClassLoader()); But I still got same error message: PutSQL[id=94d192a9-fd1d-3c59-99be-d848f8902968] Failed to process session due to java.sql.SQLException: Cannot create PoolableConnectionFactory (ERROR 103 (08004): Unable to establish connection.): org.apache.nifi.processor.exception.ProcessException: java.sql.SQLException: Cannot create PoolableConnectionFactory (ERROR 103 (08004): Unable to establish connection.) My setup: Database Connection URL : jdbc:phoenix:zk4-xxx.ax.internal.cloudapp.net,zk5-xxx.ax.internal.cloudapp.net,zk6-xxx.ax.internal.cloudapp.net:2181:/hbase-unsecure Database Driver Class Name: org.apache.phoenix.jdbc.PhoenixDriver Damien
... View more
03-21-2018
04:56 PM
All, I've resolved the issue - in the file "src/main/resources/META-INF/services/org.apache.nifi.controller.ControllerService" I previously had it as "org.apache.nifi.phoenix.service.PhoenixDBCPService" and after changing it to "org.apache.nifi.phoenix.service.PhoenixConnectionPool" NiFi was able to startup and I was able to use/access my custom processors.
... View more
03-13-2017
02:11 PM
1 Kudo
@Paras Mehta Unless the SQL statement in the content of each of your FlowFiles is identical, a new transaction will be created for each FlowFile rather then multiple FlowFiles in a single transaction. This is because of how the putSQL is designed to do batch inserts. The SQL command may use the ? to escape parameters. In this case, the
parameters to use must exist as FlowFile attributes with the naming
convention sql.args.N.type and sql.args.N.value, where N is a positive
integer. The sql.args.N.type is expected to be a number indicating the
JDBC Type. The content of the FlowFile is expected to be in UTF-8
format. So consider the case where you are inserting to a common DB table. In order to have every insert statement be exactly identical you will need to create attributes sql.args.N.type and sql.args.N.value for each column. sql.args.1.type ---> 12 sql.args.1.value ---> bob sql.args.2.type ---> 12 sql.args.2.value ---> smith sql.args.3.type --> 12 sql.args.3.value --> SME And so on.... You can use UpdateAttribute and maybe ExtractText processors to set these attributes on your FlowFiles. The you can use a ReplaceText processor to replace the content of your FlowFile with a common INSERT statement like below: INSERT INTO mydb("column1","column2","column3") VALUES(?,?,?) Now every FlowFile will have identical content and batch inserts will work in a single transaction. The "?" are replaced by the values from the attributes sql.args.1.value. sql.args.2.value, sql.args.3.value, and so on.... Thanks, Matt
... View more
05-31-2016
04:39 PM
Yes - you can run the scripts however you like. Keep in mind there may be a task that requires you to use Tez. But if nothing is mentioned specifically in the task instructions, then you can run your Pig and Hive scripts using Tez or not. I would recommend using Tez though for every task when applicable. Like you said - why waste precious exam time. I will take offense to the "weak machine" comment. The instances we use are extremely large for the small amount of processing that happens on them - c3.4xlarge EC2 instances. The datasets on the exam are purposely small so that time is not wasted in processing a lot of data. The longest queries you run will take less than 90 seconds, and that is w/out using Tez.
... View more