Member since
10-24-2017
17
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1095 | 02-06-2022 02:30 AM |
01-09-2024
04:55 PM
We're currently looking at upgrading NiFi from 1.11 to 1.22.0. During the past NiFi upgrade we encountered a bunch of breaking changes in processor behavior and found that there was no way to safely upgrade without going through and checking the behavior of every single flow in the new version. We have dozens of flows in production so this was pretty cumbersome. To possibly make this upgrade a bit easier, I was considering including the old NAR files along with the new NAR files so that we would have the option of using old 1.11 processor versions for our existing flows, avoiding any problems coming from new behavior in the 1.22.0 versions of processors. NiFi doesn't complain when you do this and even seems to have some built-in features to support it. But I eventually encountered errors that looked like they were coming from incompatible Java classes. One error came from the ExecuteSQL processor complaining about SNAPPY compression for its avro file output, something about the class org.xerial.snappy. We also saw that the UpdateAttribute processor was losing the configurations in the "Advanced" UI. I certainly don't understand much about how class loaders are supposed to work, but I thought that NAR files were supposed to be isolated in some way so that you wouldn't have problems with different NARs using incompatible versions of the same Java class. I did read that you are supposed to put extra NAR files in a different directory other than the default lib directory, but that didn't seem to help. Does anyone have experience doing this successfully?
... View more
Labels:
- Labels:
-
Apache NiFi
02-06-2022
02:30 AM
This was a result of a bug in my code and not anything to do with Hive itself - please ignore.
... View more
02-12-2021
01:44 AM
1 Kudo
Looks like the files are available here: https://repo.hortonworks.com/content/repositories/releases/org/apache/nifi/nifi-hive-nar/
... View more
08-10-2018
08:31 AM
Some more digging yielded that it is safe to apply this setting globally, and that the issue has been resolvedin hive 2
... View more
12-18-2018
03:46 PM
If you already have a TGT available in cache, then below code worked for me import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.security.UserGroupInformation;
class HiveBeelineTest {
public static void main(String args[]) throws ClassNotFoundException, SQLException {
Class.forName("org.apache.hive.jdbc.HiveDriver");
Configuration conf = new Configuration();
conf.set("hadoop.security.authentication", "Kerberos");
UserGroupInformation.setConfiguration(conf);
Connection con = DriverManager.getConnection(
"jdbc:hive2://mynode.foobar.com:10000/default;principal=hive/mynode.foobar.com@FOOBAR.COM")
Statement sqlstatement = con.createStatement();
ResultSet result = sqlstatement.executeQuery("select test_column from test_table");
while (result.next()) {
System.out.println(result.getString(1));
}
}
}
... View more
10-24-2017
08:34 PM
@Karl Fredrickson, what i mean to say at 4:11 is file creation time stamp in the directory For Example:- bash# hdfs dfs -ls /user/yashu/test_fac/
Found 1 items
-rwxr-xr-x 3 hdfs hdfs 8 2017-10-24 04:11 /user/yashu/test_fac/000000_0 in this example 000000_0 file got created at 2017-10-24 04:11(time stamp). But the processor runs at 4:20 that means above 000000_0 file is going to listed in 4:20 run. if the last modified date is earlier than 4:00 but someone put the files at 4:11? then ListSFTP won't create flow files because it will only pulls new files that got created after the state value.
... View more