Member since
12-21-2015
57
Posts
7
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2158 | 08-25-2016 09:31 AM |
01-23-2018
03:59 AM
We have seven different branches/lines of flow and at the terminal processors, we want to merge the output FlowFiles. These seven branches starts from a singe GenerateFlowFile, which in turn is triggered to run 3 times a day (1 AM, 7 AM and 12 PM). However, the execution time of the branches are different, so we need for each one of them to finish. At the merging part (funnel + MergeContent processor), we use the batch-time (trigger timestamp) as an attribute in order to bundle the 7 output FlowFiles. Things work out well for the most part, until one time, one of the branches has some mistake that it outputted two FlowFiles for same cycle instead of one. So for example, these FlowFiles were bundled together, while Branch 7 is not yet done: FlowFile1 FlowFile2 FlowFile3 FlowFile4 FlowFile5 FlowFile6 FlowFile6 Assuming we use a FlowFile attribute identifying the branch name, how can we bundle the same batch-time FlowFiles while guaranteeing that all branches are covered in the bundle?
... View more
Labels:
- Labels:
-
Apache NiFi
11-08-2017
09:22 AM
I am prototyping a flow where a JSON will be split to fragments then after some intermediate processing, will be routed to Wait and Notify processors (basically they diverge after some time after some number of processes). On the side of Wait, I want all fragments to be released almost simultaneously once Notify that the last fragment ( max(fragment.index) = fragment.count - 1 ) have arrived. My assumption is that fragment.index follows the processing sequence right after the SplitJSON (is this correct?). If the previous assumption is correct, let us assume that the intermediate processes after SplitJSON up to Notify preserves such ordering. How do I configure Wait and Notify for it to work this way? Is there a better solution than what I am thinking?
... View more
Labels:
- Labels:
-
Apache NiFi
08-18-2017
04:16 AM
I have an InvokeHTTP processor and the REST endpoint it calls requires PATCH method. However, when we send the FlowFile (we get the authentication part OK), we are getting: method PATCH must have a request body For the InvokeHTTP processor, I'm pretty sure that the we configured the Send Message Body to true (which is the default) and Content-Type as application/json. We enabled the endpoint to accept POST request, for the sake of testing and we've got the JSON payload accepted. So I wonder if there is an issue if we use PATCH.
... View more
Labels:
- Labels:
-
Apache NiFi
08-04-2017
02:09 AM
Thanks for your answer. We have NiFi below v1.2.0. Is there a workaround while we are not yet upgrading our cluster?
... View more
08-03-2017
08:40 AM
I am working on a workflow inside a Process Group. The flow inside the Process Group ends with an update on SalesForce API, but what this Process Group outputs is the original input. I will only send to the Output Port once I get a successful Response from the processor calling the SalesForce API. What should I do within the Process Group to to have this result?
... View more
Labels:
- Labels:
-
Apache NiFi
08-03-2017
03:57 AM
Yea, it looks like I'm using an older version that does not have that feature. The documentation I'm looking at is version 1.3.0.
... View more
07-25-2017
07:26 AM
According to JoltTransformJSON documentation, the JoltSpecification properties supports Expression Language. What I want is to use the FlowFile attribute containing the Jolt spec. Unfortunately, this is invalid as signified by the caution symbol. Is using attribute for Jolt specification property allowed?
... View more
Labels:
- Labels:
-
Apache NiFi
07-05-2017
06:13 AM
Is it possible to place a NiFi processors name or id to a flowfile that it outputs? For example a Processor InvokeHTTP I named as CallRESTEndpoint, the flowfile that would come out of it will have attribute, like origin:CallRESTEndpoint? Or if possible to create a chain of origin that will append the processor's name or id to this attribute. I need this in error / failure handling. If you can share a better idea in tracing errors / failure, that'd be a good idea. (But please answer the original question also, if there is one.)
... View more
- Tags:
- nifi-processor
Labels:
- Labels:
-
Apache NiFi
06-21-2017
07:48 AM
We have a custom query that we wish to put inside the ExecuteSQL processor, but this one proves a bit tricky: SELECT * FROM mytable AS OF TIMESTAMP TO_TIMESTAMP('${now():format('yyyy-MM-dd HH:mm:ss')}', 'YYYY-MM-DD HH24:MI:SS'); The processor is CRON driven where we run it every 15 minutes. The expression language inside the first single quotes is NiFi's expression language to get the current time in the format specified. Note that the expression language: ${now():format('yyyy-MM-dd HH:mm:ss')} itself uses single quotes. I tried substituting double quotes for the Oracle query, but Oracle won't accept the syntax as valid. So how do we substitute the intended date-time string inside the first single-quotes?
... View more
Labels:
- Labels:
-
Apache NiFi
06-19-2017
11:30 AM
In NiFi, is there an existing FlowFile attribute that can tell when it exited (or when it was outputted by) a certain processor? I am trying to measure end-to-end processing time of a workflow.
... View more
Labels:
- Labels:
-
Apache NiFi
06-07-2017
03:09 AM
1 Kudo
We have an SFDC REST endpoint (let's call this UpdateSFDC) where we stream and post database changes, in JSON format, to reflect it to SalesForce object store. Before we get to that endpoint, we need to Login to SalesForce first to this endpoint: https://test.salesforce.com/services/oauth2/token So we use InvokeHTTP (let's call this Login2SFDC) and appends the parameters to the Remote URL field in the processor: https://test.salesforce.com/services/oauth2/token?grant_type=password&client_id=<client_id>&client_secret=<client_secret>&username=<username>&password=<password + key> This will respond with a token which we need to extract and add as an attribute in every JSON format flowfile (let's call this CreateSignedRequest). Here's a NiFi schematic: The Login2SFDC is configure this way: However, this is quite inefficient because for every JSON flowfile, we need to login to SalesForce everytime, which will add overhead, not to mention the risk that Login2SFDC REST endpoint will eventually throw some sort of a max limit request reached. Rather, we want to login once and as long as the token is valid (not yet expired), then we will just add the token as an attribute to the JSON flowfile and go to UpdateSFDC REST endpoint straight (via CreateSignedRequest). So I will convert the Login2SFDC processor from InvokeHTTP to ExecuteScript. I have 3 questions for this problem: 1. How do I this in ECMAScript and/or Python? When I use their respective native HTTP client functionalities, they do not seem to work. Should I be importing and using Java libraries (like HttpClient) instead in which case, the scripting language is simply a wrapper to Java libraries? 2. I am successful in getting a response from Login2SFDC REST endpoint when in NiFi. I tried writing a corresponding Java code (using same exact endpoint and credentials) using HttpClient library: CloseableHttpClient client = HttpClients.createDefault();
String urlStr = "https://test.salesforce.com/services/oauth2/token?grant_type=password&client_id=<client_id>&client_secret=<client_secret>&username=<username>&password=<password + key>";
HttpPost httpPost = new HttpPost(urlStr);
String jsonPayload = "{\"qty\":100,\"name\":\"iPad 4\"}";
StringEntity entity = new StringEntity(jsonPayload); httpPost.setEntity(entity);
httpPost.setHeader("Accept", "application/json");
httpPost.setHeader("Content-type", "application/json");
CloseableHttpResponse response = client.execute(httpPost);
System.out.println(response.getStatusLine().getStatusCode());
InputStream is = response.getEntity().getContent();
String responseStr = IOUtils.toString(is, "UTF-8");
System.out.println(responseStr);
client.close(); but I get this response: Status Code 400 {"error":"invalid_grant","error_description":"authentication failure"} Am I missing something in the Java code? 3. Is there a better NiFi flow design for this kind of problem? Hi @Matt Burgess, can you help?
... View more
Labels:
- Labels:
-
Apache NiFi
05-26-2017
05:50 AM
We have a VM with JDK 7 and a number of things depend on it. We are installing NiFi v1.2 but this one is compatible with JDK 8 only. We cannot remove JDK 7 because we cannot risk to break the dependency. Is it possible to install JDK 8 and point NiFi to use this, without changing the JAVA_HOME variable pointing to JDK 7?
... View more
Labels:
- Labels:
-
Apache NiFi
05-25-2017
09:46 AM
We have an Oracle DB and we run nightly batch job but when that job would end varies. Now we have a NiFi workflow getting data from this Oracle DB but should not be during the batch job run. How do we turn off/on the entry point processor (say ExecuteSQL) whenever the Oracle Batch job is running/has stopped? We thought of using Oracle DB trigger and then invoke a HTTP request within that trigger when the job would end, but is there a processor in NiFi where you can put up a simple web service / REST endpoint and then use that to trigger or turn on the ExecuteSQL processor?
... View more
Labels:
- Labels:
-
Apache NiFi
04-25-2017
08:37 AM
We have a system composed of many databases and tables and we want to use NiFi to query these tables based on our requirements. Since NiFi's QueryDatabaseTable processor is statically linked to a single table, what we intend to do is to dynamically generate many processors of this kind to match the number of our tables in our different systems. Is this possible using ExecuteScript processor (or anything similar)?
... View more
Labels:
- Labels:
-
Apache NiFi
04-18-2017
09:03 AM
We have two tables in Hive, the first one, let's call it Table A, is where the raw data is being dumped. The data in Table B is derived from Table A after some HiveQL transformation. (I understand there are no stored procedures in Hive). When new data will be inserted on Table A, how do we automatically trigger the process of doing the same transformation and appending the result on Table B?
... View more
Labels:
- Labels:
-
Apache Hive
03-21-2017
01:21 PM
Thanks for the tip Matt. I am now in the part where I have two InvokeHTTP processors, the first for login or getting the access token (let's call this SFDCLogin), and the second is where the SFDC REST API is configured for posting changes to SalesForce's ObjectStore (let's call this SFDCUpdate). I having a hard time doing the plumbing on this. The idea is, the FlowFiles (wherever they come from, assuming its GetFile processor) would move to SFDCUpdate and then if rebuffed due to missing or expired access token, will go to SFDCLogin processor and do whatever it needs to do and then go to SFDCUpdate. Note that I only need to pass through SFDCLogin once, once authenticated; not multiple times. Any custom logic, attribute, property, or expression language to make this work? I also tried using the ExecuteScript processor using ECMAScript or Python, and do the REST API calls there on the login part. However, I'm hitting a roadblock on XMLHttpRequest (in ECMAScript) or requests (in Python) as it seems I need to point to their libraries. Should I be using a Java library (jar of HttpComponents perhaps) instead as these scripting languages are meant to wrappers only? Or can I point to their respective libraries natively?
... View more
03-16-2017
08:34 AM
Do you have this issue resolved? Can you tell me your solution if you manage to solve it?
... View more
03-15-2017
06:27 AM
We used the InvokeHTTP processor in NiFi on Salesforce REST API (POST action, to effect changes in customer data). Salesforce requires authentication in using of its REST endpoints. Is the InvokeHTTP the right processor for this requirement? How do we maintain authenticated connection / session to REST API?
... View more
Labels:
- Labels:
-
Apache NiFi
03-14-2017
08:53 AM
1 Kudo
We have a number of databases, mostly in Oracle and MS SQL Server, which were designed prior without timestamp fields; as a result, Sqoop cannot be used for incremental batch load. In addition, some real-time use-case requirements forced us to look into streaming solutions for Change Data Capture. We POC'd a database for each. To interface with the databases and get the changed data (or delta), for Oracle, we used XStream (11g and before) and for MS SQL Server, we enabled CDC tables. We wrote custom Java code and transform the delta units (using GSON) to JSON String and use that as payload to Kafka Producer, which eventually will be consumed by/into our HDP cluster as changes to corresponding Hive database/tables. To make our solution maintainable, we are switching to NiFi, but as we are new to this technology, we are still in research stage. Can anyone propose a NiFi version solution something similar to what we've done above (interface with CDC mechanism of Oracle, then delta units to JSON, then produce in Kafka, then update Hive tables)? What are the processors to be used?
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
02-01-2017
07:50 AM
It works now when I use "jdbc:hive2://<my-vm-hostname>:8443/;ssl=true...". However, when I use 'localhost', '127.0.0.1' and even <my-vm-ip>, I get this message: 17/02/01 15:38:51 [main]: ERROR jdbc.HiveConnection: Error opening session
org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLPeerUnverifiedException: Host name '<my-vm-ip>' does not match the certificate subject provided by the peer (CN=<my-vm-hostname>, OU=Test, O=Hadoop, L=Test, ST=Test, C=US)
at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
...
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Host name '<my-vm-ip>' does not match the certificate subject provided by the peer (CN=<my-vm-hostname>, OU=Test, O=Hadoop, L=Test, ST=Test, C=US)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:465)
...
... 30 more
Error: Could not establish connection to jdbc:hive2://<my-vm-ip>:8443/;ssl=true;sslTrustStore=/var/lib/knox/data-2.5.0.0-1245/security/keystores/gateway.jks;trustStorePassword=knox_123;transportMode=http;httpPath=gateway/default/hive: javax.net.ssl.SSLPeerUnverifiedException: Host name '<my-vm-ip>' does not match the certificate subject provided by the peer (CN=<my-vm-hostname>, OU=Test, O=Hadoop, L=Test, ST=Test, C=US) (state=08S01,code=0)
17/02/01 15:38:51 [main]: ERROR jdbc.HiveConnection: Error opening session
org.apache.thrift.transport.TTransportException: javax.net.ssl.SSLPeerUnverifiedException: Host name '<my-vm-ip>' does not match the certificate subject provided by the peer (CN=<my-vm-hostname>, OU=Test, O=Hadoop, L=Test, ST=Test, C=US)
at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
...
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Host name '<my-vm-ip>' does not match the certificate subject provided by the peer (CN=<my-vm-hostname>, OU=Test, O=Hadoop, L=Test, ST=Test, C=US)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:465)
...
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Host name '<my-vm-ip>' does not match the certificate subject provided by the peer (CN=<my-vm-hostname>, OU=Test, O=Hadoop, L=Test, ST=Test, C=US)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:465)
...
at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
... 24 more
Error: Could not establish connection to jdbc:hive2://<my-vm-ip>:8443/;ssl=true;sslTrustStore=/var/lib/knox/data-2.5.0.0-1245/security/keystores/gateway.jks;trustStorePassword=knox_123;transportMode=http;httpPath=gateway/default/hive: javax.net.ssl.SSLPeerUnverifiedException: Host name '<my-vm-ip>' does not match the certificate subject provided by the peer (CN=<my-vm-hostname>, OU=Test, O=Hadoop, L=Test, ST=Test, C=US) (state=08S01,code=0)
Is there anyway to connect using <my-vm-ip> in order for a remote machine to access it? The <my-vm-hostname> is not network visible, it's just locally known within the VM's /etc/hosts. Thanks for all your help. Appreciate it.
... View more
01-30-2017
11:36 AM
Alright the HDFS part seems to be working, but I have to use guest:guest-password I thought it is something I can add to Unix and Ranger as users (which I named guest as well, with a different password; which is why it won't work). If I understand correctly, this guest should be a Knox specific username, right? But how about Hive? It still won't work, even though I used guest:guest-password. I am getting the same result as the original. Thanks for your help.
... View more
01-30-2017
11:07 AM
I already pasted the default.xml above. I tried using the admin:admin-password (earlier I didn't, as I was confused) : curl -iku admin:admin-password -X GET 'https://localhost:8443/gateway/admin/api/v1/version' and I now get HTTP/1.1 200 OK
Date: Mon, 30 Jan 2017 11:04:24 GMT
Set-Cookie: JSESSIONID=1qfuugnqxqf1hf4exm5kpvkmd;Path=/gateway/admin;Secure;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/admin; Max-Age=0; Expires=Sun, 29-Jan-2017 11:04:24 GMT
Content-Type: application/xml
Content-Length: 170
Server: Jetty(9.2.15.v20160210)
<?xml version="1.0" encoding="UTF-8"?>
<ServerVersion>
<version>0.9.0.2.5.0.0-1245</version>
<hash>09990487b383298f8e1c9e72dceb0a8e3ff33d17</hash>
</ServerVersion>
... View more
01-30-2017
10:55 AM
Then there is something wrong with Knox tutorial in Hortonworks. The knox_sample.xml file does not exist even in the Sandbox VM and using touch will simply create an empty file. What would be the admin username and password here? Is it Ranger's or Knox's admin? Here's the output of ps aux | grep ldap : knox 562103 0.0 0.2 5400580 33376 ? Sl Jan25 2:59 /usr/lib/jvm/java-1.7.0-oracle/bin/java -jar /usr/hdp/current/knox-server/bin/ldap.jar /usr/hdp/current/knox-server/conf
root 675001 0.0 0.0 103320 868 pts/0 S+ 18:32 0:00 grep ldap
And here's the default.xml : <topology>
<gateway>
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<name>sessionTimeout</name>
<value>30</value>
</param>
<param>
<name>main.ldapRealm</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
</param>
<param>
<name>main.ldapRealm.userDnTemplate</name>
<value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.url</name>
<value>ldap://<my-vm-hostname>.com:33389</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
<value>simple</value>
</param>
<param>
<name>urls./**</name>
<value>authcBasic</value>
</param>
</provider>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
<provider>
<role>authorization</role>
<name>XASecurePDPKnox</name>
<enabled>true</enabled>
</provider>
</gateway>
<service>
<role>NAMENODE</role>
<url>hdfs://<my-vm-hostname>:8020</url>
</service>
<service>
<role>JOBTRACKER</role>
<url>rpc://<my-vm-hostname>:8050</url>
</service>
<service>
<role>WEBHDFS</role>
<url>http://<my-vm-hostname>:50070/webhdfs</url>
</service>
<service>
<role>WEBHCAT</role>
<url>http://<my-vm-hostname>:50111/templeton</url>
</service>
<service>
<role>OOZIE</role>
<url>http://<my-vm-hostname>:11000/oozie</url>
</service>
<service>
<role>WEBHBASE</role>
<url>http://<my-vm-hostname>:8080</url>
</service>
<service>
<role>HIVE</role>
<url>http://<my-vm-hostname>:10001/cliservice</url>
</service>
<service>
<role>RESOURCEMANAGER</role>
<url>http://<my-vm-hostname>:8088/ws</url>
</service>
</topology>
... View more
01-30-2017
10:04 AM
The knox_sample topology seems to be empty based from the tutorial. The guide just instructs the user to use touch command. The result of the command you gave is: HTTP/1.1 401 Unauthorized
Date: Mon, 30 Jan 2017 09:56:48 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/admin; Max-Age=0; Expires=Sun, 29-Jan-2017 09:56:48 GMT
WWW-Authenticate: BASIC realm="application"
Content-Length: 0
Server: Jetty(9.2.15.v20160210)
... View more
01-30-2017
08:38 AM
I've been following the Hortonworks tutorial on all things
related to Knox.
tutorial-420: http://hortonworks.com/hadoop-tutorial/securing-hadoop-infrastructure-apache-knox/ tutorial-560: http://hortonworks.com/hadoop-tutorial/secure-jdbc-odbc-clients-access-hiveserver2-using-apache-knox/ Our goal is use Knox as a gateway or single point of entry for
microservices that intend to connect to the cluster using REST API calls. However, my Hadoop environment is not the HDP 2.5 Sandbox but an
HDP 2.5 stack built through Ambari on a single node Azure VM, so the
configurations may differ and that should partially explain why the tutorials
may not work. (This is a POC for a multinode cluster build) I’m
testing the WebHDFS and Hive part. For Ranger, I created a temporary guest
account that has global access to HDFS and Hive, as well as copying the
GlobalKnoxAllow configured in the Sandbox VM (I'll take care of the more fine-grained Ranger ACL later). We didn't setup this security in LDAP mode though, just the plain Unix ACL. I also created a sample database
named microservice, which I can connect to and query via beeline. From tutorial-420, Step 1: I started the Start Demo LDAP Step 2: touch /usr/hdp/current/knox-server/conf/topologies/knox_sample.xml
curl -iku guest:<guestpw> -X GET 'http://<my-vm-ip>:50070/webhdfs/v1/?op=LISTSTATUS' This is what I get: HTTP/1.1 404 Not Found
Cache-Control: no-cache
Expires: Mon, 30 Jan 2017 07:23:17 GMT
Date: Mon, 30 Jan 2017 07:23:17 GMT
Pragma: no-cache
Expires: Mon, 30 Jan 2017 07:23:17 GMT
Date: Mon, 30 Jan 2017 07:23:17 GMT
Pragma: no-cache
Content-Type: application/json
X-FRAME-OPTIONS: SAMEORIGIN
Transfer-Encoding: chunked
Server: Jetty(6.1.26.hwx)
{"RemoteException":{"exception":"FileNotFoundException","javaClassName":"java.io.FileNotFoundException","message":"File /webhdfs/v1 does not exist."}}
Step 3: curl -iku guest:<guestpw> -X GET 'https://<my-vm-ip>:8443/gateway/default/webhdfs/v1/?op=LISTSTATUS' This is what I get: HTTP/1.1 401 Unauthorized
Date: Mon, 30 Jan 2017 07:25:51 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Sun, 29-Jan-2017 07:25:51 GMT
WWW-Authenticate: BASIC realm="application"
Content-Length: 0
Server: Jetty(9.2.15.v20160210)
I jumped to tutorial-560: Step 1: Knox is started Step 2: In Ambari, hive.server2.transport.mode is changed to http from binary and Hive
Server 2 is restarted. Step 3: SSH on my Azure VM Step 4: Connect to Hive Server 2
using beeline via Knox [root@poc2 hive]# beeline
!connect jdbc:hive2:// <my-vm-ip>:8443/microservice;ssl=true;sslTrustStore=/var/lib/knox/data-2.5.0.0-1245/security/keystores/gateway.jks;trustStorePassword=knox?hive.server2.transport.mode=http;hive.server2.thrift.http.path=gateway/default/hive This is the result: Connecting to jdbc:hive2://<my-vm-ip>:8443/microservice;ssl=true;sslTrustStore=/var/lib/knox/data-2.5.0.0-1245/security/keystores/gateway.jks;trustStorePassword=knox?hive.server2.transport.mode=http;hive.server2.thrift.http.path=gateway/default/hive
Enter username for jdbc:hive2://... : guest
Enter password for jdbc:hive2://... : ********
17/01/30 15:28:21 [main]: WARN jdbc.Utils: ***** JDBC param deprecation *****
17/01/30 15:28:21 [main]: WARN jdbc.Utils: The use of hive.server2.transport.mode is deprecated.
17/01/30 15:28:21 [main]: WARN jdbc.Utils: Please use transportMode like so: jdbc:hive2://<host>:<port>/dbName;transportMode=<transport_mode_value>
17/01/30 15:28:21 [main]: WARN jdbc.Utils: ***** JDBC param deprecation *****
17/01/30 15:28:21 [main]: WARN jdbc.Utils: The use of hive.server2.thrift.http.path is deprecated.
17/01/30 15:28:21 [main]: WARN jdbc.Utils: Please use httpPath like so: jdbc:hive2://<host>:<port>/dbName;httpPath=<http_path_value>
Error: Could not create an https connection to jdbc:hive2://<my-vm-ip>:8443/microservice;ssl=true;sslTrustStore=/var/lib/knox/data-2.5.0.0-1245/security/keystores/gateway.jks;trustStorePassword=knox?hive.server2.transport.mode=http;hive.server2.thrift.http.path=gateway/default/hive. Keystore was tampered with, or password was incorrect (state=08S01,code=0)
0: jdbc:hive2://<my-vm-ip>:8443/microservi (closed)>
The same thing I get even if replace
hive.server2.thrift.http.path=cliserver (based from hive-site.xml). In hive-site.xml, ssl=false, so I tried substituting that on the JDBC connection URL: [root@poc2 hive]# beeline
!connect jdbc:hive2:// <my-vm-ip>:8443/microservice;ssl=false;sslTrustStore=/var/lib/knox/data-2.5.0.0-1245/security/keystores/gateway.jks;trustStorePassword=knox?hive.server2.transport.mode=http;hive.server2.thrift.http.path=gateway/default/hive This is the result: Connecting to jdbc:hive2://<my-vm-ip>:8443/microservice;ssl=false;sslTrustStore=/var/lib/knox/data-2.5.0.0-1245/security/keystores/gateway.jks;trustStorePassword=knox?hive.server2.transport.mode=http;hive.server2.thrift.http.path=gateway/default/hive
Enter username for jdbc:hive2://... : guest
Enter password for jdbc:hive2://... : ********
17/01/30 15:33:29 [main]: WARN jdbc.Utils: ***** JDBC param deprecation *****
17/01/30 15:33:29 [main]: WARN jdbc.Utils: The use of hive.server2.transport.mode is deprecated.
17/01/30 15:33:29 [main]: WARN jdbc.Utils: Please use transportMode like so: jdbc:hive2://<host>:<port>/dbName;transportMode=<transport_mode_value>
17/01/30 15:33:29 [main]: WARN jdbc.Utils: ***** JDBC param deprecation *****
17/01/30 15:33:29 [main]: WARN jdbc.Utils: The use of hive.server2.thrift.http.path is deprecated.
17/01/30 15:33:29 [main]: WARN jdbc.Utils: Please use httpPath like so: jdbc:hive2://<host>:<port>/dbName;httpPath=<http_path_value>
17/01/30 15:34:32 [main]: ERROR jdbc.HiveConnection: Error opening session
org.apache.thrift.transport.TTransportException: org.apache.http.conn.ConnectTimeoutException: Connect to <my-vm-ip>:8443 [/<my-vm-ip>] failed: Connection timed out
at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:297)
at org.apache.thrift.transport.THttpClient.flush(THttpClient.java:313)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:73)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
at org.apache.hive.service.cli.thrift.TCLIService$Client.send_OpenSession(TCLIService.java:154)
at org.apache.hive.service.cli.thrift.TCLIService$Client.OpenSession(TCLIService.java:146)
at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:552)
at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:170)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at java.sql.DriverManager.getConnection(DriverManager.java:571)
at java.sql.DriverManager.getConnection(DriverManager.java:187)
at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:146)
at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:211)
at org.apache.hive.beeline.Commands.connect(Commands.java:1190)
at org.apache.hive.beeline.Commands.connect(Commands.java:1086)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:52)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:989)
at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:832)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:790)
at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:490)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:473)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to <my-vm-ip>:8443 [/<my-vm-ip>] failed: Connection timed out
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.execchain.ServiceUnavailableRetryExec.execute(ServiceUnavailableRetryExec.java:84)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:117)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.apache.thrift.transport.THttpClient.flushUsingHttpClient(THttpClient.java:251)
... 30 more
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
... 41 more
Error: Could not establish connection to jdbc:hive2://<my-vm-ip>:8443/microservice;ssl=false;sslTrustStore=/var/lib/knox/data-2.5.0.0-1245/security/keystores/gateway.jks;trustStorePassword=knox?hive.server2.transport.mode=http;hive.server2.thrift.http.path=gateway/default/hive: org.apache.http.conn.ConnectTimeoutException: Connect to <my-vm-ip>:8443 [/<my-vm-ip>] failed: Connection timed out (state=08S01,code=0)
0: jdbc:hive2://<my-vm-ip>:8443/microservi (closed)>
Again I tried replacing hive.server2.thrift.http.path=cliserver and I get the same result. Does anyone here able to configure to Knox correctly and working?
... View more
Labels:
01-26-2017
07:59 AM
I've noticed that this version of HDP 2.5 Sandbox uses Docker container. When the VM launches I can login as root (pw: hadoop) on the default console. However, when I use ssh on other terminal, like Git Bash and Putty (which I am more comfortable using), I constantly get the message 'Permission Denied' even though I am keying-in same password. ssh root@<my-ip> -p 2222 Another thing I noticed is the hadoop commands (hive, pig, oozie, etc.) are missing and the /usr/hdp path are also not in the Linux path. I really need to get into the terminal.
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
12-06-2016
08:35 AM
We're planning to add Apache HAWQ / Pivotal HDB on our Hortonworks stack. Can this be secured with Ranger? If not, how do we have ACL or security audit in HAWQ / HDB ?
... View more
Labels:
- Labels:
-
Apache Ranger
12-01-2016
05:29 AM
Is there a single command where you can display all installed components / services in an HDP stack installation plus their respective versions (not the HDP version, but the corresponding Apache version)? I know that this info is in the release notes, in Hortonworks website, but I'm always lost looking for the specific link. Neither I can find this in Ambari.
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
11-22-2016
10:27 AM
I have set a Ranger policy enabling a certain newuser to read/write/execute only on his own home directory in HDFS, say /user/<newuser>. While the policy certainly works on his own path, however, I do not want newuser to be able to read directories and files outside its own, which still happens when I do: hadoop fs -ls / Or on some other directories. Same thing happens when newuser is logged in in Hue. How do I do this in Ranger?
... View more
Labels:
- Labels:
-
Apache Ranger