Member since
07-15-2016
30
Posts
8
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1131 | 08-10-2016 12:52 AM | |
889 | 07-19-2016 03:27 AM |
12-21-2016
06:08 AM
Thank you. I fix this problem by follow this step: http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.0/bk_ambari_views_guide/content/_configuring_your_cluster_for_files_view.html
... View more
12-21-2016
02:02 AM
when i use the files view(no kerberos). The page is blank. And the ambari-server logs as bellow
ps:the other views are normal.
Dec 21, 2016 9:57:51 AM com.sun.jersey.spi.container.ContainerResponse logException
SEVERE: Mapped exception to response: 500 (Internal Server Error)
org.apache.ambari.view.filebrowser.utils.ServiceFormattedException
at org.apache.ambari.view.filebrowser.FileOperationService.listdir(FileOperationService.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:137)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:540)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:715)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.apache.ambari.server.security.authorization.AmbariAuthorizationFilter.doFilter(AmbariAuthorizationFilter.java:196)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at org.apache.ambari.server.api.MethodOverrideFilter.doFilter(MethodOverrideFilter.java:72)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at org.apache.ambari.server.api.AmbariPersistFilter.doFilter(AmbariPersistFilter.java:47)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:109)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at org.apache.ambari.server.security.AbstractSecurityHeaderFilter.doFilter(AbstractSecurityHeaderFilter.java:109)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82)
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:216)
at org.apache.ambari.server.controller.AmbariHandlerList.processHandlers(AmbariHandlerList.java:205)
at org.apache.ambari.server.controller.AmbariHandlerList.handle(AmbariHandlerList.java:152)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:745)
... View more
Labels:
- Labels:
-
Apache Ambari
09-09-2016
06:16 AM
Thanks very much!
... View more
09-08-2016
12:43 AM
1 Kudo
Can nifi promise each of the flowfiles can be processed correctly, no missing and no duplicate ? How the nifi realize it. Thanks for your reply.
... View more
Labels:
- Labels:
-
Apache NiFi
09-08-2016
12:27 AM
Thanks you,I understand. It seems the document need to be updated.
... View more
09-07-2016
02:48 AM
2 Kudos
In document Users can determine which node is currently elected as the Primary Node by looking at the Cluster Management page of the User Interface. But there is no way to do this in nifi 1.0.0.(but nifi 0.7.0 can do it)
... View more
Labels:
- Labels:
-
Apache NiFi
09-06-2016
07:35 AM
In cluster mode of NiFi 1.0.0. I try to distribute the flowfiles from primary node to cluster nodes by using Remote Process Group. The property 'Transport Protocol' of Remote Process Group is 'RAW'. I get the error message: 'Remote instance of NiFi is not configured to allow RAW Socket site-to-site communications' How to resolve it? Thanks for your help. David
... View more
Labels:
- Labels:
-
Apache NiFi
09-06-2016
01:26 AM
Thank you!
... View more
09-06-2016
01:18 AM
Thanks for your advice. I use GenerateTableFetch+ExecuteSQL to achieve my target.
... View more
09-05-2016
04:01 AM
1 Kudo
I ask a question in another page https://community.hortonworks.com/questions/53941/nifi-querydatabasetables-properties-fetch-size-is.html#answer-form But i am still puzzled. In nifi-1.x, 'Max Rows Per Flow File' can control the max number in a flowfile. But It seems that QueryDatabaseTable still query all the rows into the memory, the split the records into flowfiles. If the table is very hug. The nifi is easy to break down when start QueryDatabaseTable.(database is mysql) There is no rows limit in select query sql, like select * from table limit 0,100; -- mysql or select * from table where rownum<100;--oracle The Bulletin advice me to modify 'max_allowed_packet' in my.cnf. But it is useless. some of the errors: Unable to execute SQL select query SELECT * FROM employee due to java.sql.SQLException: Unknown character set index for field '25700' received from server.: java.sql.SQLException: Unknown character set index for field '25700' received from server. The QueryDatabaseTable unable to use in database with hug table. Thanks for you reply David. @Bryan Bende @Matt Burgess
... View more
Labels:
- Labels:
-
Apache NiFi
09-05-2016
01:50 AM
1 Kudo
How long/many of the flowfiles will be maintained in Data Provenance of nifi? (It seems not permanent)
... View more
Labels:
- Labels:
-
Apache NiFi
09-05-2016
12:30 AM
Thanks for your reply. I get it.
... View more
09-02-2016
09:06 AM
in nifi processor settings, there is a property Penalty duration:The amount of time used when this processor penalizes a FlowFile. what does 'processor penalizes a FlowFile.' mean?
... View more
Labels:
- Labels:
-
Apache NiFi
09-01-2016
08:01 AM
Thank you very much. It work.
... View more
09-01-2016
08:01 AM
Thank you very much. It work.
... View more
08-31-2016
08:36 AM
There is a page to enable User Authentication with Kerberos step by step for nifi 0.x https://community.hortonworks.com/articles/34147/nifi-security-user-authentication-with-kerberos.html# But nifi 1.x change the conf. I follow parts of steps as above, and then login with username/password . But I get following message: "Unable to perform the desired action due to insufficient permissions. Contact the system administrator." (it seem that the username/password has been authed by kerberos ) How to resolve? Thanks for you reply. @Jobin George: please update your article. Thanks very much.
... View more
Labels:
- Labels:
-
Apache NiFi
08-31-2016
01:00 AM
Thanks a lot
... View more
08-30-2016
03:15 AM
1 Kudo
It seems that all the users of nifi manage the flows on one canvas in a nifi instance. It's convenient for each user has there own canvas to create there own flows. How to realize? Or is there a plan for this function?
... View more
Labels:
- Labels:
-
Apache NiFi
08-18-2016
02:26 AM
I add nifi on ambari UI. In the step 6,I see the following message. Consistency check failed The configuration changes could not be validated for consistency due to an unknown error. Your changes have not been saved yet. Would you like to proceed and save the changes? In the browser(chrome console), i get this warning message(i do not know whether it leads to the check failed): historyserver_host from mapreduce2-site.xml can't be validate zookeeper_server_hosts from zookeeper-site.xml can't be validate kafka_broker_hosts from kafka-site.xml can't be validate ps:there is no mapreduce2-site.xml/zookeeper-site.xml/kafka-site.xml in my ambari server host and the code may be in app.js buildConfigsJSON: function(services, stepConfigs)
.............
if (configurations[property.get('filename').replace('.xml','')]){
configurations[property.get('filename').replace('.xml','')]['properties'][property.get('name')] = property.get('value');
} else {
console.warn(property.get('name') + " from " + property.get('filename') + " can't be validate");
}
}); if i hit processed anyway. the nifi can be installed correctly. But i do not want to get the step6 error when add service. How to solve it?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache NiFi
08-11-2016
11:13 AM
thank you!
... View more
08-10-2016
03:01 AM
1 Kudo
It seems that storm is mentioned rarely in HDF. Can nifi replace storm?
Nifi and storm is independent to each other?(no processer in nifi communicate with storm )
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Storm
08-10-2016
12:52 AM
I forget to set the property "Hadoop Configuration Files". Now it can work properly. Thanks for your reply.
... View more
08-08-2016
07:18 AM
1 I can access kerberized HDFS by the shell command: kinit -k -t /etc/security/keytabs/nn.service.keytab nn/bigdata1.domain hadoop fs -mkdir /abc
[SUCCESS] But it has error when i use PutHDFS processor. The kerberos property set as bellow
Kerberos Principal: nn/bigdata1.domain Kerberos Keytab: /etc/security/keytabs/nn.service.keytab [erros logs] WARN [Timer-Driven Process Thread-10] org.apache.hadoop.ipc.Client Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name ERROR [Timer-Driven Process Thread-10] o.apache.nifi.processors.hadoop.PutHDFS PutHDFS[id=662c5a72-cdc3-4273-a9ac-994318574391] Failed to write to HDFS due to java.io.IOException: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "bigdata1.domain/10.110.20.213"; destination host is: "bigdata1.domain":8020; : java.io.IOException: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "bigdata1.domain/10.110.20.213"; destination host is: "bigdata1.domain":8020; 2 I can access kerberized the hbase in shell by principal hbase/bigdata1.domain@EXAMPLE.COM:
kinit -k -t hbase.service.keytab hbase/bigdata1.domain@EXAMPLE.COM hbase shell
[SUCCESS] But it has error when i use HBase_1_1_2_ClientService Controller Services. The kerberos property set as bellow Kerberos Principal: hbase/bigdata1.domain@EXAMPLE.COM Kerberos Keytab: /etc/security/keytabs/hbase.service.keytab [erros logs]
RpcRetryingCaller{globalStartTime=1470640110984, pause=100, retries=1}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to bigdata1.domain/10.110.20.213:16000 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to bigdata1.domain/10.110.20.213:16000 is closing. Call id=0, waitTime=8
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) ~[hbase-client-1.1.2.jar:1.1.2]
at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3917) ~[hbase-client-1.1.2.jar:1.1.2]
.........
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to bigdata1.domain/10.110.20.213:16000 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to bigdata1.domain/10.110.20.213:16000 is closing. Call id=0, waitTime=8
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1533) ~[hbase-client-1.1.2.jar:1.1.2] at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1553) ~[hbase-client-1.1.2.jar:1.1.2]
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1704) ~[hbase-client-1.1.2.jar:1.1.2] The state of port 16000 [root@bigdata1 logs]# netstat -an|grep 16000 tcp 0 0 10.110.20.213:33047 10.110.20.213:16000 TIME_WAIT
tcp 0 0 ::ffff:10.110.20.213:16000 :::* LISTEN
tcp 0 0 ::ffff:10.110.20.213:16000 ::ffff:10.110.20.217:51349 ESTABLISHED Thanks for any help.
... View more
Labels:
07-25-2016
01:45 AM
Thanks for your reply. In your scenario, the flow file need to be splited and then merged.
In Admin Guide: NiFi keeps FlowFile information in memory (the JVM). But during surges of incoming data, NiFi "swaps" the FlowFile information to disk temporarily. I wonder the split and merge procedure will cost performance additionally or not? If so, I think it is better to route or update each lines of context in one flow file.
... View more
07-20-2016
03:59 AM
I want to route each line of the textual context(JSON) individually. So i use RouteText to realize. The context of the flowfile as bellow: {"id": 1, "telnum": "11880", "name": "david1"}
{"id": 2, "telnum": "2226666", "name": "david2"} {"id": 3, "telnum": "222777", "name": "david3"} {"id": 4, "telnum": "11996", "name": "david4"} I want the telnum match the prefix "11"(id in 1,4) can be put into a flowfile. The telnum match the prefix "222"(id in 2,3) can be put into another flowfile. In the Expression Language Guide, the jsonPath function can be used to evaluate the json value. I use ${line:jsonPath('$.telnum'):matches('222.*')} , but it can not be compiled correctly. The attribute name of each line may not be "line" How to set the Properties: Routing Strategy, Matching Strategy and the two Relationship expression for prefix "11" and "222"? Thanks for help. David.
... View more
Labels:
- Labels:
-
Apache NiFi
07-19-2016
03:27 AM
Thanks for your advice, Matt. I fix this problem by install certificate in my node server. And the url of RPG is set https://node1:7070/nifi. Then assign role to host1 by the nifi web UI.
... View more
07-18-2016
09:20 AM
At first I distribute works in a cluster NiFi instance in perspective.(By run processor on the Primary Node only, and then use site-to-site (remote process group) to distribute that listing to all nodes in the cluster.) Then I add User Authentication with Kerberos follow https://community.hortonworks.com/articles/34147/nifi-security-user-authentication-with-kerberos.html# Now i want to distribute works in this secure NiFi instance. I add the Remote Process Group with url [https://10.110.20.213:7070/nifi]. But it has warning as bellow
(10.110.20.213:7070 is the web ui of my NCM, 10.110.20.215:7070 is node) 10.110.20.215:7070
Unable to connect to https://10.110.20.213:7070/nifi due to com.sun.jersey.api.client.ClientHandlerException: java.io.IOException: HTTPS hostname wrong: should be <10.110.20.213> Where i was wrong? Thanks! @Jobin George
... View more
Labels:
- Labels:
-
Apache NiFi
07-15-2016
07:16 AM
Thanks George, I follow your steps to the step 10, when i login the url https://node1:9090/nifi 10. When asked, confirm for security exception and proceed. Now you are securely logged in as Demo user with admin privileges It seems that no need to use username/password to login with ADMIN user. But I can not login directly, the page ask for the username and password like step 11. Please help me to find out where is wrong? I use your zip of Certification Authorities and client certificates. And cert-browser.pfx has already loaded into browser. Thanks.
... View more