This might come across as a very naive question as I have never used Linux before. I was wondering where can I view the files inside the sandbox like hadoop configuration files? What is the path? Data files on the data node?? I should log onto the sandbox via the command line via root and where should i navigate to then. I was thinking of configuring a GUI (Gnome) for the sandbox, but unfortunately every simple task has proven to be more difficult than I expected. I do find answers to the problems that I face, online but unfortunately mostly the solutions would not work for me. I have trouble turning off the firewall in the sandbox (Centos 6.8) I have trouble installing Gnome because I get no package available etc. Lastly is this necessary? or if i have ambari I do not need to install the GUI? Trouble in pinging my machine from the guest. Trouble copying files to the guest machine.... conneciton refused 😞 Thanks ... View more

Hi I have deployed the HDP 2.5 inside virtualbox. I have Nifi 1.0.0 installed on my machine. Earlier I had tried to read and write files to HDFS and I had connectivity issues: PutHDFS/ GetHDFS Issue with data transfer Which is still unsolved. I thought may be I can move forward and try my luch with HIVE. I tried to execute an INSERT statement using PutHIVEQL. I think I am back to square one as am still getting the following errors: First i tried to connect to Hive inside the sandbox using the following URL jdbc://hive2://localhost:10000/default which gives the following error: 2016-12-08 13:15:42,902 ERROR [Timer-Driven Process Thread-3] o.a.nifi.dbcp.hive.HiveConnectionPool org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class 'org.apache.hive.jdbc.HiveDriver' for connect URL 'jdbc://hive2://localhost:10000/default' at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1452) ~[commons-dbcp-1.4.jar:1.4] Caused by: java.sql.SQLException: No suitable driver at java.sql.DriverManager.getDriver(DriverManager.java:315) ~[na:1.8.0_111] at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1437) ~[commons-dbcp-1.4.jar:1.4] ... 22 common frames omitted 2016-12-08 13:15:42,902 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.hive.SelectHiveQL SelectHiveQL[id=de0cad2e-0158-1000-fd58-ecc08beadcb0] Unable to execute HiveQL select query SELECT * FROM drivers WHERE event='overspeed'; due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class 'org.apache.hive.jdbc.HiveDriver' for connect URL 'jdbc://hive2://localhost:10000/default'. No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class 'org.apache.hive.jdbc.HiveDriver' for connect URL 'jdbc://hive2://localhost:10000/default' 2016-12-08 13:15:42,903 ERROR [Timer-Driven Process Thread-3] o.a.nifi.processors.hive.SelectHiveQL org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class 'org.apache.hive.jdbc.HiveDriver' for connect URL 'jdbc://hive2://localhost:10000/default' at org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:273) ~[nifi-hive-processors-1.0.0.jar:1.0.0] Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class 'org.apache.hive.jdbc.HiveDriver' for connect URL 'jdbc://hive2://localhost:10000/default' at org.apache.commons.dbcp.BasicDataSource.createConnectionFactory(BasicDataSource.java:1452) ~[commons-dbcp-1.4.jar:1.4] at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1371) ~[commons-dbcp-1.4.jar:1.4] at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) ~[commons-dbcp-1.4.jar:1.4] at org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:269) ~[nifi-hive-processors-1.0.0.jar:1.0.0] ... 19 common frames omitted Caused by: java.sql.SQLException: No suitable driver Then I checked the ambari dashboard and found the following URL: jdbc:hive2://sandbox.hortonworks.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 But now that I use this URL I do think I have moved forward but I am stuck with this error: 2016-12-08 13:20:31,101 ERROR [Timer-Driven Process Thread-10] o.a.nifi.processors.hive.SelectHiveQL SelectHiveQL[id=de0cad2e-0158-1000-fd58-ecc08beadcb0] SelectHiveQL[id=de0cad2e-0158-1000-fd58-ecc08beadcb0] failed to process due to java.lang.NullPointerException; rolling back session: java.lang.NullPointerException 2016-12-08 13:20:31,104 ERROR [Timer-Driven Process Thread-10] o.a.nifi.processors.hive.SelectHiveQL java.lang.NullPointerException: null at org.apache.thrift.transport.TSocket.open(TSocket.java:170) ~[hive-exec-1.2.1.jar:1.2.1] at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266) ~[hive-exec-1.2.1.jar:1.2.1] at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) ~[hive-exec-1.2.1.jar:1.2.1] at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204) ~[hive-jdbc-1.2.1.jar:1.2.1] at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176) ~[hive-jdbc-1.2.1.jar:1.2.1] at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) ~[hive-jdbc-1.2.1.jar:1.2.1] at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38) ~[commons-dbcp-1.4.jar:1.4] at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582) ~[commons-dbcp-1.4.jar:1.4] at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556) ~[commons-dbcp-1.4.jar:1.4] at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545) ~[commons-dbcp-1.4.jar:1.4] at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388) ~[commons-dbcp-1.4.jar:1.4] at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) ~[commons-dbcp-1.4.jar:1.4] at org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:269) ~[nifi-hive-processors-1.0.0.jar:1.0.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_111] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_111] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_111] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_111] at org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:177) ~[nifi-framework-core-1.0.0.jar:1.0.0] at com.sun.proxy.$Proxy132.getConnection(Unknown Source) ~[na:na] at org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:158) ~[nifi-hive-processors-1.0.0.jar:1.0.0] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.0.0.jar:1.0.0] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) [nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.0.0.jar:1.0.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_111] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_111] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] Can any one please help me with this? I have tried so many things but it seems it is impossible to get data through to the sandbox in the virtual machine. What should i do? any possible direction would be helpful. I have disabled the firewall. The replace text yields the following statement: INSERT INTO driveres (truckid,driverid,city,state,velocity,event) VALUES ('N02','N02','Lahore','Punjab','100','overspeeding'); which runs successfully in the ambari hive view. I have also downloaded the hive configs from ambari and added them to the connection pool. The last option for me would be to have Nifi running in the HDP. ... View more

Hi I am reading a CSV file. converting it into JSON and splitting it based on records. I try to write these files into Elastic. I get the following error: 2016-12-04 19:42:18,008 ERROR [Timer-Driven Process Thread-10] o.a.n.p.e.PutElasticsearchHttp PutElasticsearchHttp[id=ca053faa-0158-1000-8397-413cc421f011] PutElasticsearchHttp[id=ca053faa-0158-1000-8397-413cc421f011] failed to process due to org.apache.nifi.processor.exception.ProcessException: java.net.SocketException: Connection reset by peer: socket write error; rolling back session: org.apache.nifi.processor.exception.ProcessException: java.net.SocketException: Connection reset by peer: socket write error 2016-12-04 19:42:18,017 ERROR [Timer-Driven Process Thread-10] o.a.n.p.e.PutElasticsearchHttp org.apache.nifi.processor.exception.ProcessException: java.net.SocketException: Connection reset by peer: socket write error at org.apache.nifi.processors.elasticsearch.PutElasticsearchHttp.onTrigger(PutElasticsearchHttp.java:315) ~[nifi-elasticsearch-processors-1.0.0.jar:1.0.0] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.0.0.jar:1.0.0] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) [nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.0.0.jar:1.0.0] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.0.0.jar:1.0.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_101] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101] Caused by: java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_101] at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0_101] at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_101] at okio.Okio$1.write(Okio.java:80) ~[okio-1.8.0.jar:na] at okio.AsyncTimeout$1.write(AsyncTimeout.java:180) ~[okio-1.8.0.jar:na] at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:171) ~[okio-1.8.0.jar:na] at okio.RealBufferedSink.write(RealBufferedSink.java:41) ~[okio-1.8.0.jar:na] at okhttp3.internal.http.Http1xStream$FixedLengthSink.write(Http1xStream.java:286) ~[okhttp-3.3.1.jar:na] at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:171) ~[okio-1.8.0.jar:na] at okio.RealBufferedSink.write(RealBufferedSink.java:91) ~[okio-1.8.0.jar:na] at okhttp3.RequestBody$2.writeTo(RequestBody.java:96) ~[okhttp-3.3.1.jar:na] at okhttp3.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:756) ~[okhttp-3.3.1.jar:na] at okhttp3.internal.http.HttpEngine.readResponse(HttpEngine.java:613) ~[okhttp-3.3.1.jar:na] at okhttp3.RealCall.getResponse(RealCall.java:244) ~[okhttp-3.3.1.jar:na] at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:201) ~[okhttp-3.3.1.jar:na] at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:163) ~[okhttp-3.3.1.jar:na] at okhttp3.RealCall.execute(RealCall.java:57) ~[okhttp-3.3.1.jar:na] at org.apache.nifi.processors.elasticsearch.AbstractElasticsearchHttpProcessor.sendRequestToElasticsearch(AbstractElasticsearchHttpProcessor.java:166) ~[nifi-elasticsearch-processors-1.0.0.jar:1.0.0] at org.apache.nifi.processors.elasticsearch.PutElasticsearchHttp.onTrigger(PutElasticsearchHttp.java:313) ~[nifi-elasticsearch-processors-1.0.0.jar:1.0.0] ... 12 common frames omitted Any help is highly appriciated ... View more