Re: Hive query failed over JDBC.

ali_mohammadi · ‎03-07-2018

I had also been facing that issue in hive, but I solved by setting "use.hive.interactive.mode" to true and restart the Hive View to run in llap mode. thank you

ali_mohammadi · ‎03-03-2018

Dear @Girish Chaudhari, Unfortunately, I have faced this issue as well, but I could not understand the BEST ANSWER submitted by you. If it is possible, please help me. thanks a million, Ali

ali_mohammadi · ‎11-21-2017

@Jay Kumar SenSharma thanks

ali_mohammadi · ‎11-20-2017

Hi everybody, We have a cluster with 3 VM nodes on HP servers(64 GB ram and 10 TB capacity and 8 cores). Vsphere 5.5 has been used in this project to create virtual machines. Now we need some extra capacity to add to HDFS, but there is no space on the HP server and new VMs can not be constructed. The question is this: could I add a PC or a laptop with the same operating system as a new node to my cluster? Maybe, it is good to ask in this way: what kind of systems can be used together for constructing hdp clusters? thanks a million, Shanghoosh

ali_mohammadi · ‎10-14-2017

@pdarvas thanks, I am going to check it.

ali_mohammadi · ‎10-10-2017

@pdarvasi Thanks for your valuable and useful suggestion. Actually, my major problem is step one. I need to have openstack on current hp servers and also need to extend it whenever it is necessary. shall I install ESXi and then create a VM and after that install openstack? or I can install openstack on raw servers and then can create a extendable private cloud without ESXi. In fact, I am confusing with creating clusters. For example, I thought, I have to manage a cluster of servers on Vsphere, a cluster of nodes on openstack, and then a hadoop cluster on Ambari. It's still confusing for me. It would be appreciated if you could give me more information or introduce some tutorials. thanks a million, ALi

ali_mohammadi · ‎10-04-2017

Hi, I am new in making private clouds and hadoop ecosystem. We have three HP servers without ESX or openstack, and want to create a private cloud and then have an HDP cluster with Ambari. It would be appreciated, If somebody could explain the steps of creating a cloud and hadoop cluster by using (ESXi, openstack, cloudbreak, Ambari and ....). there are some tutorials to do this, but they are confusing to me. some of our requirements are as follows: 1- we must have at least 100TB HDFS capacity. 2- we should be able to add new nodes for increasing the capacity of HDFS and processing power. 3- we should be able to share the HDP cluster with multiple users. each user should be able to run own map/reduce job. Sincerely,

ali_mohammadi · ‎07-29-2017

Dear @Marco Gaido, I deleted my Json files in hdfs and then I ingested some other tweets in json format. Now, hive is working well. thanks a lot

ali_mohammadi · ‎07-26-2017

about tutorial: https://hortonworks.com/tutorial/analyzing-social-media-and-customer-sentiment-with-apache-nifi-and-hdp-search/ I have been following above tutorial and everything worked fine but when calculating whether a tweet was positive, neutral, or negative using this next Hive command : create table IF NOT EXISTS tweets_sentiment stored as orc as select tweet_id, case when sum( polarity ) > 0 then 'positive' when sum( polarity ) < 0 then 'negative' else 'neutral' end as sentiment from l3 group by tweet_id; I got the following error message: tanks a million java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1501058816878_0021_1_01, diagnostics=[Task failed, taskId=task_1501058816878_0021_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:325) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:563) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 17 more Caused by: org.apache.hadoop.hive.serde2.SerDeException: Row is not a valid JSON Object - JSONException: Unterminated string at 237 [character 238 line 1] at org.openx.data.jsonserde.JsonSerDe.onMalformedJson(JsonSerDe.java:424) at org.openx.data.jsonserde.JsonSerDe.deserialize(JsonSerDe.java:183) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:149) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:113) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:554) ... 18 more ], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:325) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:563) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 17 more Caused by: org.apache.hadoop.hive.serde2.SerDeException: Row is not a valid JSON Object - JSONException: Unterminated string at 237 [character 238 line 1] at org.openx.data.jsonserde.JsonSerDe.onMalformedJson(JsonSerDe.java:424) at org.openx.data.jsonserde.JsonSerDe.deserialize(JsonSerDe.java:183) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:149) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:113) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:554) ... 18 more ], TaskAttempt 2 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:325) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:563) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 17 more Caused by: org.apache.hadoop.hive.serde2.SerDeException: Row is not a valid JSON Object - JSONException: Unterminated string at 237 [character 238 line 1] at org.openx.data.jsonserde.JsonSerDe.onMalformedJson(JsonSerDe.java:424) at org.openx.data.jsonserde.JsonSerDe.deserialize(JsonSerDe.java:183) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:149) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:113) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:554) ... 18 more ], TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:325) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:563) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 17 more Caused by: org.apache.hadoop.hive.serde2.SerDeException: Row is not a valid JSON Object - JSONException: Unterminated string at 237 [character 238 line 1] at org.openx.data.jsonserde.JsonSerDe.onMalformedJson(JsonSerDe.java:424) at org.openx.data.jsonserde.JsonSerDe.deserialize(JsonSerDe.java:183) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:149) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:113) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:554) ... 18 more ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1501058816878_0021_1_01 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1501058816878_0021_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:2, Vertex vertex_1501058816878_0021_1_02 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1 (less...)

ali_mohammadi · ‎04-05-2017

Hi @Jay SenSharma Thank you for excellent notes. I solved the problem before reading your notes as follows: from hdfs import InsecureClient client = InsecureClient('http://localhost:50070') # for reading a file with client.read('/tmp/tweets_staging/tweets-082940117.json') as reader: features = reader.read() # for writing a file with client.write('/tmp/tweets_staging/1.json', overwrite=True) as writer: writer.write(features) but I am going to try your solution too. Regards, Shanghoosh

ali_mohammadi · ‎04-04-2017

I want to read and write files to and from a remote HDFS. I program by Pycharm in local machine and I want to connect to a remote hdfs (HDP 2.5). Is there any solution? How can I configure hdfs and how can I refer to a file in hdfs? Thanks a million, Shanghoosh

ali_mohammadi · ‎02-28-2017

thank you Actually I had clicked the checkbox. I think, I should wait for a new version of Ambari; my Ambari is 2.4 .

ali_mohammadi · ‎02-28-2017

Thank you @Artem Ervits Yes, I should wait for Ambari 2.5.

ali_mohammadi · ‎02-26-2017

Hello guys First of all, I am using HDP 2.5. In a tutorial with this address " https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.html " thanks to @Artem Ervits that he showed we can create workflows, coordinatores and bundles. In the tutorial I see a view the same as following: but I can not see them in my ambari. I just see workflows as the following photo: How can I solve the problem? thanks a million shanghoosh

ali_mohammadi · ‎02-26-2017

thanks for great tutorial. unfortunately I couldn't see the output. where can I see? And also I do not see create coordinators button 😞

ali_mohammadi · ‎02-18-2017

Hi @Raf Mohammed I have a same problem. Can you help me? I do not know about stale configs for solr. thank you

ali_mohammadi · ‎02-18-2017

Hi @kkawamura Finally I solved it. I installed Solr by using ambari (befor, Solr was in VM but it was not in sandbox) and after that I set "localhost:2181/solr " to PutSolrContentStream processor. Now I am working with Banana,Solr,Nifi and Hive so comfortable. Thanks Koji a million.

ali_mohammadi · ‎02-15-2017

Thank you @Ali Bajwa for good tutoral. I am trying this example with a difference, My nifi is local and I try to put tweets in a remote Solr. Solr is in a VM that contains Hortonworks sandbox. Unfortunately I am getting this error on PutSolrContentStream processor: PutSolrContentStream[id=f6327477-fb7d-4af0-ec32-afcdb184e545] Failed to send StandardFlowFileRecord[uuid=9bc39142-c02c-4fa2-a911-9a9572e885d0,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1487148463852-14, container=default, section=14], offset=696096, length=2589],offset=0,name=103056151325300.json,size=2589] to Solr due to org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://172.17.0.2:8983/solr/tweets_shard1_replica1; routing to connection_failure: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://172.17.0.2:8983/solr/tweets_shard1_replica1; Could you help me? thanks, Shanghoosh

ali_mohammadi · ‎02-15-2017

Thank you @kkawamura for nice summarization. I tried any case in hosts files locally and also in VM, but it didn't work. Are there any other processors in NiFi to help me? I mean, Can I work with Solr with another processor?

ali_mohammadi · ‎02-14-2017

Another thing, When I try bin/solr status It gives me Found 1 Solr nodes: Solr process 11364 running on port 8983 { "solr_home":"/opt/lucidworks-hdpsearch/solr/server/solr/", "version":"5.2.1 1684708 - shalin - 2015-06-10 23:20:13", "startTime":"2017-02-14T05:24:44.191Z", "uptime":"0 days, 3 hours, 36 minutes, 37 seconds", "memory":"101.6 MB (%20.7) of 490.7 MB", "cloud":{ "ZooKeeper":"localhost:2181", "liveNodes":"1", "collections":"2"}} and when I try bin/solr healthcheck -c tweets -z localhost:2181 It gives { "collection":"tweets", "status":"healthy", "numDocs":1, "numShards":1, "shards":[{ "shard":"shard1", "status":"healthy", "replicas":[{ "name":"core_node1", "url":"http://172.17.0.2:8983/solr/tweets_shard1_replica1/", "numDocs":1, "status":"active", "uptime":"0 days, 3 hours, 37 minutes, 5 seconds", "memory":"27.4 MB (%5.6) of 490.7 MB", "leader":true}]}]} but when I try bin/solr healthcheck -c tweets It gives ERROR: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper localhost:9983 within 10000 ms what is the default port of zookeeper ? Is it possible solr nodes can't to communicate with zookeeper?

ali_mohammadi · ‎02-14-2017

Hi Koji thank you 🙂 Yes, I want to try this example " https://github.com/abajwa-hw/ambari-nifi-service ", with a difference: my Nifi is in my local machine and I want to connect to a remote lucidworks locally. The lucidworks(solr) is in a VM(with sandbox). screenshot-from-2017-02-14-11-36-39.png(128.3 kB) I am trying to add tweets in solr.

ali_mohammadi · ‎02-13-2017

Hello @kkawamura First of all, I am new in Solr. When I tried the curl command "curl -i http://172.17.0.2:8983/solr " in VM (sandbox), I got a message as below: HTTP/1.1 302 Found Location: http://172.17.0.2:8983/solr/ Content-Length: 0 but when I try in the local machine (ubuntu 16), I don't get any messages. Also, when I try "curl -i localhost:8983/solr" locally, I get a message as follows: HTTP/1.1 302 Found Location: http://localhost:8983/solr/ Content-Length: 0 Actually, I use PutSolrContentStream processor for sending tweets to a remote Solr (lucidworks) in a VM (sandbox). I set Solr type= cloud and Solr Location= localhost:2181. NiFi is in my local machin. Unfortunately, I get an error as follows: PutSolrContentStream[id=f6327477-fb7d-4af0-e990-512eebdb1c1c] Failed to send StandardFlowFileRecord[uuid=c478c4dc-f16e-4ae7-814e-a7de0985985e,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1486893589035-10, container=default, section=10], offset=612767, length=7103],offset=0,name=77207261802126.json,size=7103] to Solr due to org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://172.17.0.2:8983/solr/tweets_shard1_replica1; routing to connection_failure: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://172.17.0.2:8983/solr/tweets_shard1_replica1 Excuse me for my poor English. Thanks a Million, Shanghoosh

ali_mohammadi · ‎02-13-2017

I am trying to connect to a remote solr but I faced with this error on server can not find: /configs/tweets/admin-extra.html and also faced to this error org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://172.17.0.2:8983/solr when I tried to connect to Solr from NiFi thank you guys

ali_mohammadi · ‎02-12-2017

Notice: some information will be lost when you do what we did. for example I had a lucidworks but now I should install it again.

ali_mohammadi · ‎02-12-2017

Dear Artem I tried again and now, I have a remote HDFS. Thanks a million

ali_mohammadi · ‎02-08-2017

Thank You Dear Artem I followed the tutorial, but when I apply the last command "docker ps", I saw something else and I couldn't to connect to VM in Nifi any more. I faced with this error: PutHDFS[id=015a1000-2532-17af-9400-6501c5ca7018] Failed to write to HDFS due to java.io.IOException: Failed on local exception: java.io.IOException: Connection reset by peer; Host Details : local host is: "shanghoosh-All-Series/127.0.1.1"; destination host is: "sandbox.hortonworks.com":8020; : java.io.IOException: Failed on local exception: java.io.IOException: Connection reset by peer; Host Details : local host is: "shanghoosh-All-Series/127.0.1.1"; destination host is: "sandbox.hortonworks.com":8020;

ali_mohammadi · ‎02-08-2017

Hi guys, First of all, I am new in Hortonworks. I have tried to connect to a remote HDP (actually my hdfs) in VM from NiFi in my local machine. I apply putHDFS processor, but unfortunately I am faced with an error. org.apache.nifi.processor.exception.ProcessException: IOException thrown from PutHDFS[id=07af2532-015a-1000-acc3-3ed53acfcc7c]: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /user/root/tmp/test.txt could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1641) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3198) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3122) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:843) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:500) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) It should be noted : I set core_site and hdfs_site XMLs on PutHdfs processor; Only a NameNode instance is running and it's not in safe-mode; There is a DataNode instances up and running, and there is no node dead; Namenode and Datanode instances are both running, but I don't know how they can communicate with each other; I check datanode and namenode logs and i didn't realized anything; I forwarded 50010 and 8020 ports by using the Virtualbox; Specified reserved spaces for DataNode instances in dfs.datanode.du.reserved is equal 200000000 and the usage of disk is as follows: My storage status in the Virtualbox is as follows: before the error i got another error the same as below: putHDFS[id=07af2532-015a-1000-acc3-3ed53acfcc7c] Failed to write to HDFS due to java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found and for solving it, I removed a property (io.compression.codecs) from core_site.xml. Maybe it has caused the error. Thanks.

Online	Offline
Last Visited	‎03-07-2018 05:32 AM

Date Registered	‎02-08-2017 05:17 AM
Last Visited	‎03-07-2018 05:32 AM
Total Messages Posted	33
Total Kudos Received	1

Re: Hive query failed over JDBC.

Re: Hive query failed over JDBC.

Re: Is it possible to add a new node with differe...

Is it possible to add a new node with different a...

Re: what are steps of easiest way to create an HDP...

Re: what are steps of easiest way to create an HDP...

what are steps of easiest way to create an HDP clu...

Re: tweets sentinment hive 2.0 problem when I try ...

tweets sentinment hive 2.0 problem when I try to c...

Re: How to connect to a remote hdfs (for example h...

How to connect to a remote hdfs (for example horto...

Re: Apache Ambari Workflow Manager View for Apache...

Re: My workflow manager is not complete. I can not...

My workflow manager is not complete. I can not cre...

Re: Apache Ambari Workflow Manager View for Apache...

Re: Solr 5.5 on Sandbox 2.5 Has Stopped Running - ...

Re: How to get rid of "Can not find: /configs/twee...

Re: Sample HDF/NiFi flow to Push Tweets into Solr/...

Re: How to get rid of "Can not find: /configs/twee...

Re: How to get rid of "Can not find: /configs/twee...

Re: How to get rid of "Can not find: /configs/twee...

Re: How to get rid of "Can not find: /configs/twee...

How to get rid of "Can not find: /configs/tweets/a...

Re: "File /user/root/tmp/test.txt" could only be r...

Re: "File /user/root/tmp/test.txt" could only be r...

Re: "File /user/root/tmp/test.txt" could only be r...

"File /user/root/tmp/test.txt" could only be repli...