Member since
02-08-2017
33
Posts
1
Kudos Received
0
Solutions
03-07-2018
05:34 AM
I had also been facing that issue in hive, but I solved by setting "use.hive.interactive.mode" to true and restart the Hive View to run in llap mode. thank you
... View more
03-03-2018
07:41 AM
Dear @Girish Chaudhari, Unfortunately, I have faced this issue as well, but I could not understand the BEST ANSWER submitted by you. If it is possible, please help me. thanks a million, Ali
... View more
11-21-2017
08:18 AM
@Jay Kumar SenSharma thanks
... View more
11-20-2017
10:33 AM
Hi everybody, We have a cluster with 3 VM nodes on HP servers(64 GB ram and 10 TB capacity and 8 cores). Vsphere 5.5 has been used in this project to create virtual machines. Now we need some extra capacity to add to HDFS, but there is no space on the HP server and new VMs can not be constructed. The question is this: could I add a PC or a laptop with the same operating system as a new node to my cluster? Maybe, it is good to ask in this way: what kind of systems can be used together for constructing hdp clusters? thanks a million, Shanghoosh
... View more
10-14-2017
06:05 AM
@pdarvas thanks, I am going to check it.
... View more
10-10-2017
06:19 AM
@pdarvasi Thanks for your valuable and useful suggestion. Actually, my major problem is step one. I need to have openstack on current hp servers and also need to extend it whenever it is necessary. shall I install ESXi and then create a VM and after that install openstack? or I can install openstack on raw servers and then can create a extendable private cloud without ESXi. In fact, I am confusing with creating clusters. For example, I thought, I have to manage a cluster of servers on Vsphere, a cluster of nodes on openstack, and then a hadoop cluster on Ambari. It's still confusing for me. It would be appreciated if you could give me more information or introduce some tutorials. thanks a million, ALi
... View more
10-04-2017
12:33 PM
Hi, I am new in making private clouds and hadoop ecosystem. We have three HP servers without ESX or openstack, and want to create a private cloud and then have an HDP cluster with Ambari. It would be appreciated, If somebody could explain the steps of creating a cloud and hadoop cluster by using (ESXi, openstack, cloudbreak, Ambari and ....). there are some tutorials to do this, but they are confusing to me. some of our requirements are as follows: 1- we must have at least 100TB HDFS capacity. 2- we should be able to add new nodes for increasing the capacity of HDFS and processing power. 3- we should be able to share the HDP cluster with multiple users. each user should be able to run own map/reduce job. Sincerely,
... View more
Labels:
08-28-2017
10:28 AM
Hi I read data from oracle to kafka with nifi and I want to analyze my records(for example sum of attributes) with spark streaming. I sum all of numbers in an rdd and want to add it by previous rdds value of summation. here, there is a problem! At the end of each stream, the type of variable that hold the sum of arrived data per 10 seconds is Dstream. how can I import this (just a number in dstream)to hive table? in the belove code, sum contains a number but in dstream format that I want to to insert it to hive. thanks a million Ali val words = lines.flatMap{ case(x, y) => y.split("\n")}
val words1=words.map{aline=> aline.split(" ")(3).replaceAll("\"","").replaceAll("}","")}
val sum= words1.map{ss => ss.toDouble}.reduce(_+_)
... View more
07-29-2017
05:33 AM
Dear @Marco Gaido, I deleted my Json files in hdfs and then I ingested some other tweets in json format. Now, hive is working well. thanks a lot
... View more
07-26-2017
11:16 AM
about tutorial: https://hortonworks.com/tutorial/analyzing-social-media-and-customer-sentiment-with-apache-nifi-and-hdp-search/ I have been following above tutorial and everything
worked fine but when calculating whether a tweet was positive, neutral,
or negative using this next Hive command : create table IF NOT EXISTS tweets_sentiment stored as orc as select
tweet_id, case when sum( polarity ) > 0 then 'positive' when sum(
polarity ) < 0 then 'negative' else 'neutral' end as sentiment from
l3 group by tweet_id; I got the following error message: tanks a million java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1501058816878_0021_1_01, diagnostics=[Task failed, taskId=task_1501058816878_0021_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:325)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:563)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
... 17 more
Caused by: org.apache.hadoop.hive.serde2.SerDeException: Row is not a valid JSON Object - JSONException: Unterminated string at 237 [character 238 line 1]
at org.openx.data.jsonserde.JsonSerDe.onMalformedJson(JsonSerDe.java:424)
at org.openx.data.jsonserde.JsonSerDe.deserialize(JsonSerDe.java:183)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:149)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:113)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:554)
... 18 more
], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:325)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:563)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
... 17 more
Caused by: org.apache.hadoop.hive.serde2.SerDeException: Row is not a valid JSON Object - JSONException: Unterminated string at 237 [character 238 line 1]
at org.openx.data.jsonserde.JsonSerDe.onMalformedJson(JsonSerDe.java:424)
at org.openx.data.jsonserde.JsonSerDe.deserialize(JsonSerDe.java:183)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:149)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:113)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:554)
... 18 more
], TaskAttempt 2 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:325)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:563)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
... 17 more
Caused by: org.apache.hadoop.hive.serde2.SerDeException: Row is not a valid JSON Object - JSONException: Unterminated string at 237 [character 238 line 1]
at org.openx.data.jsonserde.JsonSerDe.onMalformedJson(JsonSerDe.java:424)
at org.openx.data.jsonserde.JsonSerDe.deserialize(JsonSerDe.java:183)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:149)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:113)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:554)
... 18 more
], TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185)
at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:325)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"tweet_id":890076549655003136,"created_unixtime":1501045760917,"created_time":"Wed Jul 26 05:09:20 +0000 2017","lang":"en","displayname":"2ne1legend21","time_zone":"","msg":"RT hyung_rose_bts BTS_twt The cup couldnt have said it better
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:563)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
... 17 more
Caused by: org.apache.hadoop.hive.serde2.SerDeException: Row is not a valid JSON Object - JSONException: Unterminated string at 237 [character 238 line 1]
at org.openx.data.jsonserde.JsonSerDe.onMalformedJson(JsonSerDe.java:424)
at org.openx.data.jsonserde.JsonSerDe.deserialize(JsonSerDe.java:183)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.readRow(MapOperator.java:149)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.access$200(MapOperator.java:113)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:554)
... 18 more
]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex_1501058816878_0021_1_01 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1501058816878_0021_1_02, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:2, Vertex vertex_1501058816878_0021_1_02 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
(less...)
... View more
Labels:
07-26-2017
11:09 AM
@Luis Martinez hi if you have solved the problem, please let us know
... View more
07-24-2017
09:28 AM
add port 50010 like this https://community.hortonworks.com/articles/65914/how-to-add-ports-to-the-hdp-25-virtualbox-sandbox.html also you should add 50010 to VMbox
... View more
04-09-2017
11:42 AM
I am trying to implement a movie recommender system for users. I entirely implemented the ALS algorithm and now I want to get new user rates and after that to recommend some other movies. I programmed my code as follows in zeppelin to get information: %pyspark
myRatedMovies = [
(0, 1,1.0),
(0, 2,1.5),
(0, 3,1.1),
(0, 4,1),
(0,5,1.5),
(0,6,1.0),
(0, 5,5),
(0,52,5),
(0,96,5.0)
]
myRatingsRDD = sc.parallelize(myRatedMovies) I created an RDD myself and after that I append it in my train dataset and so on, but I want to have a user interface to get information. Actually I want to have a dynamic UI. I saw this page "https://zeppelin.apache.org/docs/0.6.1/manual/dynamicform.html", but it did not help me because I need to get a lot of tuples and I did not find some helpful information about command buttons on that web page. Also I know, I can use angular in zeppelin, but I am new in zeppelin and angulary, and it is very hard to me. For example I can not use the binded variables in pyspark. I would like to add user rates to an RDD when the user clicks on the button "Add My Rate" .
Thank you very much for your time and assistance in this matter. Sincerely, Shanghoosh
... View more
Labels:
04-05-2017
05:26 AM
Hi @Jay SenSharma Thank you for excellent notes. I solved the problem before reading your notes as follows: from hdfs import InsecureClient
client = InsecureClient('http://localhost:50070')
# for reading a file
with client.read('/tmp/tweets_staging/tweets-082940117.json') as reader:
features = reader.read()
# for writing a file
with client.write('/tmp/tweets_staging/1.json', overwrite=True) as writer:
writer.write(features)
but I am going to try your solution too. Regards, Shanghoosh
... View more
04-04-2017
11:25 AM
I want to read and write files to and from a remote HDFS. I program by Pycharm in local machine and I want to connect to a remote hdfs (HDP 2.5). Is there any solution? How can I configure hdfs and how can I refer to a file in hdfs? Thanks a million, Shanghoosh
... View more
Labels:
02-28-2017
04:41 AM
thank you Actually I had clicked the checkbox. I think, I should wait for a new version of Ambari; my Ambari is 2.4 .
... View more
02-28-2017
04:31 AM
Thank you @Artem Ervits Yes, I should wait for Ambari 2.5.
... View more
02-26-2017
08:13 AM
Hello guys First of all, I am using HDP 2.5. In a tutorial with this address " https://community.hortonworks.com/articles/82964/getting-started-with-apache-ambari-workflow-design.html " thanks to @Artem Ervits that he showed we can create workflows, coordinatores and bundles. In the tutorial I see a view the same as following: but I can not see them in my ambari. I just see workflows as the following photo: How can I solve the problem? thanks a million shanghoosh
... View more
Labels:
02-26-2017
07:18 AM
thanks for great tutorial. unfortunately I couldn't see the output. where can I see? And also I do not see create coordinators button 😞
... View more
02-18-2017
12:18 PM
Hi @Raf Mohammed I have a same problem. Can you help me? I do not know about stale configs for solr. thank you
... View more
02-18-2017
04:42 AM
Hi @kkawamura Finally I solved it. I installed Solr by using ambari (befor, Solr was in VM but it was not in sandbox) and after that I set "localhost:2181/solr "
to PutSolrContentStream processor. Now I am working with Banana,Solr,Nifi and Hive so comfortable. Thanks Koji a million.
... View more
02-15-2017
11:11 AM
Thank you @Ali Bajwa for good tutoral. I am trying this example with a difference, My nifi is local and I try to put tweets in a remote Solr. Solr is in a VM that contains Hortonworks sandbox. Unfortunately I am getting this error on PutSolrContentStream processor: PutSolrContentStream[id=f6327477-fb7d-4af0-ec32-afcdb184e545] Failed to send StandardFlowFileRecord[uuid=9bc39142-c02c-4fa2-a911-9a9572e885d0,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1487148463852-14, container=default, section=14], offset=696096, length=2589],offset=0,name=103056151325300.json,size=2589] to Solr due to org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://172.17.0.2:8983/solr/tweets_shard1_replica1; routing to connection_failure: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://172.17.0.2:8983/solr/tweets_shard1_replica1; Could you help me? thanks, Shanghoosh
... View more
02-15-2017
09:33 AM
Thank you @kkawamura for nice summarization. I tried any case in hosts files locally and also in VM, but it didn't work. Are there any other processors in NiFi to help me? I mean, Can I work with Solr with another processor?
... View more
02-14-2017
09:17 AM
Another thing, When I try bin/solr status It gives me Found 1 Solr nodes:
Solr process 11364 running on port 8983
{
"solr_home":"/opt/lucidworks-hdpsearch/solr/server/solr/",
"version":"5.2.1 1684708 - shalin - 2015-06-10 23:20:13",
"startTime":"2017-02-14T05:24:44.191Z",
"uptime":"0 days, 3 hours, 36 minutes, 37 seconds",
"memory":"101.6 MB (%20.7) of 490.7 MB",
"cloud":{
"ZooKeeper":"localhost:2181",
"liveNodes":"1",
"collections":"2"}}
and when I try bin/solr healthcheck -c tweets -z localhost:2181 It gives {
"collection":"tweets",
"status":"healthy",
"numDocs":1,
"numShards":1,
"shards":[{
"shard":"shard1",
"status":"healthy",
"replicas":[{
"name":"core_node1",
"url":"http://172.17.0.2:8983/solr/tweets_shard1_replica1/",
"numDocs":1,
"status":"active",
"uptime":"0 days, 3 hours, 37 minutes, 5 seconds",
"memory":"27.4 MB (%5.6) of 490.7 MB",
"leader":true}]}]}
but when I try bin/solr healthcheck -c tweets It gives ERROR: java.util.concurrent.TimeoutException: Could not connect to ZooKeeper localhost:9983 within 10000 ms what is the default port of zookeeper ? Is it possible solr nodes can't to communicate with zookeeper?
... View more
02-14-2017
08:11 AM
Hi Koji thank you 🙂 Yes, I want to try this example " https://github.com/abajwa-hw/ambari-nifi-service ", with a difference: my Nifi is in my local machine and I want to connect to a remote lucidworks locally. The lucidworks(solr) is in a VM(with sandbox). screenshot-from-2017-02-14-11-36-39.png(128.3 kB) I am trying to add tweets in solr.
... View more
02-13-2017
01:17 PM
Hello @kkawamura First of all, I am new in Solr. When I tried the curl command "curl -i http://172.17.0.2:8983/solr " in VM (sandbox), I got a message as below: HTTP/1.1 302 Found
Location: http://172.17.0.2:8983/solr/
Content-Length: 0
but when I try in the local machine (ubuntu 16), I don't get any messages. Also, when I try "curl -i localhost:8983/solr" locally, I get a message as follows: HTTP/1.1 302 Found
Location: http://localhost:8983/solr/
Content-Length: 0 Actually, I use PutSolrContentStream processor for sending tweets to a remote Solr (lucidworks) in a VM (sandbox). I set Solr type= cloud and Solr Location= localhost:2181. NiFi is in my local machin. Unfortunately, I get an error as follows: PutSolrContentStream[id=f6327477-fb7d-4af0-e990-512eebdb1c1c] Failed to send StandardFlowFileRecord[uuid=c478c4dc-f16e-4ae7-814e-a7de0985985e,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1486893589035-10, container=default, section=10], offset=612767, length=7103],offset=0,name=77207261802126.json,size=7103] to Solr due to org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://172.17.0.2:8983/solr/tweets_shard1_replica1; routing to connection_failure: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://172.17.0.2:8983/solr/tweets_shard1_replica1 Excuse me for my poor English. Thanks a Million, Shanghoosh
... View more
02-13-2017
08:07 AM
I am trying to connect to a remote solr but I faced with this error on server can not find: /configs/tweets/admin-extra.html and also faced to this error org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://172.17.0.2:8983/solr when I tried to connect to Solr from NiFi thank you guys
... View more
Labels:
02-12-2017
08:55 AM
Notice: some information will be lost when you do what we did. for example I had a lucidworks but now I should install it again.
... View more
02-12-2017
06:18 AM
1 Kudo
Hi Guys I am trying to add tweets in a remote solr but I faced with this error. PutSolrContentStream[id=f6327477-fb7d-4af0-e990-512eebdb1c1c] Failed to send StandardFlowFileRecord[uuid=34e67c1a-3a18-4280-a738-9943d6716962,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1486876718587-1, container=default, section=1], offset=53886, length=2508],offset=0,name=60294765850236.json,size=2508] to Solr due to org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://172.17.0.2:8983/solr/tweets_shard1_replica1; routing to connection_failure: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://172.17.0.2:8983/solr/tweets_shard1_replica1
I checked: NiFi is trying to connect http://172.17.0.2:8983/solr/tweets_shard1_replica1 but the tweets_shard1_replical is in below address and i can see it in my browser http://localhost:8983/solr/#/tweets_shard1_replica1 8983 and 2181 ports are added in my virtualbox. My solr server is on. All properties of PutSolrContentStream are configures the as this example "https://github.com/abajwa-hw/ambari-nifi-service" I appreciate your help with this error. Unfortunately I have some problems in playing with ports, host names and connections and I would like to read more about them, is there a good reference? thanks,
... View more
Labels: