question Re: stream data from web site using nifi in Archives of Support Questions (Read Only)

stream data from web site using nifi

nejmhadjmbarek — Fri, 19 Feb 2016 18:50:55 GMT

this is the first time that i use nifi ,so how i can stream data from web site to hdfs using nifi ; there is template for this ?? if not how i can do it ,which processor i can used it ? thx

Re: stream data from web site using nifi

aervits — Fri, 19 Feb 2016 19:01:39 GMT

@nejm hadj here are all templates https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates

Take a look at GetHTTP processor for your task https://nifi.apache.org/docs.html

Re: stream data from web site using nifi

nsabharwal — Fri, 19 Feb 2016 21:57:51 GMT

@nejm hadj

See this

https://community.hortonworks.com/repos/6119/apache-nifi-template-rpo.html or

https://github.com/hortonworks-gallery/nifi-templates

Now , let's take twitter example

https://community.hortonworks.com/repos/8587/storm-demo-to-monitor-tweets.html

https://community.hortonworks.com/repos/10062/single-view-demo.html

Re: stream data from web site using nifi

nejmhadjmbarek — Tue, 23 Feb 2016 17:52:19 GMT

@Neeraj Sabharwal

thx for your help , now i use template twitter to stream data but i get this error

Permission denied: user=root, access=WRITE, inode="/nifi/tweets/.668434832146110.json":hdfs:hdfs:drwxr-xr-x

how i can change permission to make user root can write to hdfs ,also whene i try to manage my     files  or change permission (i connect as admin  in ambari ) i get the same error for the user 
admin

Re: stream data from web site using nifi

nsabharwal — Tue, 23 Feb 2016 18:05:27 GMT

@nejm hadj That's good progress!!!

Now, Please do this

su - hdfs

hdfs dfs -chown -R root:hdfs /nifi

exit

This should fix permission issue.

Re: stream data from web site using nifi

nsabharwal — Tue, 23 Feb 2016 18:06:35 GMT

@nejm hadj For admin user

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_views_guide/content/_setup_HDFS_user_directory.html

Connect to a host in the cluster that includes the HDFS client.
Switch to the hdfs system account user.
```
su - hdfs
```
Using the HDFS client, make an HDFS directory for the user. For example, if your username is admin, you would create the following directory.
```
hadoop fs -mkdir /user/admin
```
Set the ownership on the newly created directory. For example, if your username is admin, you would make that user the directory owner.
```
hadoop fs -chown admin:hadoop /user/admin
```

Re: stream data from web site using nifi

nejmhadjmbarek — Tue, 23 Feb 2016 19:40:27 GMT

Thx it is work