Security / Operational best practices question

prakashpunj — Fri, 16 Sep 2022 10:07:17 GMT

Had some question about operational best practice:

a.) For external customers, what's the best way to allow them to upload large size data to HDFS. Uploading via flile browser web interface may not be safe, reason being if there are 10 users start uploading 20gb of data at the sametime. Server will choke ( Currently I have small 5 server cluster).

b.) Was thinking of having a jumpbox externally, where people can ssh to it and ftp their data and a cron job will then push the data to HDFS on a periodic basic. Once the upload, users can use web interface to program using Hiv/Pig

c.) Spark-Shell - Is there a way to have users initiate spark-shell from a web interface.

d.) Currently NameNode is a single point of failure. I was reading about federated service or use HA. What's recommended. I have a very small environment

e.) DataNode information, cluster information, spark job are all can be viewed from web. Is it a good practice to allow users see those information ? Issue is information is not restricted to just their information. It's open for all or none.

Thanks

Prakash

Re: Security / Operational best practices question

ahadjidj — Sat, 05 Mar 2016 00:25:07 GMT

Hi @Prakash Punj

You can use NiFi to supervise a directory and ingest each new file to HDFS (GetFile and PutHDFS processors). https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetFile/index.html
You can do Spark in a browser with Zeppelin. You can have it in Ambari with the Zeppelin view. Some tutorials here http://hortonworks.com/hadoop/zeppelin/#tutorials
To avoid a SPOF you need HDFS HA. Federation is having multiple NNs for managing very big clusters and reducing the stress on a single NN.
In Ambari you can have admin users and simple users. Simple users have less power in Ambari.

question Security / Operational best practices question in Archives of Support Questions (Read Only)

Security / Operational best practices question

Re: Security / Operational best practices question