Hi There,   Thanks for reaching out on the community. I'm Josh, and I'll help address this for you.   log4j.properties: CM is the central point of configuration for services, so the short answer is that you should adjust log4j settings using safety valves. Below is an engineering blog post with a good description of how CM works. http://blog.cloudera.com/blog/2013/07/how-does-cloudera-manager-work/ When a CM agent for a host heart beats to Cloudera Manager, Cloudera Manager sends back processes that should be running, and the related config files, one of which is the log4j.properties, for that service and role. From here, the CM agent makes a run time directory for these config files and references those. For instance, the agent will make a directory like the one bellow for a namenode role: / var / run / cloudera - scm - agent / process / 879 - hdfs - NAMENODE/ this is why editing config files in on the OS has no effect, and is not recommended.    Enabling audit logging: To enable audit logging for a service without navigator, you would want to set the appropriate log4j settings in the appropriate safety valve for that service. Let's use HDFS as an example. Cloudera Manager has a configuration property for HDFS labeled "NameNode Logging Advanced Configuration Snippet (Safety Valve)". This is the one you want to put your log4j settings in. Once you've put your settings in, it will insert those into the log4j.properties it sends over to the agent in heartbeats. The specifics for enabling vanilla hadoop HDFS audit logging can be found bellow: http://apprize.info/security/hadoop/7.html   Considering all of this info, bear in mind that Navigator takes care of all of this for you, as well as adding additional features. For instance, HDFS audit logs can be very bulky and cumbersome by themselves and include many operations that aren't very helpful from an auditing standpoint. Navigator is able to apply event filters to an audit log, store relevant audits, and index them for further searching. Therefore, I highly recommend enabling navigator when doing so becomes feasible.   Please let me know if you have any other questions. Cheers     ... View more

Hello, In CDH 5.8, queries can be exported from a user's home directory and imported to another users in JSON format. Steps bellow 1. Go to "my documents" on old user. (house icon) 2. Select the queries you would like to export. If they're in a folder, you can export the whole folder or use "cntrl click" to select multiple queries. Another option is to drag select them. 3. Click the "download" icon on the action bar at the top of the file browser. This will download the queries as a json file or multiple json files. 4. Log into the new account and go to "my documents" house icon. 5. Click the "upload" icon in the action bar. 6. Select a json file in the file browser. json files can only be uploaded one at a time. 7. Click import to finish the process. ... View more

Howdy,   Josh here, from Cloudera. Thanks for reaching out on this.   As far as verifying whether or not your outlined configuration would work, the short answer would be perhaps.   You might have already seen it, but I'll point to this blog post as a reference. It's a good read, and includes a matrix for deciding the specs for your cluster's nodes. If you look, you'll see the configuration you're proposing is in the neighborhood of a " Light Processing Configuration", but for every other configuration listed, it starts to fall short. As long as you don't make a fully stacked cluster with every service imaginable(it seems like you don't intend on doing that), the "Light Processing" config could suffice. You can also check out this other community post to get a better idea of how speccing your cluster could pan out in terms of how many nodes you would want.    So, in short, perhaps. Let me know if this helps or if you have any other questions.   Cheers, Josh ... View more

Howdy,   Thanks for your question. It can be quite jarring to see two columns in your output when du normally only has one column, but fear not, there is an explanation for everything. 🙂   I found a similar post discussing this topic. From looking at it, it's clear that it took some digging to get to the bottom of this, but if you look towards the bottom, you'll see a link to the source code explaining the output format. Eureka!   Anywho, the output states that the first two columns are formatted like this: [size]     [disk space consumed]   but what does this mean? the " size " field is the base size of the file or directory before replication. As we know, HDFS replicates files, so the second field( disk space consumed) is included to show you how much total disk space that file or directory takes up after it's been replicated. Under the default replication factor of three, the first two columns of a 1MB file would theoretically look like this.   1     3  M   The fun part is that we can actually use this info to infer the replication factor HDFS is using for these particular files, or at least the amount of replication the file is curently at. If you look at the first line of your output, you'll see the initial size as 816 and the disk space usage as 1.6 K. Divide 1.6K by 816 bytes , and you get 2 (roughly), which would indicate a replication factor of two, and you'll notice this math is consistent with the other entries in the output. Good times.   Armed with this knowledge, you can now use du tool to its full potential, both for informative and troubleshooting purposes. Let me know if this info was helpful or if you have any other questions. 🙂   Cheers ... View more

Howdy,   Thanks for your question. This similar post m a y b e u s e f u l f o r y o u r s i t u a t i o n. It indicates that it's important to spin up the VM via the "import appliance" option rather than making a blank VM and pointing it to the image.    If you haven't already, try starting the VM with the "import appliance option". Here's a nifty guide for doing it .   If that doesn't work, our community knowledge area also has a useful article outlining some common issues to consider.  It also wouldn't hurt to try updating your installation of Virtualbox , as it seems to be a couple versions behind. Let me know how it goes and if you need further assistance.   Cheers ... View more

Howdy,   I'm just going to jump in and give you as much info as possible, so strap in. There's going to be a lot of (hopefully helpful) info.   Before I get started, and I state this toward the end too, it’s important to know that all of this info is general “big picture” stuff, and there are a ton of factors that go into speccing your cluster (use cases, future scaling considerations, etc). I cannot stress this enough. That being said, let’s dig in.   I'm going to answer your questions in order. In short, yes. we generally recommend bare metal(“node” = physical box) for production clusters. You can get away with using VMs on a hypervisor for development clusters or POCs, but that’s not recommended for production clusters.  If you don’t have the resources for a bare metal cluster, it’s generally a better idea to deploy it in the cloud. For cloud based clusters, I recommend Cloudera Director , which allows you to deploy cloud based clusters that are configured with hadoop performance in mind.   2. It's not simply a question of how many nodes, but what the specs of each node are. We have some good documentation here that explains best practices for speccing your cluster hardware. The amount of nodes depends on what your workload will be like. This includes how much data you'll be ingesting, how often you'll be ingesting it, and how much you plan on processing said data. That being said, Cloudera Manager makes it super easy to scale out as you're workload grows. I would say the bare minimum is  5 nodes (2 masters, 3 workers) You can always scale out from there by adding additional worker and master nodes.   3 and 4. These can be answered with this nifty diagram (memory recommendations are RAM):         This comes from our article on how to deploy clusters like a boss , which covers quite a bit. Additional info on the graphic can be found toward the bottom of the article. If you look at the diagram, you'll notice a few things:  - The concept of master nodes, worker nodes, and edge nodes. Master nodes have master services like namenode service, resource manager, zookeeper, journal nodes, etc. If the service keeps track of tasks, marks changes, or has the term "manager" in it, you usually want it on a master node. You can put a good amount on single nodes because they don't do too much heavy lifting.  - The placement of DB dependent services. Note that cloudera manager, hue, and all servers that reference a metastore are installed on the master node with an RDBMS installed. You don't have to set it up this way, but it does make logical sense, and is a little more tidy. You will have to consider adding a dedicated RDBMS server eventually, because having it installed on a master node with other servers can easily cause a bottleneck when you’ve scaled enough.  - The worker node(s). this diagram only has one worker node, but it’s important to know that you should have at least three worker nodes for your cluster to function properly, as the default replication factor for HDFS is three. From there, you can add as many worker nodes as your workload dictates. At its base, you don't need many services on a worker node, but what you do need is a lot more memory, because these nodes are where data is stored in HDFS and where the heavy processing will be done.  -The edge node. It's specced similarly, or even lower, than master nodes, and is only really home to gateways and other services that communicate with the outside world. You could add these services to another master node, but it's nice to have one dedicated, especially if you plan on having folks access the cluster externally.   The article also has some good info on where to go with these services as you scale your cluster out further.   One more note. If this is a Proof of Concept cluster, I recommend saving sentry for when you put the cluster into production. When you do add, do note it’s a service that uses an RDBMS. Some parting thoughts: When you're planning a cluster, it's important to stop and evaluate exactly what your goal is for said cluster. My recommendation is to start only with the services you need to get the job done. You can always add and activate services later through cloudera manager. If you need any info on each particular service and whether or not you really need it, check out the below links to our official documentation: Hive Pig (apache documentation) Zookeeper HDFS Hue Oozie Sqoop and sqoop2 Yarn Sentry And for that matter, you can search through our documentation here .   While this info helps with general ideas and “big picture” topics, you need to consider a lot more info about your planned usage and vision to come up with an optimal setup. Use cases are vitally important to consider when speccing a cluster , especially for production. That being said, you’re more than welcome to get in touch with one our solutions architects to figure out the best configuration for your cluster. Here’s a link to some more info on that.   This is a lot of info, so feel free to take your time digesting it all. Let me know if you have any questions. 🙂   Cheers   ... View more