Hi @tja  Other community members may weigh in with their opinions, but I believe the answer to the first and last questions is, of course, "it depends on the job". The most suitable use cases for Sqoop center on bulk structured data transfer between RDBMSs and HDFS. Sqoop takes the commands you provide at the CLI and internally generates MapReduce tasks to execute your desired data movement, with HDFS as either the source or destination.   While you can do some of the same types of (simple) things with either Spark or Sqoop, Spark and Sqoop are not interchangeable tools. You can do a lot more with Spark than you can with Sqoop because Spark gives you a full-blown programming language (Scala) along with a set of libraries that support a fairly complete distributed processing framework. The "T" part of ETL is going to be a lot easier to tackle in Spark than by using Sqoop and you will probably encounter tasks that are nearly impossible to complete with Sqoop that are fairly straightforward to address in Spark code, assuming that you have the requisite software development background.   While I have not done any performance comparisons between a batch job in Sqoop to an equivalent job written in Spark (and I haven't read anybody else's work on that topic), just engaging in a bit of logical deduction from first principles would lead me to expect a considerable performance advantage in using Spark over Sqoop import (given a sufficiently large data set) because in Spark you can leverage in-memory processing capabilities which should, in theory, out perform MapReduce.   Yes, unfortunately the Sqoop PMC voted in June to retire Sqoop and move the responsibility for its oversight to the Attic. That does not mean that Apache Sqoop as a tool has lost all value. Cloudera still ships it as part of Cloudera Runtime, still fully supports Sqoop and responds to new feature requests coming from customers, and there's no plan to change this. The change in status at Apache could mean that the software has reached maturity "as is" and still has its uses. But end-user development of a complete new ETL pipeline is probably not one of them.     ... View more

@jiyo  The explanation provided earlier by @Shelton isn't wrong, but I thought I would follow-up and provide some context I think would be helpful.   In your original question, you wrote that you "got an email saying that the username is my Google email address." This email likely was referring to the username you would use to log into the Cloudera Community, not the username you would use to access Cloudera's private repositories where the binaries for Cloudera's distributions of Hadoop and/or Ambari are now located. As he pointed out and hopefully you are now aware, Cloudera modified its download policies and the binaries you are seeking to download are now only available in a private repository. If not, please see the announcement here: Transition to private repositories for CDH, HDP and HDF.   The credentials to access this private repository are not generally the same ones to access Cloudera's website or the Cloudera community. The same announcement describes new patch releases of Ambari which are required to access Cloudera’s private repositories, which now contain these new and existing releases.   The reason you're getting the error message from the invocation of the wget command is that the credentials to use to access the aforementioned private repositories does not depend upon your Google email address, and the command is never actually getting to the point of accessing the host archive.cloudera.com. The HTTP 301 redirects you're seeing are responses from one of google's web servers, and not Cloudera's.     ... View more

@jiyo    Maybe. You wrote that "it doesn't work properly because of "@" in username". Why do you say that "it doesn't work properly"? What is the exact error message you're receiving?     ... View more

Hi @Cisco94 , Yes, we are no longer making the Cloudera Quickstart VM available for download (and haven't since March of 2020) because it was old and outdated as it was based on CDH 5.13, which went out of support in the Fall of last year. If you're a registered Cloudera partner, you can probably open a request with your partner contact at Cloudera and still get a copy of it.   I'm curious as to why anyone would ask you to do a demo of CDH at this point in time, or why you are interested in showing your client a distribution which does not include the up-to-date releases of the various Hadoop ecosystem components. Cloudera's current distribution, since the Fall of 2020, is Cloudera Data Platform (or CDP); a Trial Version of CDP Private Cloud Base Edition of Cloudera Data Platform can easily be downloaded and installed from the "downloads" section of Cloudera's website.     ... View more

Hi @M_Shash  You didn't indicate how/where you downloaded and installed the software you used to originally establish your "Hortonworks Data Flow cluster managed by Ambari" from, but your installation of Ambari probably isn't set up to supply the authentication credentials. As you've indicated you are aware, earlier this year Cloudera announced new versions of Ambari which have the ability to access Cloudera’s private repositories for HDP/HDF installation and upgrade. Please see the announcement here: Transition to private repositories for CDH, HDP and HDF.   The same announcement has extensive links to documentation on installing/upgrading HDF using Cloudera’s private repository.   If this is for a new cluster, you should seriously consider upgrading to 3.5.2. HDF 3.5.2 is the last version of HDF that Cloudera will provide, and Cloudera requires customers to be on HDF 3.5.1 or HDF 3.5.2 before migrating to CFM on CDP.     ... View more

Hi @roshanbi  Just perusing the Hue user documentation, I read this: Scheduler: The application lets you build workflows and then schedule them to run regularly automatically. A monitoring interface shows the progress, logs and allow actions like pausing or stopping jobs.   ... View more

Hi @Exor , I am guessing that the key part of the log for this issue is this:     2021-06-15 08:39:14,498 INFO NodeConfiguratorThread-6-0:com.cloudera.server.cmf.node.NodeConfigurator: Using key bundle from URL: https://archive.cloudera.com/cm6/6.0.1/allkeys.asc 2021-06-15 08:39:14,960 INFO NodeConfiguratorThread-6-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: node5: Setting COPY_FILES as failed and done state       The log doesn't indicate it, but the server archive.cloudera.com is probably returning an HTTP 401 error because authentication is required. You didn't provide the version of Cloudera Manager you're using, but I would guess that the last time you attempted this operation, you were not challenged for authentication by this particular host at Cloudera and so you're wondering what changed recently.   The answer is probably that your installation of Cloudera Manager isn't set up to supply the authentication credentials. Earlier this year, Cloudera announced new versions of Cloudera Manager 6.x that are required to access Cloudera’s repositories. Please see the announcement here: Transition to private repositories for CDH, HDP and HDF.   The same announcement describes the new patch releases of Cloudera Manager, which are required to access Cloudera’s private repositories, which now contain the new and legacy releases and other assets such as those necessary to add a new host to an existing CDH cluster.     ... View more

It would be helpful to community members inclined to answer your question if you included which version of Hue you're using, and from which distribution (i.e., HDP, CDH or CDP) you installed it from.     ... View more