Reply
New Contributor
Posts: 2
Registered: ‎11-05-2013

Using CM with a local parcel repository (no internet access)

Hi

I meet an issue using CM without internet access. I created a local parcels repository following http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.6.2/Cloudera-Manager-Install....


I have a .parcel and a .parcel.sha file in /opt/cloudera/parcel-repo. I also checked that these 2 files are accessible and readable by cloudera-scm user.

I suppose however that I missed something. When I check in the CM interface under Hosts > Parcels, I do not find any thing and the only message I have is "No clusters found".

 

Moreover, if I try to install CDH using parcels, it fails because the installer try to use archive.cloudera.com repository. Maybe tthis happened because my local setting is wrong. Any idea ?

 

Thank in advance for help.

regards

Cloudera Employee
Posts: 79
Registered: ‎08-29-2013

Re: Using CM with a local parcel repository (no internet access)

Hi easyoups!

 

I find this method to be rather useful myself, as it helps rule out any unexpected permissions/owner/group on the parcel and sha files by serving up the files via HTTP and then causing Cloudera Manager to handle creation of the .parcel.sha file and putting correct permissions on them:

 

0. Obtain any .parcel files AND the full manifest.json file you require, from archive.cloudera.com. 

Examples: 

http://archive.cloudera.com/cdh4/parcels/4.4.0/CDH-4.4.0-1.cdh4.4.0.p0.39-el6.parcel

http://archive.cloudera.com/cdh4/parcels/4.4.0/manifest.json

1. Create a temporary dir from which we can serve up the files on the same node where Cloudera Manager runs, or a node within the local network that the CM server node can easily reach. My example is /tmp/parcelserve on localhost.

2. # mv CDH-4.4.0-1.cdh4.4.0.p0.39-el6.parcel manifest.json /tmp/parcelserve

3. # cd /tmp/parcelserve

4. # python -m SimpleHTTPServer 8080

   - this spins up a tiny HTTP server on localhost, listening on 8080. Change this to any nonprivileged port that's free.

5. Go to the CM UI > Parcels > Edit Settings and add a "Remote Parcel Repository URL" as http://localhost:8080. Save Changes on the page.

6. Refresh the parcel UI. If necessary click "Check For New Parcels" (CM4.6+). [IFF CM4.5.x, lower the parcel check interval from 1hr to 1min and then wait 1min].

7. The Parcel UI should now reflect CDH-4.4.0-1.cdh4.4.0.p0.39-el6.parcel available "remotely" and the Download button will be clickable. The download will happen fairly rapidly since, in my instance, it's local to the system. This then copies it administratively into /opt/cloudera/parcel-repo on the node where Cloudera Manager runs. Its permissions and .sha will be handled automatically.

 

Now, that said, it sounds like maybe you are building out a new cluster and may not even have a cluster yet defined in Cloudera Manager? If that's the case you could use this same http://localhost:8080 (or any other cluster-internal http server) as your Parcel Repository during the initial install wizard instead of having it attempt to reach out to archive.cloudera.com. Choose the radio button to provide your own URL.

 

Regards,

--

Highlighted
New Contributor
Posts: 2
Registered: ‎11-05-2013

Re: Using CM with a local parcel repository (no internet access)

Thank for your answer...

 

I tried your solution and simulated a local httpd that answered to http://archive.cloudera.com/cdh4/parcels/latest/ and so on...

It should work...

 

When I "check for new parcels", the request on the access.log of my apache server :

tail -f /var/log/httpd/access_80_log

192.168.2.201 - - [06/Nov/2013:00:11:29 +0100] "GET /cdh4/parcels/latest/manifest.json HTTP/1.1" 200 14438
192.168.2.201 - - [06/Nov/2013:00:11:30 +0100] "GET /impala/parcels/latest/manifest.json HTTP/1.1" 200 1700
192.168.2.201 - - [06/Nov/2013:00:11:30 +0100] "GET /search/parcels/latest/manifest.json HTTP/1.1" 200 2516

 

I also see a trace in of that check in /var/log/cloudera-scm-server/cloudera-scm-server.log.

2013-11-06 00:11:29,928 INFO [Parcel Update Service:parcel.ParcelUpdateService@59] Updating parcel state based on local filesystem state and remote repos
2013-11-06 00:11:29,935 INFO [Parcel Update Service:components.LocalParcelManagerImpl@105] Found files under /opt/cloudera/parcel-repo

The second message is stranger for me because I have nothing in the parcel-repo directory and no file is being downloaded

ll /opt/cloudera/parcel-repo
total 0

 

A new suggestion ?

 

Regards

Announcements