Tuesday, February 26, 2013

Raspberry Pi as a transparent squid caching proxy

Developing Openstack Heat means spending a fair amount of time building and customizing bootable cloud images. A lot of this time is spent waiting for RPMs, debs and tarballs to be downloaded by a vanilla guest OS running inside a VM. And given that I work from home with an average broadband connection in a remote country in the South Pacific, the result is some frustrating wait times.

Since the same packages are often being repeatedly downloaded, I would benefit from some local caching. This seemed like a good excuse to use a Raspberry Pi. I went with a Raspberry Pi B running Raspbian. The aim was to set it up as a bridge and run a transparent squid proxy between eth0 (the inbuilt network interface) and eth1 (a USB ethernet dongle).

Once I'd completed the initial installation, I installed the following:
$ apt-get install squid3 bridge-utils

eth0 and eth1 were set up to bridge on my network, and an iptables rule was set to direct any port 80 traffic that passes through the bridge to squid's default port.

The following changes were made to the squid configuration file. Since I'm interested in caching larger files the maximum_object_size has been set to 512MB. My Raspberry Pi is running on a 16GB SD card; for now I have configured cache_dir to use 8GB of that.

And did this actually help my image building time? Using diskimage-builder I ran an Ubuntu customization where the source image file was already cached locally. The first run populated the squid cache with apt repository packages and the second run had a hot squid cache. The build time went from (mm:ss) 04:20 to 01:20 which I'm pretty happy with.

Doing the same with heat-jeos (which is based on oz) managed to get some cache hits on the second run, but had little impact on the (mm:ss) 22:30 build time.


Günter said...

It looks to me that you updated your kernel to include iptables. I installed the wheezy image and it does not include iptables.

Unknown said...

Is there any particular reason you choose to create a bridge interface for intercepting? Rather than simply redirecting all traffic as most people do with two interfaces? What is the benefit of doing it that way, and if so, what am I missing out on?

Heres how I do it usually.

-A PREROUTING -i eth5 -p tcp -m tcp --dport 80 -j DNAT --to-destination
-A PREROUTING -i eth5 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3128

Brad Nightingale said...

Interesting post. Have you considered using WCCP on the CISCO and setting up squid as an intercepting proxy? Do you think this is a viable idea? Also what was your performance like in the end? Any hit on this?

Squidblacklist said...

Squidblacklist.org is the worlds leading publisher of native acl blacklists tailored specifically for Squid proxy, and alternative formats for all major third party plugins as well as many other filtering platforms. Including SquidGuard, DansGuardian, and ufDBGuard, as well as pfSense and more.

There is room for better blacklists, we intend to fill that gap.

It would be our pleasure to serve you.


Benjamin E. Nichols

Random Ponderings said...

Yearss ago inthe 56k modem era I used JANA to give my loal network a common cache and it taught me two things :

1) the quality for browser caching has sucked more and more as faster connects and unlkmited bandwidth became more common .... western - especially US - developers tend to act as if all the word has unlimited T1 acces

2) every router sold ought to come with a cache

Unknown said...

jual bantal jual bantal jual bantal jual bantal jual bantal jual bantal jual bantal jual bantal jual bantal

sangkar jual sangkar jual sangkar jual sangkar jual sangkar jual sangkar jual sangkar jual

cd anne jelita brenda anne caca tasya ruth
bantal web bantal bantal silikon pembicara hellow
bumbu bumbu bumbu bumbu bumbu distro distro

Unknown said...

Hi Steve, thanks for this blog post. I applied it to a Raspberry Pi B and it works as described.

Kind regards

S Kris said...

Hi Steve

Sometimes being in a much smaller South Pacific island then you, I need a web caching device like this.

I understand the tech re 2 NICs one internal one external. The external (USB) NIC I'd connect to my router. The internal NIC I'd connect to a LAN switch so the 4 PCs in my home office will grab their dynamic IP address off the Raspberry and then benefit from the Squid cache on the Raspberry since it becomes their gateway. Perhaps the cache can be a USB flash drive instead, so we could easily upgrade to a larger one for Windows Upgrades (which are getting ever larger in size!) if needed.

The problem is I have NO clue on how to get this started.

Is there any chance you could configure an SD card ready-to-go I could purchase from you?




Unknown said...

Thank you for another fantastic article. Where else may just anybody get that kind of information in such a perfect means of writing? I've a presentation subsequent week, and I'm on the look for such info. facebook login in

Lynna Conner said...

This article is an appealing wealth of informative data that is interesting and well-written. I commend your hard work on this and thank you for this information. You’ve got what it takes to get attention. visit website

Matias said...

It was a very good post indeed. I thoroughly enjoyed reading it in my lunch time. Will surely come and visit this blog more often. Thanks for sharing. privacyonline

Mona martin said...

I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. https://getmoreprivacy.com/

James harper said...

They’re also cast as heels, allegedly crafted due to the McMahon family’s belief that the far right cost Linda the election. A tag feud between these two seems a natural fit, with Darren Young cast as the good guy. lemigliorivpn

Ethan Ryan said...

I needed to thank you for this phenomenal read!! I unquestionably adored each and every piece of it. I have you bookmarked your site to look at the new stuff you post. vpnveteran

Michael Smith said...

Particular interviews furnish firsthand message on mart size, industry trends, ontogeny trends, capitalist landscape and outlook, etc. meer informatie

Aaron Tyler said...

You guys are writing some Amazing tips. Thanks for sharing this. Totally Awesome Post Please Keep Posting Regularly.
echobeat earbuds review, chargeboost reviews, liporing review , doc socks, livewave antenna review

Editor said...

Thank you for sharing this Information.
I also found Various useful links related to Devops, Docker & Kubernetes

Kubernetes Kubectl Commands CheatSheet

Introduction to Kubernetes Networking

Basic Concept of Kubernetes

Kubernetes Interview Question and Answers

Kubernetes Sheetsheat

Docker Basic Tutorial

Linux Sar Command Tutorial

Linux Interview Questions and Answers

Docker Interview Question and Answers

OpenStack Interview Questions and Answers