Tuesday, February 26, 2013

Raspberry Pi as a transparent squid caching proxy

Developing Openstack Heat means spending a fair amount of time building and customizing bootable cloud images. A lot of this time is spent waiting for RPMs, debs and tarballs to be downloaded by a vanilla guest OS running inside a VM. And given that I work from home with an average broadband connection in a remote country in the South Pacific, the result is some frustrating wait times.

Since the same packages are often being repeatedly downloaded, I would benefit from some local caching. This seemed like a good excuse to use a Raspberry Pi. I went with a Raspberry Pi B running Raspbian. The aim was to set it up as a bridge and run a transparent squid proxy between eth0 (the inbuilt network interface) and eth1 (a USB ethernet dongle).

Once I'd completed the initial installation, I installed the following:
$ apt-get install squid3 bridge-utils

eth0 and eth1 were set up to bridge on my network, and an iptables rule was set to direct any port 80 traffic that passes through the bridge to squid's default port.

The following changes were made to the squid configuration file. Since I'm interested in caching larger files the maximum_object_size has been set to 512MB. My Raspberry Pi is running on a 16GB SD card; for now I have configured cache_dir to use 8GB of that.

And did this actually help my image building time? Using diskimage-builder I ran an Ubuntu customization where the source image file was already cached locally. The first run populated the squid cache with apt repository packages and the second run had a hot squid cache. The build time went from (mm:ss) 04:20 to 01:20 which I'm pretty happy with.

Doing the same with heat-jeos (which is based on oz) managed to get some cache hits on the second run, but had little impact on the (mm:ss) 22:30 build time.


Günter said...

It looks to me that you updated your kernel to include iptables. I installed the wheezy image and it does not include iptables.

Unknown said...

Is there any particular reason you choose to create a bridge interface for intercepting? Rather than simply redirecting all traffic as most people do with two interfaces? What is the benefit of doing it that way, and if so, what am I missing out on?

Heres how I do it usually.

-A PREROUTING -i eth5 -p tcp -m tcp --dport 80 -j DNAT --to-destination
-A PREROUTING -i eth5 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 3128

Brad Nightingale said...

Interesting post. Have you considered using WCCP on the CISCO and setting up squid as an intercepting proxy? Do you think this is a viable idea? Also what was your performance like in the end? Any hit on this?

Squidblacklist said...

Squidblacklist.org is the worlds leading publisher of native acl blacklists tailored specifically for Squid proxy, and alternative formats for all major third party plugins as well as many other filtering platforms. Including SquidGuard, DansGuardian, and ufDBGuard, as well as pfSense and more.

There is room for better blacklists, we intend to fill that gap.

It would be our pleasure to serve you.


Benjamin E. Nichols

Random Ponderings said...

Yearss ago inthe 56k modem era I used JANA to give my loal network a common cache and it taught me two things :

1) the quality for browser caching has sucked more and more as faster connects and unlkmited bandwidth became more common .... western - especially US - developers tend to act as if all the word has unlimited T1 acces

2) every router sold ought to come with a cache

Unknown said...

jual bantal jual bantal jual bantal jual bantal jual bantal jual bantal jual bantal jual bantal jual bantal

sangkar jual sangkar jual sangkar jual sangkar jual sangkar jual sangkar jual sangkar jual

cd anne jelita brenda anne caca tasya ruth
bantal web bantal bantal silikon pembicara hellow
bumbu bumbu bumbu bumbu bumbu distro distro

Unknown said...

Hi Steve, thanks for this blog post. I applied it to a Raspberry Pi B and it works as described.

Kind regards

S Kris said...

Hi Steve

Sometimes being in a much smaller South Pacific island then you, I need a web caching device like this.

I understand the tech re 2 NICs one internal one external. The external (USB) NIC I'd connect to my router. The internal NIC I'd connect to a LAN switch so the 4 PCs in my home office will grab their dynamic IP address off the Raspberry and then benefit from the Squid cache on the Raspberry since it becomes their gateway. Perhaps the cache can be a USB flash drive instead, so we could easily upgrade to a larger one for Windows Upgrades (which are getting ever larger in size!) if needed.

The problem is I have NO clue on how to get this started.

Is there any chance you could configure an SD card ready-to-go I could purchase from you?




Unknown said...

Thank you for another fantastic article. Where else may just anybody get that kind of information in such a perfect means of writing? I've a presentation subsequent week, and I'm on the look for such info. facebook login in

Lynna Conner said...

This article is an appealing wealth of informative data that is interesting and well-written. I commend your hard work on this and thank you for this information. You’ve got what it takes to get attention. visit website

Matias said...

It was a very good post indeed. I thoroughly enjoyed reading it in my lunch time. Will surely come and visit this blog more often. Thanks for sharing. privacyonline

Mona martin said...

I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site. https://getmoreprivacy.com/

James harper said...

They’re also cast as heels, allegedly crafted due to the McMahon family’s belief that the far right cost Linda the election. A tag feud between these two seems a natural fit, with Darren Young cast as the good guy. lemigliorivpn

Ethan Ryan said...

I needed to thank you for this phenomenal read!! I unquestionably adored each and every piece of it. I have you bookmarked your site to look at the new stuff you post. vpnveteran

Michael Smith said...

Particular interviews furnish firsthand message on mart size, industry trends, ontogeny trends, capitalist landscape and outlook, etc. meer informatie

Aaron Tyler said...

You guys are writing some Amazing tips. Thanks for sharing this. Totally Awesome Post Please Keep Posting Regularly.
echobeat earbuds review, chargeboost reviews, liporing review , doc socks, livewave antenna review

Editor said...

Thank you for sharing this Information.
I also found Various useful links related to Devops, Docker & Kubernetes

Kubernetes Kubectl Commands CheatSheet

Introduction to Kubernetes Networking

Basic Concept of Kubernetes

Kubernetes Interview Question and Answers

Kubernetes Sheetsheat

Docker Basic Tutorial

Linux Sar Command Tutorial

Linux Interview Questions and Answers

Docker Interview Question and Answers

OpenStack Interview Questions and Answers

CORONA 19 said...

This Website is great for group chat and conversation. Thanks for sharing with us. like this website so much it's really awesome.I have also gone through your other posts too and they are also very much appreciate able and I'm just waiting for your next update to come as I like all your posts.

The Other Track
Radiation Stopper Pro
Top gadgets to buy in 2020
Q Grip Wax remover
UV Clenizer Zoom Disinfectant robot
X watch smartwatch
Keysmart key oranizer

CORONA 19 said...

This Website is great for group chat and conversation. Thanks for sharing with us. like this website so much it's really awesome.I have also gone through your other posts too and they are also very much appreciate able and I'm just waiting for your next update to come as I like all your posts.

Best Gadget Review
DartleType Laser Keyboard Review
sleepconnection anti snore review 2020

Anonymous said...

Lucky Luke - Up to £300 Welcome Bonus - THTOPbet
Lucky Luke - happyluke Up to £300 Welcome Bonus – Trusted casino game providers. We have the 188bet best games & jackpots in the UK, bet365 with every one listed here at TopBet.

abbiegailbach said...

Casino - MapYRO
Find your perfect blend of adventure and 목포 출장샵 Vegas. 하남 출장샵 포커 고수 Casino at 공주 출장안마 Bally's - Bally's Hotel and Casino, Las Vegas, 대구광역 출장안마 NV. Casino at Bally's Hotel and Casino.

Unknown said...

Casino Bonus Codes - December 2021
No deposit bonus casino promotions. We recommend 2021 바카라 사이트 casino bonus great air jordan 7 shoes Shipping Online codes and promos for new players. We also 바카라 사이트 list new High Quality jordan 15 retro casino bonuses for December Wholesale jordan 11 retro 2021.

Anonymous said...

You can play all types of card and desk game variations and stay on line casino games change frequently on the site. Players won't be disappointed at the range of games and 1xbet korea other options at the on line casino. Justspin Casino is all about ongoing deposit bonuses and free spins to get you started.

rRPxuutAGPnG said...

They respect playing guidelines and age restrictions, offering a superb actual money gaming experience in a safe surroundings devoted to players' welfare and safety online. I requested Eyal what distinguishes mobile games or courting apps from slot machines. 88 Fortunes® 바카라사이트 is the perfect slot experience for players to check their luck.