Archiving all web traffic

Today I’m going to walk through a setup on how to archive all web (HTTP/S) traffic passing over your Linux desktop. The basic approach is going to be to install a proxy which records traffic. It will record the traffic to WARC files. You can’t proxy non-HTTP traffic (for example, chat or email) because we’re using an HTTP proxy approach.

The end result is pretty slow for reasons I’m not totally sure of yet. It’s possible warcproxy isn’t streaming results.

  1. Install the server
    # pip install warcproxy
  2. Make a warcprox user to run the proxy as.
    # useradd -M --shell=/bin/false warcprox
  3. Make a root certificate. You’re going to intercept HTTPS traffic by pretending to be the website, so if anyone gets ahold of this, they can fake being every website to you. Don’t give it out.
    # mkdir /etc/warcprox
    # cd /etc/warcprox
    # sudo openssl genrsa -out ca.key 409
    # sudo openssl req -new -x509 -key ca.key -out ca.crt
    # cat ca.crt ca.key >ca.pem
    # chown root:warcprox ca.pem ca.key
    # chmod 640 ca.pem ca.key
  4. Set up a directory where you’re going to store the WARC files. You’re saving all web traffic, so this will get pretty big.
    # mkdir /var/warcprox
    # chown -R warcprox:warcprox /var/warcprox
  5. Set up a boot script for warcproxy. Here’s mine. I’m using supervisorctl rather than systemd.
    command=/usr/bin/warcprox -p 18000 -c /etc/warcprox/ca.pem --certs-dir ./generated-certs -g sha1
  6. Set up any browers, etc to use localhost:18000 as your proxy. You could also do some kind of global firewall config. Chromium in particular was pretty irritating on Arch Linux. It doesn’t respect $http_proxy, so you have to pass it separate options. This is also a good point to make sure anything you don’t want recorded BYPASSES the proxy (for example, maybe large things like youtube, etc).

Mail filtering with Dovecot

This expands on my previous post about how to set up an email server.

We’re going to set up a few spam filters in Dovecot under Debian. We’re going to use Sieve, which lets the user set up whichever filters they want. However, we’re going to run a couple pre-baked spam filters regardless of what the user sets up. Continue reading

Installing email with Postfix and Dovecot (with Postgres)

I’m posting my email setup here. The end result will:

  • Use Postfix for SMTP
  • Use Dovecot for IMAP and authentication
  • Store usernames, email forwards, and passwords in a Postgres SQL database
  • Only be accessible over encrypted channels
  • Pass all common spam checks
  • Support SMTP sending and IMAP email checking. I did not include POP3 because I don’t use it, but it should be easy to add
  • NOT add spam filtering or web mail (this article is long enough as it is, maybe in a follow-up)

Continue reading

Dependency Resolution in Javascript

Sometimes I have a bunch of dependencies. Say, UI components that need other UI components to be loaded. I’d really just like to have everything declare dependencies and magically everything is loaded in the right order. It turns out that if use “require” type files this isn’t bad (google “dependency injection”), but for anything other than code loading you’re a bit lost. I did find dependency-graph, but this requires the full list of components to run. I wanted a version would you could add components whenever you wanted–an online framework.

My take is here:

It has no requirements, and is available on npm as dependencies-online.

Archiving Twitch

Install jq and youtube-dl

Get a list of the last 100 URLs:

curl${TWITCH_USER}/videos?broadcasts=true&limit=100 | 
  jq -r '.videos[].url' > past_broadcasts.txt

Save them locally:

youtube-dl -a past_broadcasts.txt -o "%(upload_date)s.%(title)s.%(id)s.%(ext)s"

Did it. youtube-dl is smart enough to avoid re-downloading videos it already has, so as long as you run this often enough (I do daily), you should avoid losing videos before they’re deleted.

Thanks jrayhawk for the API info.

SQL views

I decided I wanted to show (restricted) data views on the web in table form. Specifically, ‘stylish.db’ is a database provided by a chrome plugin. Here’s an example script, stylish.view, which displays the contents of that. It contains a comment saying which database it’s a query on, together with the query.

-- stylish.db
SELECT style, code, GROUP_CONCAT(section_meta.value) as 'website(s)' FROM
 (SELECT AS style,
 sections.code AS code, AS sections_id
 FROM styles INNER JOIN sections ON sections.style_id =
LEFT JOIN section_meta
 ON section_meta.section_id = sections_id
GROUP BY style;

The cool part here is that none of this was specific to stylish. I can quickly throw together a .view file for any database and put it on the web. Continue reading