This year they were stored in a box on my shelf.
Will report back in 1 more year when I test the fourth 🙂
One of my archiving and backup contingencies is taking one screenshot per minute. You can also use this to get a good idea of how you spend your day, by turning it into a movie. Although with a tiling window manager like I use, it’s a headache to watch.
I send the screenshots over to another machine for storage, so they’re not cluttering my laptop. It uses up 10-20GB per year.
I’ll go over my exact setup below in case anyone is interested in doing the same:
/bin/screenlog
GPG_KEY=Zachary
TEMPLATE=/var/screenlog/%Y-%m-%d/%Y-%m-%d.%H:%M:%S.jpg
export DISPLAY=:0
export XAUTHORITY=/tmp/XAuthority
IMG=$(\date +$TEMPLATE)
mkdir -p $(dirname "$IMG")
scrot "$IMG"
gpg --encrypt -r "$GPG_KEY" "$IMG"
shred -zu "$IMG"
The script
/tmp/XAuthority. It doesn’t need any unusual permissions, but the default location has some random characters in it./etc/cron.d/screenlog
* * * * * zachary /bin/screenlog
20 * * * * zachary rsync --remove-source-files -r /var/screenlog/ backup-machine:/data/screenlog/laptop
30 * * * * zachary rmdir /var/screenlog/*
That’s
~/.profile
export XAUTHORITY=/tmp/XAuthority
I mentioned /bin/screenlog needs to know where XAuthority is. In Arch Linux this is all I need to do.
qr-backup is a program to back up digital documents to physical paper. Restore is done with a webcam, video camera, or scanner. Someday smart phone cameras will work.
I’ve been making some progress on qr-backup v1.1. So far I’ve added:
--restore, which does a one-step restore for you, instead of needing a bash one-line restore process--encrypt provides password-based encryption--instructions to give more fine-tuned control over printing instructions. There’s a “plain english” explanation of how qr-backup works that you can attach to the backup.--note for adding an arbitrary message to every sheetv1.1 will be released when I make qr-backup feature complete:
v1.2 will focus on adding a GUI and support for Windows, Mac, and Android. Switching off zbar is a requirement to allow multi-platform support, and will likely improve storage density.
I made a new project called qr-backup. It’s a command-line program to back up any file to physical paper, using a number of QR codes. You can then restore it, even WITHOUT the qr-backup program, using the provided instructions.
I’m fairly satisfied with its current state (can actually back up my files, makes a PDF). There’s definitely some future features I’m looking forward to adding, though.
My current project is to archive git repos, starting with all of github.com. As you might imagine, size is an issue, so in this post I do some investigation on how to better compress things. It’s currently Oct, 2017, for when you read this years later and your eyes bug out at how tiny the numbers are.
Let’s look at the list of repositories and see what we can figure out.
That’s about everything I can get from the repo names. Next, I downloaded all repos named dotfiles. My goal is to pick a compression strategy for when I store repos. My strategy will include putting repos with the name name on the same disk, to improve deduplication. I figured ‘dotfiles’ was a usefully large dataset, and it would include interesting overlap–some combination of forks, duplicated files, similar, and dissimilar files. It’s not perfect–for example, it probably has a lot of small files and fewer authors than usual. So I may not get good estimates, but hopefully I’ll get decent compression approaches.
Here’s some information about dotfiles:
Even though these numbers may not be representative, let’s use them to get some ballpark figures. If each repo had 600 objects, and there are 68.6 million repos on github, we would expect there to be 56 billion objects on github. At an average of 8,884 bytes per object, that’s 498TB of git objects (164TB on disk). At 40 bytes per hash, it would also also 2.2TB of hashes alone. Also interesting is that files represent 97% of storage–git is doing a good job of being low-overhead. If we pushed things, we could probably fit non-files on a single disk.
Dotfiles are small, so this might be a small estimate. For better data, we’d want to randomly sample repos. Unfortunately, to figure out how deduplication works, we’d want to pull in some more repos. It turns out picking 1000 random repo names gets you 5% of github–so not really feasible.
164TB, huh? Let’s see if there’s some object duplication. Just the unique objects now:
All right, that’s 2:1 disk savings over the existing compression from git. Not bad. In our imaginary world where dotfiles are representative, that’s 82TB of data on github (1.2TB non-file objects and 0.7TB hashes)
Let’s try a few compression strategies and see how they fare:
Throwing out everything but the objects allows other fun options, but there aren’t any standard tools and I’m out of time. Maybe next time. Ta for now.
This one’s a quickie. Just a second of my config to record all bash commands to a file (.bash_eternal_history) forever. The default bash HISTFILESIZE is 500. Setting it to a non-numeric value will make the history file grow forever (although not your actual history size, which is controlled by HISTSIZE).
I do this in addition:
#~/.bash.d/eternal-history
# don't put duplicate lines in the history
HISTCONTROL=ignoredups
# append to the history file, don't overwrite it
shopt -s histappend
# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTFILESIZE=infinite
# Creates an eternal bash log in the form
# PID USER INDEX TIMESTAMP COMMAND
export HISTTIMEFORMAT="%s "
PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND ; }"'echo $$ $USER \
"$(history 1)" >> ~/.bash_eternal_history'
Today I’m going to walk through a setup on how to archive all web (HTTP/S) traffic passing over your Linux desktop. The basic approach is going to be to install a proxy which records traffic. It will record the traffic to WARC files. You can’t proxy non-HTTP traffic (for example, chat or email) because we’re using an HTTP proxy approach.
The end result is pretty slow for reasons I’m not totally sure of yet. It’s possible warcproxy isn’t streaming results.
# pip install warcproxy
# useradd -M --shell=/bin/false warcprox
# mkdir /etc/warcprox
# cd /etc/warcprox
# sudo openssl genrsa -out ca.key 409
# sudo openssl req -new -x509 -key ca.key -out ca.crt
# cat ca.crt ca.key >ca.pem
# chown root:warcprox ca.pem ca.key
# chmod 640 ca.pem ca.key
# mkdir /var/warcprox # chown -R warcprox:warcprox /var/warcprox
#/etc/supervisor.d/warcprox.ini [program:warcprox] command=/usr/bin/warcprox -p 18000 -c /etc/warcprox/ca.pem --certs-dir ./generated-certs -g sha1 directory=/var/warcprox user=warcprox autostart=true autorestart=unexpected
Install jq and youtube-dl
Get a list of the last 100 URLs:
curl https://api.twitch.tv/kraken/channels/${TWITCH_USER}/videos?broadcasts=true&limit=100 |
jq -r '.videos[].url' > past_broadcasts.txt
Save them locally:
youtube-dl -a past_broadcasts.txt -o "%(upload_date)s.%(title)s.%(id)s.%(ext)s"
Did it. youtube-dl is smart enough to avoid re-downloading videos it already has, so as long as you run this often enough (I do daily), you should avoid losing videos before they’re deleted.
Thanks jrayhawk for the API info.
Previous work:
I wanted (for fun) to see if I could get data stored in paper formats. I’d read the previous work, and people put a lot of thought into density, but not a lot of thought into ease of retreival. First off, acid-free paper lasts 500 years or so, which is plenty long enough compared to any environmental stresses (moisture, etc) I expect on any paper I have.
Optar gets a density of 200kB / A4 page. By default, it requires a 600dpi printer, and a 600+dpi scanner. It has 3-of-12 bit redundancy using Golay codes, and spaces out the bits in an okay fashion.
Paperback gets a (theoretical) density of 500kB / A4 page. It needs a 600dpi printer, and a ~900dpi scanner. It has configurable redundancy using Reed-Solomon codes. It looks completely unusable in practice (alignment issues, aside from being Windows-only).
Okay, so I think these are all stupid, because you need some custom software to decode them, which in any case where you’re decoding data stored on paper you probably don’t have that. I want to use standard barcodes, even if they’re going to be lower density. Let’s look at our options. I’m going to skip linear barcodes (low-density) and color barcodes (printing in color is expensive). Since we need space between symbols, we want to pick the biggest versions of each code we can. For one, whitespace around codes is going to dominate actual code density for layout efficiency, and larger symbols are usually more dense. For another thing, we want to scan as few symbols as possible if we’re doing them one at a time.
Aztec From 15×15 to 151×151 square pixels. 1914 bytes maximum. Configurable Reed-Solomon error correction.
Density: 11.9 pixels per byte
Data Matrix From 10×10 to 144×144 square pixels. 1555 bytes maximum. Large, non-configurable error correction.
Density: 13.3 pixels per byte
QR Code From 21×21 to 177×177 square pixels. 2,953 bytes maximum. Somewhat configurable Reed-Solomon error correction.
Density: 10.6 pixels per byte
PDF417 17 height by 90-583 width. 1100 bytes maximum. Configurable Reed-Solomon error correction. PDF417 is a stacked linear barcode, and can be scanned by much simpler scanners instead of cameras. It also has built in cross-symbol linking (MacroPDF417), meaning you can scan a sequence of codes before getting output–handy for getting software to automatically link all the codes on a page.
Density: 9.01 pixels per byte
QR codes and PDF417 look like our contenders. PDF417 turns out to not scan well (at all, but especially at large symbol sizes), so despite some nice features let’s pick QR codes. Back when I worked on a digital library I made a component to generate QR codes on the fly, and I know how to scan them on my phone and webcam already from that, so it would be pretty easy to use them.
What density can we get on a sheet of A4 paper (8.25 in × 11.00 in, or 7.75in x 10.50in with half-inch margins)? I trust optar’s estimate (600 dpi = 200 pixels per inch) for printed/scanned pages since they seemed to test things. A max-size QR code is 144×144 pixels, or 0.72 x 0.72 inches at maximum density. We can fit 10 x 14 = 140 QR codes with maximum density on the page, less if we want decent spacing. That’s 140 QR codes x (2,953 bytes per QR code) = 413420 bytes = 413K per page before error correction.
That’s totally comparable to the other approaches above, and you can read the results with off-the-shelf software. Bam.
In a previous post I discussed how to backup android with rsync. In this post, I’ll improve on that solution so it happens when you plug the phone in, rather than manually. My solution happens to know I have only one phone; you should adjust accordingly.
The process is
First, let’s add a udev rule to auto-mount the phone when it’s plugged in and unlocked, and run appropriate scripts.
# 10-android.rules
ACTION=="add", SUBSYSTEM=="usb", ATTR{idVendor}=="18d1", ATTR{idProduct}=="4ee2", MODE="0660", GROUP="plugdev", SYMLINK+="android", RUN+="/usr/local/bin/android-connected"
ACTION=="remove", SUBSYSTEM=="usb", ENV{ID_MODEL}=="Nexus_4", RUN+="/usr/local/bin/android-disconnected"
Next, we’ll add android-connected and android-disconnected
#!/bin/bash # /usr/local/bin/android-connected if [[ "$1" != "-f" ]] then echo "/usr/local/bin/android-connected -f" | /usr/bin/at now exit 0 fi sudo -u zachary DISPLAY=:0 /usr/bin/notify-send "Android plugged in, please unlock." sudo -u zachary /usr/local/bin/android-mountfs sudo -u zachary DISPLAY=:0 /usr/bin/notify-send "Mounted, backing up..." /usr/bin/flock /var/lock/phone-backup.pid sudo -u zachary /usr/local/bin/phone-backup-xenu sudo -u zachary DISPLAY=:0 /usr/bin/notify-send "Backup completed."
# !/bin/sh # /usr/local/bin/android-disconnected #!/bin/sh sudo -u zachary DISPLAY=:0 /usr/bin/notify-send "Android unplugged." sudo -u zachary /usr/local/bin/android-umountfs
We’ll add something to mount and unmount the system. Keeping in mind that mounting only works when the screen is unlocked we’ll put that in a loop that checks if the mount worked:
#!/bin/sh
# /usr/local/bin/android-mountfs
android_locked()
{
ls /media/android 2>/dev/null >/dev/null
[ "$?" -eq 2 ]
}
jmtpfs /media/android # mount
while android_locked; do
fusermount -u /media/android
sleep 3
jmtpfs /media/android # mount
done
#!/bin/sh # /usr/local/bin/android-umountfs fusermount -u /media/android
The contents of /usr/local/bin/phone-backup are pretty me-specific so I’ll omit it, but it copies /media/android over to a server. (fun detail: MTP doesn’t show all information even on a rooted phone, so there’s more work to do)