I did a survey of the cost of buying hard drives (of all sorts), CDs, DVDs, Blue-rays, and tape media (for tape drives).
Here are the 2020-01 results: https://za3k.com/archive/storage-2020-01.sc.txt
Changes this year
- I excluded Seagate drives (except where they’re the only drives in class)
- Amazon’s search got much worse, and they started having listings for refurbished drives
- Corrected paper archival density, added photographic film
- Added SSDs (both 2.5″ and M.2 formats)
- Prices did not go up or down significantly in the last 6 months.
Some conclusions that are useful to know
- The cheapest option is tape media, but tape reader/writers for LTO 6, 7, and 8 are very expensive.
- The second-cheapest option is to buy external hard drives, and then open the cases and take out the hard drives. This gives you reliable drives with no warrantee.
- Blu-ray and DVD are more expensive than buying hard drives
Most of this should apply for any linux system, other than the driver install step. Also, I believe most of it applies to DeckLink and Intensity cards as well.
My main source is https://gist.github.com/afriza/879fed4ede539a5a6501e0f046f71463. I’ve re-written for clarity and Debian.
- Set up hardware. On the Intensity Pro 4K, I see a black screen on my TV when things are set up correctly (a clear rectangle, not just nothing).
- From the Blackmagic site, download “Desktop Video SDK” version 10.11.4 (not the latest). Get the matching “Desktop Video” software for Linux.
- Install the drivers. In my case, these were in
After driver install,
lsmod | grep blackmagic should show a driver loaded on debian.
You can check that the PCI card is recognized with
lspci | grep Blackmagic (I think this requires the driver but didn’t check)
- Update the firmware (optional).
sudo BlackmagicFirmwareUpdater status will check for updates available. There were none for me.
- Extract the SDK. Move it somewhere easier to type. The relevant folder is
Blackmagic DeckLink SDK 10.11.4/Linux/includes. Let’s assume you move that to
- Build ffmpeg from source. I’m here copying from my source heavily.
- Get the latest ffmpeg source and extract it. Don’t match the debian version–it’s too old to work.
wget https://ffmpeg.org/releases/ffmpeg-4.2.tar.bz2 && tar xf ffmpeg-*.tar.bz2 && cd ffmpeg-*
- Install build deps.
sudo apt-get install nasm yasm libx264-dev libx265-dev libnuma-dev libvpx-dev libfdk-aac-dev libmp3lame-dev libopus-dev libvorbis-dev libass-dev
PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig" ./configure --prefix="$HOME/ffmpeg_build" --pkg-config-flags="--static" --extra-cflags="-I$HOME/ffmpeg_build/include -I$HOME/ffmpeg_sources/BMD_SDK/include" --extra-ldflags="-L$HOME/ffmpeg_build/lib" --extra-libs="-lpthread -lm" --enable-gpl --enable-libass --enable-libfdk-aac --enable-libfreetype --enable-libmp3lame --enable-libopus --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-nonfree --enable-decklink
make -j $(
sudo cp ffmpeg ffprobe /usr/local/bin/
- Use ffmpeg.
ffmpeg -f decklink -list_devices 1 -i dummy should show your device now. Note the name for below.
ffmpeg -f decklink -list_formats 1 -i 'Intensity Pro 4K' shows supported formats. Here’s what I see for the Intensity Pro 4K:
[decklink @ 0x561bd9881800] Supported formats for 'Intensity Pro 4K':
ntsc 720x486 at 30000/1001 fps (interlaced, lower field first)
pal 720x576 at 25000/1000 fps (interlaced, upper field first)
23ps 1920x1080 at 24000/1001 fps
24ps 1920x1080 at 24000/1000 fps
Hp25 1920x1080 at 25000/1000 fps
Hp29 1920x1080 at 30000/1001 fps
Hp30 1920x1080 at 30000/1000 fps
Hp50 1920x1080 at 50000/1000 fps
Hp59 1920x1080 at 60000/1001 fps
Hp60 1920x1080 at 60000/1000 fps
Hi50 1920x1080 at 25000/1000 fps (interlaced, upper field first)
Hi59 1920x1080 at 30000/1001 fps (interlaced, upper field first)
Hi60 1920x1080 at 30000/1000 fps (interlaced, upper field first)
hp50 1280x720 at 50000/1000 fps
hp59 1280x720 at 60000/1001 fps
hp60 1280x720 at 60000/1000 fps
4k23 3840x2160 at 24000/1001 fps
4k24 3840x2160 at 24000/1000 fps
4k25 3840x2160 at 25000/1000 fps
4k29 3840x2160 at 30000/1001 fps
4k30 3840x2160 at 30000/1000 fps
Capture some video:
ffmpeg -raw_format argb -format_code Hp60 -f decklink -i 'Intensity Pro 4K' test.avi
The format (
format_code) will vary based on your input settings. In particular, note that
-raw_format uyvy422 is the default, which I found did not match my computer output. I was able to switch either the command line or the computer output settings to fix it.
- I’m not running any capture, but passthrough isn’t working. That’s how the Intensity Pro 4K works. Passthrough is not always-on. I’d recommend a splitter if you want this for streaming.
- ffmpeg won’t compile. Your DeckLink SDK may be too new. Get 10.11.4 instead.
- I can see a list of formats, but I can’t select one using -format_code. ffmpeg doesn’t recognize the option. Your ffmpeg is too old. Download a newer source.
- When I look at the video, I see colored bars. The HDMI output turns on during recording. The Intensity Pro 4K outputs this when the resolution, hertz, or color format does not match the input. This also happens if your SDK and driver versions are mismatched.
I’m experimenting with using Jekyll in place of wordpress. If you want you can check out https://blog.za3k.com/jekyll/ which containly my weekly review process.
If and when I do migrate, all the posts here will be magically migrated and the URLs will stay the same so links don’t break.
Summary of “the life-changing magic of tidying up”:
Marie Kondo writes the “KonMari” method. The book ends up being as much about her mistakes in learning how to tidy as it is about how to tidy. The book conveys a certain positive energy that makes me want to recommend it, but the author also brings that energy in reaction to a kind of previous stress which accompanied tidying, which she does not seem to have completely dropped–if you are mysteriously anxious and feel you MUST discard everything after reading her book, this may be why.
The primary point she makes is meant to cure it: Decide what to keep and what to discard by physically touching each item, and asking if it brings you joy.
The rest of the method:
- Positivity. Everything in your house loves and wants to help you. If it is time to send off some of the items on their next adventure, this is no reason to be sad or anxious. You had a great time meeting, and they and you were both happy.
- Tidy all at once (at least by category, but preferably in a multi-day binge).
- Physically gather the category in once place, touching everything and asking if it brings you joy.
- Find out what you’ll keep and discard before putting things away or organizing.
- Organizing: ??? [I didn’t get any big takeaways here].
Marie Kondo’s best advice is realizations from her past mistakes–the sort of methods which seems reasonable to try, but end up being wrong for subtle reasons. They are:
- Tidy by category, not place. Otherwise, you won’t realize everything you have.
- “Storage” is storing things neatly, and lets you have more and more things. This is different than tidying, which is about bringing things in harmony, and having only things you love. Becoming better at “storage” can make you unhappy.
She also has encountered her clients making mistakes. For each category of things (clothes, books, etc) there are many reasons clients may not want to throw something out. Most of the book is meant to illustrate why these things are useless, and why throwing them out is okay and will make you happier.
The fun part is that many clients were more confident and more in touch with what they valued and who they wanted once they had only possessions they loved.
za3k.com was the site of a DDoS attack. I’m pretty sure this was because my wordpress installation was compromised, and the hacker who took control of my server was herself DDoSed.
More updates to come, but the short story is that I’ll be formalizing my install and eventually containerizing + hardening everything
Sep, Oct, Nov 2014: Vietnam.
A year ago, I left my job at Streak and moved to Vietnam. I felt like I needed change. Vietnam ended up being wonderful; I was really glad I travelled with my friends Richard and Kathy, which ended up making the experience a hundred times better than it would have been otherwise. The basic environment was: everything is cheap, I newly have endless free time, I was automatically prompted by my friends in the evenings and sometimes during the day to go on small novel adventures involving physical activity, and I had little internet access. This is probably my perfect environment, and I was functioning very well (the vietnamese diet also has small, well-balanced meals which might have helped). For some reason, I was also able to intensely single-task. [I’d like to write more about what Vietnam is like, but this article is quite long enough as it is]
While I was in Vietnam, I made a to-do list. The to-do list had all the burning projects I actually wanted to do. I’ve ended up accomplishing most of them, at a rate of one every week or two, and it’s a decent summary of what I’ve been doing since. Two things made the to-do list a success. First, it had BIG tasks. These are projects like my recent “set up an IRC server” or “start a publishing company”. Because of that, I don’t get bogged down in minutae, and the tasks are always motivating. I find I function better when I try to carefully plan around having any logistics. The second reason, which I realized today, is that I was very careful to only include tasks I was planning to do (subtly different than tasks I wanted to do). The list was descriptive, not normative, although it certainly included some things like doing taxes I wasn’t wild about.
Looking at my journal and it really only starts up again in March, so I’m going to organize this post in terms of the to-do list. There are a couple items that don’t fit:
- I started dating my wonderful pet, Lealend, while I was in Vietnam. I went to visit them for a month in Puerto Rico where they live. This is very very important to me (the most important thing that happened in the last year), but I don’t usually write about things that personal on my blog so I’m not doing to say much. I’ve been emotionally maturing a lot by being with them.
- Conventions. I went to DEF CON, which was probably the best single week this year so far. I’m definitely going again next year. I attended a mirix [paper] in the South Bay, which ended up being stressful for transportation-related reasons but really good while I was there. I’m planning on going to Burning Man this year as well.
- I started contracting, that’s how I’ve been alive for a year. I’ve been doing some work for Zinc and Paul Christiano on a workflowy clone, mostly. I work two hours a day average.
- I’ve been developing a minecraft modpack [I’ll write more about this when it’s stable], and recently taken an interest in livestreaming.
Now on to the to-do list.
Success: Success but obsolete
Description: I set up a printserver. It’s a little raspberry pi that talks to my printer, because getting printers set up is a pain and I don’t want to do it all the time. It went great, it saved me a ton of hassle to have it automatically print out my daily agenda every morning, and to just be able to transfer documents over with ‘scp’.
Future plans: Unfortunately, my printer died and we only recently got a new one. I need to set it up with the new server. I could also make printing completely automatic when new files show up with scp (right now it’s manual so I can switch out paper, but my roommates would be happier with scp I think).
Project: Set up my phone so dropping/losing it isn’t horrible
Success: Partial success
Description: I wanted to root and then automatically backup
my phone. I did figure out how to do as much backing up as I can, and it is automatic. Unfortunately it turns out most of the filesystem (including SMS) just isn’t available over Media Transfer Protocol
which android uses to display files, so I had to special case the things I desperately needed backed up. I’d prefer the state of the world let me back up everything on the phone, but that’s as much work as I’m willing to do.
Project: Get digital copies of all books I own
Description: I got digital copies of all books I own via a combination of pirating, buying copies, and getting the books scanned by a service. I did not get rid of the physical books.
Project: Switch to private email
Success: Not done
Description: I get a little nervous entrusting Google (or any third party) with the ability to read, lose, or add restrictions on what I can do with my email. I want to set up my own email address (email@example.com
) and have it be my main point of contact. My email does work, but I can’t send outgoing email, and I haven’t switched everything over to it for that reason.
Project: Download ArXiV
Description: As an archive nut
, I worry that the ArXiV
collection, one of the nicer collections of scientific papers I access regularly, might someday go down or get censored. I downloaded a copy and stashed it away somewhere. Unfortunately ArXiV’s licenses they get papers under doesn’t permit redistribution, so I can’t publicly host it. (This was really cool but I had to decide whether I was going to publicly mention, since it’s a legal gray area)
Future plans: Someone (not me) should host a torrent. Contact me and I can get you a copy.
Project: Pack and unpack storage bins (trip to vietnam)
Description: Okay I know this sounds stupid, but I spent about a month packing up to go to Vietnam, and all my physical stuff has stayed organized ever since. That’s a really big change for me.
Description: Back up all my personal data from my iPhone, clear the contents, and sell it.
Project: NNTP over tor
Success: Didn’t do
Description: I run a private newsserver, and I wanted to let people access the newsserver (and anything else on that physical server) over tor. I decided the newsserver was too dead to bother with, and I didn’t feel enthusiastic about setting up tor, so I dropped the project.
Future plans: I don’t care about the original project, but if there’s a compelling stimulus, I want to set up tor for my server to learn how and leave flexibility.
Project: Post pdfmailer website
Description: I wanted people to be able to get a physical copy of a pdf document they had mailed to them. I think this project was an especial success, because I’d been failing at an over-engineered version of this off and on for a year. I decided to have the website email me instead of trying to do everything automatically, and ended up getting the books to be a factor of 10 cheaper or so by going with a publisher with no API.
Future plans: I’d like to popularize the website more. I think there are also some small technical improvements to be made. I’m not going to automate things unless it starts using up a lot of my time to process requests myself.
- [Censored project involving an arbitrage opportunity I haven’t cornered]
- Project: Back up email
Project: Flatten backups
Success: Good enough
Description: Oh just go read the XKCD
. Now imagine you’ve been archiving computers onto other computers for 15 years, and buy cheap laptops.
- Project: QR codes for ebooks
Project: Business cards
Success: Not done
Description: Make some personal business cards
Project: QR Punchcodes
Success: Didn’t do
Description: So you know how QR codes can contain any data? That means you could show them to a camera and the camera could run any code. Like, code to wait for another couple of QR codes, or to print out some more QR codes…
[Censored project involving an arbitrage opportunity I haven’t cornered]
[Censored project involving a mildly illegal thing]
Project: Make a desk out of cardboard
Description: I want to make a desk out of cardboard, because it sounds fun. I’m proud of doing the design right here. I’ve finished mocking it out of cardstock, and actually noticed a lot of flaws and fixed the design instead of hoping them away. Now I mostly have to get the cardboard and make it, should be fun.
Project: Write about paper backups
Project: Sort physical scans
Description: As part of packing up all my possessions to go to Vietnam, I scanned every physical document I own (and mostly threw them out). Twenty years of stuff is a lot of stuff, but I eventually sorted it all out. I’ve been increasingly finding that a flat folder structure ends up working out best for me in the long term, so that’s what I used.