It’s november, and I’ve decided this month that I’m going to do 30 projects in 30 days. It’s an all-month hack-a-thon!
Today’s project is the Hack-A-Blog. (demo, source).
Check out the link above to try out the live demo. I’m proud of getting this one done in time. I think the next days will be easier, as I figured some things out already.
I’ve been prototyping an e-ink laptop.
I’m not the first, there have been many other such devices before. I came up with the idea independently, but the specifics are heavily inspired by the Ultimate Writer by NinjaTrappeur in 2018. Similar to him, my use case is typing without distractions, and reading books. E-ink displays are quite slow to update, so I don’t think it can serve as a general purpose computer. Here’s a video of it in action. It operates at one frame per second.
The electronics are not fully done. They need better secured, and I’m going to redo the cabling and power back.
I’m not the best woodworker, but I’m slowly learning. Here are pictures of case and lid action.
On the software end, shout outs to:
There’s a few things I’d like to polish still–even as a prototype this isn’t fully done.
There’s also no default software, but that’s a feature. A prototype is for figuring out how I want the interface to work, and what I want it to do.
Parts list
Power budget (at 5V):
A real-life test showed 5-6 hour battery life. Theory says (13Wh/battery * 4 batteries / 2.7 watts)=20 hours battery life. I’m investigating the discrepancy. In theory, swapping for a Pi Zero W and a better keyboard would give 72-hour battery life.
qr-backup v1.1 is released. qr-backup is a command-line Linux program. You can use it to back up a file as a series of QR codes. You can restore the QR codes using a webcam or scanner, and get back the original file.
The main features of qr-backup are ease-of-use and futureproofing (restore does not require qr-backup).
Please report any bugs on github. Once this is stable, I will do the first pip/package manager release. To test the alpha, check out the latest code using git.
See also USAGE and extensive FAQ.
New features in v1.1:
qr-backup --restore restores using the webcamqr-backup --restore IMAGE IMAGE IMAGE restores from scanned imagesP.S. As a special request, if anyone is on OS X, let me know if it works for you?
I recently wrote a program that records all tty activity. That means bash sessions, ssh, raw tty access, screen and tmux sessions, the lot. I used script. The latest version of my software can be found on github.
Note that it’s been tested only with bash so far, and there’s no encryption built in.
To just record all shell commands typed, use the standard eternal history tricks (bash).
One of the more useful things I did was to start logging all my technical problems. Whenever I hit a problem, I write an entry in problem-log.txt. Here’s an example
2022-08-02
Q: Why isn't the printer working? [ SOLVED ]
A: sudo cupsenable HL-2270DW
// This isn't in the problem log, but the issue is that CUPS will silently disable the printer if it thinks there's an issue. This can happen if you pull a USB cord mid-print.I write the date, the question, and the answer. Later, when I have a tough or annoying problem, I try to grep problem-log.txt. I’ll add a note if I solve a problem using the log, too.
This was an interesting project to look at 5 years later. I didn’t see benefits until 1-2 years later. It does not help me think through a problem. It’s hard to remember to do. But, over time it’s built up and become invaluable to me. I hit a tricky problem, and I can’t immediately find an answer on the web. I find out it’s in problem-log.txt. And, someone’s written it exactly with my hardware (and sometimes even my folder names) correctly in there. Cool!
Here’s another example:
2018-10-21
Q: How do I connect to the small yellow router?Not every problem gets solved. Oh well.
I scan each and every piece of paper that passes through my hands. All my old to-do lists, bills people send me in the mail, the manual for my microwave, everything. I have a lot of scans.
scan-organizer is a tool I wrote to help me neatly organize and label everything, and make it searchable. It’s designed for going through a huge backlog by hand over the course of weeks, and then dumping a new set of raw scans in whenever afterwards. I have a specific processing pipeline discussed below. However if you have even a little programming skill, I’ve designed this to be modified to suit your own workflow.
The input is some raw scans. They could be handwritten notes, printed computer documents, photos, or whatever.
The final product is that for each file like ticket.jpg, we end up with ticket.txt. This has metadata about the file (tags, category, notes) and a transcription of any text in the image, to make it searchable with grep & co.
---
category: movie tickets
filename: seven psychopaths ticket.jpg
tags:
- cleaned
- categorized
- named
- hand_transcribe
- transcribed
- verified
---
Rialto Cinemas Elmwood
SEVEN PSYCHOPAT
R
Sun Oct 28 1
7:15 PM
Adult $10.50
00504-3102812185308
Rialto Cinemas Gift Cards
Perfect For Movie Lovers!
Here are some screenshots of the process. Apologizies if they’re a little big! I just took actual screenshots.
At any point I can exit the program, and all progress is saved. I have 6000 photos in the backlog–this isn’t going to be a one-session thing for me! Also, everything has keyboard shortcuts, which I prefer.
First, I clean up the images. Crop them, rotate them if they’re not facing the right way. I can rotate images with keyboard shortcuts, although there are also buttons at the bottom. Once I’m done, I press a button, and scan-organizer advanced to the next un-cleaned photo.
Next, I sort things into folders, or “categories”. As I browse folders, I can preview what’s already in that folder.
Renaming images comes next. For convenience, I can browse existing images in the folder, to help name everything in a standard way.
I tag my images with the type of text. They might be handwritten. Or they might be printed computer documents. You can imagine extending the process with other types of tagging for your use case.
Printed documents are run through OCR. This isn’t actually done yet, but it will be easy to plug in. I will probably use tesseract.
I write up all my handwritten documents. I have not found any useful handwriting recognition software. I just do it all by hand.
The point of scan-organizer is to filter based on tags. So only images I’ve marked as needing hand transcription are shown in this phase.
At the end of the whole process, I verify that each image looks good, and is correctly tagged and transcribed.
One of my archiving and backup contingencies is taking one screenshot per minute. You can also use this to get a good idea of how you spend your day, by turning it into a movie. Although with a tiling window manager like I use, it’s a headache to watch.
I send the screenshots over to another machine for storage, so they’re not cluttering my laptop. It uses up 10-20GB per year.
I’ll go over my exact setup below in case anyone is interested in doing the same:
/bin/screenlog
GPG_KEY=Zachary
TEMPLATE=/var/screenlog/%Y-%m-%d/%Y-%m-%d.%H:%M:%S.jpg
export DISPLAY=:0
export XAUTHORITY=/tmp/XAuthority
IMG=$(\date +$TEMPLATE)
mkdir -p $(dirname "$IMG")
scrot "$IMG"
gpg --encrypt -r "$GPG_KEY" "$IMG"
shred -zu "$IMG"
The script
/tmp/XAuthority. It doesn’t need any unusual permissions, but the default location has some random characters in it./etc/cron.d/screenlog
* * * * * zachary /bin/screenlog
20 * * * * zachary rsync --remove-source-files -r /var/screenlog/ backup-machine:/data/screenlog/laptop
30 * * * * zachary rmdir /var/screenlog/*
That’s
~/.profile
export XAUTHORITY=/tmp/XAuthority
I mentioned /bin/screenlog needs to know where XAuthority is. In Arch Linux this is all I need to do.
I just wrote the first pass at youtube-autodl, a tool for automatically downloading youtube videos. It’s inspired by Popcorn Time, a similar program I never ended up using, for automatically pirating the latest video from a TV series coming out.
You explain what you want to download, where you want to download it to, and how to name videoes. youtube-autodl takes care of the rest, including de-duplication and downloading things ones.
The easiest way to understand it is to take a look at the example config file, which is my actual config file.
Personally, I find youtube is pushing “watch this related” video and main-page feeds more and more, to the point where they actually succeed with me. I don’t want to accidentally waste time, so I wanted a way to avoid visiting youtube.com. This is my solution.
qr-backup is a program to back up digital documents to physical paper. Restore is done with a webcam, video camera, or scanner. Someday smart phone cameras will work.
I’ve been making some progress on qr-backup v1.1. So far I’ve added:
--restore, which does a one-step restore for you, instead of needing a bash one-line restore process--encrypt provides password-based encryption--instructions to give more fine-tuned control over printing instructions. There’s a “plain english” explanation of how qr-backup works that you can attach to the backup.--note for adding an arbitrary message to every sheetv1.1 will be released when I make qr-backup feature complete:
v1.2 will focus on adding a GUI and support for Windows, Mac, and Android. Switching off zbar is a requirement to allow multi-platform support, and will likely improve storage density.
Recently I wrote a scraper. First, I downloaded all the HTML files. Next, I wanted to parse the content. However, real world data is pretty messy. I would run the scraper, and it would get partway though the file and fail. Then I would improve it, and it would get further and fail. I’d improve it more, and it would finish the whole file, but fail on the fifth one. Then I’d re-run things, and it would fail on file #52, #1035, and #553,956.
To make testing faster, I added a scaffold. Whenever my parser hit an error, it would print the filename (for me, the tester) and record the filename to an error log. Then, it would immediately exit. When I re-ran the parser, it would test all the files where it had hit a problem first. That way, I didn’t have to wait 20 minutes until it got to the failure case.
if __name__ == "__main__":
if os.path.exists("failures.log"):
# Quicker failures
with open("failures.log", "r") as f:
failures = set([x.strip() for x in f])
for path in tqdm.tqdm(failures, desc="re-checking known tricky files"):
try:
with open(path) as input:
parse_file(input)
except Exception:
print(path, "failed again (already failed once")
raise
paths = []
for root, dirs, files in os.walk("html"):
for file in sorted(files):
path = os.path.join(root, file)
paths.append(path)
paths.sort()
with open("output.json", "w") as out:
for path in tqdm.tqdm(paths, desc="parse files"): # tqdm is just a progress bar. you can also use 'for path in paths:
with open(input, "r") as input:
try:
result = parse_file(input)
except Exception:
print(path, "failed, adding to quick-fail test list")
with open("failures.log", "a") as fatal:
print(path, file=fatal)
raise
json.dump(result, out, sort_keys=True) # my desired output is one JSON dict per line
out.write("\n")