mon(8)

I had previously hand-rolled a status monitor, status.za3k.com, which I am in the process of replacing (new version). I am replacing it with a linux monitoring daemon, mon, which I recommend. It is targeted at working system administrators. ‘mon’ adds many features over my own system, but still has a very bare-bones feeling.

The old service, ‘simple-status‘ worked as follows:

  • You visited the URL. Then, the status page would (live) kick of about 30 parallel jobs, to check the status of 30 services
  • The list of services is one-per-file in a the services.d directory.
  • For each service, it ran a short script, with no command line arguments.
  • All output is displayed in a simple html table, with the name of the service, the status (with color coding), and a short output line.
  • The script could return with a success (0) or non-success status code. If it returned success, that status line would display in green for success. If it failed, the line would be highlighted red for failure.
  • Scripts can be anything, but I wrote several utility functions to be called from scripts. For example, “ping?” checks whether a host is pingable.
  • Each script was wrapped in timeout. If the script took too long to run, it would be highlighted yellow.
  • The reason all scripts ran every time, is to prevent a failure mode where the information could ever be stale without me noticing.

Mon works as follows

  • The list of 30 services is defined in /etc/mon/con.cf.
  • For each service, it runs a single-line command (monitor) with arguments. The hostname(s) are added to the command line automatically.
  • All output can be displayed in a simple html table, with the name of the service, the status (with color coding), the time of last and next run, and a short output line. Or, I use ‘monshow‘, which is similar but in a text format.
  • Monitors can be anything, but several useful ones are provided in /usr/lib/mon/mon.d (on debian). For example the monitor “ping” checks whether a host is pingable.
  • The script could return with a success (0) or non-success status code. If it returned success, the status line would display in green for success (on the web interface), or red for failure.
  • All scripts run periodically. A script have many states, not just “success” or “failure”. For example “untested” (not yet run) or “dependency failing” (I assume, not yet seen).

As you can see, the two have a very similar approach to the component scripts, which is natural in the Linux world. Here is a comparison.

  • ‘simple-status’ does exactly one thing. ‘mon’ has many features, but does the minimum possible to provide each.
  • ‘simple-status’ is stateless. ‘mon’ has state.
  • ‘simple-status’ runs on demand. ‘mon’ is a daemon which runs monitors periodically.
  • Input is different. ‘simple-status’ is one script which takes a timeout. ‘mon’ listens for trap signals and talks to clients who want to know its state.
  • both can show an HTML status page that looks about the same, with some CGI parameters accepted.
  • ‘mon’ can also show a text status page.
  • both run monitors which return success based on status code, and provide extra information as standard output. ‘mon’ scripts are expected to be able to run on a list of hosts, rather than just one.
  • ‘mon’ has a config file. ‘simple-status’ has no options.
  • ‘simple-status’ is simple (27 lines). ‘mon’ has longer code (4922 lines)
  • ‘simple-status’ is written in bash, and does not expose this. ‘mon’ is written in perl, all the monitors are written in perl, and it allows inline perl in the config file
  • ‘simple-status’ limits the execution time of monitors. ‘mon’ does not.
  • ‘mon’ allows alerting, which call an arbitrary program to deliver the alert (email is common)
  • ‘mon’ supports traps, which are active alerts
  • ‘mon’ supports watchdog/heartbeat style alerts, where if a trap is not regularly received, it marks a service as failed.
  • ‘mon’ supports dependencies
  • ‘mon’ allows defining a service for several hosts at once

Overall I think that ‘mon’ is much more complex, but only to add features, and it doesn’t have a lot of features I wouldn’t use. It still is pretty simple with a simple interface. I recommend it as both good, and overall better than my system.

My only complaint is that it’s basically impossible to Google, which is why I’m writing a recommendation for it here.

Cron email, and sending email to only one address

So you want to know when your monitoring system fails, or your cron jobs don’t run? Add this to your crontab:

Now install a mail-sending agent. I like ‘nullmailer‘, which is much smaller than most mail-sending agents. It can’t receive or forward mail, only send it, which is what I like about it. No chance of a spammer using my server for something nasty.

The way I have it set up, I’ll have a server (avalanche) sending all email from one address (nullmailer@avalanche.za3k.com) to one email (admin@za3k.com), and that’s it. Here’s my setup on debian:

Now just run echo "Subject: sendmail test" | /usr/lib/sendmail -v admin@za3k.com to test and you’re done!

Mail filtering with Dovecot

This expands on my previous post about how to set up an email server.

We’re going to set up a few spam filters in Dovecot under Debian. We’re going to use Sieve, which lets the user set up whichever filters they want. However, we’re going to run a couple pre-baked spam filters regardless of what the user sets up. Continue reading

Installing email with Postfix and Dovecot (with Postgres)

I’m posting my email setup here. The end result will:

  • Use Postfix for SMTP
  • Use Dovecot for IMAP and authentication
  • Store usernames, email forwards, and passwords in a Postgres SQL database
  • Only be accessible over encrypted channels
  • Pass all common spam checks
  • Support SMTP sending and IMAP email checking. I did not include POP3 because I don’t use it, but it should be easy to add
  • NOT add spam filtering or web mail (this article is long enough as it is, maybe in a follow-up)

Continue reading

XP Boot USB Stick

Most of the following taken from : http://www.msfn.org/board/topic/151992-install-xp-from-usb-without-extra-tools/, just modified to include syslinux support.

Let me know if there are any omissions; it an XP installer bluescreens on boot for me so I can’t actually test.

  1. Obtain an XP iso file
  2. Format drive with one FAT parition, marked bootable.
  3. Edit /mnt/syslinux.cfg:
  4. Boot from the USB stick

Roll-your-own git push-to-deploy, and markdown support

Today I added support for development of za3k.com using git:

and markdown support, via a cgi markdown wrapper someone wrote for apache (yes, I’m still using Apache).

Edit: I ended up wanting support for tables in markdown, so I used Ruby‘s redcarpet markdown gem (the same thing Github uses, supports this style of tables as well as code blocks).

CGI support via http://blog.tonns.org/2013/10/enabling-markdown-on-your-apache.html

Archiving twitter

(Output)

I wanted to archive twitter so that I could

  1. Make sure old content was easily available
  2. Read twitter in a one-per-line format without ever logging into the site

twitter_ebooks is a framework to make twitter bots, but it includes an ‘archive’ component to fetch historical account content which is apparently unique in that it 1) works with current TLS and 2) works the current twitter API. It stores the tweets in a JSON format which presumably matches the API return values. Usage is simple:

I ran into a bug with upstream incompatibilities which is easily fixed. Another caveat is that the twitter API only allows access 3200 tweets back in time for an account–all the more reason to set up archiving ASAP. Twitter’s rate-limiting is also extreme (15-180 req/15 min), and I’m worried about a problem where my naive script can’t make it through a list of more than 15 accounts even with no updates.

 

 

Android backup on arch linux

Edit: See here for an automatic version of the backup portion.

Connecting android to Windows and Mac, pretty easy. On arch linux? Major pain. Here’s what I did, mostly via the help of the arch wiki:

  1. Rooted my phone. Otherwise you can’t back up major parts of the file system (including text messages and most application data) [EDIT: Actually, you can’t back these up over MTP even once you root your phone. Oops.]
  2. Installed jmtpfs, a FUSE filesystem for mounting MTP, the new alternative to mount-as-storage on portable devices.
  3. Enabled ‘user_allow_other’ in /etc/fuse.conf. I’m not sure if I needed to, but I did.
  4. Plugged in the phone, and mounted the filesystem:

    The biggest pitfall I had was that if the phone’s screen is not unlocked at this point, mysterious failures will pop up later.
  5. Synced the contents of the phone. For reasons I didn’t diagnose (I assume specific to FUSE), this actually fails as root:

Archiving github

GitHub-Backup is a small project to archive github repos to a local computer. It advertises that one reason to use it is

You are paranoid tinfoil-hat wearer who needs to back up everything in triplicate on a variety of outdated tape media.

which describes why I was searching it out perfectly.

I made a new account on my server (github) and cloned their repo.

Despite being semi-unmaintained, everything mostly works still. There were two exceptions–some major design problems around private repos. I only need to back up my public repos really, so I ‘solved’ this by issuing an Oauth token that only knows about public repos. And second, a small patch to work around a bug with User objects in the underlying Github egg:

Then I just shoved everything into a cron task and we’re good to go.

Edit: There’s a similar project for bitbucket I haven’t tried out: https://bitbucket.org/fboender/bbcloner