This post is now at http://honeypot.net/2011/10/27/stop-the-e-parasite-act/
This post is now at http://honeypot.net/2011/10/11/making-dos-usb-images-on-a-mac/
Mom: I’ve been to Florida.
Kid: When?
Mom: A long time ago. I went on a cruise to the Bahamas.
Kid: With Daddy?
Mom: No…
Kid: Then with who?
Dad: Yeah. With who?
Mom: Umm… Tim.
Kid: Who’s Tim?
Dad: Yeah. Who’s Tim?
Mom: He was… Mommy’s special friend ten years ago.
Dad: Wait, weren’t we together then?
Mom: deer in the headlights
Dad: stony silence the rest of the flight
A fascinatingly bleak look at post-Soviet Siberia
You know how it goes: you open your iPhone, launch the App Store app, wait a few seconds, press the Updates button, then wait for ages while your phone tries to find any updated apps to download. I can’t figure out why that process is so ridiculously slow and thought I’d try reproducing the process for myself.
I wrote the little Python simulator below. It creates a database of a million iPhone apps and their version numbers. Then it builds a sample inventory of 1,000 apps installed on an iPhone (which is probably way more than the average). Finally, it repeatedly checks that inventory of installed apps to find the ones that have updates.
Assumptions:
That is, if a user has version 123 of app “42”, and the current version of app “42” is 124, then the user has an old version of the app.
Here’s my simulator:
#!/usr/bin/env python
import json
import random
import time
# Pretend there are 1,000,000 apps in the iPhone app store.
numberofapps = 1000000
# Create a test database of app serial numbers and their version
# numbers.
currentversion = {str(appid): random.randrange(1000) for appid in range(numberofapps)}
# Pretend that a user has 1,000 apps installed on their iPhone. Build
# a database of each of those apps' serial numbers and
# versions.
localversion = {str(random.randrange(numberofapps)): random.randrange(1000)
for _ in range(100)}
# Serialize that database as a JSON object suitable for transmission
# to the remote server. This would only *have* to be done whenever an
# app is installed or updated, but will only take a few microseconds
# anyway.
localversionjson = json.dumps(localversion)
def checkversion(submittedversion):
"""Accept a serialized dict of app serial numbers and versions
from a client application. Deserialize it, compare each app's
version to the database of most recent versions, and return a
serialized dict of the results."""
clientverion = json.loads(submittedversion)
updatedversion = {appid: currentversion[appid]
for appid, userversion in clientverion.items()
if userversion < currentversion[appid]}
return json.dumps(updatedversion)
# Now check all of the versions of each app installed on our user's
# hypothetical iPhone a few thousand times.
print 'Looking for updated versions'
start = time.time()
for _ in range(10000):
checkversion(localversionjson)
print 'Elapsed:', time.time() - start
Even though my program skips the step of actually transmitting the list of installed apps across the Internet, the code performs all the other “expensive” steps that a real service would have to do.
And on my MacBook Pro, it does so about 7,000 times a second.
My slow, unoptimized service would be able to handle about 600 million iPhone users checking for updates once a day each.
Presumably Apple does not run their App Store on a single MacBook, and they’ve likely spent some time and effort actually trying to make it fast. So why haven’t they managed to do it? Why does it take more than a fraction of a second to check for new app updates?
Ari: I know that boy.
Gabby: Who is he?
Ari: A gross boy from my school.
Woman next to Ari: That’s my son.
I have a community-oriented bulletin board running on phpBB3. I want to limit its users to local residents - or at least residents of my own country. I have nothing against people living elsewhere; it’s just that someone living in Chile probably won’t be interested in my little website in a small town in the USA unless they’re wanting to post spam to it. Here’s how I did it.
Note: I’m using Apache 2.2 on a FreeBSD server. The details for other combinations will be a little different but very similar.
Country IP Blocks is a website that lists the network blocks associated with each country. While this isn’t guaranteed to be 100% accurate, it’ll be pretty close. After digging around on the site, I found that the list of all network blocks in the United States is at https://www.countryipblocks.net/e_country_data/US_allow.txt. I downloaded this to my Apache server’s Includes directory:
# cd /usr/local/etc/apache22/Includes
# curl -O https://www.countryipblocks.net/e_country_data/US_allow.txt
The User Control Panel, or UCP, is the phpBB page that handles registration and profile editing and all that other good stuff. By limiting access to that page (located at /ucp.php on my site) to only the network blocks in the list I downloaded in the first step, I block anyone outside the US from registering an account. They can still read my public forums - they just can’t participate.
I edited the Apache configuration file for my domain to add this clause:
<Location /ucp.php>
Order Deny,Allow
Deny from All
Include etc/apache22/Includes/US_allow.txt
Allow from ::
</Location>
The <Location> tag says that the following settings will only apply to the /ucp.php page.
The Order Deny,Allow directive tells Apache to obey any “Deny” directives by default, and then override those results with any matching “Allow” directives it finds.
The next line, Deny from All means exactly that: block all users unless their IP address is listed in one of the blocks I downloaded in the first step.
The fourth line instructs Apache to slurp in all of the allow from ... lines from the network list file. I chose this approach over putting the instructions in a more common .htaccess file for two reasons:
.htaccess file, those directives are reloaded every time a visitor tries to view a page - any page. As of this moment, there are nearly 40,000 allow from directives in the network block list and that’s a lot of extra work for something that won’t change very often. By Includeing its contents directly into the Apache configuration, those directives are only loaded and interpreted once whenever the webserver starts. The downside is that I’ll have to remember to restart Apache when or if I ever update the network block list.Next is a directive to allow all IPv6 visitors. If I ever start receiving spam from IPv6 clients, I’ll reconsider that option.
The trailing </Location> line says that we’re done with this snippet.
Soon after restarting Apache, “deny” messages started filtering into my site’s error log:
[Wed Jun 29 14:10:16 2011] [error] [client 77.93.2.81] client denied by server configuration: /usr/local/www/phpBB3/ucp.php, referer: http://norfolkboard.com/ucp.php?mode=register
Interpretation: someone in Latvia tried unsuccessfully to register an account on my site. While it’s remotely possible that a local resident is visiting European relatives and suddenly wanted to check my site, the odds are much greater that I just blocked a would-be spammer from opening an account.
So, I can’t find a single iPhone checkbook app that knows the difference between “cleared” and “reconciled” transactions. In a nutshell: “cleared” is a temporary status that can be easily removed, while “reconciled” is a permanent status that means “this transaction matched an item on my bank statement, it’s accounted for, and it’s part of the permanent record”. In database terms, “cleared” items are part of an open transaction, while “reconciled” items have been committed.
This isn’t just a nitpicky, technical difference. It’s important. Suppose you’re using a checkbook app that doesn’t know the difference, and you have 10,000 cleared items in your history. You’re diligent and your cleared balance exactly matches your bank statement’s closing balance. When your next statement comes, you sit down to clear all the new items - but you can’t get the balances to match up. Now what? How do you undo all the changes you just made so that you can get back to your correct starting balance and try again? What if you accidentally un-check a previously reconciled transaction? You get to break out last month’s statement and pull your hair out while you try to fix it.
Compare this with a real transactional system where items stay reconciled once you’ve marked them that way. Imagine the above scenario. When you discover that you’ve messed something up, you press the “unmark cleared items” button (however it’s labeled) and all the temporary “cleared” marks are removed, rolling you back to your previous, correct starting balance. When you find and fix the problem and clear all the items on your statement, you press the “reconcile cleared items” button (however it’s labeled) and all the “cleared” statuses are upgraded to “reconciled”. Your statement is now completely reconciled and set in stone, and you’re forever done with it.
This is extremely important when regularly clearing your checkbook against your online statement as it progresses through the month. I do this weekly but the online version isn’t my official statement, and I’ve seen items appear and disappear from it. Bank charges me $15 for something? I add it to my register. Bank realizes it was in error and removes it? I don’t notice and it sits on my register until I discover that my checkbook won’t balance.
So why doesn’t any iPhone checkbook app seem to support this? Every serious desktop app knows the difference, and it’s not really all that complicated or difficult to implement. I want the same thing on my iPhone and I’m willing to pay for it.
Update July 1, 2011: Catamount Software’s “Money” made a big step toward getting this right by adding a reconciling view. It’s still not perfect:
It doesn’t show the running balance as you check off items. Instead, it only shows how much your reconciled balance differs from the ending balance you enter. My bank statement looks a little like this:
Starting balance: $1000.00
Amount Balance Description
------- ------- --------------------
+100.00 1100.00 Paycheck
- 50.00 1050.00 Grocery Store
- 40.00 1010.00 Gas Station
+ 75.00 1085.00 Bonus
If Money displayed the running balance as I checked off items, I could tell at a glance whether I’d selected all the correct items so far (and that the amounts of those items was entered correctly). For example, if I start reconciling and select two items, and my running balance is anything other than $1050.00, then I’ve made a mistake.
But that said, it’s still a lot closer to the ideal than any other checkbook app I’ve tried, and I commend Catamount for adding this important feature. With a couple of tweaks it could be very useful.
I decided to try the Qtile tiling window manager after hearing so much about it at PyCon. It’s kind of a pain in the neck to install the first time, though, and I thought I’d save everyone else from the mistakes I made along the way.
Starting with the official installation instructions, here are the modifications I made to that process to get it running:
Skip manually compiling libxcb. The version that ships with Ubuntu 10.10 is sufficient.
To install xpyb, first run
sudo aptitude install xcb-proto libxcb1-dev python-xcbgen
Replace the ./configure --disable-drm2 step with ./autogen.sh.
Before installing cairo, install a newer version of libpixman:
git clone git://anongit.freedesktop.org/git/pixman.git
cd pixman
./autogen.sh
make
sudo make install
Next, install the development packages needed to compile it:
sudo aptitude install libxcb-render0-dev libpng12-dev libfontconfig1-dev libxrender-dev
Now you have to choose which X11 library to use. libx11 is the standard. lib11-xcb is the young upstart. I started with libx11-xcb, but the Thunderbird email program wouldn’t start until I reverted to libx11. It may be working by the time you read this.
To use libx11, run this (noting that we still have to enable xcb for other stuff):
sudo aptitude install libx11-dev
./autogen.sh --enable-xcb # Instead of ./configure!
If you’d like to expirement with libx11-xcb, run:
sudo aptitude install libx11-xcb-dev
./autogen.sh --enable-xlib-xcb # Instead of ./configure!
Yep: replace ./configure with ./autogen.sh.
To start Qtile from the login screen, I create a ~/.xsession file:
#!/bin/sh
xmodmap ~/.xmodmap
~/.config/qtile/session-startup &
exec /usr/local/bin/qtile
The session-startup script sets the screen’s background picture, sleeps for a little bit to give Qtile a chance to start, then runs a few handy applets and other programs:
#!/bin/sh
qiv -z ~/Pictures/Backgrounds/NATURE-SunsetBlissII_1600x1200.jpg
sleep 3
nm-applet &
gnome-power-manager &
kupfer --no-splash &
dropbox start
Ta-da! Qtile on Ubuntu 10.10 without excessive hair-pulling.
I usually use Ubuntu on my desktop at work. I decided to try Arch Linux a few days ago, but changed my mind and wanted to go back to Ubuntu. Easy, right? Nope.
First, I should mention that my desktop has 3 hard drives and an SSD, and I use LVM to manage all of them. I also wanted to wipe out the root filesystem and replace it with Ubuntu while leaving /home and a couple of other filesystems intact.
I started by downloading Ubuntu’s bootable USB image. Ha! Just kidding. They don’t really make one. Instead, you have to download an ISO image and use a special GUI tool to extract the files from it and put them on a USB flash drive. That program kept crashing to the point that I got impatient and burned the ISO to a blank CD. Once the burning program copied and verified the CD, I rebooted into the installer. After wiping away Arch Linux, the installer complained of broken packages and crashed. What? I rebooted the CD and ran the verification program, and sure enough: the CD was full of defects. Great. At that point, I had an unbootable system and no easy way to fix it.
Should this ever happen to you, this is what I did:
Post Mortem
Ubuntu could have avoided this whole mess by providing a downloadable, ready-to-use USB image. Arch Linux is tiny in comparison, but they ship a hybrid image that you can burn to a CD or copy to a USB drive without modification. Seriously, Ubuntu, WTF? CDs are so last decade.
Beyond that, it’s insane to require a GUI tool to create an installer. When I was staring at a recovery terminal with nothing but a few command line tools available and no way to convert that ISO into a USB drive image, I wasn’t feeling the love.
Brasero burned and successfully verified a defective CD. Had the verification failed at that step, I would have thrown the coaster away and burned a new one. The half-working, half-broken disc was I ended up with was perhaps the worst possible outcome of that procedure.
I should have created a new root volume in LVM to use for Ubuntu, then deleted the old Arch Linux volume once I was satisfied that the new one worked.
Reinstalling Linux the morning after returning home from PyCon might not have been the best idea.