Dazed and confused in the age of new media

We've all been talking for a long time about how the old media establishment of a few astronomically popular bands will crumble and give way to a plethora of smaller acts. That's all well and good, and you can see evidence of it in the declining record sales and the rise of music blogs like discodust.blogspot.com and discoworkout.com. The trend is more and more people are spending more time listening to music and searching for talented artists and sharing them with their friends.

However, my experience is that I discover an artist I like, and I'm unable to purchase their music through the large channels I'm familiar with. Let's take Kill the Noise, for example, who is a DJ somewhere in California who happens to have made a remix I like. I go to his site, and it's all about booking information for getting him to come and perform for you. He has several of his tracks in a Flash music player at the top of the page. I'm listening to it as I write this post, and it's good. There are even links to a handful of downloads, some for free and others for pay. But I can't find a single link to an album for purchase.

It makes sense: the album isn't a format that fits the model of new media. Why should an artist release 8 or so tracks all at once in a seemingly arbitrary collection? Why not publish as soon as each track is finished to keep the buzz alive, rather than waiting two years for new music? However, here I am with my wallet, trying to figure out how to purchase roughly $10 worth of music without spending too much time looking for it. I guess there's no place for this in the new media regime. In order to find all the new hip stuff, you've got to have boatloads of time to comb the music blogs and download hit singles.

How to configure smartmontools on an Ubuntu system

So I'm having some hard drive troubles, and I'm not sure what the cause is. I've had two incidents so far, each of which is precipitated by my music hanging and the Amarok UI freezing and then unfreezing up to 30 seconds later. The first time was quite catastrophic, and I had to restore from my previous night's backup. However, I took the opportunity to do a complete system reinstall, which blew away my cron backup script configuration. So unfortunately, this time around, I didn't have a backup to restore from.

When I noticed the symptoms, I immediately made an incremental backup and rebooted to run fsck. Predictably, fsck reported a few nasty filesystem errors, but it mostly had to do with some unmodified files checked into version control and some system fonts.

Anyway, this experience prompted me to finally setup smartmontools so I can keep track of drive errors and identify if the failure is a controller problem, a connection problem, or a drive problem.

There are many ways to configure smartmontools/smartd, and you can read about them in the man page and across the Internet. These instructions are, in my opinion, the ones that require the least modifications to what Ubuntu sets up for you by default.

So first, do the usual sudo apt-get install smartmontools .

This installs postfix, which I configured as an Internet Site, and left with most of the defaults filled in.

The package comes with an init script that lives at /etc/init.d/smartmontools and is set to run at the defaults runlevel when installed. However, if you try running it manually via sudo /etc/init.d/smartmontools , you will notice that it prints nothing and fails to start a smartd background process. It turns out that you need to edit /etc/default/smartmontools and uncomment the lines they have there, as shown below:

# Defaults for smartmontools initscript (/etc/init.d/smartmontools)
# This is a POSIX shell fragment

# List of devices you want to explicitly enable S.M.A.R.T. for
# Not needed (and not recommended) if the device is monitored by smartd
#enable_smart="/dev/hda /dev/hdb"

# uncomment to start smartd on system startup
start_smartd=yes

# uncomment to pass additional options to smartd on startup
smartd_opts="--interval=1800"

Especially without start_smartd=yes, nothing will work!

Now, before we're off to the races, we want to go check out /etc/smartd.conf, where most of the real configuration we care about happens. Here is the relevant section of my configuration, with comments about each option.

# DEVICESCAN: Scan for ATA and SCSI devices.  All lines after this one are
#   ignored.
# -o on: Enable online data collection.
# -S on: Enable automatic attribute autosave.
# -s (S/../.././02|L/../../6/03): Do a short self-test every night between
#   2-3am and a long self-test every Sunday between 3-4am.
# -H: If the device says it's not healthy, send mail.  This occurs if any
#   prefail attributs are past their thresholds.
# -l selftest: Send mail if, since the last check, a self test has found
#   additional errors.
# -l error: Send mail if the error log has new errors.
# -f: Send mail if the disk has "failed", ie it's total usage is above the
#   threshold set by the manufacturer.  Indication of age, not check failures.
# -m: Address to which to send email.
# -M...: Script to run to send mail and do other things.

# This line is broken up for posting, but I would remove it from the actual
# config file.
DEVICESCAN -o on -S on -s (S/../.././02|L/../../6/03) -H -l selftest -l error -f \
  -m reid.kleckner@gmail.com -M exec /usr/share/smartmontools/smartd-runner

The idea behind this configuration is to be a little bit less verbose than the default of -a, which monitors changes in various parameters, but still generate an email when things go wrong. The results of the self-tests are stored on the hard drive, so if you're having trouble with a drive later, you can check out its bill of health and test record with smartctl -a.

Portable Native Client

Hey, this is cool:

http://blog.chromium.org/2010/03/native-client-and-web-portability.html
http://nativeclient.googlecode.com/svn/data/site/pnacl.pdf

This is the first I've seen it disclosed publicly, but the summary is that Google plans to make Native Client (NaCl) portable with some kind of verifier on LLVM bitcode. Now, if you're familiar with LLVM, then you may have heard the claim that LLVM bitcode from a C compiler is not platform neutral. The argument is that you always have things like #ifdefs that check for the target architecture and that most C compilers (maybe including Clang? I'm not sure) lower things like sizeof(int) in the frontend. Therefore, they claim, it is essentially impossible to compile C and C++ down to a portable representation.

However, it seems doable for NaCl, because you're not dealing with arbitrary programs. In their writeup, they explicitly say you should eliminate things like architectural #ifdefs. They also state up front that sizeof(int) == sizeof(long) == sizeof(void*) == 4. I'm not sure how that fits in with x86_64. They also state that the data model assumes that integers are little endian, which isn't exactly portable either.

Still, it's a cool project.

"Blocks," or "lambdas in a non-sexp language"

Last week there was a lot of discussion around tav's proposal to add Ruby blocks to Python. Eventually, the proposal went to python-ideas, and got ground into the dust by all of the objections. The idea has been proposed before, and has always met with strong resistance. The argument against anonymous pseudo-expression functions relies on the idea that Python already has powerful syntax for doing 90% of the things that Ruby and Scheme use blocks and lambdas for, and Guido seems to prefer it that way. In the end, I think Guido's right about blocks in Python, but in another language with different scoping rules, blocks might be a great idea.

Consider the Ruby blocks examples on the c2 wiki. The first three examples, which are iteration, callback registration, and resource management, all have distinct syntaxes in Python, while in Ruby they all use blocks:

Iteration:

# Ruby:
collection.each do |element|
   ...
end

# Python:
for x in collection:
   ...

# Ruby:
numbers = [1,2,3,4]
squares = numbers.map {|n| n*n }

# Python:
numbers = [1, 2, 3, 4]
squares = [n * n for n in numbers]

Callback registration:

# Ruby:
button.on_click do |event|
   ...callback code...
end

# Python:
# (decorators are more general, but this is a common use case
# exemplified by Django filters and tags.)
@button.on_click
def raise_dialog(event):
   ...callback code...

Resource management:

# Ruby:
File.open(filename) do |file|
   ...read from file...
end

# Python:
with open(filename) as file:
   ...read from file...

The consensus on the python-ideas list was that the dedicated for-loop and context-manager syntax is more readable, because no matter what object you're iterating, you have a big fat keyword on the line start telling you how the next code block is going to be executed, instead of one syntax stretching to try and cover multiple unrelated use cases. The verdict could also be interpreted as another instance of the "there should be only one way to do it" philosophy of Python. Currently, def is the only way to create a function that can contain statements, and decorators cover many of the higher-order function use cases. Introducing another syntax for those tasks goes against the grain.

So if blocks aren't good for Python, where do they work?

First of all, I think blocks in Ruby are kind of broken. I've seen many people talk about the elegance of the Ruby block syntax, and I just don't buy it. Why all the puncuation and magic ampersand-arguments? What the hell is up with optional parentheses on function calls? That's friggin' crazy when you're working with function values. It's almost as bad as Common Lisp having separate namespaces for functions and values. The scoping rules are also crazy. Because there's no variable declarations, you can modify names in enclosing scopes by accident. Python deals with this via the new 'nonlocal' statement. Also, the whole DSL craze and the role of blocks in that is just kind of strange to me. So forget that stuff. What I like about Ruby blocks is that they are an innovative way to do non-neutered lambdas in statement-oriented languages without dangling parenthesis.

Blocks occupy this weird middle zone between functional programming and stateful languages, because in functional languages or Lisps statements are either not allowed or are parentheses-wrapped expressions that you can stick anywhere you want anyway. Blocks are especially relevant in whitespace sensitive languages like Python and Ruby, where jamming a statement into an expression is awkward grammatically. Reia is a good case study for what happens if you try to force statements into expressions. So blocks are a little innovation to move the statements out of the expression and into a following block of code. In Lisp, the trailing parenthesis would be no big deal, but in statement-oriented languages it really messes up your grammar.

So what's the point of this stupid syntax hack so you can write multi-statement lambdas in stateful langauges? I think the reason that new Ruby programmers are so much in awe of blocks is because they haven't been properly exposed to first class functions before. A lot of them are web developers, and aren't interested in those high falutin' ideas about functional programming. I think that the key to teaching someone functional programming is lambda. Without the ability to embed executable code into an expression and pass it off to another function, you're left gesticulating wildly about how functions are values like numbers, strings, and lists. Lambda can really demonstrate that, as they say in 6.001, "the value of a lambda is a procedure." Once you've internalized that idea, you're ready for higher-order functions and the rest.

So while I think that in the existing Python ecosystem it makes sense to not have blocks, it makes it harder to teach and use higher-order functions. There's something to be said for the Ruby way of doing all of those examples above. They all use the same mechanism, and that's another kind of "there should be one obvious way to do it" in action.

In conclusion, if you're designing a new non-functional or whitespace sensitive language and you like the power of lambdas, blocks are probably a good way to express them. Patching them into Python now, however, would probably take away from the simplicity of the language.

Automatic __repr__ and __eq__ for Data Structure Classes

For the compilers class project that I'm working on, I recently wrote a couple of classes that do simple structural equality and automatic __repr__ generation. We have a lot of simple data structure IR classes in our project, and they all need __eq__ and __repr__ for testing and debugging. Structural equality is easy; all you have to do is introspect on __dict__ and see that the attributes match recursively. Automatic __repr__ is more difficult, though, because to produce valid Python source, you have to know the order of the arguments to the instance's __init__ method. Fortunately, with the inspect module, you can call inspect.getargspec(self.__init__) and get that information. We use the simple convention for our IR nodes that the arguments to __init__ all become attributes of the same name, so you can then use the argument names and getattr to generate the reprs of the subnodes. Good times!

Alternatives to django-media-bundler

Apparently I didn't do a good enough job Googling for other projects when I wrote django-media-bundler. If you Google for django concatenate javsacript or django minify javascript there are a number of other projects that do similar things, each with different tradeoffs:

• django-mediacat: To use this tool, you descibe your JS packages in Django models using the django-admin interface. You then configure a URL to point at the mediacat.views.cat view function, and pass it the package names you want as GET arguments. mediacat then caches the resulting file in the database model, and sends the appropriate ETags and Last-Modified tags to make the browser cache the request contents. Last updated: 2008-11-02
  
• django-compress: This tool can be configured to automatically regenerate its bundles by checking the file modification times of the original source files. Obviously, if you're using a content distribution network or a cookieless static domain, this feature might not work out of the box, but if you're a small-time operation, this is nice. It also allows versioning the filenames of the bundles so that when you have new bundles they will bust the browser cache. You can also use the YUI compression tools if available, and finally django-compress has a templatetag that will source the compressed script if compression is enabled, or the individual source files if disabled. Finally, I'd like to point out that this project looks the most mature, as it was started on 2008-04-28, and it has a well-written wiki and not just a README. Last updated: 2008-12-05
• django-compact: Looks like it's not quite finished yet, but so far it looks like its main feature is the templatetags that will link to individual script sources or the bundle. Last updated: 2009-01-30
  
• django-assets: Has Jinja 2 templatetags, and supports automatic regeneration of bundles based on file mtime. Bundles are defined inline in the templates instead of in a central configuration. This seems less good to me, because you want to keep the number of different bundles small, so that the user only has to download one script bundle. Sometimes you want more than one because a particular page has a lot of JS, but usually you want all JS to be cached after the first page load. Last updated: 2009-02-08
  
• django-assetpackager: This tool supports cache busting by putting the bundle generation timestamp in the bundle filename. It also has a templatetag that will source individual scripts in debug mode and just the bundle in production mode. Last updated: 2008-06-21
• Finally, django-media-bundler: While somewhat unrelated to concatenating and minifying JavaScript and CSS, my project supports image spriting, which is a pain to do by hand. It also employs an interesting little heuristic 2-D bin packing algorithm to try and arrange the images into a square-ish rectangle of minimal area. It has templatetags like a couple of the others, but it doesn't have cache busting. That's an important feature I'd like to add. The auto-regeneration I'm not convinced about, because it breaks down when you're not using a simple single-server setup. Also, it means that your templatetag has to do a bunch of file system calls while its rendering the template, which isn't a terrible idea, but it feels less than perfect. Last updated: 2009-02-15

Having browsed the source trees of each of these projects, in my (biased, of course) opinion the best tools here are django-compress and django-media-bundler. django-compress is mature and has the most JS & CSS bundling features, while the media-bundler is simpler (which can be good) and has image spriting.

django-media-bundler now supports sprites!

Over the last week I've been working on adding image spriting support to django-media-bundler. For the uninitiated, image spriting is a technique that Google, Yahoo, and other fast web sites use to speed up page load times. What these web sites do is to combine all of their small icon images into one medium size image, and then use CSS background image offsets to display each icon individually from the master. For small icon graphics, the overhead of the HTTP requests dwarfs the size of the actual image, so this speeds things up drastically.

With the help of the Python Imaging Library, I was able to read the images, measure their dimensions, and paste them together into the master image. However, given icons of arbitrary size, it's not clear what is the best way to lay out the master image. Having just taken an advanced algorithms course, this problem seemed like a variation of the bin-packing problem. The bin-packing problem is NP-hard, but once you have a name for something, it's a lot easier to Google up some simple heuristic algorithms to solve the problem.

Finally, I had to figure out how to get the sprites into the page. I found that in audio-enclave we use images in all sorts of interesting ways that make it difficult to abstract away the spriting behind a template tag. In the end I generated a set of CSS rules with the background image and offsets and decided to let the user figure out how to display the images. Working in the sprites was, for our project, more pain than it was worth, and I had to break a couple of nice CSS abstractions to force a DIV node into an element which had none before.

Anyway, implementation details aside, now audio-enclave has excellent front-end performance! Check out these Firebug net tab screenshots:

Before spriting:



After spriting: