More fun with Clojure lazy sequences

Recently I was writ­ing some code to ingest new entries for my church search web­site. The appli­ca­tion is writ­ten in Clo­jure, but the down­load­ing is done with an exter­nal script that down­loads new entries and saves them as files in a direc­to­ry, but at a slow rate. I wanted to be able to spec­ify a range of entries to down­load, and have the script run and then have my appli­ca­tion ingest the files that the script gen­er­ates. I wanted this to work con­cur­rent­ly, with mul­ti­ple down­load­ers and mul­ti­ple ingesters run­ning at the same time. And I wanted all of this to be trig­gered from Clo­jure.

This is basi­cally a con­cur­rency prob­lem so it’s a good thing I was try­ing to do it in using Clo­jure. The most obvi­ous solu­tion would be to launch mul­ti­ple down­load­ers, and then have a thread which watches the direc­to­ry, feed­ing filepaths to ingester threads using chan­nels and core.a­sync. Unfor­tu­nate­ly, I’m stuck using an older ver­sion of Clo­jure for this project for the moment and core.a­sync is not avail­able. Thank­ful­ly, lazy sequences can be used as a sub­sti­tute for chan­nels in a pinch… Read More

Parsing an AJAXy website.

Pars­ing a web­site used to be rather straight­for­ward affair: You down­load the web­pages and then you poke through the HTML with reg­u­lar expres­sions until you find what you’re look­ing for. You could set up a pretty good data extrac­tor going in Perl rather quick­ly. Later e replaced the reg­u­lar expres­sions with proper HTML parsers, but on the whole the process remained rather sim­ple.

These days how­ev­er, it’s rarely so sim­ple. AJAX, which is essen­tially the prac­tice of hav­ing the web­page com­mu­ni­cate with the server through javascript, has become so com­mon we don’t even call it AJAX any­more. Now we just call it ‘The way things are done.’(Calling a web­site ‘dy­nam­ic’ is another alternative.) This is nice in a way because it allows web appli­ca­tions to be much more respon­sive and inter­ac­tive. A mod­ern web­site feels more like a client pro­gram than a down­loaded doc­u­ment. On the other hand, this makes it much harder to parse web­pages… Read More

The Network is not Free

When attempt­ing to scale soft­ware sys­tems, that is, make it pos­si­ble for the sys­tem to take on a higher load or more users, peo­ple usu­ally take on of two strate­gies. ver­ti­cal scal­ing and hor­i­zon­tal scal­ing. Ver­ti­cal scal­ing is the prac­tice of buy­ing newer and faster com­put­ers on which to run the soft­ware while hor­i­zon­tal scal­ing is the prac­tice of sim­ply buy­ing more com­put­ers on which to spread the soft­ware across. Between the two, hor­i­zon­tal scal­ing is often inter­preted as hav­ing the bet­ter scal­ing poten­tial. Unfor­tu­nate­ly, this is not always the case and can lead to some naive attempts to solve very intractable prob­lems… Read More