... So We Built One
Sean Miller's old tech blog, to February 2009

2009-02-13: ImageScience, FreeImage, and Dreamhost PS

I wanted to use ImageScience (with the patch that adds support for rotation; ETA: now one-stop thanks to Dave Myron) to do some image manipulation in a Rails app, which I’m currently testing on a Dreamhost PS server.  ImageScience requires FreeImage, which loads on OS X easily with “sudo port install freeimage”, but on the PS server you can follow the instructions on the ImageScience page until you get to the point where you would execute “sudo make install”, which you can’t do.

Here’s my workaround:

1. On a local machine, get the FreeImage code:

cvs -z3 -d:pserver:anonymous@freeimage.cvs.sourceforge.net:/cvsroot/freeimage login
cvs -z3 -d:pserver:anonymous@freeimage.cvs.sourceforge.net:/cvsroot/freeimage co -D 2007-01-01 -P

2. Copy it across to the Dreamhost PS

3. On the Dreamhost PS:

make

4. Modify the Makefile.gnu file:

change:

INCDIR = /usr/include
INSTALLDIR = /usr/lib

to

INCDIR = {homedir}/.packages/include
INSTALLDIR = {homedir}/.packages/lib

and change:

install:
install -m 644 -o root -g root $(HEADER) $(INCDIR)
install -m 644 -o root -g root $(STATICLIB) $(INSTALLDIR)
install -m 755 -o root -g root $(SHAREDLIB) $(INSTALLDIR)
ln -sf $(SHAREDLIB) $(INSTALLDIR)/$(VERLIBNAME)
ln -sf $(VERLIBNAME) $(INSTALLDIR)/$(LIBNAME)
ldconfig

to:

install:
install -m 644 -o (your_user} -g {your_group} $(HEADER) $(INCDIR)
install -m 644 -o {your_user} -g {your_group} $(STATICLIB) $(INSTALLDIR)
install -m 755 -o {your_user} -g {your_group} $(SHAREDLIB) $(INSTALLDIR)
ln -sf $(SHAREDLIB) $(INSTALLDIR)/$(VERLIBNAME)
ln -sf $(VERLIBNAME) $(INSTALLDIR)/$(LIBNAME)

5. Run

make install

6. Since you don’t have direct access to ldconfig, add to the LD_LIBRARY_PATH in .bash_profile

export LD_LIBRARY_PATH="{homedir}/.packages/include:{homedir}/.packages/lib:$LD_LIBRARY_PATH"

7. Since Passenger won’t be going through your .bash_profile and seeing that LD_LIBRARY_PATH, also add it to your config/environment.rb file (further discussion here):

ENV['LD_LIBRARY_PATH']=”:{homedir}/.packages/include:{homedir}/.packages/lib”

Once that extended LD_LIBRARY_PATH is active, FreeImage will be accessible and ImageScience will work as expected.

Tags: ruby.

2008-11-26: ActiveRecord::Base.connection.execute(… multiple statements against a mysql db ...)

Out of the box, you can’t run this against a mysql db from Rails:

ActiveRecord::Base.connection.execute(”
    update table_name set field1=’value1′ where field2=’value2′;
    update table_name set field1=’value3′ where field2=’value4′;
    … many many more statements …
“)

In almost all cases it would be better to use ar-extensions with the duplicate key error update trick. Some colleagues of mine, who invented a plugin to invalidate find_by_sql unless you provide an excuse for each use, would put it more strongly than that.

But if, bearing in mind all the good reasons not to drop to sql from Rails, you still want to do it, and you want to execute multiple statements at once against a mysql db, you need to make a change to active_record/connection_adapters/mysql_adapter.rb. This is described in a different context in the Rails wiki, but the nutshell is, you have to add the flag for CLIENT_MULTI_STATEMENTS (65536) to the connection, so (in 2.1.0) the line:

ConnectionAdapters::MysqlAdapter.new(mysql, logger, [host, username, password, database, port, socket], config)

becomes

ConnectionAdapters::MysqlAdapter.new(mysql, logger, [host, username, password, database, port, socket, 65536], config)

Then, instead of getting the error

ActiveRecord::StatementInvalid: Mysql::Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘update table_name set field1=’value3′ where field2=’value4” at line 1: update table_name set field1=’value1′ where field2=’value2′;update table_name set field1=’value3′ where field2=’value4′ ...

the updates will work as expected.

Tags: rails, ruby.

2008-09-29: Code smell detectors for Ruby

A few weeks ago we were discussing the difficulty of building code smell detectors for Ruby. Since then, Roodi and reek have stepped into the breach. I haven’t tried them yet, but I’ll report back when I do!

Tags: ruby.

2008-07-26: John Steel, "Perfect Pitch"

I’m reading Jon Steel’s Perfect Pitch: The Art of Selling Ideas and Winning New Business, and, like Nick Sieger’s remarks about jazz, some of it seems uncannily relevant to programmers. There’s even a reference to “pair advertising”:

There’s a reason why copywriters and art directors work in pairs, and it’s not because writers can’t draw and art directors are illiterate. Ideas get uncovered more quickly when people dig together. And the ideas get better much more quickly when they are shared and debated by a small group of people who like and respect each other.

Words for every shop to live by. Steel also resounds Obie Fernandez’s remarks about the importance of being willing to say “no” to a contract if you aren’t going to be able to do your best work or be proud to put the project in your portfolio, and devotes a whole chapter to Tom Preston-Werner’s conceptual algorithm of setting aside dedicated thinking time.

For anyone else suffering withdrawal symptoms from the stimulating and wide-ranging discussions at RubyFringe, this could be just the book.

Tags: agile, books.

2008-07-20: RubyFringe, Day Two

Geoffrey Grosenbach spoke from his university background in philosophy and interest in music, and wondered interestingly if developers are less aware that modern methodologies derive from the scientific method because they’re stuck in the implementation phase, without necessarily being involved in coming up with the original ideas or validating the experiments. The link from Spinoza to cloud computing was a long but entertaining stretch. Invoking the Sapir-Whorf hypothesis as a good reason to do a desktop app instead of a web app every so often (or vice versa), for the wider perspective, struck home.

John Lam spoke of how you gravitate to the same problems over and over, drawing lines from IronRuby to his early days metaprogramming in assembler on a Commodore 64 and building bridges between components, and of how you can change your mind, remembering his early paper on how dynamic languages suck. He explained his move to Microsoft on the grounds that he really wanted to build stuff that people would use: you can build stuff in open source, but brilliance alone won’t guarantee that people use it.

Hampton Catlin spoke about Jabl, which he cheerfully billed “the language you will hate”, his attempt to fix javascript (which he sees as a good if verbose general-purpose language but a terrible browser language that needs frameworks to fix it). Jabl will compile into javascript, is backed by JQuery and provides a compact DSL for client-side DOM manipulation. It look neat and quite compact (a good deal more compact than the equivalent JQuery, and a great deal more compact than the equivalent raw javascript), and as soon as there’s a compiler it will be well worth checking out.

Only the video, when it comes out on InfoQ, will be able to do justice to Giles Bowkett’s high-energy performance, which in addition to demonstrating his Archaeopteryx MIDI software was an impassioned call to do what you love and build what you want. Make profit by providing a superior service at the same price point as the competitors: we’re still banging the drum of making your offering less painful than the competition. Also, Leonardo da Vinci was an artist who didn’t ship and so, by “release early, release often” standards, a failure, there were several images of Cthulhu, references to using a lambda to determine which lambda to use (a meta-strategy pattern), and Bowkett’s thought that it was irresponsible in Ruby not to use the power the language gives you. He added that Archaeopteryx may be insane but it’s also great, and “insanely great” he can live with, and that while he may not be able to say (as Jay Phillips used to of Adhearsion) “my career is Archaeopteryx”, being able to say “my career includes Archaeopteryx” is still very good.

Damien Katz spoke about the long and hard road that led to him being the guy building the cool stuff, in his case CouchDB. It felt in part like a counterpoint to Bowkett’s story: took longer to pay off, but he did what he really wanted to do on the chance that someone might recognize it, and pay off it finally did. Katz also noted that while it was easier to get something working in Java than in Erlang, it was easier to get it working reliably in Erlang than in Java.

Reginald Braithwaite spoke about rewrite, which enables rewriting Ruby code without opening and modifying core classes. This makes it easier to scope changes to specific parts of your code. Reg likes the power of opening classes, but doesn’t think it sustainable. If you open a class in a gem, all the downstream users will be affected, whether they want the change or not. Reg also mused on adverbs, like .andand. or .not., modifiers that affect the verbs, the methods, instead of the more usual noun/object orientation. (Reg has since blogged a riff between his talk and Giles’s.)

Tom Preston-Werner spoke about conceptual algorithms, not applied to coding patterns but to higher-level thinking patterns. He gave an extended example of the full circle of the scientific method and its effectiveness in debugging, noting the usefulness of Evan Weaver’s bleak_house plugin in tracing memory leaks. He previewed the upcoming git feature gist, basically pasties backed by github. He discussed some other algorithms more briefly. Memory Initialization starts with disregarding everything you think you know about a problem and working from first principles (it worked for George Dantzig). Iteration is exemplified by the James Dyson new vacuum cleaner design, which after 5126 failures became a resounding succes. Dyson remarked, taking us right back to Nick Sieger’s jazzers, “Making mistakes is the most important thing you can do”. Breadth-First Search relies on the fact that there are over 2500 computer languages, and learning some others just for the different perspective is useful. (He shared a tip from his time learning Erlang and being weirded out by the syntax: just say to yourself, over and over in an affirming manner, “the syntax is OK”. And that helps you get past it and get on to the good stuff.) Imagining the Ideal Solution feels like a cousin of TDD: if you’re not sure how to implement the solution, write out your best case of what the god config file (say) or the Jabl syntax (say) would be, and then proceed to implement that. And Dedicated Thinking Time was an unexpected reflection of a point I first came across in Robertson Davies: it’s important to set aside time for just thinking about things. Sure it’s simple, but it’s sometimes ridiculously hard to implement.

Blake Mizerany spoke about Sinatra, one of the Ruby web app microframeworks, built (in line with Jeremy McAnally’s thoughts yesterday) because Rails was too slow for his needs. He noted some apps in Sinatra: git-wiki (a wiki served by Sinatra and backed by git), down4.net, and the admin interfaces of heroku. He also suggested that for RESTful resources, rest-client (which he called “the reverse Sinatra”) and Sinatra was all you needed, making ActiveResource look like overkill and curl look cumbersome. To complaints that Sinatra wasn’t MVC, he noted that he thought it was, actually, but in any case, insisting on a web framework being MVC felt like over-eager application of patterns instead of starting with the problem. Asked what Sinatra was tailored for, he said back end services or other small bits of functionality. If you’ve got more than a few screens of Sinatra code, it’s probably time to move to another framework.

Leila Boujnane spoke about the importance of being good, and being useful, and making the world a better place by making something that people want. Don’t start with a business model: start by building something useful, and build it well, because a crappy product upsets both the customers and the people working on it. Making the customers feel good about the product is not only good in itself, it’s the least limiting approach, because if your product doesn’t do that on its own you need to cheat or fool people into thinking it’s terrific. In a way, we’re back where the conference started: find a pain point and build something useful to fix it, to everyone’s benefit.

Oh, and the Toronto Erlang group was founded over lunch.

High praise to the folks at Unspace for making the conference happen. It was thought-provoking and loads of fun.

Joey deVilla’s more expansive account: one, two, three.

Tags: conferences.

2008-07-19: Moving Past BlueCloth

A few months ago I started a project which involved transcribing hundreds of thousands of words from handwritten notes (so that I could use the excellent Ultrasphinx full-text search on them). I needed a clean fast markup system, and since they were my notes, it had to be able to handle nested blockquotes, because there were quotations within quotations within quotations. RedCloth/Textile failed the “works in two minutes out-of-the-box” test on nested blockquotes, so I went with BlueCloth/Markdown.

I tripped over a bug right away: BlueCloth didn’t format two-character words (so my in italics above would still have been _my_ in the output). This was an easy fix, so I changed my local copy of the gem and kept going, wondering whether I should put the fixed version on github. (Since this was a personal project, I could ignore my qualms that there had been no work on the gem since 2005 because it did everything I needed, once I fixed it.)

And then a month and a half ago, Ryan Tomayko wrote Moving Past BlueCloth, revealing three newer faster and better implementations of Markdown. I started a new job at the same time, so I haven’t checked them out yet, but I look forward to doing that very soon, and retiring my patched BlueCloth. Not this weekend though, because it’s RubyFringe.

Tags: link.

2008-07-19: RubyFringe, Day One

Jay Phillips talked about Adhearsion, his framework for integration VoIP building on top of and simplifying the use of Asterisk. He noted that it can be surprisingly profitable to find a pain point a bit off the beaten track and fix it. This chimed well with Zed Shaw’s remark from QCon London 2007 that if you want to succeed beyond your wildest dreams, you should find a problem and solve it in a way which is less painful than any of the competing alternatives.

Dan Grigsby noted that entrepreneurship was easier for a developer than perhaps it had ever been, because the sweet spot for team size was about 3. If you’re a really good programmer, all you need on the code side is one or two other really good programmers. This matches Pete McBreen’s point in Software Craftsmanship, that a smaller team of really good people will probably be better and more productive than a larger team. The trick that corporations don’t seem to have solved yet is finding that small great team in the first place.

Grigsby added that a more pragmatic test-driven approach to the developer’s dilemma of picking the winners among all the possible project ideas is to build a whole bunch of narrowly-focussed projects, send them out, measure them to within an inch of their lives, and quickly kill off the failing ones. For a similar amount of effort as honing one or two projects for a much longer period before sending them out, you’re left with a couple of projects that are less well developed but you already know they are successful.

Grigsby’s engaging talk continued into more explicitly market-focused directions, discussing finding hacks in the market for the disproportionate reward, suggesting that there are marketing techniques that match Ruby’s better-known programming claim to 10x better productivity, and that exploiting non-obvious relationships was among them.

Tobias Lütke spoke about memcached.

Yehuda Katz spoke about several neat projects, from Merb and DataMapper (coming to 1.0 this summer so almost no longer Edge-y) to sake (for system-wide rather than project-specific rake tasks), thor (scripting in Ruby), YARD (a neat-looking better Ruby documentation tool), and johnson (a ruby-javascript bridge).

Luke Francl spoke on testing being over-rated. He made clear that testing is great, it’s just that there are a lot of other things we need to do (code inspection, usability testing, manual testing), some of which will find different bugs and none of which will find all the bugs. As he pointed out, you can have the most complete set of unit tests imaginable, and it won’t tell you if your app sucks. Sitting and watching a user interacting with the app may be the only way of being sure of that. Francl also pointed out that in addition to writing code and writing tests, developers are good at criticizing the code of others, and the best measure of code quality may be not rcov stats but the number of WTFs per minute in a code review. And, as a side benefit, knowing that your code will be subject to a code review will probably make you tighten up your code in anticipation. (Francl’s paper was also notable for following the Presentation Zen advice of having clear uncluttered slides and a handout that was an efficient text summary of the presentation that bore no relation to the slides at all.)

Nick Sieger spoke about Jazzers and Programmers. Go read his summary and savour the quotations (from a different field but still deeply appropriate) from great jazz musicians. “Learn the changes, then forget them” (Charlie Parker). Or the idea of The Real Book (the body of tunes that all jazz musicians learned and created a shared vocabulary that allowed musicians to play together on their first meeting) as the GoF for jazzers in the 70s.

Obie Fernandez spoke about marketing and practical details of contracts. Not only about accepting contracts but about rejecting them, noting that a key acceptance criterion should be if you’re going to want to show off the project when it’s done and keep it in your portfolio. Following on from patterns and The Real Book, he noted the importance when setting up a consultancy of defining projects and of having names for them, and of writing down your client criteria and acceptance criteria and keeping them constant. Noted Secrets of Power Negotiating, Predictably Irrational, and Seth Godin’s Purple Cow as books to check out.

Matt Todd spoke about the importance of just going ahead and making mistakes and learning from them. (Going back to one of Sieger’s quotations, as Ornette Coleman put it, “It was when I found out I could make mistakes I knew I was on to something.”)

Jeremy McAnally spoke of how we should resist the urge to make Rails do everything, even outside its comfort zone where it would be a lot simpler and more maintainable to, say, just write some Ruby code and hook it up to with Rack. How Rails is designed for database-backed web apps, and taking it outside of its 80% comfort zone, while perfectly possible, may not be the best solution, and if you find yourself in a place where the joy:pain ratio has flipped or where you find you’re writing more spaghetti code to fit the solution into Rails than you would be to write the solution on its own, it’s probably time to switch.

Zed Shaw briefly presented some dangerous ideas and then went on to an impromptu music-making session. He left us with the thought that while he was done with Ruby, he was never going to stop coding, because code was the only artform that you could create that other people could then play with.

Joey deVilla’s more expansive account: one, two.

Tags: conferences.

2008-05-28: CruiseControl.rb, rspec stories, and rake SpecTask

Yesterday rspec-1.1.4 was released, and one of the fixes was that rspec’s story runner now returns an exit code of 1 instead of 0 on a failing scenario. Since I needed to get our rspec stories running in our build, I grabbed the change. It didn’t seem to work straight out of the box, but that turned out to be because rake’s SpecTask swallows the exit code.

Running

ruby stories/all.rb

works and returns an exit code of 1 on a failing scenario, as does wrapping the call in a rake task like

namespace :spec do
  desc “Run all spec stories”
  task :stories do
    ruby “stories/all.rb –format plain”
  end
end

but wrap the stories in a rake SpecTask instead and while they’ll still run, and the log will show failed scenarios, the task will always return an exit code of 0, so a failing scenario will not break the build.

Thanks to Ben Mabey and Pat Maddox for answering questions on rspec-users.

Tags: rspec.

2008-05-04: Oliver Steele’s DBContent plugin

http://osteele.com/archives/2008/04/db-content-rails-plugin

Adds rake tasks db:content:dump, db:content:load, and db:content:archive.

I looked for it because I’d added a lot of data through a web interface and wanted an easy and Rails-friendly way of keeping it and restoring it. This did precisely what I needed.

It is limited to MySQL databases, and Windows users who don’t have gzip on the path will want to edit tasks/db.rake and remove the .gz from the .sql.gz filenames — if the gz is there it’ll (try to) compress the files, but if it isn’t it won’t.

Tags: rails.

2008-03-18: Unobtrusive Javascript

Quite possibly one of those things that will seem obvious at the other end of the morning, but to get unobtrusive_javascript working with Rails 2.0.2 (out of the box it blows up with a “wrong number of arguments (2 for 1)” error), you need to modify the plugin’s lib/tag_helper_patches.rb file, line 17,

def tag_options(opts)

to include an optional second argument

def tag_options(opts, escape = false)

because the method being over-ridden in tag_options can be called with a non-optional second argument. (The second argument is not used in the unobtrusive_javascript version of the method, so it doesn’t matter what value we assign to it here.)

Edited to Add

Sure enough, the other end of the morning reveals a discussion from the builder of ujs from June 2007 (http://www.danwebb.net/2007/6/16/the-state-and-future-of-the-ujs-plugin) about deprecating the ujs plugin and just using the lowpro.js code on its own. And since everything I was trying to do still worked with that and I didn’t have to modify routes.rb to do it, that’s fine by me.

So, a better starting point?

Grab a recent Prototype (http://www.prototypejs.org/download) (1.6.0.2 as opposed to Rails default 1.5.1_rc3)
svn co the latest Low Pro (http://svn.danwebb.net/external/lowpro/trunk) (0.5, as opposed to 0.1 from ujs)

Check out Dan Webb’s Low Pro blog filter (http://www.danwebb.net/lowpro/), the Google Group (http://groups.google.co.uk/group/low-pro), and a pair of introductory articles from Jarkko Laine (http://jlaine.net/2007/8/3/from-rails-ajax-helpers-to-low-pro-part-i)

Edited Much Later (18 July 2008) To Add

There’s a “full and frank discussion” raging at my current workplace about whether unobtrusive javascript is a good practice or a code smell. Me, I like it. I used it on a plugin I built as a solo project and I really liked the extra separation of concerns it enforced. Your html becomes just content, even onclick calls are woven in from elsewhere. I will admit that if you’re introducing it into a codebase with twenty other developers you’ll need to make sure they all know about it too, else bad things are likely-to-inevitable when a less familiar developer tries to debug it. Given a bright group of developers, though, I’d still choose to use it.

Tags: javascript.

2006-09-09: Alexander talks back

Ever wonder how Christopher Alexander felt about being the poster child for software design patterns? In his foreword to Richard Gabriel’s book he answers the question, and poses another. Are our designs as good as Chartres? If not, why not?

In fact, this is so interesting and challenging that I’d like to make a fuller extract, which I hope will be within fair use rules:

In my life as an architect, I find that the single thing which inhibits young professionals, new students most severely, is their acceptance of standards that are too low. If I ask a student whether her design is as good as Chartres, she often smiles tolerantly at me as if to say, “Of course not, that isn’t what I am trying to do. … I could never do that.”

Then I express my disagreement, and tell her: “That standard must be our standard. If you are going to be a builder, no other standard is worthwhile. That is what I expect of myself in my own buildings, and it is what I expect of my students.” Gradually, I show the students that they have a right to ask this of themselves, and must ask this of themselves. Once that level of standard is in their minds, they will be able to figure out, for themselves, how to do better, how to make something that is as profound as that.

Two things emanate from this changed standard. First, the work becomes more fun. It is deeper, it never gets tiresome or boring, because one can never really attain this standard. One’s work becomes a lifelong work, and one keeps trying and trying. So it becomes very fulfilling, to live in the light of a goal like this.

But secondly, it does change what people are trying to do. It takes away from them the everyday, lower-level aspiration that is purely technical in nature, (and which we have come to accept) and replaces it with something deep, which will make a real difference to all of us that inhabit the earth.

I would like, in the spirit of Richard Gabriel’s searching questions, to ask the same of the software people who read this book. But at once I run into a problem. For a programmer, what is a comparable goal? What is the Chartres of programming? What task is at a high enough level to inspire people writing programs to reach for the stars? Can you write a computer program on the same level as Fermat’s last theorem? Can you write a program which has the enabling power of Dr. Johnson’s dictionary? Can you write a program which has the productive power of Watt’s steam engine? Can you write a program which overcomes the gulf between the technical culture of our civilization, and which inserts itself into our human life as deeply as Eliot’s poems of the wasteland or Virginia Woolf’s The Waves?

There’s no question that progress towards finishing a project in a week is more easily and more objectively measurable, and more obviously attractive to the business side. But for the longer-term but still important purpose of getting and keeping the best developers, I think we can’t afford to lose sight of this other axis either.

Tags: quality.

2006-08-28: I never heard So musical a discord, such sweet thunder

... but, when it isn’t a question of a pack of Spartan hounds tearing a bear to shreds, poor listening skills are more a problem than a theatrical entertainment.

I found this post that chronicles eight common barriers to good listening and suggestions to get past them quite interesting. I will not, like Dante in the Purgatorio, go through them all and explain which ones I am particularly prey to, but I know all too well that I’m not in cleared-to-throw-stones territory.

Tags: communications.

2006-05-06: Waterfall as Smokescreen and the Surprisingly Long History of XP

I was reading the Thoughtworks blogs and found a neat reference to Craig Larman’s keynote speech at Agile India 2006. Larman discussed the oft-repeated comment that agile is not suited to things like large defence projects, but when Larman checked out Sapolsky’s 1972 book on the history of the Polaris system, found that Sapolsky had said that “critical path analysis and PERT charts were used as a smokescreen to placate senior management while engineers got on with their iterative development process”. (Quoting the blog entry, not the original book.)

I haven’t tracked down Larman’s 2006 paper (since the conference was today and yesterday it may not be up yet), but here’s a link to a paper he co-wrote in 2003 which chronicles iterative development going back into the 1930s and something very like XP popping up in 1957.

Tags: agile.