My remarks on quality management in general are, naturally, applicable to software – if only because it's the industry in which I've formulated my opinions on the subject. Unlike (more mature) engineering disciplines, the software industry has a remarkably relaxed attitude to quality management; and a remarkably wide gulf between industry common practice and best practice, not just in how it produces software but in how it manages the quality of its products. As such, it is an especially entertaining industry to study.
The software industry is remarkably large for its age, or young for its
size: it's a giga-dollar industry doing something simply incomprehensible to the
world of as little as twenty years before I was born. Indeed, there arguably
wasn't such a thing, as an independent industry, until well into my lifetime:
until the 1960s (and, to a large extent, even well into the 1980s), computer
programs were inseparable from the computers on which they ran; they were either
written by their users or supplied, as part of
the computer, by the
computer manufacturer. Its very youth, along with the strong tendency towards
youth in its workforce, makes it unusual – but then, the consumer
electronics industry is of comparable age yet much less peculiar, so youth isn't
the whole story of what's odd about the software industry.
The most singular thing about the software industry – and arguably the most important towards making it peculiar – compared to other industries of similar scale is that it is (almost) all design.
Consider the modern consumer electronics industry, for contrast: as in pretty much any industry, the company whose logo you'll see on the end product employs someone clever to design a new product. An important factor they take into account, in the design, is making sure it can be mass-produced at a price per unit that'll let them make a profit. Once they've completed the design, their company might do the manufacturing itself, but the trend these days is to farm out this low profit-margin business to someone else. In any case, crucially, completing the design doesn't give you a working product: you have to manufacture it, and the cost of manufacturing each unit is a decisive factor in your profitability; there are also non-trivial costs involved in distribution, albeit these have less impact on profitability (because your competitors are paying just the same as you for it). Furthermore, in the design phase, it likewise costs money to get a prototype made so that you can play with it and refine your design.
In the software industry, in contrast, the manufacturing
process for
a prototype – compile, link, package – is taken care of by standard
tools, with a negligible unit cost, that make it easy to prepare and play with
prototypes; the manufacturing
process for the final product is just to
copy the last prototype, at a zero unit cost, and this step is generally
integrated into the distribution process. At its most expensive, this last
involves burning the data to a CD or DVD, which costs very little, and
delivering it to the customer: and even the tiny costs of this are rapidly being
eliminated in favour of simply letting the user download from a web
server. Running a web server is not free, but it's very cheap. Crucially,
almost all of the cost of designing, manufacturing and delivering a software
product is in the design phase, which is to say the development and refinement
of the software.
One of the principal errors that managers from other industries make, when
dealing with the software industry, is in looking at the development of software
as manufacturing
(because it's the bulk of the cost) and trying to manage
software developers the way they would manage production-line workers, instead
of managing them as designers collaborating on an intricate and complex
system. Such managers tend to think of production-line workers as cattle (a
mistake in its own right) to be herded (i.e. bullied and cajoled, limiting their
choices so as to drive them into behaving the way the manager desires): when
they try the same approach to programmers, it fails spectacularly, leading to
the famous description of managing programmers as
being like herding cats
. The
fact that software is (almost) all design makes a similarly huge difference to
how you manage its quality.
The great miracle of software is just how much a remarkably small team, or even a single programmer, can achieve – and how fast they can achieve it. Inconveniently, however, this doesn't scale so well as one might hope. When everyone in the team can get their head round the whole of the end result they'll be producing, it's possible for each of them to ensure the part they're doing is going to work well with all the other parts. They'll have to discuss how the parts are going to work together, but their understanding of each others' parts helps keep such discussion from taking up much time.
Unfortunately, for a larger project, it gets harder for team members to hold
all of the design in their heads: so discussions with peers take longer and,
what makes this really crucial, the number of such discussions grows as the
square of the size of the team and/or the number of sub-systems. That can be
mitigated by modularising the design: break the system up into such parts as to
leave relatively little interdependence between parts and to keep those
interdependencies clean and simple. Then those not working directly on any
given module only need to know the external façade it presents
(its application programming interface
, or API), not its internal
details; they can concentrate their attention on the details of the modules on
which they do work.
As the system grows, all the same, the complex tangle of interdependencies among its parts shall slow you down – and create opportunities for bugs to creep in and make their nests in the places you least expect them. More importantly, if you fail to acknowledge that you're going to have to change the way you work, as your product grows, you'll be managing the process in ways that work poorly: and that's going to hurt quality in ways that'll surprise you – which means your customers will notice it before you do. How you respond to that will be decisive to your survival as a project.
When it comes to managing quality, a formal approach calls for keeping track of work that needs to be done, recording what work has been done and retaining an audit-trail of the details. This can as readily be done in software as on the carcasses of dead trees and, when done in software, it integrates particularly well with the tools for working on software. Indeed, the tools commonly used for working on software actually make some parts of a formal process happen automatically.
For many kinds of software, it is possible to set up automatic testing that ensures a large proportion of the problems that might otherwise be neglected (essentially, if a test is too boring to do, humans on your project shall probably not do it as often as your process needs it done: but a test-bot doesn't get bored). Although automatic testing (a long-standing industry best practice) is sadly not yet universal, folk are noticing how useful it can be (so at least it's getting to be a moderately common practice, albeit not yet standard practice) and this helps to ensure that project teams know about problems, at least the ones for which it's practical to test automatically, as soon as they arise.
Any even half-way sensible software project keeps track of the bugs that
have been found, one way or another – easy, free, off-the-shelf,
reasonably good bug tracking systems are readily available (and some of their
paid-for competitors are good, too) so widely used, but even if you're just
keeping the bug reports in one mail folder until fixed, then moving them to
another, you're tracking bug reports. Any bug-tracking system at least provides
some form of identifier for each bug (even if it's just
the Message-Id header of an e-mail), by which comments in code and
the descriptions of automated tests can refer to them. A good bug-tracking
system can be a great aid to planning, organizing and tracking work,
particularly if it's really an issue
-tracking system, capable of
distinguishing between causes (errors in code), symptoms (problems experienced
by users; several of these may share one cause, some may result from the
perverse interactions of several causes), feature requests and a few other types
of stuff we (may) need to deal with
, along with the threads of
discussions among involved parties about what the correct behaviour should be,
the various ways to achieve that, the possible impact on other things and so
on.
Any even half-way sensible modern software development project also uses
some form of version tracking: whether that is an archive of snapshots or a more
sophisticated version control system, it provides for developers to be able to
rewind time to anywhere in history and it enables delivery teams to work on a
side-branch of the history into which they only take changes necessary to their
delivery, while the primary history of the project marches forward with
development that might introduce bugs. A simple archive of snapshots suffers
from the problem that, when someone changes code, it merely records what
changed, not why it changed (the developer might include a comment beside the
changed code, explaining the change; but this may seem superfluous when making
the change): in even the most basic version control system, each change records
a check-in comment
, in which developers record what they changed and
why. After using such a system for any amount of time, developers get quite
religious about providing informative check-in comments: when a programmer
wonders why some particular piece of code is done the way it is, the
version-control system can report the history of changes that lead to it being
so, along with the associated check-in comments explaining why it is so
– or, at least, if they do, it's usually easier to make sense of the code;
if they don't, the developer investigating the code wishes they did, so learns
why it's useful to write good check-in comments. In practice, non-antique
version control systems make it easy for developers to get one another's
changes, so developers generally do use one modern version control
system or another, even if the official version tracking system of the project
is cruder. Consequently, most software development projects – no matter
how informally managed they are – naturally collect a record of what's
changed and why.
When a project has split the parts of its code-base up into modules –
components with relatively few interdependencies, via APIs, as described above
– it becomes possible for those responsible for one module to direct
everyone else to use a particular version, while they get on with developing
some new feature, reorganizing what's there or doing other disruptive work whose
intermediate stages are unsuitable for use by other teams. A version tracking
system thus provides a mechanism for light-weight internal releases
of
parts of a system that insulate other parts from transient breakage that
individual teams tend to need to cause in the process of getting from one known
good state to a better state. By providing tools to facilitate such internal
releases, and ensuring those tools ask those making the release suitable
questions and include suitable information in the internal announcement e-mail
for the release, one can arrange for the easiest way to do an internal release
to incidentally do all the things a quality management process needs. Placing
most of the release information on a web-page accessible to developers provides
a practical place to record issues found with a release after it has been
released; this might be integrated into the bug-tracking system, for
example.
Even in an informally managed project, such tools are used and ensure there is a data-trail showing who did what, why and when. With only a little effort, one can arrange for quality management to be very unintrusive, from the point of view of those doing the work.
The software industry exhibits a very wide range of degrees of formality in its quality management practices; and, at opposite extremes of that range, quite different secondary axes for variations with significant impact on the quality of the results. As it happens, I'll argue below, the debate over levels of formality is a side-show: the real controlling factor in software quality management is the recruitment process.
The software industry is noted for a high degree of informality in other respects, so it should be no surprise that much of its quality management is likewise informal. The software industry has some fine examples of delivering high quality with remarkably light-weight management: but users are all too well aware that parts of the industry often simply neglect quality.
The central axiom of the informal approach to managing software quality is
that you've got a bunch of bright, motivated people who want to make a great
product: if you (mostly) get out of their way, they'll do that. The
1990s tech bubble
demonstrated how badly that can go wrong: if you just
throw money at programmers, pamper them and hope for miracles, you're
doomed. Not that I've anything against throwing money at programmers; and a bit
of pampering can be welcome, too: but you do have to pay attention to whether
we're actually doing anything useful (amazingly enough).
The informal approach tends to work best with small teams, at least in
proprietary development: but free software (a.k.a. open source) projects have
demonstrated that it can work on a larger scale too. The proprietary case may
just be an artifact of the changes in managerial structure investors tend to
demand when a proprietary software company grows past the stage where it can
survive without outside investors: up to that point, the team has very little
management, most of which consists of programmers, respected by their peers, who
are actively involved in reviewing – and competent to review – the
work of the rest. Projects that have made this work on a larger scale have
typically done so by ensuring that programmers who have the respect of other
programmers have authority over what changes make it into the released product:
they build up a hierarchy of such gate-keepers, each of whom controls what goes
into a sub-system and propagates those changes to the king pin who controls the
whole. As long as the authority rests in the hands of able programmers who've
earned the respect of their peers, and the gate-keepers maintain a healthy
mutual respect and constructive attitude towards potential contributors, they
can make this work. (Eric Raymond's
essay The
Cathedral and the Bazaar
describes this approach in some detail.)
Variations on this model (the sketch just given is loosely based on the Linux
kernel project) have enabled various free projects – and even some
proprietary ones – to retain an informal structure for their development,
while keeping a reasonable degree of control over the quality of the results,
long after they've grown too large for any one participant to fully understand
every part of the system.
Of course, lurking beneath the surface of such a mechanism – at least
if it's producing decent-quality results – you'll find that systematic
testing is going on – each gate-keeper typically does at least
some testing of incoming contributions and gets some testing done by other
participants to the project. Some of these last may indeed be explicitly in
testing rôles and some users may be following the bleeding edge
to
provide such testing less formally. Automated testing (see below) can also help
greatly. Problems may also be found by other developers in the project, as the
changes propagate back out; and it works best when the difference between
developers and users is blurred – if the product is something developers
would be using even if they weren't working on it, they'll be doing testing just
by making normal use of it. The process, for ensuring the changes that make it
into the final result don't damage its quality, is still informal: essentially,
the gate-keepers are trusted by other participants to chose wisely. There is no
formal requirement that they run a specific test-suite, save the results to an
archive, compare them with prior results or log any regressions. The process is
informal: it is based on trust and respect, which have generally been earned;
and it is made to work by having a well-structured (albeit informal) working
process that implicitly integrates quality management, albeit barely
visibly.
Another system I've seen working well, in a project where everyone could directly check in changes to all parts of the system, is to have an automatic test system with a broad barrage of tests running all the time. Every time it completes a test set, it does a fresh build and starts a fresh test set. If the full set of tests takes too long to run, make routine builds at regular intervals and, each time one is ready, have the test system remember where in the tests it is, shut down, then start up again where it left off, but using the new build. Every time someone fixes a bug, they should add a regression test to the system, that shall fail if the bug is re-introduced; every time someone extends what the system can do, they should add tests that verify the system is doing those things right; indeed, every time someone notices something that should be tested and isn't, they should add tests. When a test fails, the system should automatically register a bug for the failing test, or re-open a prior bug, and hassle (at least) the people who most recently changed (apparently) relevant code. When they've worked out what actually caused the failure, they can tell the system which code change was the actual cause (thereby adding the affected code to what the system shall think may be relevant next time this test regresses) in the course of fixing the problem. Modern version control systems come with features that make it possible to automate hunting for exactly which change caused a given test to fail, so this system could, today, be implemented rather better than I saw it being done (well) in the late 1980s. There is, of course, no reason why this approach cannot be combined with a gate-keeper system; and it should be fairly obvious that the combination would work well.
There are, of course, also informal approaches to quality that don't work. Some projects' testing doesn't get much past what would be termed a smoke-test by anyone who takes testing seriously: that is, run the program, do some basic trivial stuff with it, check the results aren't obviously wrong. If it doesn't crash during that, ship it ! That actually works well if those to whom it's shipped know there may be issues and that the shipment is merely one of an on-going sequence of frequent deliveries (so any bugs they report shall be fixed soon): but it's all too often done as the first delivery after a long development phase, with delusions of being a final delivery. That's hopelessly irresponsible; but, when the customer demands unrealistic results unrealistically soon, there may be no time for testing; ultimately, the irresponsibility in such a case lies not with the developers but with the customer who asked too much or (more usually) the salesman who promised it.
Sadly, the caricature in the last paragraph is uncomfortably close to being
industry standard practice (note that that this is emphatically not the
same as industry best practice). Even badly-run projects do rather
more testing than that, but those planning projects are all too apt to ignore
the then we'll test it, fix the bugs we find and repeat that until we think
it's good enough
phase of their project or, at the very least, assume it'll
be a small fraction of the total. It's probably around half of the total, in
practice. When the plan doesn't provide enough time to do this part of the
project well, the project ends up rushing to a late delivery which hasn't been
adequately tested or bug-fixed.
Aside from neglecting testing, and planning for it, the most obvious way projects are wont to fail is by lacking adequate structure. If every developer can check in changes on every part of the code, in the version that all their peers are using, there are grave risks of mayhem, particularly when a developer has no clear way to know who to ask for guidance on how to change some part of the code with which the developer is unfamiliar. Good developers are capable of the humility needed to offer a respected peer a suggested change and explanation of why it's needed, and glad to be spared the horrific responsibility of making a change that might break something they don't know about – which is what a gate-keeper system provides. The problems with letting all project members change all parts of the system can be mitigated by making sure you have good staff and by having a good automated testing system; but it remains that even good staff benefit by each part of the code having someone identified with it as being who to go to with questions about that code – if you don't have gate-keepers, it's still worth identifying experts for each part of the code.
Amazingly, smart people do sometimes manage to produce working products with (under the circumstances) remarkably few bugs in them, even when management neglects to plan a testing/bug-fixing phase and expects all developers to work on all parts of the code, without even designating experts to take responsibility for particular components. On the other hand, such unstructured informal approaches, particularly when attempted on larger projects, routinely produce nightmares of inefficiency, endless project delays, cost over-runs, vast amounts of bugs and, ultimately, maintenance nightmares. The fear of these leads many, particularly corporate managers, to seek some more formal way to manage projects and their quality.
At the formal end of the spectrum, one sees very different vices and virtues: management may have more control over how the job is done, but its control over how good the end result is varies just as much.
Just as what drives many corporate types to seek a formal approach is the
nightmare stories they've heard of the write it, ship it, hope it's good
enough
informal approach, one thing that scares many in the software
industry away from formal quality management is the horror stories that
circulate from businesses that have imposed a heavy burden of bureaucracy on
programmers – typically without solving the quality problems at which they
aimed. The fundamental cause of this is that the one part of the process
management least often actually realizes must be subject to stringent
and bureaucratic quality processes (if you're going to do that for any part of
the process) is management itself. If random acts of drive-by management can
divert resources from a project, without those responsible for the project's
success even being notified of it, then projects won't succeed. People doing a
job whose results are needed by others must have the authority to refuse to be
dragged off that job, or at least to require the dragger (no matter how high and
mighty) to go via the managers responsible for managing the grabbee's time.
If a manager can demand weekly written reports from underlings detailing
progress, in detail well beyond his ability to comprehend, along with analysis
of how progress relates to plan (this means the manager is trying to cover his
posterior: underlings are scared of saying things aren't going well, so the
manager gets a stack of documents proving
that he'd diligently checked
that all was going well, so it's not his fault
if things go wrong), then
projects shall fall behind plan by the amount of time needed to comply with this
edict (when it happened to me, I carefully documented all I was asked to
document, including that the project was slipping behind schedule by the amount
of time it took to write the report: thereby ensuring the manager five layers up
in the stratosphere was actually covering his posterior with a Kick ME
sign, in the event of my project missing deadlines; my immediate managers loved
signing off on my reports). Of course, there is a place in a formal project
management regime for progress reports, but they should only contain the level
of detail actually needed to track project progress; and they should probably be
produced by someone a layer or two of management above where the work is
actually done – albeit signed off by those doing the job, with some
mechanism for them to introduce corrections. The people actually doing
the work shouldn't be kept from doing it by the process of reassuring management
that they're doing it.
There are endless other ways that formal process management can impede the
process of getting the job done, and formal quality management
ensures
bad products; if you want an over-view of some of the more egregious examples,
read Dilbert. I'm told laughter is also
good for your health, so it won't just be educational.
The thing to note, I contend, about bad formal quality management is that it is not, in fact, managing quality: almost invariably, it's designed to manage risk and, rather than managing the projects's risk of failure, it's being subverted to manage some manager's risk of being blamed, or some supplier's risk of being sued, when the failure happens – which, sadly, it almost certainly shall. While paper-trails do have a place in good project management and quality management, their proper function is to enable one to discover flaws in one's process – not to discover which person to blame for failure, nor to provide anyone with protection from such blame.
All the same, it is possible to have a full-fledged serious quality management system for software without it crushing the spirit or enthusiasm of those who do the work. Indeed, because the software industry routinely uses tools that can incidentally serve quality management goals, it is particularly well suited to systematic quality management and, when it is done right, it is enormously liberating and satisfying to work with. All of the bureaucracy a formal system involves can be mediated by software (saving a lot of trees) and integrated into the processes natural to software. If the formal system is implemented with care, it becomes the easiest way for folk to do their jobs; and the overhead added (having to record why you did what, on top of just doing it) contributes value that developers soon learn to appreciate.
As with any quality management régime, one records what is done and why. For changes to code, this is typically built into the normal process of software development – version control check-in comments, module release notes – although it may be constructive to have a process of peer review to teach developers how to write good check-in comments. Extending this to things other than code – e.g. plans, coding standards, test suites and the documents that describe standard working practices – can to some degree be achieved by placing the documents involved under a version-control system, as for code; which is generally a smart thing to do in any case, for all the same reasons as motivate doing so for code.
Typical quality management processes include documents specifying how to do various tasks. Among other things, the documents specifying how to manage projects shall typically specify that, at various points in a project, certain tasks are performed and their completion is verified in standardized ways. For example, the process should usually require an early mile-stone at which a project plan shall have been drawn up and reviewed by those who are to do the work, to determine whether the amounts of work to be done seem feasible in the amounts of time projected. Certain standard features of such a plan should be checked – for example, the time allocated for testing and bug-fixing should normally be at least as great as that for primary development; if a project deviates from this, it should at the very least include a clear justification for its unbridled optimism. As the project progresses, progress shall inevitably deviate from the plan: regular reviews should be conducted to identify such deviations, revise the plan accordingly and discuss the changes with those who shall be affected. In particular, if a project drops behind plan, those to whom it is to be delivered should be consulted about whether they can endure the anticipated delay or would prefer to keep to schedule at the expense of dropping some of the work originally planned.
On project completion, there should be a review of how the project went and how it deviated from its initial plan: at the very least, the deviations should be recorded. When any predictable pattern emerges among projects as to how they tend to deviate from plan, this can be used to revise the guidance given to project managers when drawing up plans, to help them better anticipate what is actually likely to happen. In general, for each part of the process, one should periodically review how actual work has deviated from best practice. Where such deviation has lead to trouble, the review should investigate how, in future, the circumstances that lead to such deviations, or the trouble that flowed from them, can be avoided. Where deviations are found to be harmless, one should consider whether the standard process should be revised to allow such deviation.
One of the innovations first popularized by the Linux kernel project – but eminently practical even in closed projects – is a hybrid approach, in which a development branch with informal quality management takes in all the shiniest newest improvements the project's gate-keepers can be persuaded to try out (and tends to be somewhat unstable) while a stabilization branch only accepts changes after they've been well tested and had their bugs fixed, i.e. following a formal quality management process. Those working on the two branches share a common version control system, if one is in use, and provide one another with changes – bug fixes flowing both ways and new features flowing to stabilization once they are ready for it.
One of the major virtues of such a hybrid is that it avoids a major
frustration developers experience in formally managed projects – being
held back from making change they think of, due to management fearing it may
cause instability and delay a release; this is at its worst when management in
any case accepts other changes which do have exactly those effects,
putting back the after the release
moment that the developer has been
told to wait for – by giving developers somewhere they can play, the
development branch, without hurting the release branch. In particular, when a
developer can thereby show that their change doesn't introduce the instability
management feared, it may become practical to include the change in the release
branch after all. Having a development playground, in which to demonstrate that
the change is safe, gives the developer an avenue for allaying management's
fears: it also lets the developer express a good idea while it's fresh and the
developer is feeling fired up about it – without such a playground, formal
quality management runs the risk of stifling creativity and motivation.
The focus on the development branch is exploring how to make the future as excellent as possible, while that on the stabilization branch is to have a product that end users can rely on today. Typically, indeed, there is scope for a few stabilization branches: when major changes have been happening in the development branch, it is often better to leave the prior stabilization branch without the major changes (it just takes in bug fixes from now on) and start a fresh stabilization branch off the development branch. Once this is comparably stable, it makes the prior stabilization branch redundant, but the adoption of the major changes likely makes it less stable for a time, during which those whose primary concern is stability can remain with the old branch. Generally, a stabilization branch can take in new features if their implementation was achieved by incremental, evolutionary change: but, where revolutionary change has been involved, it is time for a new stabilization branch.
This approach is particularly valuable to system integrators – that
is, folk who are taking the project's product and including it as part of a
greater whole. To them, stability is crucial: shiny new features may be nice to
have but they're not as important. They may also be more concerned about
keeping to some predictable release schedule, or meeting some dead-line, than
about the exact feature set that they'll be including – particularly if
they decided to include the project's product based on what was available
already, not on promises of future features. When a free software project
(typically due to lack of developers) only maintains a development version,
access to its source code makes it possible for system integrators to maintain
their own stabilization branches: naturally, this works best if they collaborate
closely with the upstream
project over such changes.
In particular, within such a model, it becomes possible for
the upstream
(component) developers to have releases, each based on one
stabilization branch, characterized by predetermined feature sets (but typically
with only rather loosely predicted release dates) while the downstream
(user-facing) project has releases characterized by a predictable schedule
(albeit with the final feature-set tending only to be determined near to each
release). Such an approach has been shown to work very well in free projects: I
have seen it work well in proprietary projects too.
There are, of course, also unsuccessful hybrids. Actual industry common
practices do manage to integrate the worst features of both the formal and
informal approaches, including: insanely bureaucratic approaches to managing
quality
that leave the actual management of actual quality unattended so
that quality is, at best, informally managed; and informal approaches to quality
management punctuated with bouts of drive-by bureaucracy. Informal approaches,
at least in industry, have a bad tendency to be wrecked by management not giving
technical staff the authority to decide issues that matter – management
insists on some new experimental feature being included in a delivery before it
works well enough; or makes developers get management permission for every
change, even when the developer knows the change is safe and necessary to fix
bugs; or refuses to schedule a major re-write of a tangled mess of legacy
code. There are sometimes good reasons for such interference, but management
needs to be more imaginative about finding ways to deal with the problem while
still letting developers be confident that their judgment is respected.
You might have noticed, particularly when I described how an informal process can work well, that a central factor in what makes it work well is the selection of who the gate-keepers were. This should be a good clue to the fact that a large part of what actually matters, when it comes to managing quality, is having good people in the right rôles.
The main reason why the software industry so often fails miserably at quality management is a failure to appreciate the importance of managing the quality of the recruitment process. It is possible to get useful work out of a team of predominantly mediocre programmers supporting a few really talented peers (although you'll have to watch out for ego issues in both groups); and even to adapt, deploy and maintain, with only mediocre programmers, software from an upstream supplier who has the talented people to create it. All the same, there are a lot of people in programming jobs who have no clue – they fall so far below mediocre that you shouldn't hire them even if you're desperate for programmers; they actually do more harm than good.
The principal reason is that recruiters all too often don't even try to assess competence by direct inspection of candidates' work. They rely on paper qualifications and prior job history rather than giving candidates a real task in programming to do and watching how they perform. Partly they do this because those responsible for recruitment – human resources and managers – are not programmers so wouldn't know what to look for: to remedy that, you need to involve the competent programmers already on your staff. Mostly I suppose they do this because paper qualifications and prior job history seem to work pretty well for most other jobs, so they assume it'll do the same for programming jobs. The shocking surprise you get if you institute a reasonable programming test is that this turns out not to be true. There are various reasons for this.
The candidate's prior job history may look nice, but how do you know the
candidate actually did good work in those jobs ? If their boss didn't know
how to assess coding competence, they may all too easily have taken the credit
for work they bodged together badly, that only really worked well once someone
more competent had fixed it up. There are people floating round the software
industry who have a knack for moving on just as, or slightly before, their
managers catch on to the fact that they're useless (or worse) and have merely
done a good job of seeming to get something useful done. The reference you get
from their prior boss may glow due to the boss not having yet caught on; or
because the boss really doesn't want the bureaucratic grief involved in having
to actually fire someone, so is happy to gloss over their deficiencies if that's
what it takes to get rid of them. Either way, they get a pay rise because their
new employer imagines they need to offer that to entice them away from their old
job. If they're good at sounding useful to management
they'll likely get
quite good pay rises while they're in a job, too – at least until
their boss catches on to their inutility, but by then they're moving on –
so their salaries go up faster than those of the competent people who are fixing
up the messes they left in their wake. When you're hiring someone whose salary
has been rising faster than the industry average, over the course of their last
few jobs, you just know you're getting a good one, right ?
But surely you can trust their University diploma ? Well, at least
it's authentic, no doubt. Unfortunately, it doesn't necessarily imply
competence. For one thing, group work is a great way for an inept student to
get credit for work done by their competent peers – just like the cuckoos
flitting from nest to nest in the industry at large, in fact; it's great job
training for them. For another, universities aren't necessarily doing any
better in their recruitment processes; and, even when they are, their
staff have often only worked in research and teaching, so lack industry
experience that informs a different set of priorities for software quality. So
the university staff who assessed the student's competence may be no better at
such assessment than your recruiters. Now throw in the wide-spread dogma that
the students signing up for any course have abilities normally distributed about
some good passing grade, with only a small fraction deserving to fail; combine
that with at least some ICT
course credit as a prerequisite of graduating
in practically anything these days, so that programming-related courses have an
artificially inflated enrollment of students whose talents lie elsewhere; is it
really any surprise, then, if some students are getting passing – or even
good – grades in software courses despite having no clue at all
about how to program ?
Ultimately, however, the real problem is that there are a lot of people whose assessment of their own competence with computers is wildly wrong. That goes in both directions: there are reasonably able programmers who, absent a nurturing peer group of more experienced competent programmers, fail to appreciate that they're actually quite good at their jobs and, if they'd just be a bit more confident, are capable of great work. There are also many people who have delusions of competence because they simply lack the skill to assess their own work realistically and appreciate that it's a ghastly mess that barely works, thanks to the Dunning-Kruger effect (original paper: PDF). There are so many other things to blame, when things go wrong, that people fail to realize it's their own ineptness – not the suckiness of their (doubtless sucky) operating system, not a malware infection messing up their computer's behaviour, not the user's pet peeve about some policy the system administrator has imposed – that causes things to go wrong. Lest you have forgotten: 'tis a bad workman that blames his tools.
Let there be no mistake, though: an appalling proportion of those who look good on paper and sound good at interview, for programming jobs, turn out to be completely incapable of actually programming. I know this from bitter experience because I've had past jobs in which I've had to clean up the messes those guys made: and, in my present job, I'm one of the assessors for the test we systematically use as a filter for candidates. It's not a very fancy test, it's not a very clever test and it doesn't require brilliance or exceptional talent. The test was designed by one of our managers, who started as a programmer, for one purpose only: to weed out the candidates who can't program. There is a sub-population, of the applicants whose tests I review, which has a reasonably even distribution across the competence range between those who'd probably be OK with a bit more training to those we're eager to hire before anyone else does. Unfortunately, that sub-population (of which we hire about half) is only a small fraction of the total: the vast majority of those who make it past the earlier checks (their CV looks good, they sound smart and confident at interview) are simply incapable of doing a good job. I and my fellow assessors struggle to stay motivated to keep up with assessments – a crucial job since, without it, we'll fail to accept the good candidates – because it's simply too depressing, sometimes, to grind through a day of evaluating unbearable stupidity.
Recruiters need to unlearn one of the lessons that I remember most
mistrusting when the teachers responsible for career advice taught me it, back
when I was a teenager: confidence is everything
. Confidence can no
doubt help a candidate but (outside jobs in Sales and Marketing, where it's a
real job-relevant talent) it mostly does so because it encourages the recruiter
to believe the confidence is sincere and justified (this is how confidence
tricks work, after all): unfortunately, it is all too often a mask worn on the
advice of those career advisers or, in so far as it is sincere,
delusional. Furthermore, many competent people – particularly if they
have yet to work in an environment where other competent people affirm their
competence and help them to recognize and develop their abilities – lack
the person-to-person confidence that's all an interviewer can see: and they've
been told (over and over again) that confidence is everything (it
is not; at best it is one thing) so, knowing they lack confidence,
guess where their confidence goes ? Down ! Thanks careers advisers,
that really works well – especially for the candidates who sincerely and
genuinely want that job (so they care all the more about it and are all the more
miserable about the fact that they won't get it because they lack confidence),
as compared to the ones who view it just as a stepping stone on the way to some
other job later on.
That advice needs to be revised. Tell job-seekers that confidence can help them, by all means, – because it can – but don't tell them that it's more important than being qualified, competent and eager to do the job – because, in fact, confidence isn't more important than any one of those things. Tell recruiters, on the other hand, to mistrust their instinctive tendency to favour confident candidates, especially when recruiting for technical jobs. Geeks tend to have a hard time of it, socially, in school: until they get into a fulfilling job which respects and rewards their competence, they are apt to have limited social confidence – which is the type of confidence you can actually see, in an interview – yet they are apt to be the candidates you really need to hire. Conversely, while confidence may be a good sign in a candidate, be aware that there are various exceptions; I'll just illustrate with one. In technical work, especially, you need staff who are willing to let go of their idea and let things go someone else's way: in short, you need staff who can put their ego aside while they get on with a job that depends heavily on extensive collaboration. The ones who can't do that often are confident at interview: even if they're good at the job (which isn't necessarily so, for all that they believe it) their egos can be the liabilities that make your projects fail.
You can't manage the quality of your products unless you can manage the quality of your staff: and, to do that (as for anything else whose quality you want to manage), you need to be able to assess (i.e. measure) the quality of the staff you have – and, of course, you need to actually exercise that ability, comparing the results to what you expected. If it turns out that your staff aren't as competent as your recruitment process lead you to believe, then you need to amend your recruitment process (you do, of course, apply your quality management regime to that, too) so that it assesses competence, rather than trusting the unreliable indicators that have lead it astray in the past. If your managers and recruiters aren't competent to do such assessment, look for those of your staff whose peers respect their competence: and ask them to help you assess competence.
You also won't get good quality software unless each part of the code is controlled by a programmer (or small group of programmers who work well together) who takes (or take) pride in that part of the code being good and has (or have) confidence that their judgment, as to what should be done to the code, is respected. To that end, you need to identify the programmers who are actually interested in each part of the code and, among these, those who are competent and sensible enough to handle that responsibility: then you need to actually give them that responsibility and authority – and treat them with the respect they've earned.
Written by Eddy.