Tools for Assessing Software Developers

It’s been a while since I last wrote on the subject of how to hire great software developers and weed out any applicants that aren’t experienced enough for the more senior positions within your team. Given the advent of new tools that are available to conduct such interviews, I felt it was worth updating my previous advice on the subject.

Skype is probably the single biggest game-changer in technical recruiting in recent years. Particularly if distance is an issue, using Skype to conduct interviews is a no-brainer.

Previously, phone screens were the de facto best way of carrying out an initial sift of shortlisted candidates. And to be honest they were never that good of a predictive indicator. What’s different about Skype is that, provided the candidate in question has an IDE at home (and most experienced developers do) you can use it to quickly screen candidates’ coding ability. There’s nothing like seeing someone actually using an IDE right from your very first ‘meeting’ to get a feel for whether the experience they profess to have on their CV actually translates into meaningful skills that they’re capable of applying to realistic business problems.

Skype allows you and the candidate to see one another. For the hirer, that enables you to get feedback from any non-verbal cues about their interest in the job and aptitude for same. It also allows you to screen-share, so you can see what they’re typing in real time in their IDE. In those respects, Skype is even better than trying to conduct a similar process in person, because you don’t need to crowd around a laptop screen or use a projector to be able to see them at work.

So, by all means don’t rule any interesting CVs out on the mere grounds that the applicant doesn’t have a webcam, a development setup at home, or a fast enough internet connection to facilitate a video call. But if they do have those assets available it makes it much easier to confirm their ability in a matter of minutes, before either party has invested any great amount of time in the process. 

The second biggest innovation in recent years, in my opinion, is Github. It’s always been desirable for candidates to provide code samples as a means of demonstrating their skill. However, previously you could never be sure that any work submitted was a candidate’s own. Most candidates are honest. Just occasionally, however, you’d identify someone that had provided an impressive ‘code sample’, but who it later transpired couldn’t programme a tenner out of a cash machine. Wherever they had plagiarised such samples from, it was clear that they didn't actually understand them themselves. (Such antics are quite probably how this guy here got his job.) It’s a waste of both of your time if you only discover this fact when it comes to sitting down in front of a laptop at interview and you ask the candidate to take you through their solution, only to find they can’t explain the first thing about how it works or why certain design choices have been made.

Github aids candidates’ credibility by being a freely-available online source control solution, that verifiably identifies the authors of any content submitted. Not only can you freely download any complete solutions that have been placed there, but you can see the individual check-ins that went in to producing each solution and the thought processes indicated by the comments associated with same. If you know what you’re looking at, those fine details tell you much more about a candidate than a mere CV full of buzzwords and all the glowing references in the world ever could. And unlike copying whole solutions you didn't write yourself, forging a history of the individual check-ins that go in to making up a complete solution is all but impossible.

With Github, you can also confirm a demo project’s creation date. This is important. Do you ever get the impression that candidates’ CVs are merely re-wordings of your job spec? This is in some ways understandable, and arises from the fact that the standard advice jobseekers are given is to tailor their CVs to highlight relevant experience. But still, as a hiring manager you sometimes would prefer to see what a candidate felt their own strengths were, before they knew what you were actually looking for. Github gives you that insight. If you’re looking for someone that has experience in Technology ‘X’, being able to see that they’ve completed a project using that technology some months before your particular requirement even came up is a pretty convincing demonstration that the candidate actually does know what they’re talking about when it comes to the subject concerned*.

(* That said, outside of specialist contracting roles, where you do expect new hires to hit the ground running from day 1, hiring software developers should rarely if ever merely be about hiring a particular skillset. It’s always better to instead hire for aptitude and attitude, and train for skill when you need to. Because new technologies come up all the time, and it’s no good hiring one-trick ponies that are incapable of keeping up with constantly-emerging technologies. Or, worse still, people that may be gifted as individuals but whose personality problems render them unsuitable for teamwork. You can teach people with the right aptitude and temperament almost any technical skill they need to know. The best ones will be capable of constantly improving themselves. But you can’t teach them not to try and use their one golden hammer to solve every single problem they come across. And you can’t teach them not to be an arrogant control freak that alienates their peers.)

The above are great ways to identify talent. That said, I know from working with a great many talented software developers over the years, that a lot of them don’t have the time to work on open source projects on Github whilst they’re fitting a family life around about being great assets to their existing employer. And some of them live in places where the internet connection is slow, making Skype a difficult option.

So, for people for whom Skype and Github aren’t options, there is a Plan ‘B’ you can use. A less-preferable secondary approach that also works is to conduct an initial phone screen using a stock list of questions. I’m loathe to suggest an undue correlation between merely knowing the answers to some coding trivia questions and actual meaningful ability as a software developer. One is merely knowledge, the other is a demonstration of actual intelligence. However, there are just some basic things that you should know about any language or technology you profess to be proficient in, and that knowledge can be used as a baseline check if need be.

E.g., for a junior level C# developer, I’d expect them to know:

•  Q. What is an Abstract Class?
  
  A. One that is only intended to be used as the base for other classes, but that may not be directly instantiated in and of itself.

• Q. Tell me three basic variable types in the common type system?
  
  A. Take your pick from any of these.

• Q. What are the scopes you may use to limit Field/Property visibility, and to what extent do they make these aspects of a class visible?
  
  A. Public, Private, Protected, Internal and Protected Internal.
  (NB: I wouldn’t fault anyone for failing to name that last as a distinct scope in its own right, whose limit is a combination of that afforded by ‘Protected’ and ‘Internal’.)

The key thing is that there are no trick questions here that would require knowledge of obscure parts of the .Net framework. Candidates may or may not not happen to have used certain discrete parts of the 4000-plus namespaces in the .Net Framework, but good developers could easily look up and utilise any part of the Framework if they needed to with only a couple of hours research. Asking about the features of a specific namespace is therefore pretty meaningless. The questions above instead just concern basic, core features of the C# language. Anyone that has used C# at all should be reasonably expected to be aware of them.

Questions like these don’t help you identify whether someone is a great developer or not. Seeing how candidates write actual code using a real IDE is the only thing that enables you to do that. These questions are purely intended as a baseline negative check to help you identify any manifestly-unqualified candidates where the other preferred means of confirming ability mentioned earlier are unavailable.

For more senior C#  developers, I’d expect them to know more advanced, but still core, features of the language. E.g., :

• Q. What is a Delegate?
  
  A. A pointer to a function. They’re used in lots of ways, but a typical example would be to assign a handler to an event.

• Q. What’s the difference between a class and a struct?
  
  A.  The first is a Reference type, the second is a Value type. This affects where they’re stored in memory, and how assignment between variables of the types concerned work.  

• Q. Can you explain the principles of object oriented design?
  
  A. I’m looking for them to be able to succinctly tell me what Inheritance, Abstraction, Polymorphism and Encapsulation are.  

For a Lead Developer or Architect, I’d expect them to be able to speak meaningfully about:

•  Can you describe some Design Patterns? (e.g., please explain what Singleton is, What is the Decorator pattern? Tell me about a time when you used them?)

• What are your thoughts on Inversion of Control / Dependency Injection? What about Test Driven Development? Do you always use them on every solution?* If not, what criteria do you use when deciding whether to expend the additional effort? What are the limitations of IoC? Which of the 22 plus frameworks that presently exist have you encountered on live projects?
  (* FWIW, I personally believe that using these presently-fashionable methodologies and techniques on every single project is about as misguided as never using them.)

•  What is an abstract class?*
  (* The observant will notice that this last question is the same question used for junior developers. It’s amazing how many Architects can recite high-level summaries of chapters from the Gang of Four, but who’ve lost touch with how coding actually works in the trenches. It gets more difficult as your career develops to keep in touch with the front line, but my personal belief is that you can only lead great developers if you actually share their pain by hitting a keyboard yourself once in a while. You certainly shouldn’t exhibit any signs of Hero Syndrome or micro-managerial tendencies by needing to be involved in writing every line of code yourself, and you shouldn’t try to do developers’ thinking for them. You need to entrust and empower those you lead by allowing them the freedom to get on with any tasks you delegate to them using their own skill. However, it is important to implement a particular feature yourself every so often, purely to keep your own skills current in an ever-changing technical landscape. Otherwise you only lose touch with emergent technologies. A clear sign that you aren’t getting enough personal keyboard time is when you begin to lose the basic knowledge that even junior developers working under you are expected to possess.)

For any one topic that I consider myself experienced enough to assess others in, I have a list of about 200 such questions that represent basic knowledge I’d expect most people to know at each level. During an initial phone screen, selecting two or three such questions as baseline checks is the next best alternative to using Skype or Github to assess whether there’s any potential.

I wouldn’t lose sleep over anyone getting any one individual question wrong. (Especially if they’re honest enough to admit they don’t know a particular fact. The very best people show awareness of things they don’t presently know, whilst less skilled individuals are often paradoxically unaware of their own current limitations. That inability to perceive their own present weaknesses leads to them failing to ever improve. This is known as the Dunning-Kruger Effect.) I still prefer actually seeing a person code using Skype, Github or even YouTube in preference to using coding trivia as an initial screening tool, but phone screens using basic questions to eliminate candidates is the next best option for the initial sift of candidates that invariably apply to almost any openly-advertised technical position. You can apologise to the ones that find it ridiculously easy afterwards, and explain the reasoning behind your using such simple baseline checks.

Skype and Github are better options because they represent positive checks for ability, whilst asking baseline questions is merely a negative check to identify the absence of basic knowledge. However, if a candidate can’t answer any of the simple baseline questions appropriate to their level of seniority, that’s clearly someone that you won’t take forward to interview.

For anyone that attends an in-person interview, I’d always recommend seeing them code using an actual IDE. (If you’ve seen them do so via Skype previously, obviously you can skip this step). The best way to do this is to attach a projector to a laptop that’s loaded up with a full IDE and an internet connection, and watch them work. I once had a hiring manager tell me that they used pen and paper coding exercises instead “because they didn’t want the candidate to have access to Intellisense, and all those other ‘cheats’ that a full IDE provides”. No, I don’t understand the logic behind that one either. I found myself wondering if they’d ask a prospective master carpenter to bang in nails wearing a blindfold, and decide from how swollen their thumbs were afterwards which was the ‘best’ at their craft.

Just like when you’re using Skype, you can record candidates’ efforts to build a quick solution using free tools like CamStudio recorder if you like. That approach can be very useful if you work in a large organisation and have a wider selection committee that will need to review the interview later on. It can also feel a little like an unfriendly interrogation, though, so you need to decide what’s right for your own organisational culture. Personally, I’d only record a coding test if there were a need to show the recording to other members of your recruitment panel afterwards. And I would explain to the candidate that the purpose was to save them having to demonstrate their ability multiple times to different people.

It’s important to make clear that the problem you’re asking them to solve constitutes realistic work, but not real work on an actual business problem. The first activity is a meaningful test of their skill. The second would merely represent unpaid work, and that would risk making you look like a freeloader. One problem I’ve seen used in the past and that I thought was a pretty fair baseline check read something like this:

“Design a system that allows you to model shapes as objects. Each shape should be capable of outputting a text description of itself. The description given in each case will be:

‘I am a _________. I have ____ sides, and ____ corners. My colour is ______. Their lengths are _______.’

There will be appropriate Properties in any classes you use to model such shapes to store the information to be supplied in the blanks in the above description.

You can implement this solution using any UI you like. Have specific classes that describe the shapes ‘triangle’, ‘square’, ‘rectangle’ and ‘circle’”

A developer should be able to come up with a simple design that has a base (possibly abstract) class that provides any shared Properties like colour, numSides, etc. They can either implement a Method in that abstract class to allow a string description to be output, or they can override the default ToString method. Classes describing the specific shapes requested should be inherited from this base. Extra points for having the perception to make appropriate properties/fields read only in more specific classes (i.e., you don’t want consumers to be able to create a triangle with four sides). Points too for using inheritance where appropriate (e.g., realising that a square is just a more specific instance of a rectangle.)  Nothing too taxing, and no trick questions or tasks that would take an unreasonable amount of time. Just a simple problem to allow developers to show that they’re not a non Fizz-buzzer.

As this is a blog about assessment tools, it’s worth mentioning ‘online’ tests like ProveIT, Brain Bench, and Codility. These ‘tests’ fall into two main categories:

• Tests that attempt to assess ability based on being able to instantly-recall knowledge of obscure parts of particular frameworks.

• Tests that try to assess an actual ability to write code, but not using an actual IDE.

My opinion on using obscure trivia to assess problem-solving ability is well-documented. I’m with Einstein on this one, who when asked what the speed of sound was once said that:

“[I do not] carry such information in my mind since it is readily available in books. ...The value of a college education is not the learning of many facts but the training of the mind to think.” *

[ * New York Times, 18 May 1921 ]

I don’t consider memorising a lot of obscure and easily-obtainable facts to be a good indicator of programming ability. Nor do I consider not being able to recall such facts at will to be an indicator of a lack of ability. Developers have Google and reference books available on the job. I’m therefore only concerned with testing those aspects of a developer’s ability that those tools can’t provide.

That leaves those online ‘tests’ that attempt to assess coding skill, such as Codility. There’s nothing wrong with the basic idea of getting candidates to write code as a demonstration of their existing ability and potential. However, there’s a big difference between writing code using an actual IDE, and attempting to write code using a web browser (which is how Codility works). In a real IDE, you have Intellisense, code snippets, meaningful object navigation (e.g., if you place the carat on the usage of a class or property in Visual Studio and use the F12 key, it’ll take you to where that class/property is implemented), colour coding of keywords and objects, compilation checking as you type, etc, etc. Codility advocates believe that because that assessment tool has a “compile solution now” button at the bottom of the browser window that amounts to the same thing. It simply doesn’t. Going back to my earlier analogy about inappropriate ways to assess carpentry skills, you’ve merely gone from using a blindfold to asking the candidate to wear sunglasses in a dimly-lit room.  

Codility tests run in a web browser

The main problem with Codility et al, however, is simply this. They don’t give you anything that you don’t also get by watching a candidate solve a real problem using a real IDE. Because of this, you invariably find that these tools are preferred by interviewers that don’t possess skills in the language concerned themselves. Such interviewers don’t use an IDE / laptop with a projector approach, because they simply wouldn’t understand what it was they were looking at. By using Codility instead, they’re generally looking for an ‘easy’ way to understand whether a given solution is ‘right’ or ‘wrong’, without having to go to the trouble of understanding why such a value judgement has been arrived at themselves. Good candidates are aware of this, and the best of them will be concerned that if you only understand how good they are because some automagically-marked test tells you what to think, how are you going to be able to fairly assess their performance on the actual job in the absence of such feedback?

Everyone knows that good interviews are a two-way street. Candidates are assessing you and your organisation just as you are assessing them. Sending a signal that you don’t understand what it is that they do can damage your credibility and your employer/manager brand considerably. So, if you’re not technical yourself (and some managers aren’t), I’d generally recommend instead asking one of your existing staff that you trust to be able to make a meaningful assessment of a candidate's ability to accompany you when assessing candidates’ technical fit.

A second problem with Codility, in my opinion, is that solving discrete problems using technology in the real world rarely works in such black and white terms as a solution being ‘more’ or ‘less’ right than other approaches. There are generally a great many ways to satisfy any one problem. Which one(s) is/are ‘correct’ is all about context. Tests that focus on an overly-narrow set of criteria when determining success may not always identify the best candidate, even if they identify someone that produces the fastest solution, or the one that uses the least (or most) lines of code to solve a problem. e.g., if someone were to use the line  123 << 4  to get the result 1968 instead of writing  123 * 16  , that might be the genius you need to optimise nanoseconds on calculations within the firmware for a graphics card, or they might just be That One Guy that writes unreadable code that produces hard to find bugs. (Mostly, though, they’ll just be someone that doesn’t realise low-level arithmetic optimisations like bitwise operators are largely meaningless in languages like C#, where high-level code is converted at compile time into optimised MSIL before being converted into even more optimised machine code specific to the hardware it’s running on.)

You can try Codility for yourself here, and I'd strongly recommend that you do so if you're considering using it to fairly assess candidates. It's not enough just to get someone else to look at the test for you, unless you ask your chosen guinea pig to work under the exact same time constraints as candidates will be asked to work to. That also means they only get one shot at the test, just like candidates.

In the interests of debunking The Emperor's New Code, when I tested Codility out as an assessment tool I found that I didn't produce a 100% solution myself first time in the time allowed. I therefore felt it'd be unfair to ask candidates to do something that I myself couldn't.

I doubt that many people could produce an 'optimal' result in the timeframe allowed, particularly when you don't get to see the criteria that will be deemed to constitute an 'optimal' solution before submitting your answer. When they only have a short window to think about the problem, candidates will be inclined to focus on providing a solution that works rather than one that shaves milliseconds off of the runtime. And even where candidates do provide an 'optimal' solution, there doesn't seem to be much allowance for readability in the simplistic percentage score returned.

I suspect that most 100% results that users might see from this tool may be best explained by the fact that there are many solutions to the tests published online, and some candidates will be inclined to copy one of those.

This deliberately-obscure and unreable solution scores 100% (Full-size view available here)

This shorter and more readable solution also scores 100%

My overall conclusion: companies that let computer algorithms select the best people to work for them rather than the other way round may well be disappointed by the results.

Amazon Kindle: One of the Best Tools in a Technical Pro's Arsenal

As any professional knows, you can't get by in technology without constantly keeping your skills up to date with emergent technologies. Of the various ways to do this, one of the best is to choose some detailed technical books in those fields that are presently of interest. Preferably ones that have been written by people that are actual hands-on developers themselves, rather than those by authors that merely have an academic interest in the subject under discussion. Wrox Press does an excellent line of books called Programmer to Programmer, which are very useful in this regard.

Once you've absorbed the material, and have used the technologies concerned on a few live projects, these books become an excellent source of reference material, which you can refer back to time and time again. If you're anything like me, you'll have a bookshelf at home that looks something like this:

RP's old technical library

The only downside to Wrox books (and any other type of reference manual) is their weight. It's typically handy to have those books that are relevant to the discrete areas you're working in at a given time to hand as you tackle a real project. Given that even a few books combined generally weigh around 2-3kg, that can be a real strain on your back, especially if you're already carrying a laptop and other bits and pieces around too. 

Around about two years ago, I bought an Amazon Kindle, for the express purpose of carrying around these tomes without dislocating my spine. The passive e-ink display of the basic Kindle device is great for reading in bright sunlight:

Physical Kindle Device

And there's a handy little management tool on the Amazon website that allows you to keep your books safe and available at all times, wherever you may be:

Amazon's "Manage Your Kindle" tool

(View full size here)

The battery on a physical Kindle device lasts for a very long time (at least a month of continual use between charges), and the device itself has room for loads and loads of books. There's also a free PC-for-Kindle application that allows you to view the e-books you've purchased on any PC. And for Android users, there's a nifty free Kindle-for-Android app that allows you to view your books on your tablet:

Kindle for Android tablet, Library View

 or on your Android phone:

Kindle for Android phone

  

These latter applications are particularly useful if a book you happen to be using utilises colour to convey meaning:

Colour-Coded Keywords

Coloured Graphics

                    (View full size)                                                  (View full size)

However, the best thing about using Kindle as your technical library, which all technical departments should take heed of, is that you need only buy each book once. If you've got a department full of 20 developers (within which there will undoubtedly be a huge overlap of skillsets), it makes sense to buy one copy of each book, and allow each developer access to that library of virtual books via their own PC. An organisation that wants to make effective use of technology will also probably find it profitable to fork out for a few Tablets and physical Kindle devices for taking into meetings. In terms of increased profitability for individual developers, they'll pay for themselves in no time.

Code Reviews

One of the practices that’s become semi-common within development teams over the past ten years or so has been the observance of formalised management/peer reviews of team members’ work, known as Code Reviews. The specific format these reviews take, and who carries them out, varies quite a bit from workplace to workplace, and project to project. This article covers some of the approaches I’ve encountered within various working environments, and provides some personal viewpoints on what works and what doesn’t when carrying out reviews. 

---

When I first started writing software professionally, in the late 1980’s, it was a fairly solitary activity. Development ‘Teams’ tended to be comprised of at most two or three individuals. Often, only one of those people would be writing the actual code, and it was rarely the case that anyone other than that single coder would take an interest in the actual implementation of the solution (as distinct from a user-level understanding of the functionality being developed). Code reviews were therefore fairly superfluous to the development process – solutions either worked, or they didn’t, and it was usually fairly easy to see what was or wasn’t working without having to enter into too much analysis of the subject.

In recent years, since around about the turn of the millennium, it’s become more popular to have larger development teams, comprised of more specialised (and less broadly-skilled) individuals. The reasons for this change in approach are many and varied, but perhaps one of the main drivers is that there are so many more development technologies available now in comparison to those early days, and having larger teams allows different parts of those larger and more diverse teams to better focus on more specific parts of the solution. It’s common nowadays to find 3-4 developers, a DBA (sometimes more than one, especially in the Public Sector), Business Analysts (including some that don’t have a coding background), Project Managers, and specialist users with expert knowledge of the problem being solved, all working together within dedicated teams aimed at producing a single software product. There are some advantages to this more modern approach, including greater support for individuals within the team and an increased ability to fill gaps left by any departing members, though there are some disadvantages too. Specifically, a corollary of working within those larger and more diverse teams is that, when working on multiple fronts consecutively, there is more chance of any one particular area of work getting out of sync with the rest of the team’s activity. By increasing the separation of concerns, development teams also multiply their potential points of failure. This is where Code Reviews come in.

Code Reviews, when done correctly, can have multiple benefits to a properly-functioning and well-motivated team. They can help individuals to communicate their ideas with, and engage in mutual learning from the experience of, their peers and technical management, thereby facilitating a two-way process of learning and mutual understanding. They can make the product being developed more robust and maintainable, since it stands to reason that the more members of the team that know about the logic behind the wider implementation (including parts of that implementation they may not have worked on directly themselves), the easier it’ll be for any one developer to take over another’s work if the demands of the team (sickness, holidays, re-prioritisation, etc) dictate. Perhaps most importantly, though, Code Reviews enable individual developers to understand the impact that their own work has within the context of the wider team’s direction. They enable individual members of the team to contribute their own part of the wider effort in conjunction with rather than in isolation from the concurrent activity going on around about them. In this way, they encourage each member of the team’s work to be part of a widely-understood set of greater goals, rather than just leaving each individual to contribute their part of what eventually becomes a patchwork quilt of loosely-aligned ideas. 

If carried out incorrectly, on the other hand, Code Reviews can be extremely damaging to a team’s morale. If conducted in the wrong spirit, advice passed on during code reviews can seem, from the perspective of the recipient, to merely represent unnecessary interference in, unwelcome distraction from, and unjustified criticism of, their work. They can create a climate of fear that stifles individual initiative. If done badly enough, they can breed resentment that ultimately leads otherwise skilled people to (correctly or not) conclude that they can’t do right for doing wrong, sapping their confidence and motivation, and diminishing the team’s productivity.

So, how best do them properly? Well, in my humble opinion from two decades of being on the receiving and delivering end of same, here are some of the things that I’ve seen work worst and best during my career:

1) Keep It Private

I’ve found it best to always give reviews about individual developers’ work directly to the person concerned, face-to-face and in private. If more than one individual is involved in a particular piece of discrete work, it’s of course fine to have a conversation together with all of the parties concerned aside from the wider team, but any discussion of individuals’ work should never be done in front of uninvolved members of the team. It’s human nature to perceive even constructive criticism as a negative experience, where that constructive criticism is delivered before an audience of the recipient(s)’ peers. The self-same feedback that would be perceived as well-meaning advice if delivered in private, can sound like the harshest criticism if given in public; context is everything. I once knew a lead developer that not only gave code reviews in public at individual developers’ desks in front of the whole development office, but who also later wrote about those individuals’ work in clearly-identifying terms on his internet-facing public blog. I guess he must have mistakenly thought of the internet as an anonymous medium and considered his blog in much the same way that other people might think of their private diaries. Ouch. My advice is to never behave in this way, however tempted you may be to vent frustration and however much you may feel that an individual deserves it. Criticising your peers (or, even worse, your direct reports, who look to you for even-handed guidance and leadership) in public is never a classy way to behave. Doing so is guaranteed to immediately lose you credibility with the very people whose work you should be trying to positively influence – who wants to work for someone that appears only to be interested in embarrassing and deriding them in public? What you should aim to achieve out of each Code Review is a more motivated and better-informed team member, and an insight into that team member’s thought processes for yourself; it’s plain to see that you’re unlikely to achieve either of those outcomes by merely seeking to criticise or denigrate people in public.

Code Reviews are either Win-Win or they are Lose-Lose for the parties involved; by providing any constructive criticism you have to make in private, you’ll help ensure the former rather than the latter outcome.

2)  It’s Not All About The Negatives

One of the things that can make Code Reviews become a morale-sapping chore rather than the productive exercise they should be is if a perception is allowed to develop amongst the team that they’re merely used to provide oversight of those being reviewed, to pick up on any areas where there’s room for improvement. For experienced technical professionals (who are most often experts in their particular field, and are regularly the most experienced member of the team in whatever it is that they individually specialise in) this can sometimes feel like a wholly-negative experience. Even where the technical reviewer is of equal experience in a given technology, it’ll certainly be the case that the person being reviewed has a greater knowledge of the solution as it was developed, since they will be the very person that built the thing. They’ll know precisely why they chose implementation ‘X’ over possible alternative ‘Y’, and can resent it if someone with ten minutes’ understanding of their work chooses to focus exclusively upon perceived negatives in preference to recognising the bigger picture. It’s not that reviewers should be afraid of giving relevant feedback where necessary – they should by all means do so, that’s one of the things they’re there to do – but it is highly beneficial to the process if that feedback can be delivered in a way that avoids alienating the skilled people whose initiative and expertise the reviewer should be seeking to encourage and nurture. One of the key things that makes for such an effective review is that the reviewer should make clear that any constructive criticism being passed on is being given within the context of a generally acceptable performance by the person being reviewed (if indeed that is the case, which it should be in all bar exceptional circumstances: if the reviewer was responsible for mentoring and briefing the person whose work they are now reviewing, and the reviewer has been keeping an eye on source control during the development process, any major problems discussed during a code review shouldn’t come as a total surprise to either the reviewer or the developer – unaddressed major problems should have been spotted and raised well in advance of a retrospective code review). Showing appreciation for the many positives of your direct reports’ efforts within the wider context of a rounded and balanced review of what they have produced goes a long way to lending credibility to any suggestions for improvement you may have to make.

3) Don’t Nit Pick

This is related to but subtly different from It’s Not All About The Negatives above. The first and foremost consideration when you’re giving a review should be whether the work you’re reviewing actually does what it was functionally intended to by the developer, and envisaged to by any specification document. If the solution does meet those criteria, that’s the biggest single tick in the box, whether or not you may have implemented things subtly differently yourself. Within the context of that big picture, a rounded review may also encompass some more esoteric developer-level issues such as the coupling and cohesion evident in the design, how well abstraction of concerns has been achieved, whether the developer has adhered to any design patterns that had been set in advance, the readability of the code, whether the developer has included meaningful comments, etc. However, it’s advisable to avoid commenting on trivial issues, such as the variable names used, or the developer’s use of white space (unless there are glaring and inconsistent gaps that make the code hard to read). The only time it’s appropriate to comment on something as subjective and individual as variable names is if the developer is using generic names like String1, String2, etc, instead of labels that make it clear what the variable is actually for. Perceived wisdom on the ‘best’ naming conventions for Variables, Classes, etc varies wildly from workplace to workplace, and even from project to project within the same workplace. On the whole, it’s best not to focus too much on minor details like these during formal reviews. If you make the mistake of being over-controlling, whatever you may ‘gain’ from that approach in terms of control and code consistency, you invariably lose in terms of individual initiative and the developer’s pride of ownership in their work (and the inner motivation to do their best that professional pride brings). Trying to micromanage every aspect of a person’s work is a sure-fire way to stifle their initiative, and ruin their self-confidence, which in the long run reduces the team's productivity. It’s far more beneficial if developers walk away from code reviews with an accurate sense that the reviewer trusted their judgement enough not to sweat the small stuff, and only bothered them with feedback that was worth their time and the reviewer’s time.

4) Use The Review To Communicate Across The Team

I mentioned above that one of the most beneficial aspects of Code Reviews are that they allow individuals within the team to gain a broader perspective on where their work falls into the bigger picture of concurrent activity going on within the wider team. Code Reviews can be a good time to discuss other areas of the wider solution with developers that are uninvolved with those areas, and to explain where the work being reviewed is intended to fit in with those separate parts of the wider team’s activity. This helps provide an element of useful redundancy within the team (in case developers need to cover for one another or take over one another’s responsibilities for a time), and helps communicate a greater understanding of why certain strategic decisions have been made. From the perspective of the reviewer (who will generally be the lead developer or development manager), Code Reviews also help communicate intent from individual developers back to those guiding the overall solution; as touched upon earlier, good reviews should involve a two-way flow of useful information and feedback.

5) Write It Down

Code Reviews should be formally documented, and held confidentially for later reference only by the reviewer and the person whose work was the subject of the review. This formal record of the review should ideally only be 1-2 pages long, will contain less than 500 words in most cases, and no more than 1000 words in any event. The written outcome should be a list of bullet points of any positive and negative observations to be raised, that the reviewer brings two copies of to the meeting and uses to guide the discussion, then e-mails to the person being reviewed after the meeting. It’s advisable not to send this document in advance, since the brief nature of a bulleted list can be taken out of context if the issues it pertains to aren’t given context by being elaborated upon in person. Another reason for waiting until after the meeting to mail the document to the recipient of the review is that it allows you to incorporate any feedback from the meeting itself into the formal record, and to note any specific outcomes that are expected (such as any remedial work that may have been  requested). If you’re feeling brave and you’d like to encourage openness and communication, it’s also an idea to include a section at the bottom of the document headed : “And how did I do?”. This encourages the recipient of the review to give you any feedback they wish on any aspect of the process. Any given project needs to have one distinct individual that is entrusted with responsibility for the overall direction of the solution, and that individual will usually be the person conducting the code reviews (though I have seen working environments where developers have been asked to review one another’s code on a regular basis, which I don’t think worked particularly well - developers like to be able to focus on their own work in the main, and they often perceive it as a distracting and unnecessary chore if they’re instead tasked with regularly commenting on their peers’ work; perhaps most importantly, though, developers’ time is too valuable to waste on administrative tasks that the technical leadership of the team should be shouldering for themselves). Meaningful communication between members of the team is the key to making sure that the main technical decision maker knows everything they need to know to do their job effectively. If communication is truly to be bi-directional, in a code review it’s advisable to document any feedback from the person being reviewed alongside any critique the reviewer may have made of their work.

---

So much for Code Reviews, and the best and worst of how I’ve seen them employed in various places. I of course welcome any reviews of the above that anyone out there may care to make. ;)

How to Recruit and Vet Technical Professionals

Over the years, I’ve had several experiences of being on either side of the recruitment table. At different times, I’ve been the hiring manager looking to augment the team I’ve been leading with a skilled technical professional, and at other times I’ve been that external technical professional, with skills, experience and insights to offer the team I’d be joining. Those experiences on either end of the equation have led me to some conclusions about the best way to go about ensuring that each party to the process gets what they intend out of the deal. For the hiring manager, that’s a new and well-motivated team member that’s able to bring some desired skills into the team, and for the candidate it’s the opportunity to meaningfully demonstrate what they can do and determine what the opportunity has to offer them. Those goals aren’t in competition with one another; a “win, win” is what each party should be looking for when it comes to matching the right people to the right roles. It’s only when each party gets what they’re looking for that everyone involved (the hirer, the hired, and any intermediaries) can consider that the process has been truly successful. This article contains some thoughts on how each of the various parties to the process can help ensure a mutually-beneficial outcome.

One of the difficulties with recruiting technical professionals is that you will sometimes find yourself in the position of assessing people that have skills you don’t yet have. For intermediaries like recruitment consultancies, HR departments, and for non-technical hiring managers, you can substitute the “sometimes” in that last sentence for ‘always’, and the “do not yet have” for ‘will never be interested in personally acquiring’. That’s merely a fact of the different types of value those parties add to the process; similarly, you can be sure that most prospective technical hires will never be overly interested in finer details of employment legislation, employer branding, marketing, or abstract management theory. How, then, if you’re not a technical professional yourself, do you go about determining whether the person in front of you has what it is that you’re looking for? Before answering that question, here are some likely approaches that might seem reasonable at the time, but which, depending on how they’re carried out, may not provide the outcome you intended:

1)     Sending an intermediary to do your first interview for you

Whatever else you do, as a hiring manager my advice is not to make the mistake of letting someone else represent your company or department to prospective new employees. Your HR department and any external recruitment consultancies you may engage can certainly take the marketing burden out of sourcing suitable candidate CVs for you, and they can also provide a screening service to ensure that you don’t need to deal with the initial sift of wholly-unsuitable applicants yourself, but they’re not there to conduct any actual interviews or assessments on your behalf.

Typically, what an agency will do when they receive a brief from a hiring manager is they’ll advertise for suitable candidates using the job spec you provide on several of the industry-specific online recruitment websites such as JobServe, etc. The agency will remove your company’s name and identifying details from their listing, so that any less-reputable candidates and competitor agencies wont attempt to contact you directly. The recruitment consultant will sift the CVs they receive in response to their advert, will discount any outright unsuitable applicants (e.g., you’re looking for a Java guru, and it turns out the candidate has made coffee once), and they will conduct a brief telephone screen with each short-listed candidate to make sure that their stated skills, salary aspirations (or daily rate, for a Contractor) match with what you’re looking for. A good HR professional or external recruiter will also be able to give you as hiring manager advice on the state of the skills market, average salaries that other organisations are paying for roles similar to the one you’re recruiting for, and will be skilled at extending and negotiating a mutually-beneficial package when it comes time to do that. Make no mistake about it, though: for all the value they can add, your recruitment consultant or HR professional isn’t in a position to conduct an in-depth technical interview or make a recruitment decision for you, and they’re not there to represent what you have to offer to the candidate – that’s your job.

Interviews are a two-way street, and if you make the mistake of letting your department be represented at a first interview by someone that doesn’t even work in your department, don’t be surprised if the most marketable candidates fail to be impressed by having to run a gauntlet of your intermediaries before even getting to meet an actual decision maker, and opt out of your process before you get a chance to assess them.

2)     Getting the candidate to write some code using a pen and paper.

It’s certainly useful to get a candidate to write code at some stage in the interview process (and this is in fact something that many good developers look for in the recruitment processes they’re subjected to – an employer that tests their ability to write code not only gives them a chance to prove themselves, but also gives them some confidence that their colleagues-to-be will be as skilled as they are, having come through a similar process themselves: more on this later). In my experience, though, it’s never a good idea to ask a developer of any level of experience to write code using a pen and paper. Doing so is akin to asking a skilled chef to cook you a meal.......with their feet. Which is to say, it’s simply the wrong tool for the job, and not in any way indicative of the relevant skills you should be looking for.

An analogy I use when highlighting why pen and paper is the wrong medium for assessing an ability to write useful code is the telephone. Up until the mid-nineties, most people were able to remember lots and lots of their friends’ telephone numbers off by heart. It wasn’t an uncommon nor an especially useful skill, it was just a somewhat abstract and unusual ability that everyone had spontaneously acquired because of the limitations of the way the telephone medium worked back then. These days, in the age of near-ubiquitous use of mobile phones, less and less people choose to remember such meaningless lists of numbers off by heart. Instead, everyone tends to use the standard address book function on their mobile phone to attach meaningful text labels to the people with whom they wish to communicate, and look up those labels whenever they need to make a call. You often find people looking up numbers on their mobiles these days, even when they’re using a desktop phone, because it’s become the modern paradigm of how to efficiently access that category of information. The essential skills of how to use a telephone and how to have a meaningful conversation with another person that you can only hear but not see are still the same today as they were in years gone past; it’s merely the tertiary skills surrounding the medium that have evolved.

With reference to the above sociological changes that have come about as a result of advances in telephony technology, asking a developer to write code with a pen and paper is a bit like attempting to assess a person’s ability to use a telephone by asking them to write down all of the telephone numbers they know off of the top of their heads. In much the same way as telephone handsets have evolved, development environments have changed a lot in the past 15 years too. Round about that same time in the mid-nineties that everyone in society possessed ubiquitous Rainman-esqe abilities to recall lists of essentially meaningless numbers in order to be able to communicate using the telephone, developers once had to remember most of the syntax of the languages they worked with in their heads. These days, in the age of smart Integrated Development Environments (IDEs), that’s not the case any more. With the advent of Intellisense, Code Snippets, Templates, and Keyboard Shortcuts, there are many more efficient ways of coding today than there were even five years ago, and new tools are becoming available all the time. These days, if developers want to insert a Constructor into a class they’re designing in C#, they don’t need to recall the exact format of such a code fragment from memory, they can just type ctor, hit tab, and all of the syntax for a parameterless Constructor that is meaningful within the context of the class they are creating is inserted for them. The real skill isn’t in remembering the syntax, it’s in knowing what a Constructor is, when to use one, and in understanding why a parameterless one can be overtly implemented in a class but not in a struct. When you can’t quite remember which namespace one of the 3,500-odd pre-defined .Net Framework classes resides within, the latest development environments will help you by suggesting ‘Imports / using’ directives that you may have missed. When you create a custom class in a project, provided you’ve been smart enough to structure your project logically using folders, the Visual Studio development environment will create meaningful namespaces for you. These cues, and many others, allow developers to concentrate on the important part of their job – creating innovative, robust and reliable solutions to well-defined problems – and relegates the non-cognitive and repetitive parts of the process to the place where they belong: the subconscious.

Asking a developer to undertake a pen-and-paper coding exercise never reveals any of that relevant knowledge, insight or craft. Instead, it obstructs many good developers from demonstrating the meaningful work they can do inside a real development environment that takes care of the repetitive, irrelevant and non-creative parts of the process, and lets them focus on the meaningful work they’re actually good at: producing useful, stable, innovative solutions.

3)     Trivial Pursuit

Nearly as bad as using pen-and-paper coding, is having a list of trivia questions associated with the skill of interest, and asking the prospective developer a pre-set list of abstract multiple-choice questions. Generally speaking, interviewers do this when they don’t have an in-depth knowledge of the topic themselves, because it provides an easy way for them to understand whether the answers they receive are ‘right’ or ‘wrong’, without the interviewer having to understand why they may be right or wrong themselves.  It’s not advisable to go down this route, for two reasons. Firstly, to be blunt, it’s pretty disrespectful to the interviewee; if you don’t understand the answers to the questions you’re asking yourself, how can you be certain that the questions and the way you’re framing them makes any sense to someone that is actually knowledgeable on the subject? Go down this route, and it’ll quickly become clear to an experienced candidate that you don’t know what you’re talking about, and are merely repeating what’s written on a piece of paper in front of you parrot-fashion. That in turn will ruin their confidence in your ability to assess them meaningfully, and even if they do ‘pass’ your test, you may very well find the best candidates will opt out of your process as a result. Secondly, even if the developer isn’t put off by being quizzed by someone that clearly doesn’t understand the subject themselves, a process based on the ability to recall abstract facts fails to address the fundamental difference between knowledge and intelligence. I’ll take an intelligent developer that has yet to acquire knowledge in a given area over one that has learned to recite stock answers to set questions but has not one iota of ingenuity and creativity about them any day of the week, and twice on Sunday.

In short, if you use set lists of coding trivia that you don’t really understand the answers to yourself in a misguided attempt to assess your candidates, whilst you may end up with someone that would do very well on a VB Pub Quiz Team, you’re just as likely to end up with an individual that couldn’t implement an IPissup interface within a Brewery class if their lives depended on it. You may find to your horror, after they come on board, that they are not after all the learned experts you supposed, but had merely read the same three articles on Wikipedia as you used when compiling your list of questions.

Finally, I would strongly advise against using ‘online tests’ to assess candidates. Such ‘tests’ tend to have all of the abstractness, lack of context, and general irrelevance of the in-person version of the Meaningless Trivia Test, but are even less personal than the in-person variety.

4)  Getting the candidate to do some real work

Getting close to the way I recommend doing things now, but with caveats. The best developers, like any skilled professional, are happy to demonstrate that they can do the work you’re asking them to do. They welcome employers (or customers, in the case of contractors) exercising meaningful due diligence that allows them to demonstrate they’re the right person for the job. As I mentioned earlier, this is something that many of the best people actively look for in a recruitment process, because as well as giving them an opportunity to prove themselves, it also gives them confidence that their colleagues will be every bit as on their game as they are. Good people want to work with other good people, from whom they can learn and share knowledge, and alongside whom they can grow and develop as ever-more experienced professionals. The fly in the ointment is that there are a small number of less-reputable organisations out there, also advertising jobs, that are only too happy to ask for samples and free work, without ever intending to recruit. For such organisations, unfortunately, it’s a ‘creative’ way of saving on their training budget by getting an external professional to give some free demos under the auspices of an interview, or of getting small projects done for free. For this reason, you need to be careful; you don’t want to chase good people away by coming across as a freeloader.

One good way of avoiding your organisation’s honourable intentions being misconstrued, is to not make providing a sample of the candidate's work the very first stage in the process. Interview the person first, and invest some of your time to show them a little of what you have to offer them as an employer, before asking them to invest their time in providing free work for you. That approach immediately makes you stand out as an organisation that’s serious about recruiting. Organisations that are instead only looking for freebies, generally make it obvious by asking prospective candidates to invest their time in providing a demo solution after only a short telephone screen, and well before the organisation has demonstrated that it has anything of substance to offer the right candidate. Less reputable organisations will also have the cheek to ask for second and third samples of work, and changes to work submitted, never attempting to explain why earlier demos didn’t satisfy their ostensible purpose of demonstrating the candidate’s technical ability. I advise those on the hiring side of the recruitment table never to behave this way; not only will it be entirely obvious to people with marketable skills that you’re merely freeloading, but the best people can and will inform their equally-skilled professional colleagues about their experience with your process, and it’s very likely that you’ll be left with only sub-standard applicants should you become serious about recruiting at any point in the future. Reputations are hard to win, and easy to lose; for candidates, for employment brands and for internal and external recruiters of all flavours alike.

So much for the ‘wrong’ ways to go about finding the right people to join your fold. Now, the right way. As touched upon above, the best people will welcome you engaging in due diligence. There’s no better way to do this than by getting your existing technical staff to speak with the prospective new member of the team. Ideally, on your side of the interview table, as a minimum, you’ll have your lead developer or development manager (whatever nomenclature you use within your particular organisation, they should be the person your most senior management are most likely to go to when they want an opinion on how anything technical should be tackled within your organisation).

Invite your prospective new employee in to speak with you and your lead tech informally for a first interview (or a telephone screen, if distance is an issue), and don’t beat them up during that initial interview. The first interview should be to establish just two things: one, whether the person in front of you is manifestly unqualified for the work concerned, and two, to give them a taste of what you have to offer and to let them know what you expect in return. That’s it for the first meeting; those are the only goals. The aim isn’t to establish whether they’re exactly right for your organisation just yet, and it’s not to make any hasty decisions based on limited information. During your conversation, ask them to tell you about something particularly innovative that they’ve worked on recently, or better still their pet project; every developer has at least one on the go, and, whilst they’ll mostly be commercially irrelevant to your business, they’ll tell you a lot about how well your prospective new colleague is able to communicate the concepts they understand to others.

As mentioned earlier, I strongly advise against asking coding trivia questions from a set list, but it’s perfectly acceptable to have your internal technical expert ask the candidate to explain more about the nuts and bolts of the candidate’s pet project or most recent work. This is why you have your best existing internal technical resource on your side of the interview table; they should be able to probe and question during a discussion of the design, without coming across as needing to be the smartest person in the room. Your technical expert should avoid the temptation to ask the candidate any ‘trick questions’, that rely on obscure technical knowledge that they’ve only just encountered themselves, but it’s OK to probe a little to see how deep the candidate’s knowledge of underlying concepts is. It’s also OK to ask about newer technologies that the candidate may or may not have encountered; if they’ve not, you’re looking for them to tell you so rather than for them to waffle or bluff their way through – knowing what they don’t know is a good trait in a developer. Your internal technical expert may not know everything that the candidate does about the subject at hand, especially if the candidate is intended to add a skill to your team that it hasn’t yet mastered, but there should be enough common ground for your tech lead to determine whether the candidate outright doesn’t know their subject area.

After that first interview, ask suitable candidates to participate in a coding task. You can choose to allow your candidate to do this at home, or within your premises; it shouldn’t matter which to you, although bear in mind that some developers wont have access to all of the tools at home that they’re used to having in their workplace. The developer should have access to every resource they’ll have available on the actual job, including the internet, any reference books they want, MSDN, etc. They should also have a reasonably-powerful development machine, similar to the one they’d be using if they were to be recruited. Make sure that the problem you ask them to solve is realistic, and can be developed by someone with reasonable skill in the area you’re recruiting for within 3-4 hours max. Ideally, get one of your existing developers that wasn’t involved in designing the exercise to try out the problem first, to be sure it’s of a reasonable level of difficulty. Make it obvious to your candidate that you’re not asking them to do unpaid commercial work, by presenting them with an abstract problem that is clearly unrelated to your business (e.g., perhaps ask them design a Windows Service that writes an entry to the event log every time the sum of the minutes in the current time is a prime number, or something similar). It’s not usually possible to present candidates with a real commercial problem, because even if it’s something your team has actually tackled within a set time frame, chances are you brought existing knowledge to your understanding of the problem that brand new candidates wont be able to draw upon.

After your candidate has completed their solution, get them to talk through their design with you, and review it with them in the same open and non-critical way that you’d conduct internal code reviews on real, commercial pieces of work. Doing this will not only tell you a lot about how the candidate approaches technical tasks, it will additionally give you an insight into how well they work with others as part of a mutually respectful and co-operative team. If done properly, it’ll also give the candidate a good insight into your culture, and help you both to decide whether or not you’re a good fit for one another. What you should be looking for in an ideal candidate isn’t just a window onto their existing knowledge, but their ability to apply what they know to new problems, whether they have the honesty and self-awareness to recognise what they don’t yet know, and, of key importance, whether they have the initiative to find things out as and when the need arises. Those are all key traits that a set list of coding trivia questions or a faceless online test would never tell you.

After your ‘technical review’ with the candidate, discuss your findings internally amongst the recruitment team. Subsequently, if you decide to proceed, have one, final interview with the candidate, to review all that you’ve learned about one another in a post-mortem type setting. Ask them about their experience of your process, discuss any issues that you’d like to clarify with them, and be prepared to take any questions they’ll have of you. This meeting is about rounding out what you know about one another, and finalising any missing details.

Once you’ve completed the above process, your internal ‘recruitment team’ (consisting of the hiring manager, your internal technical expert, and any advisers from HR that you’ve consulted) should be in an excellent position to decide whether the person in front of you is a right fit for your needs or not. Don’t be tempted to conclude that just because you’ve come all the way through the process, you need to extend an offer; psychologists call that phenomenon ‘justification of effort’. Remember that your process is designed to give you the information you need to make an informed decision, and to give a favourable impression of your organisation to everyone that has taken the time to apply. It’s not merely about picking the last person standing.

Put any candidates that have made it to the end of your process into three categories: first choice, any second choices, and any unselected candidates. Contact unselected candidates right away, to thank them for their time, and to tell them they haven’t been selected. Keep any second choices in reserve whilst your external recruiter extends and negotiates a mutually-agreeable offer with your preferred candidate (hiring managers should never do this part themselves – negotiation is skill that most hiring managers don’t have, and, even when they do, this process is still at its most effective when conducted via an intermediary). If an offer can’t be agreed with the preferred candidate, approach any second preferences and try to come to an agreement with them.

At the end of the process, let every candidate that hasn’t been selected know the outcome, and, if they’d like to know, the reasons for your decision. The classiest way of doing this is for the hiring manager to call each individual personally, to thank them for their time, and to express regret that they weren’t the right person on this occasion, followed up by a letter to confirm the news. Many hiring managers shy away from doing this, perhaps through fear that the candidate will react badly. In my experience, though, most if not all professional candidates really appreciate it when the organisations in which they’ve invested their time and effort take the time to conclude the process properly. In years of informing people in person when they’ve not got a job, I’ve never once had the experience of a candidate reacting with anything other than courtesy and appreciation that I had enough respect to let them know the outcome myself, when I called to let them know the decision. If you’re unfortunate enough to experience a candidate being aggressive, or treating the call like a debate rather than a simple communication of your decision, feel free to end the call quickly – you’re not there to be their career counsellor, and you’re not there to justify yourself beyond respectfully communicating your decision, and your reasons for making that decision if the candidate is interested in hearing them. Conclude matters this way, and you’ll always have a steady stream of candidates for jobs that you may advertise in the future; word gets around about which organisations treat candidates well and which don’t, and it’s amazing how often that candidate that wasn’t quite right for you yesterday, has exactly the right skills for the position that comes up tomorrow, or knows someone that does. End things well, and you’ll always have the ability to open dialogue again if it becomes relevant to do so.