NDepend

Improve your .NET code quality with NDepend

A Test Coverage Primer for Managers

Managing Blind

Let me see if I can get in your head a little bit.  You manage a team of developers and it goes well at times.  But, at other times, not so much.  Deadlines slide past without a deliverable, weird things happen in production and you sometimes get deja vu when a bug comes in that you could swear the team fixed 3 months ago.

What do you do?  It’s a maddening problem because, even though you may once have been technical, you can’t really dive into the code and see what’s going on.  Are there systemic problems with the code base, or are you just experiencing normal growing pains?  Are the developers in your group painting an accurate picture of what’s going on?  Are they all writing good code?  Are they writing decent code?  Are any of them writing decent code?  Can you rely on them to tell you?

As I said, it’s maddening.  And it’s hard.  It’s like coaching a sports team where you’re not allowed to watch the game.  All you can do is rely on what the players tell you is going on in the game.

A Light in the Darkness

And then, you light upon a piece of salvation: automated unit tests.  They’re perfect because, as you’ll learn from modern boilerplate, they’ll help you guard against regressions, prevent field defects, keep your code clean and modular, and plenty more.  You’ve got to get your team to start writing tests and start writing them now.

But you weren’t born yesterday.  Just writing tests isn’t sufficient.  The tests have to be good.  And, so you light upon another piece of salvation: measuring code coverage.  This way, not only do you know that developers are writing tests, but you know that you’re covered.  Literally.  If 95% of your code base is covered, that probably means that you’re, like, 95% good, right?  Okay, you realize it’s maybe not quite that simple, but, still, it’s a nice, comforting figure.

Conversely, if you’re down at, say, 20% coverage, that’s an alarming figure.  That means that 80% of your code is not covered.  It’s the Wild West, and who knows what’s going on out there?  Right?  So the answer becomes clear.  Task a team member with instrumenting your build to measure automated test coverage and then dump it to a readout somewhere that you can monitor.

Time to call the code quality issue solved (or at least addressed), and move on to other matters.  Er, right?

Fools’ Gold?

I’ve managed software developers in the past, so I understand the disconcerting feeling that comes from not really being able to tell what’s happening under the covers.  These days, I’m living a little closer to the keyboard, and my living comes largely from coaching developers, and doing IT management consulting and gap analysis.  And a common gap is managers thinking that test coverage correlates a lot more strongly with well written, well tested code than it actually does.

Don’t get me wrong.  In a nice, clean code base, you’re likely to see extremely high test coverage figures.  But many managers get the cause and effect relationship backward.  Really well crafted code causes high coverage.  High coverage does not necessarily cause really well crafted code.

To understand the disconnect, consider the relatively common situation of purchasing a home.  In the USA, part of this process is a (generally) mandatory home inspection in which a licensed home inspector examines the house in detail, making note of any deficiencies, problems, dangers, etc, and giving a general report on what kind of shape the house is in.

Now, you’re no carpenter, and you certainly don’t understand what all the home inspector is doing.  But, you want to know if he’s doing a good job.  So, you introduce a metric.  You make note of how many rooms in the house he enters during his inspection.  If he doesn’t cover at least 90% of the rooms, he’s not doing great work.  If he only covers 20%, he’s terrible.

But the same problem arises.  The fact that he wandered through every room in the house does not mean he did a good inspection.  If he did a thorough inspection, the result will be that he covered a high percentage of rooms.  If he covered a high percentage of rooms, he may or may not have done a thorough inspection.

There Are No Shortcuts

So, what if you timed the inspector?  And, what if you introduced another metric in addition to code coverage to make it harder to game or to be wrong?  Well, that might help a bit, but there are no easy answers, really.  If you want to know whether the inspector does good work, hire another inspector or two, and triangulate.  If you want to know whether a test suite is good or not, you need to have someone with experience do a lot more digging and inspection of the code.  I know you don’t want to hear this because you’re picturing costs, but knowledge isn’t cheap.

I assess code bases a lot, and when looking at test suites, here are some questions I ask (and then answer):

  • What is the average number of asserts per method?  Way less than 1 means you might have useless tests.  Way more than 1 means you might have convoluted, brittle tests.
  • What percentage of test methods have a number of asserts other than 1?  This might indicate a flawed approach to testing.
  • What is the average size of a test method?  Large test methods tend to indicate brittle tests that people will abandon.
  • Do tests often refer to static/global state?  This may indicate an unreliable test suite.
  • Do tests have a lot of duplication?  This may indicate poorly written tests or even a poor design of production code.

I could go on, but you get the idea.  An important takeaway is that I say “may” and “might” in all of my explanations because, you know what?  Life isn’t simple.  These things may exist in your test code, and there may be a perfectly reasonable explanation.  Just as test coverage may or may not be indicative of code quality.

Detecting whether things are on the right track or not isn’t easy.  But the more metrics you have, and the more you use them as triggers for further investigation instead of quality gates, the better equipped you and your group will be.

A Software Architect’s Best Friend

To this day, I have a persistent nightmare about my time in college.  It’s always pretty similar.  I wake up and I have a final exam later in the day, but I’m completely unprepared.  And I don’t mean that I’m unprepared in the sense that I didn’t re-read chapter 4 enough times.  I mean that I realize I haven’t attended a single lecture, done a single homework assignment, or really even fully understood what, exactly, the course covers.  For all intents and purposes, I’m walking in to take the final exam for a class I haven’t taken.  The dream creates a feeling that I am deeply and fundamentally unprepared for life.

I’ve never encountered anything since college that has stuck with me quite as profoundly; I don’t dream of being unprepared for board meetings or talks or anything like that.  Perhaps this is because my brain was still developing when I was 20 years old, or something.  I’m not a psychologist, so I don’t know.  But what I do know is that the closest I came to having this feeling was in the role of software architect, responsible for a code base with many contributors.

If you’re an architect, I’m sure you know the feeling.  You’ve noticed junior developers introducing global variables on a few occasions and explained to them the perils of this practice, but do you really know they’re not still doing it?  Sure, you check from time to time, but there’s only one of you and it’s not like you can single-handedly monitor every commit to the code base.  You do like to go on vacation at least every now and then, right?  Or maybe you’ve shown your team a design pattern that will prevent redundancy and standardize what had been a hodgepodge approach.  You’ve monitored the situation for a few commits and noticed that they’re not getting it quite right, but you can live with what they’re doing… as long as it doesn’t get any further off the rails.

You get the point.  After all, you’re living it.  You don’t need me to tell you that it’s stressful being responsible for the health of a code base that’s changing faster than you can track in any level of detail.  If you’re anything like me, you love the role, but sometimes you long for the days when you were making your presence felt by hacking things and cranking out code.  Well, what if I told you there were a way to get back there while hanging onto your architect card… at least, to a degree?

You and I both know it — you’re not going to be spending 8+ hours a day working on feature implementations, but that doesn’t mean that you’re forever consigned to UML and meetings and that your hacking days are completely over.  And I’m not just talking about side projects at night.  It’s entirely appropriate for you to be writing prototyping code and it’s also entirely appropriate for you to write little plugins and scripts around your build that examine your team’s code base.  In fact, I might argue that this latter pursuit should be part of your job.

What do I mean, exactly, by this?  What does it mean to automate “examining your team’s code base?”  You might be familiar with this in its simplest forms, such as a build that fails if test coverage is too low or if there are compiler warnings.  But this concept is extremely extensible.  You don’t need to settle for rudimentary information about your team’s code.

NDepend is a tool that provides you with a lot of information out of the box as well as hooks with which to integrate that information into your team’s build.  You can see architectural concerns at a high level, such as how much you depend on external libraries and how internally coupled your modules are.  You can track more granular concerns such as average method complexity or class cohesion.  And you can automatically generate reports about these things, and more, with each build.  All of that comes built in.

But where things get really fun and really interesting is when you go beyond what comes out of the box and start to customize for your own, specific situation.  NDepend comes with an incredibly powerful feature called CQLinq that lets you ask very custom, specific questions about your code.  You can write a query to see how many new global variables your team is introducing.  You can see if your code base is getting worse, in terms of coupling.  And, you can even spend an afternoon or two putting together a complex CQLinq query to see if your team is implementing the pattern that you prototyped for them.

And not only can you see it — you can see it in style.  You can generate a custom report with clear, obvious visuals.  This sort of visualization isn’t just decoration.  It has the power to impact your team’s behavior in a meaningful way.  They can see when their checkins are making more things green and happy and when their checkins are making things red and angry.  They’ll modify their code accordingly and basically anonymously, since this feedback is automated.  They’ll like it because automated feedback won’t feel judgmental to them, and you’ll like it because you’ll know that they’re being funneled toward good architectural decisions even when you aren’t there.  Talk about peace of mind.

You’ll spend a few days getting to know NDepend, a few days tweaking the reports out of the box, and a few days hacking the CQLinq queries to guard your code base according to your standards.  And, from there on in, you’ll enjoy peace of mind and be freed to focus on other things that command your attention.  As an architect there are a thousand demands on your time.  Do yourself a favor and get rid of the ones that you can easily automate with the right tooling.

Why Should Managers Care About Static Analysis?

I’d like to talk a bit today about how, if you’re a dev manager for a team or teams that are responsible for .NET code bases, NDepend will help you lower your blood pressure and sleep better at night.  But first, bear with me as a I introduce and explain myself a bit.  It’ll be relevant.  I promise.

There’s been a relatively unique fluidity to my career, particularly in recent years.  The first 10 years or so of my professional career featured somewhat of a standard path from developer to management, with milestone stops along the way earning me designations like “senior” developer, team lead, and architect.  At the end of my salaried career, I had earned myself a CIO role, presiding over all development and IT operations for a company.  And, while I enjoyed the leadership responsibilities that came along with that sort of role, I ultimately decided to go off on my own into consulting, where I’d have more variety and a more fluid set of responsibilities.  These days, I do a combination of development coaching, management consulting, and content creation to earn my living.

The reason I mention all of this is that I’ve had success as a developer, architect, manager, and consultant to all of the above, and that multi-disciplined success has given me unique perspective on the interaction among those roles.  It’s shown me that, above all, what you need to be successful is the ability to trust your dev team.  But trust doesn’t come easily.  You didn’t get to where you are by blindly trusting anyone who tell you a pleasant story.

Here’s your essential conundrum as a manager.  You’re managing a team of intelligent humans that are doing things you don’t understand.  You can protest and claim to understand what they’re doing, but you don’t.  I learned this in my travels as a manager, even managing a team of .NET folks who turned to me to answer their toughest technical questions.  I may have had years and years of professional development on them, but I wasn’t there in the weeds with them, dealing with the challenges that they were, day in and day out.  I had to trust them because keeping my nose in their affairs didn’t scale.  So no matter how well you might understand the surrounding technologies and principles, you don’t understand what they’re doing at any given moment.

There’s a tendency toward entropy that kicks in when you’re managing people whose labor you don’t understand.  A lot of managers in this situation have to fight a panicky impulse to regain control via micromanagement.  I can’t really tell if they’re making good decisions in their coding, but I can make sure they’re getting in no later than 9, leaving no earlier than 5, writing at least 100 lines of code per day, and keeping unit test coverage about 90%.  You may laugh, but it’s a natural impulse, particularly in the face of a slipped deadline or low quality release.  The C-suite wants answers, and you’re caught in the middle, scrambling simultaneously to provide them and to figure out how to prevent this in the future.  You didn’t get ahead in your career through passivity, so the impulse is to seize control, even if perhaps you realize it’s a misguided impulse.

NDepend provides help here.  It allows you to keep an eye on metrics that aren’t obtuse if you structure things right.  I cannot overemphasize this caveat.  Renowned systems thinker W. Edwards Deming once explained that, “people with targets and jobs dependent upon meeting them will probably meet the targets – even if they have to destroy the enterprise to do it.”  Translated into our terms for this post, what this means is that if you measure your team’s productivity by them producing 100 lines of code per day, they will produce 100 lines of code per day, even if it means creating unstoppable death stars of terrible code.  So disabuse yourself of the notion that you can use this tool to evaluate your developers’ performance, and understand instead that you can use it to identify potential trouble spots in the code.

Pair with a developer or your team’s architect, and use NDepend’s phenomenal reporting capabilities to create a big, visible report on the state of your team’s code.  Here are some ideas for things that might interest you:

  • Which sections of the code are most heavily depended on, so that I can have a heads up when they are changed?
  • How coupled is a module to the rest of the code base?  So, for instance, if I wanted to swap in a new database solution, how hard would that be?
  • Are the most troublesome areas of the code base getting worse, staying the same, or improving?
  • Is the team’s development practice generally in line with commonly accepted good ideas?
  • What does the architecture of the code actually look like (as opposed to some design document that was written and may be out of date)?
  • Are there currently any code violations that the team has identified as critical mistakes?

When I was managing teams, I certainly would have been interested in the answers to these questions for strategic planning purposes.  How dependent you are on a database or whether a feature implementation has touched a dicey section of the code is valuable information for making important decisions (maybe it’s not worth considering a database switch and maybe we should add in more testing than usual for the upcoming release).

Not only can NDepend give you this information, but it can be integrated with your build to furnish it in a very clear, easy, and visible way.  Picture it integrated with your team’s build and delivered to your inbox in digest format on a daily basis.  This is possible to do with your build tool — you could work with the developers on your team to design a build customization where you can see if an area of the code is getting worse or how much coupling there is among the modules in the code.

Imagine it.  Instead of wondering whether your developers are actually working and whether they’re making good decisions, you can partner with them to keep you in the loop on important characteristics of the code, and also to keep you out of their hair for the details it’s better for them to worry about.  You get your information and early detection scheme, and they benefit from having an informed manager that asks intelligent questions only when it makes sense to do so.  And for you, that all adds up to better sleep and lower blood pressure.