Improve your .NET code quality with NDepend

The unit test effect study, refined

The Unit Test Effect Study, Refined

About a month ago, I wrote a post about how unit tests affect (and apparently don’t affect) codebases.  That post turned out to be quite popular, which is exciting.  You folks gave a lot of great feedback about where we might next take the study.  I’ve incorporated some of that feedback and have a followup on the unit test effect on codebases.

Summarized briefly, here are the high points of this second installment and refinement of the study:

  • Eliminating the “buckets” from the last time.
  • Introducing more statistical rigor.
  • Qualifying and refining conclusions from last time.

Also, for the purposes of this post, please keep in mind that non-incorporation of feedback is not a rejection of that feedback.  I plan to continue refinement but also to keep posting about progress.

Addressing Some of the Easier Questions and Requests

Before getting started, I’ll answer a few of the quicker-to-answer items that arose out of the comments.

Did your analysis count unit test methods when assessing cyclomatic complexity, etc.?

Yes.  It might be interesting to discount unit test methods and re-run analysis, and I may do that at some point.

Can you show the code you’re using?  Which codebases did you use?

The scraping/analysis tooling I’ve built using the NDepend API is something that I use in my consulting practice and is in a private repo.  As for the list of specific codebases, I’m thinking I’ll publish that following the larger sample size study.  In the most general terms, I’m going through pages like this that list (mostly) C# repos and using their links.

What about different/better categorization of unit test quality (test coverage, bolted on later vs. written throughout vs. demonstrably test driven)?  

This is definitely something I want to address, but the main barrier here is how non-trivial this is to assess from a data-gathering perspective.  So I will do this, but it will also take time.

Think of even just the anecdotally “easy” problem of determining TDD vs. non-TDD.  I approximated this by positing that test-driving will create a certain ratio of test methods to production methods since any production method will be preceded by a test method (notwithstanding future extract method refactorings).  We could, perhaps, do better by auditing source control history and looking for a certain commit cadence (modification to equal numbers of test/production classes, for instance).  But that’s hard, and it doesn’t account for situations with large batch commits, for instance.

The upshot is that it’s going to take some doing, but I think we collectively can figure it out.

Continue reading The Unit Test Effect Study, Refined

Unit testing doesn't affect codebases the way you think.

Unit Testing Doesn’t Affect Codebases the Way You Would Think

I’ve just wrapped up another study.  (The last one was about singletons, if you’re interested.) This time, I looked at unit testing and the impact it has on codebases.

It didn’t turn out the way I expected.

I’m willing to bet that it also won’t turn out that you expected these results.  It had a profound effect on certain unexpected codebase properties while having minimal effect on some that seem like no-brainers.  But I’ll get to the specifics of that shortly.

First, let me explain a bit about expectations and methodology.

Unit Testing: The Expected Effect on Code

Let’s be clear for a moment here.  I’m not talking about the expected effect of unit tests on outcomes. You might say, “We believe in unit testing because it reduces defects” or, “We unit test to document our design intentions,” and I didn’t (and probably can’t) measure those things.

When I talk about the effect of unit testing, I’m talking about the effect it has on the codebase itself.  Let’s consider a concrete example to make it clear what I mean.  You can’t (without a lot of chicanery) unit test private methods, and you can’t easily unit test internal methods.  This creates a natural incentive to make more methods public, so we might expect heavily unit-tested codebases to feature more public methods.

This actually turns out to be true.

The rate of public methods increases with increased prevalence of unit testing.

I’ll talk more about the axes later in the post.  But for now, check out the plot and the trend line.  More unit testing means a higher percentage of public methods.  Score one for that hypothesis!

With a win there, let’s think of some other hypotheses that seem plausible.  Methods with more “going on” tend to be harder to test.  So you’d expect relatively simple methods in a highly tested codebase.  To get specific, here was what I anticipated in heavily tested codebases:

I also had some thoughts about the impact on types:

  • More interfaces (this makes testing easier).
  • Less inheritance (makes testing harder).
  • More cohesion.
  • Fewer lines of code.
  • Fewer comments.

Continue reading Unit Testing Doesn’t Affect Codebases the Way You Would Think

The Singleton Design Pattern: Impact Quantified

The Singleton Design Pattern: Impact Quantified

This post has been about a month in the offing.  Back in August, I wrote about what the singleton pattern costs you.  This prompted a good bit of discussion, most of which was (as it always is) anecdotal.  So a month ago, I conceived of an experiment that I called the singleton challenge.  Well, the results are in.  I’m going to quantify the impact of the singleton design pattern on codebases.

I would like to offer an up-front caveat.  I’ve been listening lately to a fascinating audiobook called “How to Measure Anything,” and it has some wisdom for this situation.  Measurement is primarily about reducing uncertainty.  And one of the driving lessons of the book is that you can measure things — reduce uncertainty — without getting published in a scientific journal.

I mention that because it’s what I’ve done here.  I’ll get into my methodology momentarily, but I’ll start by conceding the fact that I didn’t (and couldn’t) control for all variables.  I looked for correlation as a starting point because going for causation might prove prohibitive.  But I think I took a much bigger bite out of trying to quantify this than anyone has so far.  If they have, I’ve never seen it.

A Quick Overview of the Methodology

As I’ve mentioned in the past on this blog, I earn a decent chunk of my consulting income doing application portfolio assessments.  I live and breathe static code analysis.  So over the years, I’ve developed an arsenal of techniques and intellectual property.

This IP includes an extensive codebase assessor that makes use of the NDepend API to analyze codebases en masse, store the results, and report on them.  So I took this thing and pointed it at GitHub.  I then stored information about a lot of codebases.

But let’s get specific.  Here’s a series of quick-hitter bullets about the experiment that I ran:

  • I found this page with links to tons of C# projects on GitHub, so I used that as a “random” selection of codebases that I could analyze.
  • I gave my mass analyzer an ordered list of the codebase URLs and turned it loose.
  • Anything that didn’t download properly, decompress properly, or compile properly (migrating for Core, restoring NuGet packages, and building from command line) I discarded.  This probably actually creates a bias toward better codebases.
  • Minus problematic codebases, I built all solutions in the directory structure and made use of all compiled, non-third-party DLLs for analysis.
  • I stored the results in my database and queried the same for the results in the rest of the post.

I should also note that, while I invited anyone to run analysis on their own code, nobody took me up on it.  (By all means, still do it, if you like.)

Singleton Design Pattern: the Results In Broad Strokes

First, let’s look at the scope of the experiment in terms of the code I crunched.  I analyzed

  • 100 codebases
  • 986 assemblies
  • 5,086 namespaces
  • 72,615 types
  • 501,257 methods
  • 1,495,003 lines of code

From there, I filtered down raw numbers a bit.  I won’t go into all of the details because that would make this an immensely long post.  But suffice it to say that I discounted certain pieces of code, such as compiler-generated methods, default constructors, etc.  I adjusted this so we’d look exclusively at code that developers on these projects wrote.

Now, let’s look at some statistics regarding the singleton design pattern in these codebases.  NDepend has functionality for detecting singletons, which I used.  I also used more of its functionality to distinguish between stateless singleton implementations and ones containing mutable state.  Here’s how that breaks down:

Continue reading The Singleton Design Pattern: Impact Quantified