The Singleton Design Pattern: Impact Quantified

This post has been about a month in the offing.  Back in August, I wrote about what the singleton pattern costs you.  This prompted a good bit of discussion, most of which was (as it always is) anecdotal.  So a month ago, I conceived of an experiment that I called the singleton challenge.  Well, the results are in.  I’m going to quantify the impact of the singleton design pattern on codebases.

I would like to offer an up-front caveat.  I’ve been listening lately to a fascinating audiobook called “How to Measure Anything,” and it has some wisdom for this situation.  Measurement is primarily about reducing uncertainty.  And one of the driving lessons of the book is that you can measure things — reduce uncertainty — without getting published in a scientific journal.

I mention that because it’s what I’ve done here.  I’ll get into my methodology momentarily, but I’ll start by conceding the fact that I didn’t (and couldn’t) control for all variables.  I looked for correlation as a starting point because going for causation might prove prohibitive.  But I think I took a much bigger bite out of trying to quantify this than anyone has so far.  If they have, I’ve never seen it.

A Quick Overview of the Methodology

As I’ve mentioned in the past on this blog, I earn a decent chunk of my consulting income doing application portfolio assessments.  I live and breathe static code analysis.  So over the years, I’ve developed an arsenal of techniques and intellectual property.

This IP includes an extensive codebase assessor that makes use of the NDepend API to analyze codebases en masse, store the results, and report on them.  So I took this thing and pointed it at GitHub.  I then stored information about a lot of codebases.

But let’s get specific.  Here’s a series of quick-hitter bullets about the experiment that I ran:

• I found this page with links to tons of C# projects on GitHub, so I used that as a “random” selection of codebases that I could analyze.
• I gave my mass analyzer an ordered list of the codebase URLs and turned it loose.
• Anything that didn’t download properly, decompress properly, or compile properly (migrating for Core, restoring NuGet packages, and building from command line) I discarded.  This probably actually creates a bias toward better codebases.
• Minus problematic codebases, I built all solutions in the directory structure and made use of all compiled, non-third-party DLLs for analysis.
• I stored the results in my database and queried the same for the results in the rest of the post.

I should also note that, while I invited anyone to run analysis on their own code, nobody took me up on it.  (By all means, still do it, if you like.)

Singleton Design Pattern: the Results In Broad Strokes

First, let’s look at the scope of the experiment in terms of the code I crunched.  I analyzed

• 100 codebases
• 986 assemblies
• 5,086 namespaces
• 72,615 types
• 501,257 methods
• 1,495,003 lines of code

From there, I filtered down raw numbers a bit.  I won’t go into all of the details because that would make this an immensely long post.  But suffice it to say that I discounted certain pieces of code, such as compiler-generated methods, default constructors, etc.  I adjusted this so we’d look exclusively at code that developers on these projects wrote.

Now, let’s look at some statistics regarding the singleton design pattern in these codebases.  NDepend has functionality for detecting singletons, which I used.  I also used more of its functionality to distinguish between stateless singleton implementations and ones containing mutable state.  Here’s how that breaks down:

• 50 of the 100 codebases didn’t use singletons at all.
• 49 codebases used singletons, leaving one odd codebase that didn’t have any developer-created types.  ¯\_(ツ)_/¯
• Of the 49 singleton codebases, 32 of them used stateful singletons.
• These singleton codebases used 828 stateless singletons and 84 stateful ones.
• Just two of these codebases made use of a whopping 637 singletons (mostly stateless), accounting for 70% of total use.
• The stateful singletons were more evenly distributed, with no single codebase using more than 12% of them.

This seems to generally indicate that the singleton design pattern is pretty prevalent, but that stateful singletons are becoming less common.

My Original Hypotheses

Here were some hypotheses that I stated in the post announcing this experiment.

• Stateful singleton prevalence varies inversely with the number of unit tests in a codebase.
• But it varies directly with average method cyclomatic complexity in that codebase.
• It varies inversely with assembly relational cohesion.
• But it varies directly with average afferent coupling per type.
• And it varies directly with average type lack of cohesion of methods (LCOM).

How These Held Up

• Calculating unit test prevalence was somewhat imprecise, as my current best means of doing this was to detect methods invoking some kind of assert and calculate what percentage of a codebase’s methods do that.  For the non-singleton (NS) codebases, 1.29% of methods were test methods.  For singleton (S) codebases, this was 5.13%. And for stateful singleton (SS) codebases, the figure was 2.15%.  This data does not support my hypothesis.
• Method cyclomatic complexity looks a lot better for me. For NS, S, and SS, respectively, average method cyclomatic complexity was 1.42, 1.82, and 1.98.  SS codebases are thus about 40% more complex at the method level.
• I was wrong about relational cohesion, but, in retrospect, that was kind of a stupid hypothesis.  The relational cohesion within an assembly essentially measures how interrelated types are, and one would probably expect more inter-type relationships where global state involves itself.  Average NS, S, and SS assembly cohesion was 1.21, 1.56, and 1.62, respectively.  Interestingly, NDepend thinks the NS codebases are a little too low and the S and SS ones are okay (but just barely).
• With afferent coupling, I nailed it.  NS, S, SS average values per type are 3.35, 6.86, and 5.85, respectively.  This means that types in S codebases have more than double the coupling of NS codebases.  It’s curious that SS actually mutes the effect a bit compared to S codebases.  That’s probably worth further investigation.
• With LCOM (lack of cohesion of methods) I also got it right.  NS, S, and SS values are 0.12, 0.13, and 0.16, respectively, with SS codebases averaging a 33% decrease in cohesion.  Because singletons promote type interdependence, you might expect higher assembly cohesion with simultaneously lower type cohesion.  And that’s what we see.

Other Observations

I gathered a lot more statistics besides, so I’ll list some of these that I found interesting as well.

• Method level cyclomatic complexity has a clear trend as well.  NS, S, SS values there are 1.43, 1.82, and 1.98, respectively.  In other words, the average SS codebase method is 40% more complex than the average NS codebase, in terms of paths through the code.
• Lines of code per method shows a similar trend, with NS, S, SS values of 3.46, 4.09, and 4.25, respectively.  NS codebases have shorter, less complex methods.
• This same trend holds up with method overload counts as well, where NS, S, SS have an average of 1.58, 2.76, and 3.08 overloads respectively.  I would have thought this relationship orthogonal, so that’s interesting.
• Another interesting result comes from the number of parameters, with NS, S, SS, 1.15, 1.14, and 1.19.  This property seems mostly unaffected by the presence of singletons.
• For the sake of brevity, here are a few more metrics where NS codebases have significantly less of the value than ones with singletons: control flow nesting depth, number of locals per method, number of methods called by the average method, type rank, and number of methods the average methods calls.
• And here are a few (perhaps some surprising) ones where NS codebases have a higher value than S codebases: percentage of static methods, percent of methods that read mutable state, number of fields per type, and rate of code comments.
• And finally, some metrics where no meaningful trend seems to exist: lines of code per type, number of methods per type, type inheritance depth, and method rank.

What’s the Takeaway?

I’ve included a lot of statistics about what I did, and I’ve elided even more.  But, on the whole it’s been pretty interesting and, I think, revealing.  Open source codebases are subject to a lot of scrutiny. They’re often created and maintained by those in it for the love of the game — serious programmers.  On the whole, we’d expect these codebases to be cleaner than average.  And in terms of raw stats (compared to the enterprise codebases I evaluate), they are.

But if I’m to believe the people that claim the problem with singletons is only that people abuse them, this study doesn’t support that conclusion.  While stateful singletons (“abuse”) did exhibit exaggerated properties compared to stateless ones (“use”), both sets differ significantly from non-singleton codebases in important ways.

In general, the singleton codebases had more path complexity, more conditional nesting, more code per method, more coupling among types, more local variables, and more things in general that make it harder to reason about and maintain code.  On the flip side, they only seemed to have an advantage in having fewer class level fields, but this might be simply explained by a reliance on global collaborators.  Based on this data, you can see significant drawbacks to the pattern in aggregate with almost no arguable advantages.

Again, it’s worth remembering that this is only a look at correlations and only an attempt to replace anecdotes with at least some data.  But based on what I’ve done here, I feel comfortable reiterating my claim that the singleton design pattern has a very real cost to your codebases and your team.