NDepend

Improve your .NET code quality with NDepend

Toward Bug Free Software: Lines of Defense

Hurrah!! Last week we released NDepend v6 RTM. Once again we relied on a 2 months private beta-testing period and a one month Release Candidate period to do our best to release a polished and stable product.

I’d like to talk about our lines of defense to fix as many bugs as possible. Except a few pieces of software in the world that can afford mathematical demonstrations to prove they are bug-free (like plane and some medical ones), all other pieces of software, including NDepend, rely on an empirical approach to chasing bugs and fixing them. An empirical approach is an evidence based approach that relies on direct observations and experimentation in the acquisition of new knowledge. An empirical approach will never lead to a bug-free product, but it can help a lot in keeping the number of bugs low, and make it so that bugs happen in rare enough situations that won’t have any impact on most user’s experience.

I could write many blog posts about each line of defense, after more than a decade applying them there is so much to say, but I want this post to be synthetic.

Production Crash Logs

Crashes are due to unhandled exceptions. Unhandled exceptions are due to situations at runtime that were unexpected, this typically includes:

  • null reference access,
  • division by zero,
  • disposed object accessed,
  • invalid cast,
  • wrong method call parameters,

A bug doesn’t necessarily lead to a crash, but a crash necessarily mean that there is a bug. Certainly the most important line of defense against bugs is to log all production crashes and relentlessly fix them all. The .NET Framework offers several unhandled exception access points, including:

In some environments like Visual Studio hosting, these access points don’t work and you’ll have to write code to catch all exceptions in all possible handlers of your program.

Of course some users are disconnected from internet, or behind a proxy, and you won’t get those production crashes. Our statistics show that this concerns only 20% of users at most. So being aware of 80% of all production crashes is certainly enough to have a good measure of what’s going wrong in a production. And because Windows and .NET are highly sophisticated technologies that are constantly evolving, you can expect plenty of issues that never occurred on your team’s machines! For example, the NDepend v6 Release Candidate Period shows us that users running NDepend v6 RC on a fresh Windows 8.1 or Windows 10 install, experienced crashes because of a P/Invoked win32 method that our code calls. Oddly enough this P/Invoked method behaves differently when .NET v3.5 is not installed on the machine!

The key to successful production crash logs, is to get as many useful data as possible per log. For example here is a production crash below we had a few week ago. When the same issue lead to several crash logs, we can start doing data mining on it. Do they all occur with the same stack trace? with the same high DPI resolution? on the same Windows platform version ? on the same machine only? Notice also the stack frames improved with IL offset retrieved with the StackFrame.GetILOffSet() method. Many times in the past, this alone lead us to the root cause.

ProductionCrash

 

You’ll notice that we only log crashes, we don’t have other forms of runtime logs, like logging every major event that happen (button clicked, panel opened, analysis started…). Our experience with logging events is that ultimately we logged too much or not enough of them. In both cases, the information that could help fixi a particular issue is then lost or hard to find. We found out that having verbose crash logs was enough. Sometime we can ask a user a question, like which action did you do just before the crash, but in the vast majority of real-world cases, this information is implicitly contained in the stack trace. For the same reason we don’t use remote debugging nor Windows dump files. In our context, custom and verbose production crash logs are enough.

Code Covered by Automatic Tests and Code Contracts

Not only are production crash logs an important line of defense, but they also demand just a few days of dedicated work to set up.

Having automatic tests is the second line of defense. Contrary to production crash log,s not only does it require a lot of work, but it even changes (forever) the way you write code. After a decade of writing automatic tests, a lot of conclusions can be made. In my attempt to remain synthetic in this post, let’s try to summarize the most relevant ones in a few points:

  • The number of tests is absolutely meaningless.  When it comes to unit/automatic tests, the king measure is the percentage of code covered by tests.
  • A high percentage of code covered by tests is not enough, everything that can be checked must be checked. In almost all literature related to unit testing, you’ll read that checks are assertions in unit test code. Few actually realize that assertions in the code itself are at least as important as assertions in unit-test code. There is a scientific terminology for assertions in code: Code Contracts. The important thing about code contracts is that they must fail both at manual-test time (see later) and at automatic-test time.
  • In NDepend, we use more than 20K simple Debug.Assert() in code. These are our contracts. Debug.Assert() are removed in production code. This is ok for us since we want maximum performance and having some sophisticated assertions at runtime can significantly slow down an application. Hence we decided to sacrifice an important line of defense in the name of performance. By using MS Code Contract that could let have assertions fail in production, we could increase the number and the accuracy of production crash logs. This is a choice you must make depending of your application. Let’s precise that NDepend.API actually supports MS Code Contract for its great ability to provide active documentation to users.
  • How much coverage is enough? My answer is 100%.
    • Typically, developers don’t want to lose time writing tests to cover say, properties getter and setter. My point is twofold: typically you can write higher level tests that will cover these + if these getters and setters contain assertions, this is even better.
    • Typically, developers claim that 10% of a class is difficult to test, it takes as much effort to test this 10% as it does to test the remaining 90%. Once again, they don’t want to lose time! My point is that this 10% of code is by definition not easily testable, as a consequence this code is both complex and not-well designed, and as a consequence it is certainly highly-bug-prone. So basically, the highest bug-prone portion of code ends up being not covered by automatic tests!! This is non-sense but this is the reality in most dev shops.
    • Typically, developers say that not everything is coverable by tests and I agree. Code calling blocking methods like MessageBox.Show() is just not coverable, this is why such calls must be mocked. Some other UI code can be especially tricky to test. The approach we use for this is that we designed our UIs in a way that the underlying code can be triggered by unit tests (some would say automated by tests) and then, we mostly rely on assertions in UI code itself to catch any potential regression. Of course when possible, assertions in tests are welcome and of course such UI code is highly decoupled from non-UI logic that has its own set of tests. Doing so has been proving work for our dev shop.
    • I’ll add that when a class or a group of classes are 100% covered by tests, experience shows that the innocent fact that suddenly a coverage hole appears, often means that there is a new problem, either in the code, in test code, or  in both. More often than not, we discovered regression bugs this way that were not caught by assertions. This is why we use tooling (aka NDepend) to check that all code that used to be 100% covered must remain 100% covered.
    • Last but not least, when a bug is fixed, if the fixed code portion is already covered by tests, it is easy to write assertions specific to the fix to avoid any regression in the future. And when most classes are 100% covered, more often than not it is a matter of minutes, or even seconds, to write such assertions.
  • If your application is successful enough, the code base will grow over the years. Finally, the biggest benefit you can expect from writing coverage-oriented automatic tests, is that the number of regression issues will remain under control because it won’t be proportional to the growing size of the code base. Keep in mind that only code covered by tests whose result is asserted somehow is protected by this line of defense.

Let’s illustrate this section with the NDepend 82.6% code coverage visualized with the NDepend metric view. We abide by our rules.

Coverage

Static Analysis and Code Review

I see static analysis as unit tests, but instead of exercising the code dynamically, static analysis exercises properties that can be inferred from the code. In the previous section for example, I wrote that if a class was 100% covered by tests, it must remain 100% covered by tests.  And I even underlined that if a hole suddenly pops up in this perfect coverage, more often than not, understanding the root cause of the hole will lead to a bug fix. This illustrates how static analysis is actually a line of defense against bugs.

In the previous analogy between static analysis and unit-tests, a test is actually a code rule. NDepend makes it easy to write custom code rules, it is just a matter of writing a C# LINQ query based on a fluent API, for example:

Code rules involving code coverage and diff after setting a baseline are especially suited to hunt regression bugs. But static analysis can handle many other properties of the code and it is not only related to bugs but also to code maintainability.

  • Code metrics: for example, methods with too many loops, if, else, switch, case… end up being non-understandable, hence non-maintainable. Counting these through the code metric Cyclomatic Complexity is a great way to assess when a method becomes too complex.
  • Dependencies: if the classes of your program are entangled, effects of any changes in the code becomes unpredictable. Static analysis can help to assess when classes and components are entangled.
  • Immutability: types that are used concurrently by several threads should be immutable, else you’ll have to protect state read/write access with complex lock strategies that will end up being un-maintainable. Static analysis can make sure that some classes remain immutable.
  • Dead code: dead code is code that can be removed safely, because it is not invoked anymore at runtime. Not only can it be removed, but it must be removed, because this extra code add unnecessary complexity to the program. Static analysis can find most of dead code in your program (yet not all).
  • API breaking change: if you present an API to your client, it is very easy to remove a public member without noticing and thus, breaking your clients code. Static analysis can compare two states of a program and can warn about this pitfall.
  • API usage: some APIs are intended to be used carefully. For example, a class that hold disposable fields must be itself disposable in general, except when the disposable field lifetime is not aligned with the class instances lifetime, which then sounds like a design problem.

The list of code properties that can be exercised by static analysis is endless. And the quoted ones refer to NDepend’s capabilities, some other tools like Resharper or CodeRush offer some other sorts of static analysis to warn about micro potential issues, like if a foreach variable is accessed from a closure for example, this can lead to major problems.

Static analysis is not only about directly finding bugs, but also about finding bug-prone situations that can decrease code understanding and maintainability.

Concerning code review, I don’t have much to say. This is static analysis except that the logic is statically checked by human instead of being checked by automatic rules. Thus it is highly imperfect and time consuming, yet we still practice it because experience shows that it helps finding issues that can hardly be found otherwise. The key to code review is to do it on bug-prone code, which include refactored code and new code that haven’t reached production yet.

Manual Tests and Beta Testing

No matter how good a team is at the previously explained lines of defense, if it fails at manual tests and beta testing, the end product will be buggy and ultimately unusable.

Because not all bugs lead to obvious crashes, tests done by humans are essential. For example, only a potential user can notice an incoherent numerical result in a UI.

Manual testing is like code review, highly imperfect and time consuming, and a team cannot capitalize on it. Yet, experience shows that it helps finding issue that can hardly be found otherwise.

We mentioned previously code contracts, they work hand in hand with manual tests. When, during a manual test session, I have the chance to break an assertion, this actually makes me happy 🙂 because I know that this is a great opportunity to fix a bug before it reaches the next release line.

Manual testing actually includes user feedback. Users are paying for a product and one main goal is to offer them a bug free product. Nevertheless, de-facto users are also testers and listening carefully to a user’s bug report and relentlessly struggling to fix them is an essential line of defense. Of course this does not only apply to bugs, but also to feature improvements, new feature suggestions, documentation gap, and much more, but these are another topics.

NDepend version 6 insights

Welcome to the brand new NDepend blog! This blog will talk about NDepend of course, announcements, directions, features advanced usage, but this blog will also talk about related topics including, coding tricks, .NET static analysis, methodologies, theories behind features and .NET development tooling industry in general.

 

Choosing the right set of new features

Today I’d like to talk about the just released NDepend version 6. So far our major releases schedule followed the Microsoft Visual Studio release schedule, and Visual Studio 2015 is actually in Release Candidate period. NDepend v6 is the result of 18 months of hard work. After 11 years of development, NDepend reached a certain level of maturity. Yet there are still many new potential features and improvements possible. We found out that the most sensitive part for this version 6 was actually to chose carefully the set of new features and improvements we’d like to offer to users.  Hopefully, once version 5 was published in October 2013, we got inspiration from the Visual Studio User Voices and opened our NDepend User Voices.  As a matter of fact, version 6 new stuff reflects well the most demanded features that have just been marked as completed.

Build-Process Workflow Integration

Clearly users were asking loud for more NDepend integration into their build-process automated workflow.  So far, users were running NDepend.Console.exe, gathering the report and the information: is there any critical rule violated that should make our build fail? (this info can be obtained NDepend.Console.exe exit code). Users were asking about more information to be shown right inside their build-process tooling. There are many build-process technologies and the most demanded one were TFS, TeamCity and SonarQube. Hence we’ve collaborated with world-class experts of these technologies to deliver build-in integration. We hope that proposed integration will address most scenarios, and we are committed to support further versions and enhance the feature set.

Enhanced Visual Studio Integration

It was not surprising that users wanted more build-process integration since during the last year the bulk of our efforts were devoted to tooling for developers inside Visual Studio. NDepend was integrated in Visual Studio through the VS Addin technology because of a choice we’ve made in 2009. At that time VSIX (VS package technology) was immature and we had to support VS2008 and VS2005, hence VS Addin was the right choice. Time passing by made this initial choice less and less relevant, the VS Addin technology didn’t get any improvement from Microsoft and worth, implied a few glitches like random bug, but worth, the impossibility for VS to remember NDepend windows layout across execution. This was a pain for NDepend users and they ended up asking it loud on the User Voice to remember VS Windows Layout!

With version 6 NDepend relies on the VSIX technology to integrate in VS 2015, 2013, 2012 and 2010. (Should we mention that MS deprecated the VS Addin technology in VS 2015?). Anyway now thanks to VSIX we are able to offer a smoother and more reliable VS integration with windows layout remembering and with new features like the possibility to attach the NDepend project to a .suo file (and not only .sln file) and full support for NDepend commands keyboard-shortcut.

There were also 3 VS build featurettes proposed by various free VS extensions, and we figured out that ourselves were pretty much addicted to it. We decided to include them in the NDepend VS package, this way we will be sure to always have them at hand, and with a bit of luck, NDepend users will enjoy them too 🙂

Rule Improvements

.NET code ruling is the central feature of NDepend. We had many feedback from users about how to improve existing default code rules, mostly to reduce the number of false-positive reported. hence we’ve refactored around a third of default code rules based on their recommendations.

While doing so we figured out we have been a bit too geeky. This is great that NDepend code rules are actually editable C# LINQ queries that can contain some comments. But for many users this didn’t convey enough information about why a rule actually warns, why abiding by a specific rule will enhance their code, and how to fix a violation of the rule. Hence we’ve added the possibility to add 2 formatted texts in a NDepend rule source code comment through <Description> and <HowToFix> tags. This has been a great opportunity for us to write out about each default rule and provide users more explanations. Both in UI and in report, the user can chose to see the Description/HowToFix, or the rule C# LINQ source code.

Another need that users were asking loud in the User Voice, was the possibility to share some rule set among several NDepend projects. Typically a Company has several code bases analyzed each by a NDepend project, and what users want is a Company-level quality profile that can be applied to each code base. And taking account that NDepend makes easier than any other tool to define custom rules on .NET code, users wanted even more to define custom quality profile shareable. We addressed this need in NDepend v6 with the possibility to define rule files that can be shared among NDepend projects.

Code Metrics Visualization

NDepend already had code metrics visualization with treemaping and we decided to add coloring. The idea is that we can then visualize two code metrics, one represented by rectangle area, the other one represented by rectangle color. For example, the screenshot below shows NDepend version 6 code coverage by tests in a unique, yet convenient and accurate way.

  • Rectangles represent methods,
  • Rectangle area is proportional to # Lines of Code,
  • Rectangle color represents percentage code coverage by tests

ColoredTreemap

I hope this picture helps understand why we wanted to offer users this feature. At a glance, colored treemaping can help grasp information that would be hard to get through another mean. Despite the fact NDepend v6 code base is 82%+ covered by tests, we can identify aggregation of red rectangles that shed light on regions of code not enough covered by tests. I believe no other tooling can offer such level of details, and while developing this feature, I was really looking forward what it has to say about our own coverage & testing work!

And as usual we like to make the various NDepend features interact. Hence not only you can visualize any two of the many NDepend default code metrics set, you can visualize custom code metrics defined through C# LINQ queries, and you can also highlight the result set of a code rule or a code query. Hence if we query for complex methods, we can filter complex methods match on the colored treemap:

TreemapAndSelection

Let’s mention that since we think that code coverage by tests is one of the most important code quality data, we also added dedicated UI candies for it in code query result:

Improvements that had to be made

A product belongs to a constantly evolving environment. This means that with years, some features that were working fine need to be revisited. Several of such needed improvements / fixes have been achieved with NDepend version 6.

Support code generated by C# and VB.NET compiler: Version after version, C# and VB.NET languages increased the amount of syntax sugar offered to the user. It started with C#2 anonymous methods and iterators, followed with C#3 lambda and anonymous types and then C#5 async support. These language constructs constraint the compiler to generate extra methods and types. And since these constructs are pretty much used, numbers of these generated elements accumulate in the IL code. And NDepend mostly relies on IL code to get code information. Hence the NDepend code model included all these generated methods and types and this started annoying the user, mostly interested by its own code. NDepend now fixes this. Simply getting rid of these generated elements was not an option because they actually contains some user code. Hence we developed some heuristics to merge these generated elements in users classes and methods that contain the code that provokes the generation by the compiler. This way there is no compromise, all the code model data is preserved, including: Code metrics (# Lines of Code, Percentage Coverage, Cyclomatic Complexity…), dependencies and diff.

Better async support: The async language construct comes with a specificity that provoked some NDepend issues. The user code is actually embedded by the compiler in a generated method that overrides the virtual method IAsyncStateMachine.MoveNext(). Hence the IL code invoking this MoveNext() generated method is bound with the interface abstract method and not with the implementation. The calls toward the async method are then lost in IL … ans so in the NDepend code model. Thanks to the heuristics wrote for the previous point, NDepend analysis is now able to track properly callers of async methods.

Support for high-DPI: Nowadays hi-resolution monitors (like 4K monitors) become affordable and as a consequence, more and more developers work with high-DPI Windows settings. High DPI means that the density of pixels is higher and as a consequence more pixels are used to represent a graphical element (text, bitmap…). The NDepend UI is mostly based on Windows Form and GDI+ and these technologies don’t scale well by default when Windows is set with high DPI. Since some users started to ping us about this problem we decided to re-lift the whole NDepend UI to make it works with DPI settings from 100% to 250%. This has been the opportunity to enhance some graphical elements and to learn some Windows Form and GDI+ high-DPI practices. I’ll try to summarize these findings in a future blog post, because browsing the web, I can see that it is a pain for many developers.

Visual Studio Theming: From our production statistics most of users consume NDepend for Developer features from within Visual Studio. So far the NDepend UI themes didn’t fit perfectly the various Visual Studio default themes, this includes Light, Dark, Blue and VS2010 default themes. Hopefully NDepend menus/bars/docking… elements are based on the DevExpress WinForm library and their latest version included these themes (and some more). Hence upgrading to the highest version of this framework did the trick.

Analysis improvements: Some improvements had to be made concerning analysis. One of them concerns resolution of assemblies to analyze. A NDepend project references a set of directories and a set of assemblies names (without file extension). At analysis-time, assemblies to analyze locations are resolved from these two sets. If multiple copies of an assembly were found, either all copies had same bits and NDepend took one of them, either some copies were different and NDepend emitted an error resolving the assembly, because we use to consider this situation as a serious deployment problem. However with time, new technologies like Windows Phone or  Portable Class Libraries appeared, and they actually provoking such situation during valid scenario. Hence, when detecting such situation, NDepend v6 now uses an heuristic to resolve the proper assembly occurrence, based on the assembly version, assemblies referenced and file size.

 

 

We are very glad to have been able to pack all these in NDepend version 6.

We hope you’ll enjoy using this new version just as much as we enjoyed making it 🙂