In the previous post Case Study: 2 Simple Principles to achieve High Code Maintainability I explained that the principles layered code + high coverage ratio by test are 2 simple principles that can be objectively applied, validated and measured. When these 2 principles are applied they lead to High Code Maintainability: As a consequence the management saves money in the long term and developers are happy to work in a cleaner code base with principles easy to follow.
Large and complex UI 90%+ covered by tests
In this previous post I used the example of the new NDepend v2020.1 graph. This new tool is a large and complex UI with dozens of actions proposed to the user and drastic performance requirement (it scales live on 100.000+ elements). This graph implementation is 90% covered by tests. It is not because there is a lot of UI code that it should not been well tested. We didn’t spend a good part of our resources in writing tests just for the sake of it. We did it because we know by experience that it’ll pay off: probably a few bugs will be reported as for all 1.0 implementation although beta test phases already caught some. But we are confident that it won’t take a lot of resources to fix them. We can look forward the future confidently (like supporting properly .NET 5 that will be released in November 2020). And indeed 10 days after its 1.0 release no bug has been reported (nor logged) on this new graph although many users downloaded it: so far it looks rock-solid and we can focus on what’s next.
The picture below shows all namespace, classes and methods of the graph implementation. Smaller rectangles are methods and the color of each rectangle indicates how well a method is covered by tests. Clearly we tolerate some gaps in UI code, while non UI code like Undo/Redo actions implementations are 100% covered. Experience told us how to balance our resources and that everything does not have to be perfect to achieve high maintainability.
How did we achieve High Coverage Ratio on UI Code?
It is easy: we have a simple MVC (Model View Controller) design. Some controller classes contain the logic for all actions the user can do and those classes pilot the UI. Concretely in our scenario actions are: load/save, change group-by, change layout direction, zoom, generate a call graph for this method, change filters…
Then we wrote a test suite that first starts the UI and then invokes all actions. Each complex peculiarity of each action gets fully tested, hence complex actions get invoked several times by tests but differently each time, to make sure all scenarios get tested.
The video below shows the UI under testing: more than 40 actions get tested in less than a minute. It would take more than an hour to do all this work manually and any change in code could potentially ruin the validity of manual tests.
In such a complex UI there are many classes that are not directly related to UI. For example the grape of classes that describe the underlying model are tested separately.
As usual, a side benefit of writing tests is better design : the code gets structured in a way that makes it easy to invoke it through tests. Concretely some abstractions are introduced (that wouldn’t make sense without tests), some classes and some methods get splitted, some logic gets refined and as a result developers are happy to live in a code base where the logic is smoothly implemented.
High Coverage Ratio is not Enough: Assertions to the rescue
Typically at this point comes the remark: but code coverage is not enough, results must be asserted. And indeed, if nothing gets asserted nothing gets tested even if the code is entirely covered by tests. We want tests to fail if something can go wrong.
Of course our tests contain many assertions for example load / save actions are invoked and asserted this way:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
// Save and undo until DefaultGraphAction InvokeOnUIThread(() => Assert.IsTrue(m_GraphController.TrySaveGraphInFile(graphFilePath, "My Desc", out failureReason))); InvokeOnUIThread(() => m_GraphController.ExecuteUndoRedoGraphAction(new OnUndoTillItemAction<IGraphAction>((uint)(2 - 1)))); // -1 coz 0-based WaitUntil(stack => stack.GetUndoItems()[0] is DefaultGraphAction); Assert.IsTrue(m_GraphController.GraphContext.NodesSelected.Count == 0); // Load and make sure we have 2 nodes selected InvokeOnUIThread(() => Assert.IsTrue(m_GraphController.TryLoadGraphFromFile(graphFilePath, out string graphDescFromFile, out failureReason))); WaitUntil(() => m_GraphController.GraphContext.NodesSelected.Count == 2); Assert.IsTrue(m_GraphController.GraphContext.UndoRedoStack.UndoItemsCount == 3); Assert.IsTrue(m_GraphController.GraphContext.MessageForUser == @"Graph loaded from file: File.ndgraph My Desc"); |
But these assertions are not enough. Per definition the UI code contains tons of visual peculiarities represented by states that can be potentially corrupted. As a consequence our UI code is stuffed with thousands of assertions: everything that can be asserted gets asserted.
- A Rectangle with width/height in certain range
- The state of a node or an edge when another element gets selected (is it a caller, a callee…?).
- The current application state when a new graph is demanded by the controller.
- The graph UI contains many asynchronous computation to avoid UI freezing. This leads to many assertions to check that mutable states are not corrupted by concurrent accesses.
- …
All those states asserted would be hardly reachable from test code. However they get naturally accessed by the UI code itself so it is the right place to assert that they are not corrupted.
Btw, We still use the good old System.Diagnostics.Debug.Assert(…) for that, it has several advantages:
- It is simple.
- It is understood by tools like Roslyn/Resharper/CodeRush analyzers.
- An assertion that fails cannot be missed both when running automatic tests and when running manual tests on the Debug mode version.
- Debug assertions are removed by the compiler in Release mode: assertions are not executed in production and users get better performance. The idea is to not consider users as testers: code released in production is supposed to be rock-solid. Assertions are like scaffolding that gets removed when a building gets delivered. If there is still a bug we’ll discover it from users feedback, from production logs or from our own manual tests.
Debug.Assert(…) is enough for us and it is understandable that some other teams wants more sophisticated assertions framework. The key is to take the habit to assert everything that can be asserted when writing code (UI code or not). Each assertion is a guard that helps making the code rock-solid. Also each assertion improves the code readability. At code-review time we’ve all been wondering: can this integer be zero? can this string be empty? can this reference be null?. Hopefully C#8 non-nullable discards the last question but so many questions remain open without assertions.
Design by Contracts
This idea of stuffing code with assertions is actually an important software correctness methodology named DbC, Design by Contract, that is really worth knowing. Contracts mean much more than the usual approach with exception:
|
1 2 3 4 |
void Method(int argument) { if(argument == 0) { throw new ArgumentOutOfRangeException(); // ... } |
- Explicitly throwing an exception says: zero is tolerated, it is not a bug, but you won’t get the result you’d like, be prepared to catch some exceptions.
- Writing a contract says: don’t even thing of passing a zero value. The real type of argument is not Int32 it is [Int32 minus 0]. Ideally such violation could be caught by compilers and analyzers (and is indeed sometime caught as we saw in the screenshot above).
Conclusion
Any complex UI can be automatically tested as long as:
- It is well designed with some controllers that pilot the UI and that can be invoked from tests.
- UI code gets stuffed with assertions to make sure that no state becomes corrupted at runtime.
In short assertions embedded in code tested matter as much as assertions embedded in tests. If an assertion gets violated there is a problem, no matter the assertion location, and it must not be missed nor ignored. This powerful idea doesn’t necessarily applies only to UI code and is known as DbC, Design by Contract.
Actually in this post I added a third principle to achieve high code maintainablity and high code correctness : layered code + high coverage ratio by test + contracts