The famous Fred Brooks paper “No Silver Bullet – Essence and Accident in Software Engineering“ published in 1987 stated that: “there is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity.” The paper also stated that “we cannot expect ever to see two-fold gains every two years in software development, as there is in hardware development”.
Since 1987 developers benefited from many progresses that substantially increased their productivity: modern IDE ; higher level languages ; faster compilers ; sophisticated runtimes ; intellisense ; testing tools ; refactoring tools ; code analyzers ; code navigation tools ; code visualization tools ; source management and DevOps tools…
Nowadays we are facing some new kind of progress with AI assisted coding. AI assisted coding is already a reality with technologies like Visual Studio Intellicode introduced in my last post (At the end of this article, Mark Wilson-Thomas Program Manager, Visual Studio IntellICode, commented about Intellicode technologies and roadmap). AI assisted coding might represent the next step toward improving developers productivity. It could be well the software development silver bullet that Brooks couldn’t predict in 1987.
In this post I try to predict what could be the AI-based developer productivity tools of the (near?) future.
Augmented Developer: Intellisense on Steroid
What makes AI different than development productivity progresses listed above, is that an AI-based system can handle situations not considered by people who designed it. As a consequence it could offer more sophisticated intellisense advices. Imagine stackoverflow.com inviting itself in your code editor: Are you trying to make a function that does X? If yes here is an OSS implementation well ranked by the community or already integrated into a public library or maybe, already implemented somewhere else in the current code base.
For example when developing the Slice()
method below, how long would it take for an observer (human or AI) to realize that you are actually rewriting the .NET Base Class Library method System.String.Split()
? This could be guessed both from the Slice()
implementation under development and also from the identifier Slice
that is semantically close to Split
.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
using System.Collections.Generic; using System.Diagnostics; using System.Linq; static class Program { static void Main() { const string str = "aa||b|cc"; var result1 = str.Slice("|"); var result2 = str.Split('|'); Debug.Assert(result1.SequenceEqual(result2)); } public static string[] Slice(this string str, string separator) { var list = new List<string>(); int index = 0; int sepIndex = str.IndexOf(separator); while (sepIndex >= index) { list.Add(str.Substring(index, sepIndex - index)); index = sepIndex + separator.Length; sepIndex = str.IndexOf(separator, index); } if (index < str.Length) { list.Add(str.Substring(index)); } string[] array = list.ToArray(); return array; } } |
For several years now, everyone got used to services like Shazam, that can retrieve the name and author of a song from a few seconds listening with noise around. I guess that in the future IA will be able to shazam the algorithm you currently write to avoid you to reinvent the wheel.
Anticipating code under development will augment the developer productivity. But the development process could be augmented too. Once trained, an IA assisted IDE could trigger all sorts of tasks like running some specific tests, starting a background compilation or highlight some important facts. Modern IDEs are already good at those, certainly they will be better.
Smart Bug Finders
Bug finder tools will certainly benefit from AI. Verifying the correctness of a program has a lot of similarities with estimating how good or poor is a Chess or a Go move. And AI excels at those.
In both cases (game and bug finding) a tree is involved. The tree of game possible moves vs. the possible runtime-calls tree in a program (the code flow). The cause of hard-to-spot-bugs is often an unexpected state change made in a part of the application that is not directly related with the part where the bug effect occurs. On a side note, this is why functional programmers tell that they produce less bugs: there is no mutating state in functional programming.
There are already many tools that can investigate both static and runtime code flow. The same way they are traditional (in the sense non-AI based) algorithms investigating the tree of possible moves in the Go game. But the combinatorial explosion makes it so that those algorithms cannot handle the rapid growth of the complexity.
In 2016 AlphaGo unexpectedly beat the 18-time world Go champion Lee Sedol. This was unexpected. While the number of Chess board configuration is around 10^120, the number of Go board configuration is around 10^174. Go has 1 million trillion trillion trillion trillion more configurations than chess! To quote wikipedia on AlphaGo: “AlphaGo’s algorithm uses a combination of machine learning and tree search techniques, combined with extensive training, both from human and computer play. It uses Monte Carlo tree search, guided by a “value network” and a “policy network,” both implemented using deep neural network technology.”
How long will it take until these powerful ideas get successfully applied to the code flow of complex programs?
I’d like to put in perspective these ideas with the halting problem. Turing proved in 1936 that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist. But unlike a general algorithm, an AI is trained against real-world code, and can thus recognize suspicious patterns and investigate them. It won’t be holistic, it won’t find all possible bugs, but it will be of great help.
Smart Vulnerabilities Finders
A vulnerability is bug that manifests as an opportunity for malicious use of the product. Often vulnerabilities are not that clearly evident: they require a lot of expertise to be spotted and fixed properly. There are many sorts of vulnerabilities – like the infamous SQL injection – that are usually well detected by existing analysis tools. If in the future some AI will be able to find some tricky bugs, they will surely be able to find some tricky vulnerabilities as well. By the way, if you search on the web AI and firewall, you’ll see that a lot of research are done to increase firewall robustness thanks to AI-based systems.
Ease Large Refactoring and Migrations
Nowadays refactoring tools are powerful but they are based on rigid algorithms. They don’t adapt to new situations. They just match pre-defined patterns in your code that can be transformed into other patterns without changing the logic.
On the other hand the source code is the design: the source code thoroughly defines the logic executed at runtime without ambiguity. Thus the source code could be used as an input to an AI assisted refactoring engine that would output better code with the exact same behavior. What better code mean?
I wrote this post a while ago. Business Complexity vs. Implementation Complexity Business complexity is related to what a program does at runtime. The implementation complexity on the other hand is related to how the actual implementation solves the business needs. 2 programs can address the same need (they have the same business complexity) but their implementation can differ a lot. Does the implementation rely on things like global state? long method with high cyclomatic-complexity rank? monolithic and entangled code? overly coupled and poorly cohesive classes?… All those code smells are the roots of the implementation complexity. Costs associated with these code smells can be objectively measured through technical debt estimations. An AI could be then trained to reduce these code-smells scores while preserving the code behavior.
Another usage of AI when it comes to application refactoring would be to ease large migrations. Applications periodically need major changes to keep up with latest technologies and trends: you might need to migrate from WPF to MAUI (Multi-Platform App UI) to run your app on multi platform. Or you might want to migrate from a VB6 legacy to .NET 5 because there is no more VB6 developers to hire. Here also a well trained AI could do a better job than existing migration assistant tools.
One of the major problem with AI nowadays is that it is a black box: it produces results but no-one really knows how it did. In the case of automatic refactoring or migration the development team would end up with plenty of brand new code that nobody actually wrote. Such situation could lead to reverse-engineering and maintenance headaches! Thus the code produced would have to be very well designed by the AI before this becomes mainstream. A high level of maturity will be needed.
Automatically Generate Tests from the Code Itself
As many programmers these days I cannot do my work without writing tests to exercise my code. It takes significant time but the Return On Investment (ROI) in the long run plainly justifies it.
Writing code and writing tests are two different – but related – ways to express the same logic. The code implements the general case while tests implement sample cases.
1 2 3 4 5 6 7 8 |
// Tests Assert.IsTrue(ApplyVAT(100m, 20m) == 120m); Assert.IsTrue(ApplyVAT(1000m, 10m) == 1100m); // Code public static decimal ApplyVAT(decimal price, decimal vatPercentageRate) { return price * ((100m + vatPercentageRate) / 100m); } |
I remember the Microsoft Pex project that aimed at generating tests from the code. It seems the project is not maintained anymore. One of its creator, Peli de Halleux found a bug in our library NDepend.Path with Pex once. This was pretty cool!
I remember also taking a breakfast with Bertrand Meyer during the Build Stuff 2013 conference in Vilnius Lituania. He mentioned the EiffelStudio automatic test generation. I didn’t try this tool but I guess in 2013 it wasn’t AI based (is it now?).
These anecdotes attest that attempting to generate sample cases (tests) from the general implementation (code itself) is not a new thing. However certainly AI will help a lot in this area.
Writing tests is beneficial because it leads the developer to question what she is doing. Edge cases are often spotted this way. Relying on an AI based system to generate tests will certainly help detecting automatically not-thoroughly-handled-edge-cases. But the risk is that developers stop questioning what they are doing. This would necessarily lower their coding skills.
From plain English Sentence to Compiled Machine Code
When I was a child I learnt to program in the 80’s with my father that was a professional programmer. Once I asked him : “why can’t we just write plain-French programs to tell the machine what to do?” I understand today that it is because the source code is the design (again). You application is the sum of all its instructions. Change any instruction and you get a different application. The compiler is a determinist machine: it can guess a few things (like with typing inference) but it cannot guess any logic that the programmer don’t explicitly provide.
Nowadays (2021) AI still doesn’t have the common sense to understand human language. Once AI will reach some sort of common sense, it could start translating your plain-English sentences into the C# program you most likely want. Professional developers would still be needed to validate and eventually refine generated programs. The same way that today professional human translators are still needed to validate and eventually refine traductions generated by an AI.
Actually there is an increasingly growing market for Low Code and No Code platforms. With such platform, instead of writing code people rely on a graphical user interfaces to define the logic of their applications. This is like using a wizard to define the logic. A common benefit is that a wider range of people can contribute to the application’s development.
When I was consulting in a large bank 15 years ago I remember traders writing their own quick & dirty VBA snippets because they didn’t have the time to wait for the IT department. It is just an example but everywhere there is a demand for easing the automation process. Certainly AI will play a huge role in this area.
Summary
Visual Studio Intellicode is a good start to improve developer productivity with an AI based system. As of now, it doesn’t touch any ambitious scenarios explained but it can certainly help developers saving some time every day which is a win.
By searching on the web you will find other AI-based coding assistance solutions. Being labelled as an AI solution is an excellent marketing pitch. But from the independent comments I have read it looks like no significant breakthrough is available yet.
I understand why it is difficult to build an AI that can drive a car in any condition (level 5). Many experts are not even sure it’ll be possible. On the other hand, I don’t see significant barriers to address most of scenarios described:
- Plenty of open-source code is available to train AIs.
- A road is a physical, continuous and unexpected thing to analyze. Code is a purely logical, discrete and predictable material like a Chess game.
- The AI assisted coding R&D will certainly benefit from all the B$ it will need. Indeed after 3 decades of progress in developer productivity tooling, companies that don’t use tools to automate test, source management or code refactoring are more the exception than the rule. The demand for more coding productivity is here. The demand to ease the development process is also here as the growing success of low-code systems attest.
Hopefully the industry will still need us – pro software developers – for quite a while. But we should carefully follow the progress made in this are. When such tools will be effective our productivity will be increased significantly and our daily activity will change forever.
I think you may be missing one of the biggest: optimization.
Thanks for the article link Patrick! You’ve outlined some interesting problems that could benefit from AI-assistance for developers going forward!
I definitely don’t see AI putting pro developers out of a job; if anything the demand for developers appears to be increasing – see Amanda Silver’s recent post on this https://educationblog.microsoft.com/en-us/2021/01/software-development-in-2021-and-beyond/ . Our hope on the IntelliCode team is that like all great productivity enhancers, “AI-assistance” will continue to improve at making the tedious and repetitious/error prone parts of development more straightforward, leaving you free to spend time on creatively solving the new business problems that come up. We also see AI-assistance adding “community wisdom”/insight in a distilled form, so that when a developer needs to learn or move into a new area (as happens so often these days), the system is there ready to help put guide-rails in place to point toward success or even help with resolving issues/bugs, and help organizations and teams to succeed even when they don’t necessarily have an expert on tap to guide their peers.
To that end, we’re actively working on continuing to make code authoring (“coding with confidence”), and code issue finding (“find issues faster”) ever-easier using the tools at our disposal and new ones we’re developing. These tools include deep learning models of code that let us provide ever-improving code completions, as my data scientist colleague Shengyu describes in his blog post https://devblogs.microsoft.com/visualstudio/the-making-of-intellicodes-first-deep-learning-model-a-research-journey/ , and pattern-matching/learning-by-example tools like PROSE to capture and scale out expressions of developer intent from their live coding activities as we already do in IntelliCode Suggestions. Those alone will take us some way forward, but we have other models in research that we hope will benefit developers in new ways too.
Mark Wilson-Thomas
Program Manager, Visual Studio IntellICode
https://aka.ms/vsic
a very accurate, valuable and useful article, thank you. Good days and works.
Thanks Allan, indeed, I am quite focused on static analysis but an AI could be trained against dynamic analysis to recognize what makes applications faster and less memory hungry.
Thanks so much Mark for sharing your insight and your plans, this sounds quite promising!
Thanks, Patrick Smacchia for sharing such great article…Keep posting