In a previous post The proper usages of Exceptions in C# I explained that it is important to get as much information as possible from production crash logs. One such key information is IL Offset for each method call in a stacktrace.
The Problem of Production Crash Naked Stack Trace
One recurrent problem with exception stack trace in .NET is that one needs to either release PDB files along with assemblies installed in production or setup a symbol server. Without PDB files, you get a naked stack trace: you lose the exact source file line from where the exception popups. One problem is that usually the PDBs files weight around twice the weight of the target assemblies. Hence, releasing PDBs is usually not an option because this consumes more resources (download size / install footprint / process memory at run-time). There is also the point about Intellectual Property protection:
- not only PDB files contain extra info about your source that shouldn’t be public.
- but also if your assemblies are obfuscated their original PDB become useless.
The exact source file line from where the exception popups is not just a convenient info, it is an essential info. The typical awkward situation is when a method throws a NullReferenceException, but contains several references that can potentially be null: in such case without the exact location that caused the exception you are blind. By knowing the exact source file line from where the exception popups, it would become easy to identify the faulty null reference. Notice also that the longer your methods (in terms of lines of code), the most likely you will fall into the awkward situation described.
Rich Production Stack Trace with IL Offset
Several years ago we developed some code to gather IL offsets in stacktrace. Now our production stacktraces look like:
1 2 3 4 5 6 7 8 |
Exception.Type {System.NullReferenceException} Exception.Message {Object reference not set to an instance of an object.} Exception.StackTrace { bed.a(bqq A_0, byf A_1) L_00e3 bed.a(List`1 A_0) L_0050 bed..ctor(b16 A_0, IList`1 A_1, abc A_2) L_0032 bo0.a(Object A_0, KeyEventArgs A_1) L_0069 ... |
Thanks to such stack trace containing IL offsets, it is straightforward to infer a source code line from an IL instruction. You just have to keep in mind that if the Nth IL instruction throws the exception, you’ll get the offset of the N-1th IL instruction logged. The L_00e3 offset in stack trace translates to L_00e4 in IL code.
On the screenshot below you can see that we obfuscate our code. Hence there is an extra step to resolve obfuscated identifiers to actual source file identifiers.
Below you will find our source code and corresponding tests. It can be compiled with .NET Standard. This code implements another useful facility: we remove all localization info from the stacktrace. This way we are able to build a hash code from the string of a stacktrace. Such hash code is useful to group similar crash logs, independently from the underlying OS localization settings. We then infer the number of similar crash logs. By logging other environment info we can distinguish between situations like:
- Many users faces the same crash
- A single user faces a crash multiple times
Thus we can prioritize problems to fix from production data. We are running this code for many years. It helped us countless times to fix non-trivial and rare bugs. With such relentless bug-fix approach the number of crash reported over time significantly decrease and is getting close to zero. Having a stable product has two essential benefits:
- This avoids unnecessary friction with users.
- We don’t lose time on bugs already fixed and have more time to implement new features.
Source Code with Tests
StackTraceHelper.cs
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 |
using System; using System.Diagnostics; using System.IO; using System.Reflection; using System.Text; namespace NDepend.Product.ErrorHandling { public static class StackTraceHelper { public static string FormatStackTrace(Exception ex) { Debug.Assert(ex != null); #if DEBUG return FormatUnlocalizedStackTraceFromString(ex.StackTrace); // Get Line Number in DEBUG config #else return FormatUnlocalizedStackTraceWithILOffset(ex); // Get IL OffSet in RELEASE config #endif } #region FormatUnlocalizedStackTraceWithILOffset() //------------------------------------------------------------------------ // // FormatUnlocalizedStackTraceWithILOffset() // //------------------------------------------------------------------------ public static string FormatUnlocalizedStackTraceWithILOffset(Exception ex) { Debug.Assert(ex != null); try { //Get offset: var stackTrace = new StackTrace(ex, true); StackFrame[] stackFrames = stackTrace.GetFrames(); if (stackFrames == null || stackFrames.Length == 0) { return FormatUnlocalizedStackTraceFromString(ex.StackTrace); } var sb = new StringBuilder(); var stackFramesLength = new int[stackFrames.Length]; for (var i = 0; i < stackFramesLength.Length; i++) { var stackFrame = stackFrames[i]; var method = stackFrame.GetMethod(); var parameters = GetMethodParameters(method); var ilOffset = GetILOffset(stackFrame.GetILOffset()); sb.Append($" {method.ReflectedType.FullName}.{method.Name}({parameters}) {ilOffset} \r\n"); } return sb.ToString(); } catch (Exception) { return FormatUnlocalizedStackTraceFromString(ex.StackTrace); } } private static string GetILOffset(int ilOffset) { Debug.Assert(ilOffset >= 0); // Format to hexadecimal to have a standard display of IL instruction OffSet var ilOffsetHexString = ilOffset.ToString("X").ToLower(); // Get a Reflector-like ILOffset like "L_018e" var sb = new StringBuilder("L_"); if (ilOffsetHexString.Length < 4) { sb.Append(new string('0', 4 - ilOffsetHexString.Length)); } sb.Append(ilOffsetHexString); return sb.ToString(); } private static string GetMethodParameters(MethodBase method) { Debug.Assert(method != null); if (!TryGetParameters(method, out ParameterInfo[] parameters, out string failureReason)) { Debug.Assert(!string.IsNullOrEmpty(failureReason)); return failureReason; } Debug.Assert(parameters != null); var length = parameters.Length; var sb = new StringBuilder(); for (var i = 0; i < length; i++) { var parameter = parameters[i]; sb.Append(parameter.ParameterType.Name); sb.Append(" "); sb.Append(parameter.Name); if (i < length - 1) { sb.Append(", "); } } return sb.ToString(); } private static bool TryGetParameters(MethodBase method, out ParameterInfo[] parameters, out string failureReason) { Debug.Assert(method != null); try { parameters = method.GetParameters(); Debug.Assert(parameters != null); failureReason = null; return true; } catch (FileLoadException ex) { failureReason = $@"Exception thrown while calling MethodBase.GetParameters() Exception Type: {{{ex.GetType().ToString()}}} Exception Message: {{{ex.Message}}}"; parameters = null; return false; } } #endregion FormatUnlocalizedStackTraceWithILOffset() #region FormatUnlocalizedStackTraceFromString() //------------------------------------------------------------------------ // // FormatUnlocalizedStackTraceFromString() // // OldVersion, still used in Debug mode to get the Line number // and used in case FormatUnlocalizedStackTraceWithILOffset() has a problem! // // StackTrace lines are prefixed with " at " or " à " or " в " or " 場所 " that we need to remove // public const string EMPTY_STACK_TRACE = "Empty StackTrace"; public static string FormatUnlocalizedStackTraceFromString(string stackTraceIn) { if (string.IsNullOrEmpty(stackTraceIn)) { return EMPTY_STACK_TRACE; } var lines = stackTraceIn.Split(new char[] { '\r' }); Debug.Assert(lines != null); Debug.Assert(lines.Length >= 1); var sb = new StringBuilder(); for (var i = 0; i < lines.Length; i++) { var unlocalizedLine = UnlocalizeLine(lines[i]); if (i > 0) { sb.Append("\r\n"); } sb.Append(unlocalizedLine); } return sb.ToString(); } // 3Dec2015: Idea we could use var strAt = GetRuntimeResourceString("Word_At") ?? "at"; ,as explained in // https://github.com/gimelfarb/ProductionStackTrace/blob/master/ProductionStackTrace/ExceptionReporting.cs // This project has some code to explore PDB (we don't need) it has also some code to get assembly name (we don't really need) // --> so far this code works so don't touch it!! //private static string GetRuntimeResourceString(string id) { // var m = typeof(Environment).GetMethod("GetRuntimeResourceString", BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public, // null, new[] { typeof(string) }, null); // if (m == null) return null; // try { // return (string)m.Invoke(null, new object[] { id }); // } catch (MemberAccessException) { // return null; // } //} private static object UnlocalizeLine(string lineIn) { Debug.Assert(lineIn != null); // Can be empty lineIn = lineIn.Replace("\n", ""); // Eventually discard \n at the beginning // var strAt = GetRuntimeResourceString("Word_At") ?? "at"; int indexFirstNonWhiteSpace = 0; for (indexFirstNonWhiteSpace = 0; indexFirstNonWhiteSpace < lineIn.Length; indexFirstNonWhiteSpace++) { if (lineIn[indexFirstNonWhiteSpace] != ' ') { break; } } // lineIn is like // '\n' 'white' 'white' 'at' 'white' "xxxx // The idea is to get rid of the localizable 'at' and to transform it in // 'white' "xxxx var indexOfSecondSpace = lineIn.IndexOf(' ', indexFirstNonWhiteSpace); if (indexOfSecondSpace == -1) { return lineIn; } var lineOut = lineIn.Substring(indexOfSecondSpace, lineIn.Length - indexOfSecondSpace); return lineOut; } #endregion FormatUnlocalizedStackTraceFromString() } } |
Test_StackTraceHelper.cs
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 |
using Microsoft.VisualStudio.TestTools.UnitTesting; using System; namespace NDepend.Product.ErrorHandling { [TestClass] public class Test_StackTraceHelper { [TestMethod] public void Test_FormatUnlocalizedStackTraceWithILOffset_EMPTY_STACK_TRACE() { var ex = new Exception(); var str = StackTraceHelper.FormatUnlocalizedStackTraceWithILOffset(ex); Assert.IsTrue(str == StackTraceHelper.EMPTY_STACK_TRACE); str = StackTraceHelper.FormatStackTrace(ex); Assert.IsTrue(str == StackTraceHelper.EMPTY_STACK_TRACE); } [TestMethod] public void Test_FormatUnlocalizedStackTraceWithILOffset_RealEx() { NullReferenceException nullReferenceException = null; try { string s = null; var i = s.Length; } catch (NullReferenceException ex) { nullReferenceException = ex; } Assert.IsNotNull(nullReferenceException); var str = StackTraceHelper.FormatUnlocalizedStackTraceWithILOffset(nullReferenceException); // Don't put the IL offset coz it can change once instrument with coverage tool! Assert.IsTrue(str.StartsWith(@" NDepend.Product.ErrorHandling.Test_StackTraceHelper.Test_FormatUnlocalizedStackTraceWithILOffset_RealEx() L_00")); var str1 = StackTraceHelper.FormatStackTrace(nullReferenceException); Assert.IsTrue(str1 != str); Assert.IsTrue(str1.StartsWith(" NDepend.Product.ErrorHandling.Test_StackTraceHelper.Test_FormatUnlocalizedStackTraceWithILOffset_RealEx()")); } [TestMethod] public void Test_FormatUnlocalizedStackTraceWithILOffset_RealExInAnonymouseMethod() { NullReferenceException nullReferenceException = null; try { // Note that we test here with delegate with two parameters! var action = new Action<int, string>(delegate (int i1, string s2) { string s = null; var i = s.Length; }); action.Invoke(1, ""); } catch (NullReferenceException ex) { nullReferenceException = ex; } Assert.IsNotNull(nullReferenceException); var str = StackTraceHelper.FormatUnlocalizedStackTraceWithILOffset(nullReferenceException); // Don't put the IL offset coz it can change once instrument with coverage tool! Assert.IsTrue(str.StartsWith(@" NDepend.Product.ErrorHandling.Test_StackTraceHelper+<>c.<Test_FormatUnlocalizedStackTraceWithILOffset_RealExInAnonymouseMethod>b__2_0(Int32 i1, String s2) L_00")); Assert.IsTrue(str.Contains(@" NDepend.Product.ErrorHandling.Test_StackTraceHelper.Test_FormatUnlocalizedStackTraceWithILOffset_RealExInAnonymouseMethod() L_00")); ; } [TestMethod] public void Test1_FormatUnlocalizedStackTrace() { const string strIn = @" 在 afs.h() 在 afs.c()}"; const string strOut = @" afs.h() afs.c()}"; Assert.IsTrue(StackTraceHelper.FormatUnlocalizedStackTraceFromString(strIn) == strOut); } [TestMethod] public void Test2_FormatUnlocalizedStackTrace() { Assert.IsTrue(StackTraceHelper.FormatUnlocalizedStackTraceFromString( @" в qz.a(Window A_0, Window A_1)}") == StackTraceHelper.FormatUnlocalizedStackTraceFromString( @" in qz.a(Window A_0, Window A_1)}")); } [TestMethod] public void Test3_FormatUnlocalizedStackTrace() { Assert.IsTrue(StackTraceHelper.FormatUnlocalizedStackTraceFromString( @" в qz.a(Window A_0, Window A_1)}") == @" qz.a(Window A_0, Window A_1)}"); } [TestMethod] public void Test4_FormatUnlocalizedStackTrace() { try { throw new Exception("hello"); } catch (Exception e) { var unlocalizedStackTrace = StackTraceHelper.FormatUnlocalizedStackTraceFromString(e.StackTrace); Assert.IsTrue(unlocalizedStackTrace.IndexOf( " NDepend.Product.ErrorHandling.Test_StackTraceHelper.Test4_FormatUnlocalizedStackTrace()") == 0); } } [TestMethod] public void Test5_Faultolerant() { Assert.IsTrue(StackTraceHelper.FormatUnlocalizedStackTraceFromString( @"xxx xxx") == @"xxx xxx"); } [TestMethod] public void Test6_Faultolerant() { Assert.IsTrue(StackTraceHelper.FormatUnlocalizedStackTraceFromString( @"xxx") == @"xxx"); } [TestMethod] public void Test7_Faultolerant() { Assert.IsTrue(StackTraceHelper.FormatUnlocalizedStackTraceFromString( @" xxx") == @" xxx"); } [TestMethod] public void Test8_Faultolerant() { Assert.IsTrue(StackTraceHelper.FormatUnlocalizedStackTraceFromString( @" xxx xxx") == @" xxx xxx"); } } } |