In the penultimate post in the metrics series, I explained the reasoning for winding down this series. I also talked about performing one last experiment. Well, that last experiment is in the books, and I’m going to share the results here, today. The idea was to see what progress so far looked like, when applied to real code bases.
Before we get to the results, let’s recap the (admittedly quite incomplete) experimental formula that we’ve arrived at thus far.
T is the time in seconds, and it is a function of p, the number of parameters; n, the number of logical lines of code; f, the number of class fields; and, g, the number of globals. Let’s see how this stacks up to reality.
I picked the methods to look at basically just by poking around at random C# projects on GitHub. I tried to look for methods that were not overly jargon intensive and that minimized the factors that are not accounted for by this formula.
The First Method
First up was the following method, taken from this file.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
// Returns true if a new template is created as opposed to reusing the existing template. private bool MaybeCreateNewTemplate() { // When there are multiple available formats, the formatter uses the first format where a // formatting template could be created. while (possibleFormats.Count > 0) { NumberFormat numberFormat = possibleFormats[0]; String pattern = numberFormat.Pattern; if (currentFormattingPattern.Equals(pattern)) return false; if (CreateFormattingTemplate(numberFormat)) { currentFormattingPattern = pattern; // With a new formatting template, the matched position using the old template needs to be // reset. lastMatchPosition = 0; return true; } else { possibleFormats.RemoveAt(0); } } ableToFormat = false; return false; } |
Let’s take stock. We’ve got no parameters, so that’s easy. The method has 12 logical lines of code, 4 fields, and no globals. Plugging that into the calculation, we get an expected time to comprehend of 298.3404 seconds.
So, what was the experimental verdict? Well, our experimental method readers came in with an average of 131 seconds. So, not particularly close.
The Second Method
Now, method number 2, taken from this file.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
static void CopyFile(string path, string filename, bool bCheckDifference) { string strCWD = Directory.GetCurrentDirectory(); string strSource = Path.Combine(Path.Combine(strCWD, path), filename); string strDest = Path.Combine(strCWD, filename); if (!File.Exists(strSource)) { Debug.LogWarning(string.Format("[Steamworks.NET] Could not copy {0} into the project root. {0} could not be found in '{1}'. Place {0} from the Steamworks SDK in the project root manually.", filename, Path.Combine(strCWD, path))); return; } if (File.Exists(strDest)) { if (!bCheckDifference) return; if (File.GetLastWriteTime(strSource) == File.GetLastWriteTime(strDest)) { FileInfo fInfo = new FileInfo(strSource); FileInfo fInfo2 = new FileInfo(strDest); if (fInfo.Length == fInfo2.Length) { return; } } Debug.Log(string.Format("[Steamworks.NET] {0} in the project root differs from the Steamworks.NET redistributable. Updating.... Please relaunch Unity.", filename)); } else { Debug.Log(string.Format("[Steamworks.NET] {0} is not present in the project root. Copying...", filename)); } File.Copy(strSource, strDest, true); File.SetAttributes(strDest, File.GetAttributes(strDest) & ~FileAttributes.ReadOnly); if (File.Exists(strDest)) { Debug.Log(string.Format("[Steamworks.NET] Successfully copied {0} into the project root. Please relaunch Unity.", filename)); } else { Debug.LogWarning(string.Format("[Steamworks.NET] Could not copy {0} into the project root. File.Copy() Failed. Place {0} from the Steamworks SDK in the project root manually.", filename)); } } |
Now to get our figures for this method. This time, we’ve got 3 parameters. The method has 21 logical lines of code, but refers to no fields, and no globals. It rips into static state a lot with File I/O, but doesn’t actually trigger our formula. Plugging these figures into the calculation, we get an expected time to comprehend of 776.8937 seconds.
Unfortunately, the experiment takers logged an average time of only 117.5 seconds. That’s even more divergent than the last one. Let’s see if method 3 yields better results.
The Third Method
The third method was taken from this file.
|
1 2 3 4 5 6 7 8 9 |
public override SettingsPropertyValueCollection GetPropertyValues( SettingsContext context, SettingsPropertyCollection collection) { var settingPropertyCollection = new SettingsPropertyValueCollection(); if (collection.Count < 1) { return settingPropertyCollection; } string username = context["UserName"].ToString(); if (username.IsNotSet()) { return settingPropertyCollection; } // this provider doesn't support anonymous users if (!Convert.ToBoolean(context["IsAuthenticated"])) { ExceptionReporter.ThrowArgument("PROFILE", "NOANONYMOUS"); } // load the property collection (sync profile class) this.LoadFromPropertyCollection(collection); // see if it's cached... if (this.UserProfileCache.ContainsKey(username.ToLower())) { // just use the cached version... return this.UserProfileCache[username.ToLower()]; } // transfer properties regardless... foreach (SettingsProperty prop in collection) { settingPropertyCollection.Add(new SettingsPropertyValue(prop)); } // get this profile from the DB DataSet profileDS = DB.Current.GetProfiles(this.ApplicationName, 0, 1, username, null); DataTable profileDT = profileDS.Tables[0]; if (profileDT.HasRows()) { DataRow row = profileDT.Rows[0]; // load the data into the collection... foreach (SettingsPropertyValue prop in settingPropertyCollection) { object val = row[prop.Name]; // Only initialize a SettingsPropertyValue for non-null values if (val is DBNull || val == null) { continue; } prop.PropertyValue = val; prop.IsDirty = false; prop.Deserialized = true; } } // save this collection to the cache this.UserProfileCache.AddOrUpdate(username.ToLower(), (k) => settingPropertyCollection, (k, v) => settingPropertyCollection); return settingPropertyCollection; } |
This healthy method has 2 parameters and weighs in at 34 logical lines of code. It features 1 field and 1 global as well. According to our formula, time to comprehend should thus be 2125.0758 seconds. Wow, that’s a long time. It looks like we’re way off the rails here.
And, indeed, the experimental value was 85.5 seconds.
Wrapping Up
So, what went wrong, exactly? Well, in compiling these figures, I’d say it looks like the number of lines of code was distorting things substantially. Perhaps it’s not actually quadratic, or perhaps it is up until some maximum cap or something. There might also be other factors at play as well. The experiment was really too early for us to be expecting particularly accurate results.
The learning here is that it’s really, really hard to arrive at this figure, though I do not think it’s practically impossible to come up with a good, predictive framework. I honestly do think it’d be a reasonable project with some serious investment and effort, but I, personally, lack the time to put that in with what I’ve got going on. Perhaps I’ll take it up someday on my own and maybe even have a Kickstarter for it or something.
For those who have stuck with and continued with the series, my thanks. I’ve enjoyed this, and I think it’s an important and interesting problem; perhaps at some point, the industry will come into broader agreement. Until then, happy static analyzing to those reading.