View Issue Details
| ID | Project | Category | Date Submitted | Last Update | |
|---|---|---|---|---|---|
| 0002220 | AI War 1 / Classic | Crash/Exception | Dec 30, 2010 12:00 am | Jan 10, 2011 10:32 pm | |
| Reporter | Vinraith | Assigned To | keith.lamothe | ||
| Status | resolved | Resolution | fixed | ||
| Product Version | 4.058 | ||||
| Summary | 0002220: Reproducible, recurrent gc heap error | ||||
| Description | The host (me) gets a gc heap error after approximately 1 hour, 10 minutes of playing the attached game. Upon reloading, the game will again play fine for about an hour and ten minutes, then crash again. This has happened perhaps 5 times now with regularity. This is the first time we've experienced a recurring gc heap error in all our time playing this game. 12-23-10 4.sav marks a save after the first instance of this crash. At the time there were 41,000 total ships in the game. The "battle of lacaille" save is from later, when there are about 50,000 ships in the game, and after we've gone through several cycles of crashing and reloading. For reference, my machine is a core i7, 3 GB DDR3, running Win XP service pack 3. Also, this is the same game as that in issues 2064, 2066, and 2079, which may provide some insight. | ||||
| Tags | No tags attached. | ||||
| Internal Weight | |||||
| parent of | 0002228 | closed | Chris_McElligottPark | Possibly new error type - same game |
| has duplicate | 0002236 | resolved | Chris_McElligottPark | Game Freezes Completely |
| related to | 0002066 | resolved | Chris_McElligottPark | One Way Doormaster has Experimental Ships |
| related to | 0002079 | resolved | Chris_McElligottPark | Insane numbers of AI MRS's and Engineers |
| related to | 0002064 | resolved | Chris_McElligottPark | ErrorsReportedByEngine on new game start |
| related to | 0002221 | closed | Chris_McElligottPark | Crash |
| related to | 0002233 | closed | Chris_McElligottPark | Possibly new error type with more info - same game |
| related to | 0000177 | resolved | keith.lamothe | "Too many heap sections" crash (suspected memory leak) |
|
|
|
|
|
|
|
|
Curious if you'll still find this reproducible in 4.059, as the planet-serialization thing might help more than we expected. Also curious if you were to, say, make your autosave interval every 1 minutes instead of every 10, if you get the issue after 11 minutes instead of 110. That would help us narrow it down to if it's some sort of saving-related thing or if it's an actual gameplay thing, at least. These ones are always challenging, as the core issue is simply that you ran out of memory in the game process. That means that, in a direct sense, we can't fix the issue in most cases. All we can do is reduce the transient usage so that hopefully you don't run into it as frequently, or so that it gets pushed out beyond the time that anyone would reasonably play a session. That said, when it's this soon in the game and this repeatable, that is suspicious, and it could be some sort of runaway allocation of... something. My curiosity is if it has to do with autosaving, since you said it happens even when not much is going on. |
|
|
The next time we play I'll try switching the autosave interval and let you know. |
|
|
Just tested it under 4.058 and you were exactly right, with autosaves at 1 minute intervals it crashes after 7 minutes (so 6 autosaves and then crash on the seventh, regardless of time interval). Nice shot, Chris! Incidentally, this only happens in multiplayer. If I run that save by myself, it doesn't work. |
|
|
Wicked! That gives us something we can test for, at least. That's really great news. If Keith isn't able to look at it in the next few days (he's working a bit), I'll check it out monday and see what I can find. This may still not be easy, but if we can reproduce it in a fashion like you are able to, we should be able to figure it out, at least! |
|
|
Note: That testing was done under .058, we're testing now under .059. |
|
|
23 autosaves so far without a crash under .059! Edit: Up to 32, now. I think for all practical purposes the memory usage changes in .059 have solved this. I'm curious just when, if at all, it'll do it again if we keep playing. |
|
|
Okay... wow. I totally didn't see this one coming. That's great news, though! Keith, given this made such a difference just with the per-planet stuff, I think we should do the Player stuff (and PlayerStats stuff), too, as soon as we can. Those are the last areas of the Save process that are not Stringbuilder-based, and those specifically get a lot larger in terms of the playerstats stuff as time goes on in gametime. A shift like that might reduce transient usage even more, which could be great. |
|
|
Do you think it was that conversion from a string.concat call to a StringBuilder.append call? It seems like that would generate less objects. |
|
|
Yes, without a doubt -- that's why the change was made, and why I asked them to see what would happen here with that shift. I just didn't expect that to make as big a difference in this specific case as it clearly did. Crazy! |
|
|
OK, the answer to "how many does it take?" is 44 (crashes on the 45th). We got a crash after 44 successful autosaves, 1 minute apiece. Assuming that's consistent for any autosave interval then at the default 10 minute intervals it should never, ever come up (if we've played more than 7 hours it might be best if it DID crash). If you make further changes of that same sort, my guess is you can push this issue into the unreachable distance with any sane autosave interval. |
|
|
Wow, I really didn't think the planet string.concat calls were _that_ big a deal, and certainly it doesn't make sense that the GC wasn't reclaiming that memory between autosaves (which forcibly call the GC, I thought). I just happened to be in the code and decided to go ahead and do the conversion. I almost didn't even mention it in the release notes ;) Anyway, yea, I'll go take a look at the player serialization too. |
|
|
I'm starting to think maybe there was something else going on here, too, and the change to the serialization fixed the other bug inadvertently or something. I don't know. If this is indeed such a big deal, we need to start looking at GameCommand serialization, too. There was something I was planning to do for that, but now I've changed my mind. I think just generalized improvement-of-the-current-pattern is better than the newer Tidalis-style pattern at this stage, given the nature of AI War. Having a buffer-reader that doesn't use string.split would also be a target for improvement at some point in the future, sort of the inverse of stringbuilder. But that doesn't seem like an imminent priority, even though it would have wide applicability with savegames, gamecommands, and otherwise, since it's also fairly risky and involved to do. I'm not quite ready to call this one resolved, though -- it's very great that it took 44 autosaves to happen after it took 7 in the prior version, but that still means it crashed after 44 minutes. I'd like confirmation that it seems to be not crashing when you play with a normal 10-minute autosave, and then I think we can all rest easier. Anyway, thanks for all the testing Vinraith, that's sounding very promising! :) |
|
|
No problem. The next time we play I intend to set it back to the usual ten minutes, if anything happens there I'll let you know. If there's anything else I can do that would be useful from a diagnostic perspective, just let me know. |
|
|
Doesn't the standard C# library already come with a nice tokenizing input reader? |
|
|
Ok, the player/playerStats/snapshots changes are in for 4.060 :) |
|
|
Vinraith -- awesome. And thanks again! TechSY730 -- not in so many words, no. Our input is a lot more complex, with nested arrays of arbitrary length (and particularly arbitrary number of characters) delimited by various values (pipe, comma, and otherwise). I assume you're thinking of StringReader, which is pretty woeful compared to our needs. A tokenizer of the sort you refer to in Java is more or less what we're doing with string.split, anyhow. Bear in mind that, unlike in Java, all strings in C# are interned, which means that the benefits of a tokenizer are severely minimized in a lot of cases. Also bear in mind we're not using .NET, we're using Mono. I'm more accustomed to .NET, which handles strings WAY better internally. A lot of our recent memory problems come from switching to Mono, which has a gimped GC and string-handling by comparison to .NET, so now we're having to adjust our own code to account for that. Our biggest RAM-efficiency changes -- and they were enormous -- have come from replacing foreach statements with for statements, to give an example of what I mean. In .NET those had far less impact. Keith -- definitely great. :) |
|
|
Ouch, I didn't realize you were passing around such varied data structures in an environment with such an iffy garbage collector. I do not envy your task of "coding around" the system. But I'm sure you guys can figure it out. ;) And yea, I can see how you would want to avoid manipulation of actual string objects as much as you can if all of them are interned. Yikes! |
|
|
Well, when these data structures were designed, they were running on the best garbage collector I've yet seen. The port to unity was worth it, but it did create issues like this. |
|
|
Yea, the mono folks already have a much better GC under development, so my guess is by the time Unity 4.0 rolls around they'll be able to incorporate it and we can breathe easier. In the meantime it's just a matter of finding sufficient workarounds, and it seems like we're pretty close to there. |
|
|
Marking this as feedback pending testing in 4.060 by whoever has time. The player stuff certainly was doing a lot of string.concat, though normally it wouldn't be a terribly big deal with only 3 players (1 human + 2 AI) compared to 80-120 planets, etc. But there was a lot going on in there, to be sure. |
|
|
Dammit. Switched it back to 10 minutes per autosave, but just got a gc after 22 minutes of play. I'll see if that timing is consistent, maybe it was a one-off. It almost perfectly coincided with popping the command station on EZ Aquarii in the new save I'm attaching. Edit: Mantis shows me having changed the status of this ticket from feedback to assigned, which I most certainly didn't (all I did was add this note). Weird. |
|
|
|
|
|
OK, I've now had 1 crash at around half an hour on an autosave, and two crashes at about half an hour on manual saves. No idea what's changed. I'll try switching back to 1 minute intervals on autosave and see what impact that has. |
|
|
Yeah, consistent crashes on third autosave at 10 minute intervals. Getting the game to even let you join with 1 minute autosaves seems problematic, it'll be up briefly and then drop to "waiting for players." Particularly interesting: if MCP hosts then he gets this crash as well. However, whereas when it crashes on my machine he can still save, I locked up when he locked up. Strange. Attaching another save game, these are starting to get considerably larger than we've ever seen before, which is no doubt fueling the problem. I still wonder if it isn't partially linked to the AI's MRS and Engineer spam. |
|
|
|
|
|
OK, finally got one loaded with the 1 minute autosaves. We weren't able to test for too long, but over the span of 10 minutes (and thus ten autosaves) we got no crash, so it's not scaling directly to the "2 autosaves and a crash" behavior at 10 minute intervals. It probably is worth noting, though, that after a mere 10 minutes with 1 minute autosaves on the attached save (20110101_problems) my AIWar process showed as using 1.5 GB of memory. |
|
|
|
|
|
Which leads to an interesting question, what is the max heap size (or whatever C#'s runtime environment's equivalent is) currently set for AI war? 1.5GB is sort of a far cry from the expected 500MB or so expected size mentioned from my report about memory usage. |
|
|
Every one of the crashes has coincided with task manager reporting something in the 1.7-1.8 GB memory usage range, for what that's worth. On initial load of the last save game I posted here, I show 1.2 GB in use. It's no wonder, starting from a platform like that, that we end up over in very little time. |
|
|
OK, ran another test with 1 minute autosaves, we made it through ten, crashed on the eleventh. Switching back to ten and trying to limp over the finish line, we're so close to winning this one... |
|
|
It's interesting that when I get these crashes it locks up, but generates no log or error files. On the other hand, when MCP hosts, his crashes seem to coincide with useful log output (like the two "possibly new error type" issues attached to this one). The behavior even looks slightly different, so it's not 100% clear whether it's the same or simply something related (hence the separate tickets). |
|
|
Okay, I came up with some changes that should reduce certain types of baseline memory usage caused by the memory usage caused by having larger numbers of players and larger numbers of ships. It's basically a redesign of the rollups, letting parts of them be per-team rather than always being per-player, which literally can save hundreds of thousands of array entries. I still need to do that sort of thing for the AI thread in the morning, and there's more testing I need to do with it (it's a big change), but in my unscientific tests so far it seems to have reduced memory usage by a fair margin. I'll do better tests tomorrow when the refactoring is complete to get a better idea of what to expect in terms of RAM reduction, but hopefully reducing it will prevent or at least delay this sort of GC heap error. Normally we go for the transient RAM, but this was one case where I was able to actually optimize the working RAM set, which is comparably more rare. These will be in for 4.060. Thanks for your patience! |
|
|
Thanks Chris, glad to see you guys are on this! |
|
|
You bet! It was just down period while we weren't in the office, that's all. :) |
|
|
One additional note - be sure to look at issue 2233, as Vinraith pointed out I (the purple player aka MCP) managed to get a little extra data, a crash dump folder that is, I'm guessing, from unity. A second crash occurred right after the one in which Unity (again, a guess) dumped out the crash data and that time the adv logging caught some stuff (but no "unity dump" on the second one). Anyhow, both sets of files are attached to 2233 with notes. But since the tracker isn't filled with this kind of error (as best we can tell) our working (hip shot yet again) hypothesis is that our weird initial conditions in this ongoing game (namely a truly schitzo AI, the red one has way more ship types than it should have) might be contributing to the error. This is also the game where the (red in particular) AI is producing a ton of honey combs of death (tractor ships), decloakers, MRSes and engineers. Perhaps all of these "extra" ships are contributing to the memory drain in our case.. Anyhow, thanks again, you guys are by FARRRRR the best - actually you've really spoiled the community in many ways - i'm starting to say things like why the hell isn't EA in on xmas patching my freakin game! But seriously, you guys should take a break! |
|
|
|
|
|
Just did a fair bit of testing to make sure the GC was doing what we thought it was, etc. Attached my data in the xlsx file. Mostly I tested a late-game save from my FS-campaign from a few weeks ago, and then compared it to the Vinraith/MCP save. First, looking at just managed memory (where all our entity data is, so should be the only part changing much once the relevant textures and such are loaded). My game started at around 400 million bytes managed memory, and climbed by about 850 thousand bytes "permanent" every 2-minute autosave interval. This seems normal since the number of ships was increasing too. Each autosave I had it recording the memory before any GC ("PreGCBytes"), right after the GC ("PostGCBytes"), and then again after running the custom "trim" operation to trim down unneeded rollup space on all ships, planets, and players ("PostTrimPostGCBytes"). The first recovery number can be misleading because it isn't necessarily the first GC since the last autosave (in fact, I don't think it ever was), and the second is also a bit off because the GC itself increases the byte count somewhat (as can be seen on some cases where evidently it had _just_ been run). But the data is interesting anyway. The Vinraith/MCP game started at around 554 million bytes and climbed by about 1.5 million bytes "permanent" every 2-minute autosave interval. Since this was a normal-caps game (mine was low-caps) this doesn't seem _way_ out there, though it may indicate more sticking around than it needs to. My other tests were looking at the grand total of memory held from the outside-process perspective, using Process Explorer. My game started at around 1,161,000 KB, and was up to around 1,214,000 KB at the first autosave and did not go up over the next 2 autosaves. The Vinraith/MCP game started at around 1,389,000 KB and went up to 1,488,000 KB when I used manage-players to disable the second player, it was at 1,562,000 KB after the first autosave, 1,596,000 KB after the second, and it died (heap sections error) before the third autosave with the same 1,596,000-ish number still showing. Bear in mind that all these tests were run within the Unity Editor, which has a just-started external footprint of roughly 198,000 KB. For reference, my save had between 15k and 16k ships total, the Vinraith/MCP game had between 66k and 67k ships. Anyway, it does look like something is making the autosaves pump up the memory in one case and not the other, though it's difficult to tell due to the disconnect between actually-asking-for-memory and it-shows-up-on-an-external-counter, due to the layers of memory management that hold onto stuff "for a rainy day", etc. |
|
|
Chris, I didn't want to mess with the full-sync code while you were in there, but I just committed my addition of PrepareForGarbageCollection and DebugLogMemoryInfo methods to the Configuration class. If you want to try having it do that rollup-trim on _everything_ (on the main thread, doesn't touch the AI thread, obviously) before the GC just call Configuration.PrepareForGarbageCollection() |
|
|
Were those tests with my many various improvements from today (prior to the revised savegame format), or were they prior to that? Thanks for that other method, by the way -- I'll include that with the revised savegame format, and will have that checked in in a few moments. |
|
|
Prior to your more recent changes, I retested my save's overall footprint after that and did see something like a 30MB lower baseline. |
|
|
Cool stuff. The new savegame format is now in, too: * A revised savegame format has now been put in place, with an emphasis on using less RAM to create it. ** One happy side effect of this is that that savegame files are now about 7/8 to 4/5 of their former size. They still load in about the same amount of time, though. ** Another side effect is that the new savegames actually show their loading process as they load, now, rather than just sitting there silently on parsing data. ** At any rate, the chief purpose of this is to make the creation of savegames as well as the syncing of multiplayer network state into a lower-RAM-using process to avoid GC heap errors when players are already running near the RAM redline. The new format is vastly superior in that regard, possibly using as little as half as much RAM as before, depending on the exact circumstances of the save. ** HANG ON TO YOUR OLD SAVES! There's no risk of corrupting any old saves with this new system, but new saves might indeed turn out to be corrupt. Our testing has resulted in no corruption, but with anything this large and new, it's good to hang onto your old saves for a bit, just in case. |
|
|
I'm going to look at some of the rollups on Player, actually. Those might be another good place for some cutting. |
|
|
Sorry for sounding ignorant, but what are "rollups" in this context? |
|
|
Sub-lists or other similar data structures, counts, etc, that let us slice the enormous amount of data we have in various useful ways. It's critical for having a game like this actually be functional at all, rather than having to loop over tens of thousands of objects anytime we want to do anything. |
|
|
Ahh, thanks. I haven't heard of that class of data structures being called that. (shows how little tiered data structures are covered in modern day colleges :/) |
|
|
It stems from my database background, I tend to think of things in those sorts of terms. |
|
|
One of my favorite Far Side comics ever showed two guards standing on the newly-constructed Great Wall Of China. One remarks to the other, "Let's see that dog get in here now!" As of version 4.061, I feel kind of like that guard insofar as this bug report goes. ;) I'll be curious to hear if you still are able to experience the issue. Of course, it will still be possible to get a GC Heap error if simply too much stuff is loaded for the game to handle, but hopefully the changes in the last two versions will cumulatively be enough to keep you from having the errors in these specific scenarios. Knock on wood! |
|
|
Optimistically, I'm marking this one fixed. It's as good as we can make it, memory-use-wise, in terms of what we can do pre-5.0 (the next official, hitting later this week). We spend a good man-week of time gutting huge swathes of the application for this, and in our tests the heap memory usage -- the relevant bit here -- is something like 100mb lower, which is 1/8th of the total heap usage. And that's not even getting into the efficiency gains on autosaves in general, which were the most direct culprit of the issue. Fingers crossed that it does better for you when you get a chance to test it again -- feel free to reopen if you have to! |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| Dec 30, 2010 12:00 am | Vinraith | New Issue | |
| Dec 30, 2010 12:00 am | Vinraith | File Added: 20101229_battle_of_lacaille.sav | |
| Dec 30, 2010 12:02 am | Vinraith | File Added: 12-23-10 4.sav | |
| Dec 30, 2010 12:04 am | Vinraith | Description Updated | |
| Dec 30, 2010 12:06 am | Vinraith | Description Updated | |
| Dec 30, 2010 12:06 am | Vinraith | Description Updated | |
| Dec 30, 2010 12:54 am | Vinraith | Relationship added | related to 0002066 |
| Dec 30, 2010 12:54 am | Vinraith | Relationship added | related to 0002079 |
| Dec 30, 2010 12:54 am | Vinraith | Relationship added | related to 0002064 |
| Dec 30, 2010 12:55 am | Vinraith | Description Updated | |
| Dec 30, 2010 12:56 am | Vinraith | Description Updated | |
| Dec 30, 2010 1:58 am | Vinraith | Relationship added | related to 0002221 |
| Dec 30, 2010 10:45 am | Chris_McElligottPark | Note Added: 0007257 | |
| Dec 30, 2010 11:31 am | Vinraith | Note Added: 0007259 | |
| Dec 30, 2010 5:07 pm | Vinraith | Note Added: 0007261 | |
| Dec 30, 2010 5:08 pm | Vinraith | Note Edited: 0007261 | |
| Dec 30, 2010 5:09 pm | Vinraith | Note Edited: 0007261 | |
| Dec 30, 2010 5:11 pm | Vinraith | Note Edited: 0007261 | |
| Dec 30, 2010 5:20 pm | Vinraith | Note Edited: 0007261 | |
| Dec 30, 2010 5:22 pm | Chris_McElligottPark | Note Added: 0007262 | |
| Dec 30, 2010 5:22 pm | Chris_McElligottPark | Assigned To | => keith.lamothe |
| Dec 30, 2010 5:22 pm | Chris_McElligottPark | Status | new => confirmed |
| Dec 30, 2010 5:46 pm | Vinraith | Note Added: 0007263 | |
| Dec 30, 2010 5:48 pm | Vinraith | Note Edited: 0007263 | |
| Dec 30, 2010 6:03 pm | Vinraith | Note Added: 0007264 | |
| Dec 30, 2010 6:14 pm | Vinraith | Note Edited: 0007264 | |
| Dec 30, 2010 6:15 pm | Vinraith | Note Edited: 0007264 | |
| Dec 30, 2010 6:21 pm | Chris_McElligottPark | Note Added: 0007265 | |
| Dec 30, 2010 6:23 pm | TechSY730 | Note Added: 0007266 | |
| Dec 30, 2010 6:24 pm | Chris_McElligottPark | Note Added: 0007267 | |
| Dec 30, 2010 6:30 pm | Vinraith | Note Edited: 0007264 | |
| Dec 30, 2010 6:31 pm | Vinraith | Note Edited: 0007264 | |
| Dec 30, 2010 6:31 pm | Vinraith | Note Added: 0007268 | |
| Dec 30, 2010 6:42 pm | Vinraith | Note Edited: 0007268 | |
| Dec 30, 2010 6:43 pm | Vinraith | Note Edited: 0007268 | |
| Dec 30, 2010 6:46 pm | Vinraith | Note Edited: 0007268 | |
| Dec 30, 2010 6:46 pm | Vinraith | Note Edited: 0007268 | |
| Dec 30, 2010 7:12 pm | keith.lamothe | Note Added: 0007269 | |
| Dec 30, 2010 7:19 pm | Chris_McElligottPark | Note Added: 0007270 | |
| Dec 30, 2010 7:23 pm | Vinraith | Note Added: 0007271 | |
| Dec 30, 2010 7:43 pm | TechSY730 | Note Added: 0007272 | |
| Dec 30, 2010 7:56 pm | keith.lamothe | Note Added: 0007273 | |
| Dec 30, 2010 8:07 pm | Chris_McElligottPark | Note Added: 0007274 | |
| Dec 30, 2010 8:21 pm | TechSY730 | Note Added: 0007275 | |
| Dec 30, 2010 8:21 pm | TechSY730 | Note Edited: 0007275 | |
| Dec 30, 2010 8:22 pm | TechSY730 | Note Edited: 0007275 | |
| Dec 30, 2010 8:23 pm | TechSY730 | Note Edited: 0007275 | |
| Dec 30, 2010 8:31 pm | Chris_McElligottPark | Note Added: 0007276 | |
| Dec 30, 2010 9:11 pm | keith.lamothe | Note Added: 0007277 | |
| Dec 30, 2010 9:12 pm | keith.lamothe | Note Added: 0007278 | |
| Dec 30, 2010 9:12 pm | keith.lamothe | Status | confirmed => feedback |
| Dec 31, 2010 1:08 am | Vinraith | Note Added: 0007286 | |
| Dec 31, 2010 1:08 am | Vinraith | Status | feedback => assigned |
| Dec 31, 2010 1:10 am | Vinraith | Note Edited: 0007286 | |
| Dec 31, 2010 1:13 am | Vinraith | Note Edited: 0007286 | |
| Dec 31, 2010 1:14 am | Vinraith | File Added: 12-30-10 5.sav | |
| Dec 31, 2010 1:51 am | Vinraith | Note Added: 0007287 | |
| Jan 1, 2011 5:33 pm | Vinraith | Note Added: 0007315 | |
| Jan 1, 2011 5:35 pm | Vinraith | File Added: 2011111_red.sav | |
| Jan 1, 2011 5:57 pm | Vinraith | Note Added: 0007316 | |
| Jan 1, 2011 5:58 pm | Vinraith | File Added: 20110101_problems.sav | |
| Jan 1, 2011 6:15 pm | TechSY730 | Note Added: 0007318 | |
| Jan 1, 2011 6:35 pm | Vinraith | Note Added: 0007319 | |
| Jan 1, 2011 9:58 pm | Vinraith | Note Edited: 0007319 | |
| Jan 1, 2011 9:59 pm | Vinraith | Note Edited: 0007319 | |
| Jan 1, 2011 10:17 pm | Vinraith | Note Added: 0007322 | |
| Jan 1, 2011 11:11 pm | Vinraith | Note Edited: 0007322 | |
| Jan 1, 2011 11:37 pm | Master Cylinder Pants | Issue cloned: 0002228 | |
| Jan 1, 2011 11:37 pm | Master Cylinder Pants | Relationship added | parent of 0002228 |
| Jan 3, 2011 12:45 am | Master Cylinder Pants | Relationship added | related to 0002233 |
| Jan 3, 2011 12:48 am | Vinraith | Note Added: 0007343 | |
| Jan 3, 2011 1:57 am | Vinraith | Note Edited: 0007343 | |
| Jan 3, 2011 11:31 pm | Chris_McElligottPark | Relationship added | has duplicate 0002236 |
| Jan 3, 2011 11:36 pm | Chris_McElligottPark | Note Added: 0007424 | |
| Jan 4, 2011 1:16 am | Vinraith | Note Added: 0007429 | |
| Jan 4, 2011 1:17 am | Chris_McElligottPark | Note Added: 0007430 | |
| Jan 4, 2011 8:36 am | Master Cylinder Pants | Note Added: 0007434 | |
| Jan 4, 2011 1:43 pm | Chris_McElligottPark | Relationship added | related to 0000177 |
| Jan 4, 2011 4:57 pm | keith.lamothe | File Added: AIWarMemoryConsumptionStudy.xlsx | |
| Jan 4, 2011 5:18 pm | keith.lamothe | Note Added: 0007453 | |
| Jan 4, 2011 5:23 pm | keith.lamothe | Note Added: 0007454 | |
| Jan 4, 2011 6:12 pm | Chris_McElligottPark | Note Added: 0007465 | |
| Jan 4, 2011 6:20 pm | keith.lamothe | Note Added: 0007467 | |
| Jan 4, 2011 6:21 pm | Chris_McElligottPark | Note Added: 0007468 | |
| Jan 4, 2011 6:22 pm | Chris_McElligottPark | Note Added: 0007469 | |
| Jan 4, 2011 6:24 pm | TechSY730 | Note Added: 0007471 | |
| Jan 4, 2011 6:45 pm | Chris_McElligottPark | Note Added: 0007472 | |
| Jan 4, 2011 6:52 pm | TechSY730 | Note Added: 0007473 | |
| Jan 4, 2011 6:54 pm | Chris_McElligottPark | Note Added: 0007474 | |
| Jan 6, 2011 10:53 am | Chris_McElligottPark | Note Added: 0007590 | |
| Jan 6, 2011 10:53 am | Chris_McElligottPark | Status | assigned => feedback |
| Jan 10, 2011 10:32 pm | Chris_McElligottPark | Note Added: 0008171 | |
| Jan 10, 2011 10:32 pm | Chris_McElligottPark | Status | feedback => resolved |
| Jan 10, 2011 10:32 pm | Chris_McElligottPark | Resolution | open => fixed |
| Apr 14, 2014 9:27 am | Chris_McElligottPark | Category | Bug - Crash or Exception => Crash/Exception |
