View Issue Details
| ID | Project | Category | Date Submitted | Last Update | |
|---|---|---|---|---|---|
| 0002236 | AI War 1 / Classic | Crash/Exception | Jan 3, 2011 10:57 am | Jan 10, 2011 10:32 pm | |
| Reporter | Penguin21 | Assigned To | Chris_McElligottPark | ||
| Status | resolved | Resolution | fixed | ||
| Product Version | 4.059 | ||||
| Summary | 0002236: Game Freezes Completely | ||||
| Description | The game runs for a few game-time hours and then all of a sudden it freezes completely. The mouse can move around and I can Alt-Tab out of the program, but the game won't respond. I have to force it to close and restart up with the autosave. Twice I have noticed this occurs when it says autosaving. I didn't pay attention the other times. It might have those times too. My system: Windows 7 Professional 64 bit i7 Overclocked to 3.2 GhZ 12 GB RAM GTX 285 If you need any other information I will upload it. If I notice anything else I'll update this. | ||||
| Tags | No tags attached. | ||||
| Internal Weight | |||||
| duplicate of | 0002220 | resolved | keith.lamothe | Reproducible, recurrent gc heap error |
|
|
A couple things come to mind: - can you post a recent save from the game? - what version where you playing? Thanks :) |
|
|
Whenever the game freezes it also appears to be using at that moment very close to 1.6 GB of RAB each time. |
|
|
Yea, it's probably dying on trying to allocate more heap than it actually can. 32-bit application, etc. |
|
|
It's probably that it's actually getting a GC heap allocation error, and just not showing that if it's in fullscreen. |
|
|
I do remember something about 'GC' in an error that popped up once. Is there something that could be done to fix it? Thank you for the quick replies. |
|
|
It's something we're looking into otherwise, but it's a case of simply running out of memory space that the game can address, given that it's a 32bit application, etc. You might well have more free RAM, but it's run out of what it can allocate at about 1.6GB, basically. |
|
|
There are ways to keep the memory consumption of the game down, but it's difficult to know what to suggest in your case without the savegame itself. Can you attach it? We're also investigating the saves from the (relatively few) reports of this to see if there's any buggily-high consumption to be found. |
|
|
Keith, one thing you could test for: we do the GC.Collect right before a savegame, but given Vinraith's troubles, maybe that is just scheduling a collection, not actually running one. You could have it Debug.Log the amount of RAM in use by the GC right before the collection, and the current number of collections, and then right after, to see what happens in terms of the state right before the save. If it's not actually GC collecting right before a save, then the save process could be made into a "Save request" that only happens after the GC number has increased beyond current GC number plus one. In terms of the autosave issue, since it GC collects right before and right after an autosave, that really shouldn't ever be causing a crash unless either the above is happening; or else some memory is leaking; or else the base memory usage (after a GC collection) plus the memory used by the save process itself, is larger than the allowed heap size (which would really be a problem, eh?). I just added in some GC collections to the LoadGame process just to be safe there, but it's elsewhere that the real problem is, clearly. |
|
|
With all of these memory issues that keep cropping up no matter how much you decrease memory use or how often you call GC.Collect(), I am beginning to wonder something. Does the Unity API itself have some sort of memory leak? |
|
|
I highly doubt it. And, don't blow this out of proportion: we've got exactly 2 current issues out of thousands of players. We've had very few such issues compared to the size of the playerbase and the enormous possible numbers of configurations, etc. |
|
|
Oops, sorry about that. Just me wondering out loud. (well, wondering in text, whatever) :) |
|
|
No worries. |
|
|
|
|
|
I've uploaded a save file of a game that has frozen at least twice. I restarted AI war each time and loaded the auto save to keep playing. I don't know how easily reproducible it will be though. Like I said, it often seems to happen at random after a few hours of game play. One possible thing I thought of earlier today I'll post here to see if you guys think it could be a cause. I have a SSD as my bootable drive, so AI war and most other games are on a physical HDD that will turn itself off after inactivity. Could it possibly be that if I pause the game long enough or slow it down so that the auto save frequency is far enough apart that the drive goes into standby it encounters an error upon attempting to save? I don't know how that could account for the 1.6 GB memory allocation I keep seeing or the one time I saw the GC error, but I thought I'd mention it. Either way, thanks for the continued support. |
|
|
Okay, I came up with some changes that should reduce certain types of baseline memory usage caused by the memory usage caused by having larger numbers of players and larger numbers of ships. It's basically a redesign of the rollups, letting parts of them be per-team rather than always being per-player, which literally can save hundreds of thousands of array entries. I still need to do that sort of thing for the AI thread in the morning, and there's more testing I need to do with it (it's a big change), but in my unscientific tests so far it seems to have reduced memory usage by a fair margin. I'll do better tests tomorrow when the refactoring is complete to get a better idea of what to expect in terms of RAM reduction, but hopefully reducing it will prevent or at least delay this sort of GC heap error. Normally we go for the transient RAM, but this was one case where I was able to actually optimize the working RAM set, which is comparably more rare. |
|
|
Sounds great. Looking forward to seeing it in action. |
|
|
So, I've now put in a bunch of more changes to the AI thread, etc, as well, and the total savings in my apples-to-apples comparisons with your savegame is about 40 MB. That said, it's also counting up the transient memory slower, which is also useful, and the AI thread is in a better position to keep ship counts from getting artificially high in a few situations. Keith: I also tested to make sure that the GC runs immediately inline, and in fact it does. So nothing needed there, that's already as good as it can get. In terms of savegames, I'm rewriting the savegame format (and thus also how it does syncs) in order to hopefully make those both a lot more efficient in RAM, to make it so that the autosave is less likely to trip the heap limit size. |
|
|
Yea, I was just testing the GC thing too, and got some numbers on heap size, etc. It's not looking like the autosave is the main culprit, it's just (understandably) what throws things over the cliff it's already close to. |
|
|
Yes, that is true. I think I can halve (or better) the transient allocation needed to do an autosave, though, so that would make it less cliff-happy at least. But it won't solve the core issue in these specific savegames, which may simply be that there is too much data for it to really hang onto in a meaningful fashion. But, the various changes I've been working on at least help with that some. |
|
|
Chris, one other idea on reducing the actual working set size would be to split ForegroundUnitType into ForegroundUnitData and PlayerSpecificForegroundUnitData and replace existing objects with objects of the latter, which would in turn have a reference to the corresponding non-player-specific data. Being able to have only 1 (or I guess 2, one for main thread, one for AI thread) instead of 10 or 12 or whatever it is copies of a lot of that data would probably help. How much it would help I dunno, since I can't really get any kind of overall allocation data. |
|
|
Hmmm... it's possible, but I don't think it's a main culprit. That doesn't increase with the number of ships increasing, so I think that's a low value target (and would be really hard to do). One thing I've been thinking about is just how freaking many variables are directly on the ForegroundObject class. A lot of those are specialized and are only used by some subsets of the total number of ships in the game. Splitting out OtherObject a while back was one big win, obviously, but it also might help if we had some sub-objects for ForegroundObject that are only initialized as-needed. The RAM usage that matters is that which is per-ship, I think. That's why I'd been focused on the rollups, etc (and when you can shave forty megabytes from rollups alone, that's really... a lot of data). I would focus on other high-value stuff along these lines, honestly. |
|
|
Yea, I basically never add any instance data to ForegroundObject any more if I can possibly help it. I think I added one (if that) int during all of LotS development ;) And a while ago I went through to make sure it wasn't initializing any of the sub-lists until actually necessary, though all those null-checks does add some cpu cost. |
|
|
If you'd like I can look at pruning and/or pulling-into-a-substructure (and related tomfoolery) stuff from ForegroundObject. I think in the Vinraith/MCP save case that even removing a single Int32 would be a roughly 250KB savings on the base footprint. |
|
|
I think that would be great, yeah. Another thing we really need to do is make sure that when a battle ends, all the old targeting data is dropped, and all those targeting lists get pruned. I suspect that those might be a chief culprit of causing increases-over-time in memory with many large battles. Before AI ships go into cold storage, they should jettison all that stuff, when AI or player ships find themselves on a planet with no enemies they should do the same, etc. That last might actually be the biggest value that we can accomplish quickly, but beyond that pulling some of that stuff into substructures might be the next best step. I'm almost done with the new save format, and then I'll check that in. |
|
|
I'm think there are already checks for clearing out targeting data, but definitely worth checking. |
|
|
Another thing we could look at eventually is having the aggregate-targeting just copy the reference to the "lead" list and handle the attach and detach cases properly. Obviously that could lead to interesting bugs but could be a significant help. |
|
|
You'd certainly know better than me! |
|
|
Oops, ninja'd. Yes, I agree with that last, too -- could be both dangerous and helpful. |
|
|
One of my favorite Far Side comics ever showed two guards standing on the newly-constructed Great Wall Of China. One remarks to the other, "Let's see that dog get in here now!" As of version 4.061, I feel kind of like that guard insofar as this bug report goes. ;) I'll be curious to hear if you still are able to experience the issue. Of course, it will still be possible to get a GC Heap error if simply too much stuff is loaded for the game to handle, but hopefully the changes in the last two versions will cumulatively be enough to keep you from having the errors in these specific scenarios. Knock on wood! |
|
|
Optimistically, I'm marking this one fixed. It's as good as we can make it, memory-use-wise, in terms of what we can do pre-5.0 (the next official, hitting later this week). We spend a good man-week of time gutting huge swathes of the application for this, and in our tests the heap memory usage -- the relevant bit here -- is something like 100mb lower, which is 1/8th of the total heap usage. And that's not even getting into the efficiency gains on autosaves in general, which were the most direct culprit of the issue. Fingers crossed that it does better for you when you get a chance to test it again -- feel free to reopen if you have to! |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| Jan 3, 2011 10:57 am | Penguin21 | New Issue | |
| Jan 3, 2011 11:00 am | keith.lamothe | Note Added: 0007353 | |
| Jan 3, 2011 11:02 am | Penguin21 | Note Added: 0007354 | |
| Jan 3, 2011 11:09 am | keith.lamothe | Note Added: 0007357 | |
| Jan 3, 2011 11:33 am | Chris_McElligottPark | Note Added: 0007359 | |
| Jan 3, 2011 6:06 pm | Penguin21 | Note Added: 0007375 | |
| Jan 3, 2011 6:07 pm | Chris_McElligottPark | Note Added: 0007376 | |
| Jan 3, 2011 6:08 pm | keith.lamothe | Note Added: 0007377 | |
| Jan 3, 2011 6:17 pm | Chris_McElligottPark | Note Added: 0007378 | |
| Jan 3, 2011 6:28 pm | TechSY730 | Note Added: 0007381 | |
| Jan 3, 2011 6:32 pm | Chris_McElligottPark | Note Added: 0007384 | |
| Jan 3, 2011 6:40 pm | TechSY730 | Note Added: 0007387 | |
| Jan 3, 2011 6:40 pm | Chris_McElligottPark | Note Added: 0007388 | |
| Jan 3, 2011 8:42 pm | Penguin21 | File Added: Has frozen twice before in this game.sav | |
| Jan 3, 2011 8:48 pm | Penguin21 | Note Added: 0007404 | |
| Jan 3, 2011 11:31 pm | Chris_McElligottPark | Relationship added | duplicate of 0002220 |
| Jan 3, 2011 11:35 pm | Chris_McElligottPark | Note Added: 0007423 | |
| Jan 3, 2011 11:35 pm | Chris_McElligottPark | Assigned To | => Chris_McElligottPark |
| Jan 3, 2011 11:35 pm | Chris_McElligottPark | Status | new => confirmed |
| Jan 3, 2011 11:51 pm | Penguin21 | Note Added: 0007427 | |
| Jan 4, 2011 4:31 pm | Chris_McElligottPark | Note Added: 0007448 | |
| Jan 4, 2011 4:53 pm | keith.lamothe | Note Added: 0007450 | |
| Jan 4, 2011 5:02 pm | Chris_McElligottPark | Note Added: 0007451 | |
| Jan 4, 2011 5:47 pm | keith.lamothe | Note Added: 0007456 | |
| Jan 4, 2011 5:47 pm | keith.lamothe | Note Edited: 0007456 | |
| Jan 4, 2011 5:50 pm | Chris_McElligottPark | Note Added: 0007457 | |
| Jan 4, 2011 5:52 pm | keith.lamothe | Note Added: 0007458 | |
| Jan 4, 2011 5:54 pm | keith.lamothe | Note Added: 0007459 | |
| Jan 4, 2011 5:59 pm | Chris_McElligottPark | Note Added: 0007460 | |
| Jan 4, 2011 6:00 pm | keith.lamothe | Note Added: 0007461 | |
| Jan 4, 2011 6:02 pm | keith.lamothe | Note Added: 0007462 | |
| Jan 4, 2011 6:02 pm | Chris_McElligottPark | Note Added: 0007463 | |
| Jan 4, 2011 6:03 pm | Chris_McElligottPark | Note Added: 0007464 | |
| Jan 6, 2011 10:53 am | Chris_McElligottPark | Note Added: 0007591 | |
| Jan 6, 2011 10:53 am | Chris_McElligottPark | Status | confirmed => feedback |
| Jan 10, 2011 10:32 pm | Chris_McElligottPark | Note Added: 0008172 | |
| Jan 10, 2011 10:32 pm | Chris_McElligottPark | Status | feedback => resolved |
| Jan 10, 2011 10:32 pm | Chris_McElligottPark | Resolution | open => fixed |
| Apr 14, 2014 9:27 am | Chris_McElligottPark | Category | Bug - Crash or Exception => Crash/Exception |
