View Issue Details
| ID | Project | Category | Date Submitted | Last Update | |
|---|---|---|---|---|---|
| 0002014 | AI War 1 / Classic | Bug - Other | Dec 11, 2010 8:52 pm | Jan 7, 2011 1:16 pm | |
| Reporter | Suzera | Assigned To | Chris_McElligottPark | ||
| Status | resolved | Resolution | fixed | ||
| Summary | 0002014: Possible memory leak in load/unload or ship creation? | ||||
| Description | I started up this game and built loads of ships with cheat. I got up to about 90-100k total ships before it crash at the 1.54gb point with a heap section overflow. On reload of the save I did before the crash at 1.51gb, with all the ships intact but mostly in transports, the memory useage was reduced by 1/3. See before and afters below, as well as the save. It took forever to load. I tried loading and unloading roughly the same 10kish ships from transports in and out several times and each time it seemed to bump up a good 40-60m. Going to give it one more shot in a controlled test as soon as it loads. The process froze for 10 minutes before I could jack it up more than 300m this way with the shipyards paused. In previous games, as I have mentioned, after blowing up every AI ship and guard post on a 120 planet map, it was only after that that I got a heap section error at the same memory point. On reload that particular game went down to 812m and loaded pretty quickly. | ||||
| Tags | No tags attached. | ||||
| Internal Weight | |||||
|
|
|
|
|
|
|
|
|
|
|
This game should be pretty processor safe because the bulk of them are in transports. It will just take a long time to load. :( |
|
|
Unload-reload transport cycle did 40-60 MEGABYTES per reload of 10k ships, not kilobytes. Typo. |
|
|
Can you try switching your view to another planet and back between transport cycles? Planet switch calls a forced GC, and perhaps the memory would drop back down a bit. It's pretty normal for issuing commands to thousands and thousands of ships to involve significant allocation, but it's not normal for it to hang onto it once those commands are executed, etc. So: 1) Check memory 2) Load ships in transports 3) Check memory 4) Unload ships 5) Check memory 6) Load ships in transports 7) Check memory 8) Unload ships 9) Check memory 10) Switch view to another planet 11) Check memory Thanks :) |
|
|
I can't unload most of the transports it looks like. :) Loading a transport the first time seems like it may be taking megabytes of extra memory, something on the order of 100mb/8000ish ships, and then may never be giving it back. Going to try what you said. |
|
|
Come to think of it, the games I have memory crashed in usually involved lots of AI carriers. More than my typical game. |
|
|
Well, if you take a freshly loaded save, and give 10,000 ships 1 command, it has to allocate 1 space in the CurrentCommands list for each of those 10,000 ships. It doesn't deallocate that space after the commands are done, it just reuses it the next time that ship gets a command (unless it gets more than 1 queued command, in which case it needs to allocate more). At least I think so, I think in some cases it re-nulls the list reference to let the GC get it back. |
|
|
It wasn't allocating more memory each time on a transport load. It coincidentally did though at times. I did however leave it at 1.3gb with the shipyards off (so no new ships) and it eventually crashed from heap section overflow after about 10-20 min while I was posting elsewhere. |
|
|
Some testing on this: 1.28 GB mem usage after preparation, all docks paused. 1.33 GB after unload - around 60k ships. CPU barely alive... auto load on 20k fighters... full game death, restart 1304MB at test start, 1322MB after unload, 1341MB after loading all again, 1364MB after planet change, 1365MB after another unload, 1399MB after another load, planet change no mem usage change, 1399MB after unload, 1436MB after another load, 1467MB after planet change, 1478MB after load Data seems to be at least suspicious.. also each load action causes serious lag - it's visible on actual load operation not when command to load is executed. |
|
|
After next load mem usage went up only to 1479MB. The lag visible when loading ships seems to depend an amount of unloaded ships present and is shown as very large process time (seen it up to 6 seconds). |
|
|
Ok, that's very good data for the investigation going forward :) I won't be able to get to it til monday but that's really good, thanks. You mentioned auto-load, is that what you used for all the load operations? (It would be if I needed to load that much stuff, just checking) |
|
|
Yeah, I used that too. Loading 10k units by hand takes a while. :( |
|
|
All load test were done with auto-load and I used 10k units for them. |
|
|
One thing to note: carriers/barracks don't actually transport units in at all the same manner as human transports work, so if there's a bug in one it's unlikely to be in both. Or, perhaps more likely: it's not a bug related to either of these, if there is a bug there, but rather a bug in adding/removing ships to/from the rollups or other lists for planets. If there was a bug there, that would affect all of these cases as well as moving ships back and forth between planets, or having ships die and new ships spawn, etc. |
|
|
I'm investigating this now. Just to be sure I checked auto-load to make sure it wasn't being terribly inefficient, and it seems fine. But figuring out how to load 20,000 fighters and actually issuing that order simply takes a good bit of time and memory :) But there's another problem where lots of memory is being held onto that doesn't seem to make sense; the "everything's loaded into the transports, haven't GC'd yet" should be the high-water-mark memory wise unless something big is happening elsewhere in the galaxy, but it looks like repeating the process just kept making it go up. |
|
|
(Loads attached save) Good grief that's a lot of ships... Continuing on ;) |
|
|
Ships dying and new ships spawning after does seem to be an issue if the games I have crashed in legitiamtely are any indication. I crash AFTER blowing up tons of ships to where the population at crash is lower than the start of the game. It really seems like something is not being unloaded from memory properly in relation to ships, posts and other such entities that are created and destroyed through the game. I mostly did this to see if there would also be a compacting of memory to the point I had seen in games where I have actually crashed. There was still a memory compaction post-loading, but not nearly as big. If I had to hazard another guess at this point, something attached to every ship that is not allocated for ship instantiation itself is being used at least one time and then that memory is just NEVER freed, even on destruction, instead of there being a constantly increasing leak per ship or per some type of action. Judging from the time I just loaded it and let it sit running, the memory used in this way does not immediately get used on reload of a save, but does over time eventually pass through all the ships and use the crash-level memory amount even without any further action on the player's part. If you want to build a dozen or two ships per second for some reason, just turn the docks on. :) |
|
|
Well, Convington's Law of Diagnosis: there can always be more than one thing wrong :) I'm also going to look at having it do the ClearAllReferencesToHeap thing when destroying a ForegroundObject (ship). I already have it doing that when unloading a gamestate to avoid cruft accumulating across multiple save-load repetitions. Just been avoiding it during actual gameplay since strictly it shouldn't be necessary (the references should be being cleared elsewise) and it may hurt performance, but I think we can work through that. But I think there's also an issue with rollups never shrinking since they're really not supposed to unless we tell them to, and that means that there's a fair bit of monotonic-increase to memory consumption from those. |
|
|
I'm not sure it's something related to not clearing specifically on destruction per se, because if you just idle that save (issue no commands after pausing the docks) it should consume huge extra chunks of memory every so often. It seemed to for me anyway. Every 10 seconds or so it would add up another 40-50 mb. |
|
|
Right. Remember that the AI is pretty frequently getting new ships and perhaps fighting against minor faction stuff (particularly if you have the Zenith Cookie Monster enabled). As ships move from planet to planet the rollups have to expand to fit that ship count, and never shrink back down (in terms of heap space), etc. |
|
|
Now that the rollups are shrinking themselves as of a few versions back, are we still seeing this? |
|
|
This wasn't a true memory leak, but at any rate the various issues that were present as part of this have been resolved over the last week plus in the best way we can. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| Dec 11, 2010 8:52 pm | Suzera | New Issue | |
| Dec 11, 2010 8:52 pm | Suzera | File Added: lotsofstuffpostload1.5g.png | |
| Dec 11, 2010 8:53 pm | Suzera | File Added: lotsostuffcrash.png | |
| Dec 11, 2010 8:53 pm | Suzera | File Added: 1.51g.sav | |
| Dec 11, 2010 8:54 pm | Suzera | Description Updated | |
| Dec 11, 2010 8:54 pm | Suzera | Description Updated | |
| Dec 11, 2010 8:55 pm | Suzera | Note Added: 0006024 | |
| Dec 11, 2010 8:56 pm | Suzera | Description Updated | |
| Dec 11, 2010 8:57 pm | Suzera | Note Added: 0006025 | |
| Dec 11, 2010 8:57 pm | Suzera | Note Edited: 0006025 | |
| Dec 11, 2010 9:02 pm | keith.lamothe | Note Added: 0006026 | |
| Dec 11, 2010 9:05 pm | Suzera | Note Added: 0006027 | |
| Dec 11, 2010 9:06 pm | Suzera | Note Edited: 0006027 | |
| Dec 11, 2010 9:07 pm | Suzera | Note Edited: 0006027 | |
| Dec 11, 2010 9:08 pm | Suzera | Note Added: 0006028 | |
| Dec 11, 2010 9:16 pm | keith.lamothe | Note Added: 0006029 | |
| Dec 11, 2010 9:41 pm | Suzera | Note Added: 0006032 | |
| Dec 11, 2010 10:32 pm | orzelek | Note Added: 0006038 | |
| Dec 11, 2010 10:36 pm | orzelek | Note Added: 0006039 | |
| Dec 11, 2010 10:36 pm | keith.lamothe | Note Added: 0006040 | |
| Dec 11, 2010 10:40 pm | Suzera | Note Added: 0006041 | |
| Dec 12, 2010 5:49 am | orzelek | Note Added: 0006054 | |
| Dec 13, 2010 9:02 am | Chris_McElligottPark | Note Added: 0006088 | |
| Dec 13, 2010 9:34 am | keith.lamothe | Note Added: 0006090 | |
| Dec 13, 2010 9:34 am | keith.lamothe | Assigned To | => keith.lamothe |
| Dec 13, 2010 9:34 am | keith.lamothe | Status | new => assigned |
| Dec 13, 2010 10:05 am | keith.lamothe | Note Added: 0006091 | |
| Dec 13, 2010 10:58 am | Suzera | Note Added: 0006093 | |
| Dec 13, 2010 11:01 am | Suzera | Note Edited: 0006093 | |
| Dec 13, 2010 11:41 am | keith.lamothe | Note Added: 0006096 | |
| Dec 13, 2010 11:56 am | Suzera | Note Added: 0006099 | |
| Dec 13, 2010 12:00 pm | keith.lamothe | Note Added: 0006101 | |
| Dec 23, 2010 4:17 pm | Chris_McElligottPark | Note Added: 0006905 | |
| Dec 23, 2010 4:17 pm | Chris_McElligottPark | Status | assigned => feedback |
| Jan 7, 2011 1:16 pm | Chris_McElligottPark | Note Added: 0007700 | |
| Jan 7, 2011 1:16 pm | Chris_McElligottPark | Status | feedback => resolved |
| Jan 7, 2011 1:16 pm | Chris_McElligottPark | Resolution | open => fixed |
| Jan 7, 2011 1:16 pm | Chris_McElligottPark | Assigned To | keith.lamothe => Chris_McElligottPark |
