View Issue Details

IDProjectCategoryLast Update
0025920AI War 2Gameplay IssueDec 18, 2021 9:33 pm
ReporterChris_McElligottPark Assigned ToChris_McElligottPark  
Status resolvedResolutionfixed 
Fixed in VersionBeta 3.768 Ironman And Doomsday 
Summary0025920: Move away from threadstatic to instead have lists that are checked out of central pools.
DescriptionI'm seeing memory rise over extended play periods that is based on there being so many more threads now than there used to be. This is something that is a side effect of switching to Task.Run, which was largely a great thing. But that switch did make ThreadStatic kind of poison. I have some existing examples of the new pattern working well and quickly, so it's not a crisis. I believe there are around 180 places in the code that require edits, however, so it's at least a day or two of work on my part.
TagsNo tags attached.

Relationships

child of 0025884 resolvedChris_McElligottPark Parent: Todo prior to beta exit 

Activities

Chris_McElligottPark

Dec 17, 2021 11:59 am

administrator   ~0063498

* Removed StringBuilderCache, as that was less efficient and almost never used, and our own ArcenCharacterBuffer is a lot more efficient and does the same thing.

* All of the uses of [ThreadStatic] in Arcen.Universal have been reviewed, and have either been replaced or marked as "ok because of their infrequency of use."

* A variety of uses of ThreadStaticWrapperedPooledItem have been replaced with implementations that require you to get a temporary list from a central pool instead, and then return it.
** Because of the sheer volume of background threads, the ThreadStaticWrapperedPooledItem is not the best idea because it can rack up a huge amount of ram usage during a long game even though it's not technically a memory leak.

Chris_McElligottPark

Dec 17, 2021 12:10 pm

administrator   ~0063499

* Pathfinders no longer use ThreadStatic or ThreadStaticWrapperedPooledItem. These were huge amounts of RAM that could be orphaned on worker threads for long periods, but now it's all centrally pooled.

Chris_McElligottPark

Dec 17, 2021 12:58 pm

administrator   ~0063500

* The Core dll has moved from having 38 remaining ThreadStatic objects, of which most were used super frequently, to only having 8 remaining.
** 6 of those remaining 8 would be ideal to refactor, but are not all that important compared to what was already refactored.

Chris_McElligottPark

Dec 17, 2021 1:30 pm

administrator   ~0063501

* There are now only 4 remaining threadstatic objects in the Core dlls. These ones would be destructive to try to remove or replace.

Chris_McElligottPark

Dec 17, 2021 10:34 pm

administrator   ~0063505

* Added a new RapidAntiLeakPool, which is a lot like ConcurrentPool but with added anti-memory-leak bits added in.
** Specifically, ConcurrentPool is meant for long term or arbitrary-length times of usage of items. But unlike that, RapidAntiLeakPool is purely for temp objects/collections/etc that will be checked out and given back rapidly. This makes memory leaks very simple to detect with them.
** This is a very handy safety net, and even lets us know when things like tracing logs are leaking if tracing is enabled.

Chris_McElligottPark

Dec 18, 2021 12:40 am

administrator   ~0063507

* At this point, there were still 121 ThreadStatic items in External code. Time to keep chopping onto those.
** After several hours, it's down to 73. Oooof.
** On the plus side, I keep seeing little errors that these are fixing, although the errors were probably rarely going to pop up.

Chris_McElligottPark

Dec 18, 2021 11:00 am

administrator   ~0063510

* Okay, so starting the morning with 73 external mentions of threadstatic.
** First starting focusing on some of the dictionaries, some of which are static but not threadstatic.
** Some of the mapgen logic has been updated so that it is more robust and can properly handle capturable seed caps done in multiple passes, rather than expecting them to all be in one pass.
** Down to 67.

Chris_McElligottPark

Dec 18, 2021 12:22 pm

administrator   ~0063511

* All of the static dictionaries in External are now converted over if they need to be, whether they were threadstatic or not. 53 actual threadstatic objects remain.

Chris_McElligottPark

Dec 18, 2021 1:17 pm

administrator   ~0063514

* The various tooltip appenders and pieces all now support either regular character buffers or double character buffers.
** Lots of conversion of threadstatic character buffers of various sorts. Down to 29 threadstatic objects in External, one of which is now marked as okay. So 28 to still investigate.

Chris_McElligottPark

Dec 18, 2021 4:17 pm

administrator   ~0063515

* Fixed a code-todo item that might have been behind ships not always being willing to move from one planet to another when you clicked a planet on the galaxy map.
** This was done while fixing up some of the various threadstatic items, so a nice bonus.

* The External code is now down to (giant sigh) 3 remaining threadstatic members. Two are okay and marked as such, while the other one is already a different form of memory leak from the look of things and needs a larger refactor.

Chris_McElligottPark

Dec 18, 2021 9:33 pm

administrator   ~0063516

FFFFFFFFFFFFFFFIIIIINALLY. This was the thing on my list that I was least looking forward to out of everything I had left to do as of a week or two ago.

* Added a new ListOfLists, which is now used for writing the details of outguard when they are in tooltips.
** This solves a very minor memory leak that was present there previously, as well as removing the last of the threadstatic items that needed to go. At last!

Issue History

Date Modified Username Field Change
Dec 14, 2021 5:01 pm Chris_McElligottPark New Issue
Dec 14, 2021 5:01 pm Chris_McElligottPark Relationship added parent of 0025884
Dec 16, 2021 5:40 pm Chris_McElligottPark Relationship deleted parent of 0025884
Dec 16, 2021 5:40 pm Chris_McElligottPark Relationship added child of 0025884
Dec 17, 2021 11:59 am Chris_McElligottPark Note Added: 0063498
Dec 17, 2021 12:10 pm Chris_McElligottPark Note Added: 0063499
Dec 17, 2021 12:58 pm Chris_McElligottPark Note Added: 0063500
Dec 17, 2021 1:30 pm Chris_McElligottPark Note Added: 0063501
Dec 17, 2021 10:34 pm Chris_McElligottPark Note Added: 0063505
Dec 18, 2021 12:40 am Chris_McElligottPark Note Added: 0063507
Dec 18, 2021 11:00 am Chris_McElligottPark Note Added: 0063510
Dec 18, 2021 12:22 pm Chris_McElligottPark Note Added: 0063511
Dec 18, 2021 1:17 pm Chris_McElligottPark Note Added: 0063514
Dec 18, 2021 4:17 pm Chris_McElligottPark Note Added: 0063515
Dec 18, 2021 9:33 pm Chris_McElligottPark Assigned To => Chris_McElligottPark
Dec 18, 2021 9:33 pm Chris_McElligottPark Status new => resolved
Dec 18, 2021 9:33 pm Chris_McElligottPark Resolution open => fixed
Dec 18, 2021 9:33 pm Chris_McElligottPark Fixed in Version => Beta 3.768 Ironman And Doomsday
Dec 18, 2021 9:33 pm Chris_McElligottPark Note Added: 0063516