View Issue Details

IDProjectCategoryLast Update
0002546AI War 1 / ClassicCrash/ExceptionMar 22, 2011 11:14 pm
ReporterDraco18s Assigned ToChris_McElligottPark  
Status resolvedResolutionfixed 
Product Version4.071 
Fixed in Version5.006 
Summary0002546: Scout Exporing -> Desynch
DescriptionInitial desynch happened as I was marking a planet P4. Attempted to replicate with the following results:

Did not repro upon immediate reload, but there was a minute of "synched, not synched, synched, not synched" followed by a full desynch just as a scout explored a new planet.

Upon reload of hte same save there was another desynch as the scout entered the same planet.

Getting save.
TagsNo tags attached.
Internal Weight

Relationships

related to 0000852 resolvedChris_McElligottPark Host to Client Switch: Cannot Synch 
related to 0002460 resolvedChris_McElligottPark Unable to play multiplayer game. 
related to 0003077 resolvedChris_McElligottPark Desync 

Activities

TechSY730

Jan 18, 2011 9:58 pm

reporter   ~0009165

Could this be the fabled last lurking desync? I hope so.

Draco Cretel

Jan 18, 2011 9:59 pm

reporter  

Draco18s Game 1.sav (142,983 bytes)

keith.lamothe

Jan 18, 2011 10:02 pm

administrator   ~0009167

Thanks much :) Do you have the save from the other player(s)? And the desync error log?

Draco18s

Jan 18, 2011 10:03 pm

developer   ~0009168

Keith, see attached file, should have been uploaded a moment ago from the host.

keith.lamothe

Jan 18, 2011 10:04 pm

administrator   ~0009170

Yes, I see your save, I was asking if you had the save from the other player(s), that's generally necessary to really tell exactly what desync'd. And the desync error log gives some more easily-accessible general info that helps point us in the right direction.

Draco18s

Jan 18, 2011 10:07 pm

developer   ~0009173

Sorry, that is the other player. I was the client, do you want my save?

keith.lamothe

Jan 18, 2011 10:09 pm

administrator   ~0009176

Yes, please :)

Also, if the host (or you) have a DesyncErrors.txt, that's helpful info to have.

Draco18s

Jan 18, 2011 10:11 pm

developer  

Draco_Desynch.sav (142,039 bytes)

Draco18s

Jan 18, 2011 10:12 pm

developer   ~0009180

Last edited: Jan 18, 2011 10:19 pm

Uploaded. Will get those errors too.

And upon a new load of the game it seems to be working (stuttering "Waiting for Players"). Edit: desynched again.

Draco Cretel

Jan 18, 2011 10:17 pm

reporter  

DesyncErrors.txt (2,531 bytes)   
1/18/2011 9:52:36 PM (4.071)
-----------------------------------
Draco18s Desync Error-----------------------------------
Draco18s Sent: P0 Sc6705 Sh144 En123850 Cr209482913 Ma169364868 Kn10238 : P1 Sc6237 Sh117 En80000 Cr66849631 Ma12352956 Kn2912 : P8 Sc175237 Sh3476 : P9 Sc225047 Sh4874 : Rd1699598229
Draco18s Should Have Been: P0 Sc6705 Sh145 En123850 Cr209482913 Ma169364868 Kn10238 : P1 Sc6237 Sh117 En80000 Cr66849631 Ma12352956 Kn2912 : P8 Sc175224 Sh3476 : P9 Sc225047 Sh4874 : Rd1699598229

1/18/2011 9:54:44 PM (4.071)
-----------------------------------
Draco18s Desync Error-----------------------------------
Draco18s Sent: P0 Sc6729 Sh145 En123850 Cr222614689 Ma173731204 Kn10252 : P1 Sc6255 Sh120 En79925 Cr54911327 Ma9556556 Kn2940 : P8 Sc175276 Sh3476 : P9 Sc225047 Sh4874 : Rd768543394
Draco18s Should Have Been: P0 Sc6729 Sh144 En123850 Cr222614689 Ma173731204 Kn10252 : P1 Sc6255 Sh120 En79925 Cr54911327 Ma9556556 Kn2940 : P8 Sc175289 Sh3476 : P9 Sc225047 Sh4874 : Rd768543394

1/18/2011 9:56:00 PM (4.071)
-----------------------------------
Draco18s Desync Error-----------------------------------
Draco18s Sent: P0 Sc6705 Sh145 En123850 Cr215110817 Ma174747012 Kn10244 : P1 Sc6255 Sh120 En79925 Cr69031775 Ma10795164 Kn2924 : P8 Sc175237 Sh3476 : P9 Sc225047 Sh4874 : Rd1606595987
Draco18s Should Have Been: P0 Sc6705 Sh145 En123850 Cr215110817 Ma174747012 Kn10244 : P1 Sc6255 Sh120 En79925 Cr69031775 Ma10795164 Kn2924 : P8 Sc175250 Sh3476 : P9 Sc225047 Sh4874 : Rd1606595987

1/18/2011 10:00:10 PM (4.071)
-----------------------------------
Draco18s Desync Error-----------------------------------
Draco18s Sent: P0 Sc6711 Sh142 En123900 Cr218862753 Ma174695908 Kn10248 : P1 Sc6255 Sh120 En79925 Cr63114591 Ma10473452 Kn2932 : P8 Sc175276 Sh3476 : P9 Sc225047 Sh4874 : Rd38498012
Draco18s Should Have Been: P0 Sc6711 Sh143 En123900 Cr218862753 Ma174695908 Kn10248 : P1 Sc6255 Sh120 En79925 Cr63114591 Ma10473452 Kn2932 : P8 Sc175276 Sh3476 : P9 Sc225047 Sh4874 : Rd38498012

1/18/2011 10:12:18 PM (4.071)
-----------------------------------
Draco18s Desync Error-----------------------------------
Draco18s Sent: P0 Sc6747 Sh144 En123875 Cr258258081 Ma201649636 Kn10290 : P1 Sc6294 Sh122 En78525 Cr41018431 Ma24188 Kn3016 : P8 Sc175328 Sh3476 : P9 Sc225047 Sh4874 : Rd905727978
Draco18s Should Have Been: P0 Sc6747 Sh144 En123875 Cr258258081 Ma201576196 Kn10290 : P1 Sc6294 Sh122 En78525 Cr41018431 Ma24188 Kn3016 : P8 Sc175328 Sh3476 : P9 Sc225047 Sh4874 : Rd905727978

DesyncErrors.txt (2,531 bytes)   

Chris_McElligottPark

Jan 18, 2011 10:20 pm

administrator   ~0009187

Keith, you taking this one for now?

Oddly, it looks like the player zero metal was out of sync in this one -- man this is just a different issue every time.

TechSY730

Jan 18, 2011 10:23 pm

reporter   ~0009189

Yay. Time for more endless hours pouring through code looking for odd things. :\

I don't envy your jobs when things like this happen man. But I'm sure you guys can figure it out.

keith.lamothe

Jan 18, 2011 10:28 pm

administrator   ~0009190

I wasn't planning on running with this one right now, was just making sure the necessary files were available for whoever did. I can give it a shot if you like but I'm not sure how much longer I'll be on tonight. Currently working on a few balance issues.

TechSY730

Jan 18, 2011 10:28 pm

reporter   ~0009191

Last edited: Jan 18, 2011 10:31 pm

Am I reading the log right? Knowing how to read it may help me get clues about how to reproduce future desyncs.

Per player stuff:
Pn denotes the section for player n
Sc is raw score
Sh is number of ships
En is energy
Cr is crystal
Ma is metal

Global sutff:
Rd is ???

Chris_McElligottPark

Jan 18, 2011 10:32 pm

administrator   ~0009192

Keith: no worries. I'm working on the Tidalis release at the moment, so I guess we can look into them in the morning if nothing else.

TechSY730:

Sh is number of ships controlled by that player. All the rest of the per-player stuff is right.

Rd is the current value generated by the random number generator at the end of the turn. Basically, if those numbers match then the same number of random calls (however many) were made during the turn. If they don't match, then the code branched enough that the random function was called a different number of times between machines. So far those have always matched in this latest batch of desyncs, which is a great great thing.

TechSY730

Jan 18, 2011 10:39 pm

reporter   ~0009193

Is it possible for there to be a desync but these numbers remain consistent? Maybe like the same ship is in different places for two people but haven't killed or fired upon something, or be killed or fired yet?

Chris_McElligottPark

Jan 18, 2011 10:41 pm

administrator   ~0009194

Yes, of course. This is basically a basic hash. It doesn't _find out_ about the desync until these differ, but there has always been a little bit of time (a few seconds, sometimes, but hopefully less) before it is reflected here).

Chris_McElligottPark

Jan 18, 2011 10:57 pm

administrator   ~0009195

Draco18s: any steps for us to reproduce? Keith and I have this loaded up, and aren't seeing any stuttering or anything. For my part I've mostly just been sitting there, but I'm not sure what he's doing.

keith.lamothe

Jan 18, 2011 11:15 pm

administrator   ~0009196

I was sending out hundreds of scouts (mostly on auto-explore, but not all), until the AI reminded me that scouts can't fight off waves.

Chris_McElligottPark

Jan 18, 2011 11:16 pm

administrator   ~0009197

Well, the sql server went down for a bit, but it's back now apparently. Keith and I played the game for a bit, got beat to heck by the too-large waves that were a part of the next version (but now won't be), and never saw a desync. Keith was scouting the whole time, I didn't really scout at all. I attacked one planet, and we both defended our planets from waves. Played about 10ish minutes, never saw a waiting for players message nor a desync. :/

TechSY730

Jan 18, 2011 11:18 pm

reporter   ~0009198

Last edited: Jan 18, 2011 11:18 pm

You know things are too hard when not even the devs can handle it. :D

Draco18s

Jan 18, 2011 11:19 pm

developer   ~0009199

You guys got farther than we could. Basically the save (host) was from when the game initially desynched (so we could load back in). As soon as the scout explored THE NEXT planet we desynched every time. (It's the planet just north of my [client]'s homeworld, though not with a direct connection, IIRC).

Chris_McElligottPark

Jan 18, 2011 11:19 pm

administrator   ~0009200

Well, to be fair, we weren't really trying to play for real. More a matter of what we could stir up and get messed up. We could have fended off those waves if we'd tried. But, that said, with that crazy scenario and the too-large-waves things, it was only a matter of time before they got us fair and square, anyway.

Chris_McElligottPark

Jan 18, 2011 11:21 pm

administrator   ~0009201

Draco18s: were either of you playing your second game of the current AI War session? In other words, had both of you just started up the AI War program and connected to play, or had there been other games on one or both of the machines?

We're starting to suspect that, since we can't ever duplicate these, these are related to having some static variable that isn't getting cleared properly or something. Which would make it fail on subsequent runs but not the first, for instance.

TechSY730

Jan 18, 2011 11:24 pm

reporter   ~0009202

On that route, I have noticed that after playing a single player game, quitting that (but not the program), and then trying to host or join a multiplayer game, it doesn't work. Either I can't join, or others can't join me. Never tried a multiplayer to multiplayer though.

Draco18s

Jan 18, 2011 11:24 pm

developer   ~0009203

Last edited: Jan 18, 2011 11:25 pm

Host had played some single player before hand, otherwise no. After the third desynch he restarted his client (I attempted to be host, but was unsuccessful, as apparently my IP can't be connected to now).

TechSY: might that be 0000852 ?

TechSY730

Jan 18, 2011 11:27 pm

reporter   ~0009204

Last edited: Jan 18, 2011 11:27 pm

Yea, very similar to what I described, though it seems to happen _sometimes_ the other way around too.

EDIT: Dang, that bug has been around for a while.

Chris_McElligottPark

Jan 18, 2011 11:27 pm

administrator   ~0009205

Draco18s -- that is very interesting info. If you both restart your client, are you able to play?

TechSY730

Jan 18, 2011 11:29 pm

reporter   ~0009206

If restarting both clients fixes this, should 0000852 be added as a relationship and maybe even reopened?

Chris_McElligottPark

Jan 18, 2011 11:31 pm

administrator   ~0009207

Sure.

Draco Cretel

Jan 18, 2011 11:31 pm

reporter   ~0009208

We were not able to play ever after restarts. If we tried to start the game now it would do the same thing.

Chris_McElligottPark

Jan 18, 2011 11:34 pm

administrator   ~0009209

Just to make sure I understand: after both players had restarted at the same time? That part is important, because if only one of you did it at a time, that might have still had the issue.

Draco18s

Jan 18, 2011 11:39 pm

developer   ~0009211

No, only host had restarted. Right now we seem to be fine.

Apparently 0000852 was killing us.

Chris_McElligottPark

Jan 18, 2011 11:41 pm

administrator   ~0009212

Ok -- that's very good to know, thanks. So this sounds like it is not a true in-game desync, but rather a case where "the game desyncs after another game has been run in the same session."

That's a serious issue as well, but not nearly so much as the first one, and hopefully it will be easier to find. It also means we were looking in _totally_ the wrong parts of the code for this issue before. Though I did find some other stuff that needed to be fixed, so that wasn't a waste of time, at least.

Draco18s

Jan 19, 2011 12:08 am

developer   ~0009215

Glad to help.

keith.lamothe

Feb 8, 2011 11:20 am

administrator   ~0010118

Have you seen any more of these desyncs? We did a pretty thorough inspection of the static scope stuff (fields that could potentially persist from one game to another), and these may have been resolved since.

Draco18s

Feb 8, 2011 11:40 am

developer   ~0010124

Haven't played many games, most of those last under 2 hours.

Chris_McElligottPark

Mar 22, 2011 11:14 pm

administrator   ~0011297

Optimistically, I think that 0003077 was also the cause of this.

Issue History

Date Modified Username Field Change
Jan 18, 2011 9:57 pm Draco18s New Issue
Jan 18, 2011 9:58 pm TechSY730 Note Added: 0009165
Jan 18, 2011 9:59 pm Draco Cretel File Added: Draco18s Game 1.sav
Jan 18, 2011 10:02 pm keith.lamothe Note Added: 0009167
Jan 18, 2011 10:03 pm Draco18s Note Added: 0009168
Jan 18, 2011 10:04 pm keith.lamothe Note Added: 0009170
Jan 18, 2011 10:07 pm Draco18s Note Added: 0009173
Jan 18, 2011 10:09 pm keith.lamothe Note Added: 0009176
Jan 18, 2011 10:11 pm Draco18s File Added: Draco_Desynch.sav
Jan 18, 2011 10:12 pm Draco18s Note Added: 0009180
Jan 18, 2011 10:17 pm Draco Cretel File Added: DesyncErrors.txt
Jan 18, 2011 10:19 pm Draco18s Note Edited: 0009180
Jan 18, 2011 10:20 pm Chris_McElligottPark Note Added: 0009187
Jan 18, 2011 10:23 pm TechSY730 Note Added: 0009189
Jan 18, 2011 10:28 pm keith.lamothe Note Added: 0009190
Jan 18, 2011 10:28 pm TechSY730 Note Added: 0009191
Jan 18, 2011 10:29 pm TechSY730 Note Edited: 0009191
Jan 18, 2011 10:31 pm TechSY730 Note Edited: 0009191
Jan 18, 2011 10:32 pm Chris_McElligottPark Note Added: 0009192
Jan 18, 2011 10:39 pm TechSY730 Note Added: 0009193
Jan 18, 2011 10:41 pm Chris_McElligottPark Note Added: 0009194
Jan 18, 2011 10:57 pm Chris_McElligottPark Note Added: 0009195
Jan 18, 2011 11:15 pm keith.lamothe Note Added: 0009196
Jan 18, 2011 11:16 pm Chris_McElligottPark Note Added: 0009197
Jan 18, 2011 11:18 pm TechSY730 Note Added: 0009198
Jan 18, 2011 11:18 pm TechSY730 Note Edited: 0009198
Jan 18, 2011 11:19 pm Draco18s Note Added: 0009199
Jan 18, 2011 11:19 pm Chris_McElligottPark Note Added: 0009200
Jan 18, 2011 11:21 pm Chris_McElligottPark Note Added: 0009201
Jan 18, 2011 11:24 pm TechSY730 Note Added: 0009202
Jan 18, 2011 11:24 pm Draco18s Note Added: 0009203
Jan 18, 2011 11:25 pm Draco18s Note Edited: 0009203
Jan 18, 2011 11:27 pm TechSY730 Note Added: 0009204
Jan 18, 2011 11:27 pm Chris_McElligottPark Note Added: 0009205
Jan 18, 2011 11:27 pm TechSY730 Note Edited: 0009204
Jan 18, 2011 11:29 pm TechSY730 Note Added: 0009206
Jan 18, 2011 11:31 pm Chris_McElligottPark Note Added: 0009207
Jan 18, 2011 11:31 pm Draco Cretel Note Added: 0009208
Jan 18, 2011 11:34 pm Chris_McElligottPark Note Added: 0009209
Jan 18, 2011 11:39 pm Chris_McElligottPark Relationship added related to 0000852
Jan 18, 2011 11:39 pm Draco18s Note Added: 0009211
Jan 18, 2011 11:41 pm Chris_McElligottPark Note Added: 0009212
Jan 18, 2011 11:42 pm Chris_McElligottPark Relationship added related to 0002460
Jan 19, 2011 12:08 am Draco18s Note Added: 0009215
Feb 8, 2011 11:20 am keith.lamothe Note Added: 0010118
Feb 8, 2011 11:20 am keith.lamothe Assigned To => keith.lamothe
Feb 8, 2011 11:20 am keith.lamothe Status new => feedback
Feb 8, 2011 11:40 am Draco18s Note Added: 0010124
Feb 8, 2011 11:40 am Draco18s Status feedback => assigned
Mar 22, 2011 11:14 pm Chris_McElligottPark Note Added: 0011297
Mar 22, 2011 11:14 pm Chris_McElligottPark Status assigned => resolved
Mar 22, 2011 11:14 pm Chris_McElligottPark Fixed in Version => 5.006
Mar 22, 2011 11:14 pm Chris_McElligottPark Resolution open => fixed
Mar 22, 2011 11:14 pm Chris_McElligottPark Assigned To keith.lamothe => Chris_McElligottPark
Mar 22, 2011 11:14 pm Chris_McElligottPark Relationship added related to 0003077
Apr 14, 2014 9:27 am Chris_McElligottPark Category Bug - Crash or Exception => Crash/Exception