View Issue Details

Jump to NotesJump to History
IDProjectCategoryDate SubmittedLast Update
0024624AI War 2Crash/ExceptionMar 28, 2021 12:22 amMar 30, 2021 5:36 pm
ReporterBadgerBadger Assigned ToChris_McElligottPark  
Status resolvedResolutionfixed 
Product VersionBeta 2.764 Command Station Hotfix 
Fixed in VersionBeta 2.765 The Hunt For Frigates And Civilians 
Summary0024624: 2.764 MP errors
DescriptionNotes from tonights MP session. This was definitely the most stable session we've had, though we only had 3 clients instead of our usual 5 or 6.
I've split out host vs client errors

3/27/2021 7:28:58 PM 2.764 SINGLEP Game Version: 2.764

host errors:

<Saw this a bunch of times with a wide variety of negative numbers:
3/27/2021 7:44:05 PM 2.764 HOST Tried to write Int32 NonNeg, but passed value was: -8000, which is out of range! Value set to 0 so your save would work. FieldNameForErrors: DeathEffectCausingDamageReceivedToEntity_Data
  at System.Environment.get_StackTrace () [0x00000] in <9577ac7a62ef43179789031239ba8798>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, System.Boolean IncludeStackTrace, Arcen.Universal.Verbosity Verbosity, System.DateTime Timestamp) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, System.Boolean IncludeStackTrace, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenSerializationBuffer.AddInt32 (Arcen.Universal.ReadStyle RStyle, System.Int32 Item, System.String FieldNameForErrors) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.AIW2.Core.GameEntity_Squad.SerializeTo (Arcen.Universal.ArcenSerializationBuffer Buffer, Arcen.Universal.UltraEfficientStyle StyleForPKIDsIfNetworkSyncData, System.Boolean isDoingPartialNetworkSync) [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at AIWar2NetworkSync.Server_SendBatchOfDivergentSquadsToFix (System.Int32 clientsOfRelevance) [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at AIWar2NetworkSync.Server_SendBatchOfSyncsBasedOnCurrentSyncStage () [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.AIW2.Core.AIWar2Networking.OnServer_CheckForSendingServerBatchMessageToClients () [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.AIW2.External.SimPlannerImplementation.DoActualSimStep (System.Boolean& stalledWaitingOnConnections, System.Boolean& mayUpdateVisuals, System.String& reasonForNoVisualUpdates) [0x00000] in <530e5f0408bb463da28b713b904d5d16>:0
  at Arcen.AIW2.Core.Engine_AIW2.ProcessSimStep (System.Boolean& stalledWaitingOnConnections, System.String& reasonForNoVisualUpdates) [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.Universal.Engine_Universal.OnUpdateFromMainThread () [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.AIW2.Core.ArcenGameControllerBase.BaseUpdate () [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at ArcenGameController.Update () [0x00000] in <36858076bed84b2db389eb6da6b20ae4>:0


<This is weird; you put in defensive code so I have no idea how this can fail>
3/27/2021 7:46:34 PM 2.764 HOST Exception in entity tooltip text generation at stage 15010102:System.IndexOutOfRangeException: Index was outside the bounds of the array.
  at Arcen.AIW2.External.Window_InGameHoverEntityInfo.WriteTechThatBenefits (Arcen.Universal.ArcenDoubleCharacterBuffer buffer, Arcen.AIW2.Core.TechUpgrade upgrade, Arcen.AIW2.Core.Faction localFaction, System.Int32& debugStage) [0x00065] in <530e5f0408bb463da28b713b904d5d16>:0
  at Arcen.AIW2.External.Window_InGameHoverEntityInfo.GetTextForEntity (Arcen.Universal.ArcenDoubleCharacterBuffer buffer, Arcen.AIW2.Core.GameEntity_Base EntityBase, Arcen.AIW2.Core.Fleet+Membership MembershipBase, Arcen.AIW2.Core.GameEntityTypeData TypeDataOrNull, System.Int32 OptionalCountToShow, Arcen.AIW2.Core.Faction ForFactionOrNull, System.Byte OptionalForMarkLevel, Arcen.AIW2.Core.FromSidebarType IsFromSidebarType, Arcen.AIW2.External.ShipExtraDetailFlags DetailFlags) [0x0b93e] in <530e5f0408bb463da28b713b904d5d16>:0
  at System.Environment.get_StackTrace () [0x00000] in <9577ac7a62ef43179789031239ba8798>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, System.Boolean IncludeStackTrace, Arcen.Universal.Verbosity Verbosity, System.DateTime Timestamp) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, System.Boolean IncludeStackTrace, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.AIW2.External.Window_InGameHoverEntityInfo.GetTextForEntity (Arcen.Universal.ArcenDoubleCharacterBuffer buffer, Arcen.AIW2.Core.GameEntity_Base EntityBase, Arcen.AIW2.Core.Fleet+Membership MembershipBase, Arcen.AIW2.Core.GameEntityTypeData TypeDataOrNull, System.Int32 OptionalCountToShow, Arcen.AIW2.Core.Faction ForFactionOrNull, System.Byte OptionalForMarkLevel, Arcen.AIW2.Core.FromSidebarType IsFromSidebarType, Arcen.AIW2.External.ShipExtraDetailFlags DetailFlags) [0x00000] in <530e5f0408bb463da28b713b904d5d16>:0
  at Arcen.AIW2.External.Window_InGameHoverEntityInfo+bPanel.GetTextToRender (Arcen.Universal.ArcenDoubleCharacterBuffer buffer) [0x00000] in <530e5f0408bb463da28b713b904d5d16>:0
  at Arcen.AIW2.External.Window_InGameHoverEntityInfo+bPanel.UpdateContentFromVolatile (Arcen.Universal.ArcenUIWrapperedUnityImage Image, Arcen.Universal.ArcenUI_Image+SubImageGroup _SubImages, Arcen.Universal.SubTextGroup _SubTexts) [0x00000] in <530e5f0408bb463da28b713b904d5d16>:0
  at Arcen.Universal.ArcenUI_ImageButton.OnUpdateVolatileDataFromMainThread () [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenUI_Window.OnUpdateVolatileDataFromMainThread () [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenUI.OnUpdateVolatileDataFromMainThread () [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.Engine_Universal.OnUpdateFromMainThread () [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.AIW2.Core.ArcenGameControllerBase.BaseUpdate () [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at ArcenGameController.Update () [0x00000] in <36858076bed84b2db389eb6da6b20ae4>:0

<Saw this once when a player was trying to connect to the game partway through. This connection failed, but quitting and reconnecting worked>
3/27/2021 7:55:00 PM 2.764 HOST MP Error: SendMessageToSpecificClient returned something other than None/Ok 1 times! FRAGMENTED 3 of 4 sendResult: LimitExceeded (array 511913 / len 511913 bytes) with 0 extra attempted loops
  at System.Environment.get_StackTrace () [0x00000] in <9577ac7a62ef43179789031239ba8798>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, System.Boolean IncludeStackTrace, Arcen.Universal.Verbosity Verbosity, System.DateTime Timestamp) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, System.Boolean IncludeStackTrace, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.UniversalMessageHandler.HandleMessage (System.Int64 UniqueMessageIDFromSender, System.Boolean IsFromSelfWithoutNetwork, System.UInt32 senderNetworkID, Arcen.Universal.ArcenNetworkMessageType CoreNetworkMessageType, Arcen.Universal.ArcenDeserializationBuffer buffer) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenNetworkMessageHandler.HandleMessage (System.Boolean IsFromSelfWithoutNetwork, System.UInt32 senderNetworkID, Arcen.Universal.ArcenDeserializationBuffer buffer) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.AIW2.Core.SteamSocket+SteamServerManager.OnMessage (Steamworks.Data.Connection connection, Steamworks.Data.NetIdentity identity, System.IntPtr data, System.Int32 size, System.Int64 messageNum, System.Int64 recvTime, System.Int32 channel) [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Steamworks.SocketManager.ReceiveMessage (System.IntPtr msgPtr) [0x00000] in <1a4d0b9e0e9644d59545a7f1c42821c8>:0
  at Steamworks.SocketManager.Receive (System.Int32 bufferSize) [0x00000] in <1a4d0b9e0e9644d59545a7f1c42821c8>:0
  at Arcen.AIW2.Core.SteamSocket.DoPerUpdateLogic () [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.Universal.ArcenNetworkAuthority.DoPerUpdateLogic () [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.Engine_Universal.OnUpdateFromMainThread () [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.AIW2.Core.ArcenGameControllerBase.BaseUpdate () [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at ArcenGameController.Update () [0x00000] in <36858076bed84b2db389eb6da6b20ae4>:0



client 0
<Two errors in deserializing external data>
3/27/2021 7:44:19 PM 2.764 CLIENT ERROR: DeserializeExternalDataFrom for EXTERNAL DATA - Faction debugStage: 9000 Error: System.Exception: No DeserializeFrom_NewStyle defined for HistoryItemType Unknown
  at Arcen.AIW2.External.Fireteam+HistoryItem.DeserializeFrom_NewStyle (Arcen.Universal.ArcenDeserializationBuffer Buffer) [0x00176] in <530e5f0408bb463da28b713b904d5d16>:0
  at Arcen.AIW2.External.Fireteam.DeserializedIntoSelf (System.Int32 FireTeamID, Arcen.Universal.ArcenDeserializationBuffer Buffer, System.String ForDebugging_FactionName) [0x001b9] in <530e5f0408bb463da28b713b904d5d16>:0
  at Arcen.AIW2.External.Fireteam.DeserializeNewFrom (System.Int32 id, Arcen.Universal.ArcenDeserializationBuffer Buffer, System.String ForDebugging_FactionName) [0x00007] in <530e5f0408bb463da28b713b904d5d16>:0
  at Arcen.AIW2.External.FireteamUtility.DeserializeFireteamsAndDiscardAnyExtraLeftovers (Arcen.Universal.ArcenDeserializationBuffer Buffer, Arcen.Universal.ArcenLessLinkedList`1[ItemType] Teams, System.String ForDebugging_FactionName) [0x0006e] in <530e5f0408bb463da28b713b904d5d16>:0
  at Arcen.AIW2.External.ExternalData_AIFactionCommon.DeserializeExternalData (Arcen.AIW2.Core.Faction ParentFaction, System.Object[] Target, System.Int32 ItemsToExpect, Arcen.Universal.ArcenDeserializationBuffer Buffer, System.Boolean IsForPartialSyncDuringMultiplayer) [0x005e2] in <530e5f0408bb463da28b713b904d5d16>:0
  at Arcen.AIW2.Core.ArcenExternalDataPatternImplementationBase_Faction.DeserializeExternalDataOuter (System.Object ParentObject, System.Object[] Target, System.Int32 ItemsToExpect, Arcen.Universal.ArcenDeserializationBuffer Buffer, System.Boolean IsForPartialSyncDuringMultiplayer) [0x000a4] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.Universal.ArcenExternalData.DeserializedIntoSelf (System.Object ParentObject, Arcen.Universal.ArcenDeserializationBuffer Buffer, System.Boolean IsForPartialSyncDuringMultiplayer, System.String ExternalDataHeader) [0x000ed] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenExternalDataLookup.DeserializeExternalDataFrom (Arcen.Universal.ArcenDeserializationBuffer Buffer, System.String ExternalDataHeader, System.Boolean IsForPartialSyncDuringMultiplayer) [0x00341] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at System.Environment.get_StackTrace () [0x00000] in <9577ac7a62ef43179789031239ba8798>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, System.Boolean IncludeStackTrace, Arcen.Universal.Verbosity Verbosity, System.DateTime Timestamp) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, System.Boolean IncludeStackTrace, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenExternalDataLookup.DeserializeExternalDataFrom (Arcen.Universal.ArcenDeserializationBuffer Buffer, System.String ExternalDataHeader, System.Boolean IsForPartialSyncDuringMultiplayer) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at AIWar2NetworkSync.Client_AcceptPeriodicFactionExternalSyncDataThatJustOverrides (Arcen.Universal.ArcenDeserializationBuffer buffer) [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.AIW2.Core.AIWar2Networking.HandleMessage (System.Int64 UniqueMessageIDFromSender, System.Boolean IsFromSelfWithoutNetwork, System.UInt32 senderNetworkID, Arcen.Universal.ArcenNetworkMessageType CoreNetworkMessageType, Arcen.Universal.ArcenDeserializationBuffer buffer) [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.Universal.ArcenNetworkMessageHandler.HandleMessage (System.Boolean IsFromSelfWithoutNetwork, System.UInt32 senderNetworkID, Arcen.Universal.ArcenDeserializationBuffer buffer) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.AIW2.Core.SteamSocket+SteamClientConnectionManager.OnMessage (System.IntPtr data, System.Int32 size, System.Int64 messageNum, System.Int64 recvTime, System.Int32 channel) [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Steamworks.ConnectionManager.ReceiveMessage (System.IntPtr msgPtr) [0x00000] in <387e1e351a374f9c8eaf77f1a06cdf1b>:0
  at Steamworks.ConnectionManager.Receive (System.Int32 bufferSize) [0x00000] in <387e1e351a374f9c8eaf77f1a06cdf1b>:0
  at Arcen.AIW2.Core.SteamSocket.DoPerUpdateLogic () [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.Universal.ArcenNetworkAuthority.DoPerUpdateLogic () [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.Engine_Universal.OnUpdateFromMainThread () [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.AIW2.Core.ArcenGameControllerBase.BaseUpdate () [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at ArcenGameController.Update () [0x00000] in <14f1c75f68204a3f8c456fbd19057562>:0

3/27/2021 7:44:19 PM 2.764 CLIENT ERROR: Missing ArcenExternalDataPattern for EXTERNAL DATA - Faction with namespaceName '???'
  at System.Environment.get_StackTrace () [0x00000] in <9577ac7a62ef43179789031239ba8798>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, System.Boolean IncludeStackTrace, Arcen.Universal.Verbosity Verbosity, System.DateTime Timestamp) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, System.Boolean IncludeStackTrace, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenExternalDataLookup.DeserializeExternalDataFrom (Arcen.Universal.ArcenDeserializationBuffer Buffer, System.String ExternalDataHeader, System.Boolean IsForPartialSyncDuringMultiplayer) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at AIWar2NetworkSync.Client_AcceptPeriodicFactionExternalSyncDataThatJustOverrides (Arcen.Universal.ArcenDeserializationBuffer buffer) [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.AIW2.Core.AIWar2Networking.HandleMessage (System.Int64 UniqueMessageIDFromSender, System.Boolean IsFromSelfWithoutNetwork, System.UInt32 senderNetworkID, Arcen.Universal.ArcenNetworkMessageType CoreNetworkMessageType, Arcen.Universal.ArcenDeserializationBuffer buffer) [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.Universal.ArcenNetworkMessageHandler.HandleMessage (System.Boolean IsFromSelfWithoutNetwork, System.UInt32 senderNetworkID, Arcen.Universal.ArcenDeserializationBuffer buffer) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.AIW2.Core.SteamSocket+SteamClientConnectionManager.OnMessage (System.IntPtr data, System.Int32 size, System.Int64 messageNum, System.Int64 recvTime, System.Int32 channel) [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Steamworks.ConnectionManager.ReceiveMessage (System.IntPtr msgPtr) [0x00000] in <387e1e351a374f9c8eaf77f1a06cdf1b>:0
  at Steamworks.ConnectionManager.Receive (System.Int32 bufferSize) [0x00000] in <387e1e351a374f9c8eaf77f1a06cdf1b>:0
  at Arcen.AIW2.Core.SteamSocket.DoPerUpdateLogic () [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.Universal.ArcenNetworkAuthority.DoPerUpdateLogic () [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.Engine_Universal.OnUpdateFromMainThread () [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.AIW2.Core.ArcenGameControllerBase.BaseUpdate () [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at ArcenGameController.Update () [0x00000] in <14f1c75f68204a3f8c456fbd19057562>:0

<You put in defensive code into this c ode so you can now see which null reference it is. This was hit a bunch>
3/27/2021 7:45:27 PM 2.764 CLIENT CheckForShotAOEDetonation error at debugStage 4100: System.NullReferenceException: Object reference not set to an instance of an object
  at Arcen.AIW2.External.EntitySimLogicImplementation.CheckForShotAOEDetonation (Arcen.AIW2.Core.GameEntity_Shot ShotOrNull, Arcen.AIW2.Core.GameEntity_Squad TargetOrNull, Arcen.AIW2.Core.EntitySystem OriginSystemForShotOrNull, Arcen.AIW2.Core.ArcenSimContext Context) [0x005a6] in <530e5f0408bb463da28b713b904d5d16>:0

<A couple instances of this sort of error>
3/27/2021 7:55:47 PM 2.764 CLIENT Called ReadSplitMessage but the FragmentsReadSoFar was 1 and is not one lesss than the messageIndex of 3 that we just tried to append.
  at System.Environment.get_StackTrace () [0x00000] in <9577ac7a62ef43179789031239ba8798>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, System.Boolean IncludeStackTrace, Arcen.Universal.Verbosity Verbosity, System.DateTime Timestamp) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, System.Boolean IncludeStackTrace, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.DebugLogDestination Destination, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenDebugging.ArcenDebugLog (System.String Message, Arcen.Universal.Verbosity Verbosity) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenNetworkAuthority.ReadSplitMessage (System.Boolean IsFirstMessage, System.Boolean IsFromSelfWithoutNetwork, System.UInt32 senderNetworkID, Arcen.Universal.ArcenDeserializationBuffer buffer) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.UniversalMessageHandler.HandleMessage (System.Int64 UniqueMessageIDFromSender, System.Boolean IsFromSelfWithoutNetwork, System.UInt32 senderNetworkID, Arcen.Universal.ArcenNetworkMessageType CoreNetworkMessageType, Arcen.Universal.ArcenDeserializationBuffer buffer) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.ArcenNetworkMessageHandler.HandleMessage (System.Boolean IsFromSelfWithoutNetwork, System.UInt32 senderNetworkID, Arcen.Universal.ArcenDeserializationBuffer buffer) [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.AIW2.Core.SteamSocket+SteamClientConnectionManager.OnMessage (System.IntPtr data, System.Int32 size, System.Int64 messageNum, System.Int64 recvTime, System.Int32 channel) [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Steamworks.ConnectionManager.ReceiveMessage (System.IntPtr msgPtr) [0x00000] in <387e1e351a374f9c8eaf77f1a06cdf1b>:0
  at Steamworks.ConnectionManager.Receive (System.Int32 bufferSize) [0x00000] in <387e1e351a374f9c8eaf77f1a06cdf1b>:0
  at Arcen.AIW2.Core.SteamSocket.DoPerUpdateLogic () [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.Universal.ArcenNetworkAuthority.DoPerUpdateLogic () [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.Universal.Engine_Universal.OnUpdateFromMainThread () [0x00000] in <0dbe617830d34d3f9ea90d25adf53274>:0
  at Arcen.AIW2.Core.ArcenGameControllerBase.BaseUpdate () [0x00000] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at ArcenGameController.Update () [0x00000] in <14f1c75f68204a3f8c456fbd19057562>:0

<and>
3/27/2021 7:55:50 PM 2.764 CLIENT Called ReadSplitMessage but the FragmentsReadSoFar was 1 and is not one lesss than the messageIndex of 4 that we just tried to append.

<and then>
3/27/2021 8:12:16 PM 2.764 CLIENT BUG: entityTospawnat is null
3/27/2021 8:12:46 PM 2.764 CLIENT Exception in DoPerSecondLogic_Stage3Main_OnMainThreadAndPartOfSim for faction asd(Index 10), so will not run any more of those until a reload of the save.
System.NullReferenceException: Object reference not set to an instance of an object
  at Arcen.AIW2.External.SpecialFaction_Human+<>c__DisplayClass92_0.<HandleAutobuild>b__3 (Arcen.AIW2.Core.GameEntity_Squad commandStation) [0x0005e] in <530e5f0408bb463da28b713b904d5d16>:0
  at Arcen.AIW2.Core.EntityCollection.DoForEntities (Arcen.AIW2.Core.EntityRollupType Rollup, Arcen.AIW2.Core.GameEntity_Squad+ProcessorDelegate Processor) [0x00068] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.AIW2.Core.Faction.DoForEntities (Arcen.AIW2.Core.EntityRollupType Rollup, Arcen.AIW2.Core.GameEntity_Squad+ProcessorDelegate Processor) [0x00033] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
  at Arcen.AIW2.External.SpecialFaction_Human.HandleAutobuild (Arcen.AIW2.Core.Faction faction, Arcen.AIW2.Core.ArcenSimContext Context) [0x00107] in <530e5f0408bb463da28b713b904d5d16>:0
  at Arcen.AIW2.External.SpecialFaction_Human.DoPerSecondLogic_Stage3Main_OnMainThreadAndPartOfSim (Arcen.AIW2.Core.Faction faction, Arcen.AIW2.Core.ArcenSimContext Context) [0x00059] in <530e5f0408bb463da28b713b904d5d16>:0
  at Arcen.AIW2.Core.Faction.Safe_DoPerSecondLogic_Stage3Main_OnMainThreadAndPartOfSim (Arcen.AIW2.Core.ArcenSimContext Context) [0x00069] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0
3/27/2021 8:12:46 PM 2.764 CLIENT Unknown Exception in DoPerSecondLogic_Stage3Main_OnMainThreadAndPartOfSim for faction <main player>

<And finally>
3/27/2021 8:17:02 PM 2.764 CLIENT Exception in Client_AcceptDivergenceDataFromHost-DivergencesSection: debugStage: 1200 Exception: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
  at System.ThrowHelper.ThrowArgumentOutOfRangeException (System.ExceptionArgument argument, System.ExceptionResource resource) [0x00029] in <9577ac7a62ef43179789031239ba8798>:0
  at System.ThrowHelper.ThrowArgumentOutOfRangeException () [0x00000] in <9577ac7a62ef43179789031239ba8798>:0
  at AIWar2NetworkSync.Client_AcceptDivergenceDataFromHost (Arcen.Universal.ArcenDeserializationBuffer buffer) [0x00334] in <b94cbedd712f4d2e86b6547ccbfc8faa>:0

TagsNo tags attached.

Activities

BadgerBadger

Mar 28, 2021 2:11 pm

manager   ~0060948

Other weirdnesses.

Sometimes units appear to go to the wrong planet on MP clients. Lets say I have 3 planets in a line like A - B - C. A player whose ships were on B issued a move order to move to planet C. The host saw them moving correctly toward C, but the player reported that visually the ships appeared to go to A.

Also reported: An MP client reported that the planet screen was showing ships from an Allied faction (in this case, the Dyson) as Enemies instead of Allies. This player was playing the Necromancer.

Chris_McElligottPark

Mar 30, 2021 4:20 pm

administrator   ~0060973

Thanks! As I go:

* Properly fixed that index out of range exception that could still happen on WriteTechThatBenefits(). I was missing a -1 in there.

Chris_McElligottPark

Mar 30, 2021 4:25 pm

administrator   ~0060974

Back to the first one:

* Fixed some issues where death effect damage may have been overflowing, and in general where it would not serialize if it was negative. Now it just corrects to 0 damage rather than throwing an exception. This was during DeathEffectCausingDamageReceivedToEntity_Data mainly for multiplayer.

The other item was the second one.

Chris_McElligottPark

Mar 30, 2021 4:35 pm

administrator   ~0060975

Last host one:

* For some strange reason, it was still possible for us to hit Steam's "LimitExceeded" error when sending certain messages that include the entire world file. This was intermittent with clients in general.
** Overall it is true, we were trying to fragment savegames as little as possible, basically breaking them at around 500kb rather than something lower, knowing that Valve's limit on messages is 512k. But with our own headers being around 100 bytes or so, we still should have had plenty of room, so why exactly this was happening is not clear.
** What we've now done is shifted it so that we now target 400kb, which is VASTLY more room than needed. However, in terms of the grand scheme of sending world data, this is really paranoid in a way that should avoid the issue while not letting edge cases slip through. No headers are remotely near 112kb in size, so if we still get the error then this is something to do with how much is queued up in the steam networking buffer or somesuch.
** It's also worth noting that the difference in fragmenting 100kb one way or the other is essentially invisible in terms of performance, and this only affects initial connection.

Chris_McElligottPark

Mar 30, 2021 4:39 pm

administrator   ~0060976

The two deserializaton errors on client 0 are 100% errors in serialization in general, and I wonder what is happening there. Is that something that is easy to get to repeat? The errors that I'm actually seeing are nonsense, as they are essentially downstream from something else that was deserialized incorrectly.

It's not fast-blast data, which is a bummer, because that would have been my first guess. It's the limited faction sync data, and I am guessing that something in there has a special code branch for multiplayer that is not quite set up consistently, since this is not something affecting other parts of the game in SP. The fact that this was only hit a few times is very curious indeed.

Chris_McElligottPark

Mar 30, 2021 4:45 pm

administrator   ~0060977

* "CheckForShotAOEDetonation error at debugStage 4100" errors in multiplayer should now be fixed, or if not then it should give an even more specific error now. But knock on wood, should be fixed!

Chris_McElligottPark

Mar 30, 2021 4:48 pm

administrator   ~0060978

<and>
3/27/2021 7:55:50 PM 2.764 CLIENT Called ReadSplitMessage but the FragmentsReadSoFar was 1 and is not one lesss than the messageIndex of 4 that we just tried to append.

Holy @#$@# that's a terrifying message. This is implying... one of two things. Either:

1. Valve's messages are not arriving in an ordered fashion. Things should be way more broken if that's the case.

2. Actually when the four messages were sent and some were rejected on the host, it was one or two of the middle parts, and the first and final parts were both sent. This is actually way more likely, and would fit with the host error (maybe), so in that case this is kind of a non-bug.

But at first glance it looks terrifying. Essentially the host was blocked from sending the middle two fragments by steam, and so the client got parts 1 and 4 and was understandably confused.

Chris_McElligottPark

Mar 30, 2021 4:51 pm

administrator   ~0060979

Thanks!

* Fixed a "BUG: entityTospawnat is null" on multiplayer clients, but in the process also stopped them from trying to spawn the contents of a wave at all. That was just going to lead to strangeness and lag whenever a wave arrives. That data will be arriving by fast-blast from the host within a matter of milliseconds, and there's nothing for the client to calculate on their own.

Chris_McElligottPark

Mar 30, 2021 5:01 pm

administrator   ~0060980

Thanks!

* A HandleAutobuild() exception was happening on some multiplayer clients. It probably still will, but it will give way more informative errors now. There's some slight extra protections in there already, but probably not enough for whatever was going on.

Chris_McElligottPark

Mar 30, 2021 5:03 pm

administrator   ~0060981

Okay... that last line was also highly suspect, hmm. I... don't have a great way to fix this, so it's time to narrow things down with some literal canaries in the code. I have a good idea on how to handle that.

Chris_McElligottPark

Mar 30, 2021 5:36 pm

administrator   ~0060982

Okay, this is the last of it:

* "Canary code" has been introduced into several of the multiplayer messages. These may be taken back out later, but they really require very little processing and potentially a great amount of clarity, so we'll see.
** Essentially, using our extremely efficient sub-ASCII text encoding, we send a string at certain points in the multiplayer messages, and on the other end it tries to read that exact string. The key point here is that the client KNOWS what it should be receiving. If it receives something other than what it expects, then it will yell right away and throw an exception and stop processing that message.
** When it comes to our serialization format, it is blind and it is without markup or metadata. This is hyper efficient, but when we are reading naked bits... any string of 4 bites can maybe a byte, and any string of 16 bits can make some sort of small integer. We are reliant on clients and hosts knowing exactly what they are reading at any given time, and it's often not until we get a value that is clearly out of range later on that it actually dies. This can be VERY downstream from the actual divergence in data (we could have had 1, 10, 100, or even 1000s of reads that were invalid without knowing it).
** This sort of thing gives us an existential problem in that when we know there's an error in some sort of message, we have no way to know how to narrow it down. The only way to find the divergence is to get the client and the host to both write a log of what they sent, and then diff the two. The problem is always immediately obvious once we do that. However, we have no way of knowing when to log, when the problem is a very intermittent one. Full time logging would make the game unplayable, and we also can't retroactively log after discovering we have a problem (the host never even knows there is a problem).
** The one other way that we can find a problem like this is via manual code review, but that is extremely tedious and we could be looking through literally thousands of lines of code in dozens of files for a single send, since it could involve many external factions or even mods.
** To solve this existential problem, at certain points we can send a known unique series of ASCII characters, which get converted into a hyper-specific series of bits. We're talking about... let's say maybe a series of 72 bits. In the canary check, it would be unimaginably unlikely for all 72 bits to happen to be the right combination of on and off unless the data that has been read prior to now is all exactly correct. Thus we basically cut down dramatically on the amount of code that we have to review in order to find the problem spot, since we don't have logs.
** If the amount of code is still too much, then after the first round of canary checks, we may wind up needing to add more layers of more detailed canaries, potentially that are toggled on as a more lightweight form of message checking, since logs are really impossible to synchronize in a realtime multiplayer game. The data is just too voluminous for players to handle, and they have no way to pre-parse it for our use. The canary solution keeps things brief, to a single line of log code on just the client, and has minimal-to-no slowdown added (if we wind up adding lots of layers of canaries, that would be slowdown, and would be a toggle for when you suspect problems).
** Two things to be aware of with canaries:
*** If I make a mistake when entering a canary, then I may make multiplayer unplayable by accident. That could happen, and would just require a code change to fix it. This is difficult to test in advance of someone else firing up multiplayer, unfortunately.
*** If my canaries are all correct, multiplayer still may become unplayable if there's a bug that a canary is now finding. Essentially, an area of multiplayer might ONLY have been functional because of the nature of "any row of bits is a number," and that in turn would lead to all sorts of strange effects like... well, potentially ships going to the wrong location, or having very mysterious orders, that sort of thing. Those are all just lots of series of numbers, so its entirely plausible to have a functional -- but wrong and fill with lots of sync data request -- multiplayer game going in those circumstances. The canaries would make such a game unplayable, however, because they actually would find at least the general areas of the code and then halt reading of the message (since they know it's garbage after that, even if it's theoretically-playable garbage). This may actually mean a short-term reduction in multiplayer playability, is the bottom line. But it leads to long-term correctness, and the removal of strange MP-specific divergences that wind up sending more and more and more sync data trying and failing to fix itself. Aka over time it will also lead to less bandwidth usage.

Thanks!

Issue History

Date Modified Username Field Change
Mar 28, 2021 12:22 am BadgerBadger New Issue
Mar 28, 2021 12:22 am BadgerBadger Status new => assigned
Mar 28, 2021 12:22 am BadgerBadger Assigned To => Chris_McElligottPark
Mar 28, 2021 12:23 am BadgerBadger Description Updated
Mar 28, 2021 2:11 pm BadgerBadger Note Added: 0060948
Mar 30, 2021 4:20 pm Chris_McElligottPark Note Added: 0060973
Mar 30, 2021 4:25 pm Chris_McElligottPark Note Added: 0060974
Mar 30, 2021 4:35 pm Chris_McElligottPark Note Added: 0060975
Mar 30, 2021 4:39 pm Chris_McElligottPark Note Added: 0060976
Mar 30, 2021 4:45 pm Chris_McElligottPark Note Added: 0060977
Mar 30, 2021 4:48 pm Chris_McElligottPark Note Added: 0060978
Mar 30, 2021 4:51 pm Chris_McElligottPark Note Added: 0060979
Mar 30, 2021 5:01 pm Chris_McElligottPark Note Added: 0060980
Mar 30, 2021 5:03 pm Chris_McElligottPark Note Added: 0060981
Mar 30, 2021 5:36 pm Chris_McElligottPark Status assigned => resolved
Mar 30, 2021 5:36 pm Chris_McElligottPark Resolution open => fixed
Mar 30, 2021 5:36 pm Chris_McElligottPark Fixed in Version => Beta 2.765 The Hunt For Frigates And Civilians
Mar 30, 2021 5:36 pm Chris_McElligottPark Note Added: 0060982