Worries

So... Last September I was working on implementing the first LAN features into Striker Ball. This October I continued the work with a bit of a concern weighing on me as I built out the user interface and game flow. I was aware of an unfortunate de-sync that occurred during testing where the players were eventually in different positions on different computers so they weren't even playing the same game anymore.

When your building a LAN game in Bones you have to make sure that your game is deterministic across all computers during the LAN session. For the most part this means making sure you only use functions that output the same data given the same input. I couldn't find anything right away that suggested a possible cause for the de-sync. If it wasn't my fault there was a possibility, given Bones' limited testing, that there could be a mistake in the Bones networking code. This was the last thing I wanted at the time since it would take extra digging and very possibly was out of my wheel house.

Breaking It Down

As I finished everything else necessary for the month, I had nothing else to do but try and fix the de-sync before moving on. We really want LAN capability for our games, so I held my breath and started to dig in. From what I saw, there was an uncommon de-sync of the player's positions when colliding with each other. I tried several times to find a way to reproduce the de-sync intentionally and consistently, but it seemed pretty much random. Every time I thought I found a method to repeat it, it eventually failed to reproduce. This seemed to suggest it was performance dependent, which was an idea later reinforced when tested between two different computers (I've done most of my LAN testing on the same computer). One computer is much slower than the other and the desync happened every time. Knowing it was performance related though still didn't give us a good bearing on the root cause of the problem.

I took the next step and began making a small logging resource. My logger created a file for every session, pairing each frame that Bones considered synced with the data of my choice from that frame. After testing and getting a de-sync, I then had two files from my two connected sessions that I could easily compare to each other using VSCodium. Once I got it working, the first good sign was that I found the controller inputs were logging the same during the de-sync. This took a lot of stress off of Bones being the issue because the only thing that Bones is actually syncing is inputs. It didn't mean that Bones couldn't be the problem, but the more complicated part of Bones was not the issue.

The signs pointed to the problem being inside Striker Ball, but WHERE? I added value after value to my logger to find where it was de-syncing, and after some confirmation, I found it appeared to be only de-syncing on one frame that somehow caused a slight player positioning discrepancy. This narrowed it down a bit and made sense of some of the strange behavior I was getting. It meant that the problem wasn't my collision systems because those ran every frame. It only presented itself this way because my collisions were circles, making an indistinguishable change have a visible result. It also made sense of the odd behavior I noted early on where it seemed to either choose to de-sync from the beginning or choose not to for the rest of the session. The question still remained, what exactly needed to be fixed? So, I logged more values and got closer to the root of the problem. I found that the game countdown timer readings were the earliest thing to de-sync. After some puzzling out of the timer's odd behavior and looking at where the timer was started, it hit me.

Solutions

When starting the game session I used a local fade-out and fade-in that started the countdown when it was done. These fade functions used local timers that were not synced on the network game session, so any slight performance difference across different applications resulted in the players starting to run at different times. It was a huge relief to finally find it, and fix it myself. It was nearly a two line fix, but not quite. I just had to make sure that I didn't do anything to the session after it started. I waited to start the match until the fade-out was done, then started the match session with its' own fade-in functionality so its' timer could sync with the other applications.

I found this out only two days ago just in time for me to write this and push the latest release before the end of the month.

Issues

As ecstatic as I am over having fixed the issue, there is still lots of polish to be done, inside and out, that I can't wait to finish. I also want to take the opportunity to note some of the issues present in the game that we haven't yet gotten the chance to mention.

First, there might be some funky controls in the menu, and while we don't have keyboard controls yet, some of the menu buttons respond to the mouse and override the controller navigation. I've needed some of these things for testing and haven't had the chance to clean them up or implement all the controls properly. Once we have all the features it will be easier to focus on purifying the experience.

On some platforms, including the web version, the game skips the first input for every button on every gamepad for some reason. This is the first thing after this release I want to fix but it's not a problem with Striker Ball. The problem is either Bones or one of its' resources which makes it harder for me to get into. I may still have a haphazard solution, but if it doesn't work I'll have to wait for some help to fix it later.

Another thing is, we haven't thoroughly tested the game ourselves on different platforms and computers, so we don't know how it will perform in a lot of situations. If you find yourself trying out the game, please feel free to contact us about your experience in any capacity.

Written by, Latency: Lead game programmer at K-tech Studio