On June 9th we began to notice severe instability on PvE. The server was only able to stay online for a few minutes at a time before crashing. This began in the early evening (EST) and lasted for about 30 minutes before stabilizing. Roughly five hours later the issue returned in greater severity. After several unsuccessful hours of attempted debugging with the server open, we opted to enable the whitelist so we could perform more control and intensive testing.
With information collected we determined that the server instability was being caused by newly generated strongholds in both the Normal and Peaceful worlds. On the first generation attempt the server would crash, but on any subsequent returns to the same location after rebooting the server, the stronghold would generate correctly and the server remain online.
In the end, we determined this to be a map compatibility issue. All our maps were generated in MC 1.13.2 and converted to 1.14 compatibility, but never to 1.14.1 or 1.14.2 compatibility. To this point we had been relying on the server to make on-the-fly conversions of old map data when the chunks were loaded by users. This for some reason created issues when generating new chunks containing strongholds. To resolve this we ran map-wide conversions of all chunk data to 1.14.2 compatibility. Spigot has a launch flag, --force-upgrade, for to perform this function. The operation took roughly 3 hours to complete - currently our maps our 30GB cumulative.
Thank you to all those who helped with data collection and debugging.
We apologize to everyone for the unexpected downtime.
