It's been three weeks since we update to Minecraft 1.14 and we've still not reached the same level of stability we had for Minecraft 1.13.2. However, we have made significant progress in returning to that level of stability. So far we've;
- Disabled Towny's claimed-area mob removal task.
- Increased the interval between auto-saves.
- Migrated Towny, CoreProtect, and LWC to use a local MariaDB server.
- Disabled DiscordSRV (#gamechat) was disabled because the websocket looked like it was causing server hangs, but was added back when this was found to not be the case.
- LuckPerms was reconfigured to use a local database because the MariaDB requests looked like they were causing server hangs. LuckPerms is now back to using our network-wide MariaDB permissions database as it doesn't look like it was actually causing issues.
- Migrated PvE from a dual-core Intel based system to a quad-core Ryzen based system.
- Given PvE an extra 15GB of Disk space for world storage.
- Given PvE an extra 1.5GB of RAM.
- Optimized entity ticking schedules with Spigot so we don't overflow on as many ticks.
Outstanding Issues ### (UPDATE: See below)
Right now we're aware of one major Spigot bug that has been causing a significant amount of lag and connection timeouts. It was reported to Spigot by a plugin developer and a full technical explanation is available here.
In summary, Mojang changed how the Minecraft server handles unloaded world chunks and the entities in those chunks. In certain circumstances it's possible for a chunk to be unloaded by the server, but for the entities in that chunk to remain in server memory. These entities remain in server memory and, as far as we can tell, continue to be at least partially ticked. Over time the entity build up becomes pretty significant, we've seen as many as 13,000 entities on the server before memory resources are exhausted and the RreadTimedOut Exceptions begin.
Restarting the server clears all these entities and we're looking into writing a plugin that might be able to detect at least some of these entities and kill them off as a stop-gap fix. We've tried conventional methods like /killall and difficulty cycles, but because these entities are in limbo and not fully active, they're not removed by these conventional methods.
We're compiling the latest builds of Spigot daily and pushing the out to our servers so as soon as a fix is implemented we'll try and implement it. However, this is a high probability that a potential fix for this issue would break several core plugins like Towny and WorldGuard that rely heavily on #CraftWorld and #World functionality.
We'll continue to post updates and changes as their made and again thank everyone for sticking with us through the crashes.
