When a chunk is loaded from disk that has already been generated,
the server has to promote the chunk through the system to reach
it's current desired status level.
This results in every single status transition going from the main thread
to the world gen threads, only to discover it has no work it actually
needs to do.... and then it returns back to main.
This back and forth costs a lot of time and can really delay chunk loads
when the server is under high TPS due to their being a lot of time in
between chunk load times, as well as hogs up the chunk threads from doing
actual generation and light work.
Additionally, the whole task system uses a lot of CPU on the server threads anyways.
So by optimizing status transitions for status's that are already complete,
we can run them to the desired level while on main thread (where it has
to happen anyways) instead of ever jumping to world gen thread.
This will improve chunk loading effeciency to be reduced down to the following
scenario / path:
1) MAIN: Chunk Requested, Load Request sent to ChunkTaskManager / IO Queue
2) IO: Once position in queue comes, submit read IO data and schedule to chunk task thread
3) CHUNK: Once IO is loaded and position in queue comes, deserialize the chunk data, process conversions, submit to main queue
4) MAIN: next Chunk Task process (Mid Tick or End Of Tick), load chunk data into world (POI, main thread tasks)
5) MAIN: process status transitions all the way to LIGHT, light schedules Threaded task
6) SERVER: Light tasks register light enablement for chunk and any lighting needing to be done
7) MAIN: Task returns to main, finish processing to FULL/TICKING status
Previously would have hopped to SERVER around 12+ times there extra.
Due to some complexity in mojangs complicated chain of juggling
whether or not a chunk should be unloaded when the last ticket is
removed, many chunks are remaining around in the cache.
These chunks are never being targetted for unload because they are
vastly out of view distance range and have no reason to be looked at.
This is a huge issue for performance because we have to iterate these
chunks EVERY TICK... This is what's been leading to high SELF time in
Ticking Chunks timings/profiler results.
We will now detect these chunks in that iteration, and automatically
add it to the unload queue when the chunk is found without any tickets.
This is for 2 reasons:
1) Ensuring our log4j is mostly loaded at OUR version.
I've seen stack traces with line numbers that do not match our version. This means that some
plugin has shaded in log4j and their loaded version is mixing with ours....
So by at least trying to load a bunch of log4j classes before we load plugins, we can be
more sure mixed versions are not loading.
2) If the jar file is replaced while the server is runnimg class not found errors galore
This will preloaod a bunch of classes commonly seen to error during shutdown due to this.
The goal here is to help let the server shutdown gracefully as possible. Some plugins will
still blow up here if they access a class that hadn't been loaded yet, but goal is to at least
stop freezing the shutdown process as it does with JLine and Log4j errors requiring an external kill.
Ideally you should not replace jars while the server is running, but it is something that happens in
development for testing.
Updated test server to do a copy though to avoid this happening in Paper development.
While this method has async in it's name, it's not actually meant
to be called asynchronously.... It just means IT will load the chunk
asynchronously without blocking main.
So fix this so that if a plugin calls it async, it forces the request back to main thread.
Fixed issues where urgent and prioritized chunks didn't actually
always get their priority boosted correctly....
Properly deprioritize non ticking chunks.
Limit recursion on watchdog prints to stop flooding as much
Remove neighbor priorities from watchdog to reduce information
reduce synchronization duration so that watch dog won't block main should main actually wake up
probably fixed a deadlock risk in watchdog printing also that was leading to crashes
fixed chunk holder enqueues not being processed correctly
added async catchers in some locations that should not be ran async
Fixed upstream bug where VITAL callbacks that must run on main actually could
sometimes run on the server thread pool causing alot of these nasty bugs we've seen lately!
This build will provide massive improvements to stability as well as even faster
sync chunk load/gens now that priority is correctly set.
Fixes#3435
I believe this brings us back to stable. A lot of complexity was
learned about juggling priorities.
We were essentially promoting more chunks to urgent than really
needed to be urgent.
So this commit adds a lot more logic to juggle neighbor priorities
and demote their priority once they meet the requirements needed of
them.
This greatly improves the performance of "urgent" chunks".
Fixes#3410Fixes#3426Fixes#3425Fixes#3416
Blow up if a plugin tries to mutate visibleChunks directly and prevent them
from doing so.
Also provide a safe get call if any plugins directly call get on it so
that it uses the special logic to check pending.
Also restores ABI for the visibleChunks field back to what it was too.
Additionally, remove the stack trace from Timings Stack Corruption for any
error thrown on Minecraft Timings, and tell them to get the error ABOVE this
instead, so people stop giving us useless error reports.
Also fixes a memory leak when the source map down sizes but dest map didn't,
which resulted in lingering references to old chunk holders.
Fixes#3414
Fixed a few bugs, and made numerous improvements.
Fixed issue where a sync chunk load could have its ticket removed and the
priority ticket could expire...
Still not perfect there but better than before.
Also fixed few other misc issues such as watchdog cpu usage, chunk queue update
had risk of double enqueue due to it no longer being a set.
Added much more information about chunk state to watchdog prints.
I see some more room for improvement even, but this is much better than before.
Fixes#3407Fixes#3411Fixes#3395Fixes#3389
Mark chunks that are blocking main thread for world generation as urgent
Implements a general priority system so that chunks that are sorted in
the generator queues can prioritize certain chunks over another.
Urgent chunks will jump to the front of the line, ensuring that a
sync chunk load on an ungenerated chunk does not lag the server for
a long period of time if the servers generator queues are filled with
lots of chunks already.
This massively reduces the lag spikes from sync chunk gens.
Then we further prioritize loading order so nearby chunks have higher
priority than distant chunks, reducing the pressure a high no tick
view distance holds on you.
Chunks in front of the player have higher priority, to help with
fast traveling players keep up with their movement.
This commit also improves single core cpu scenarios in that we will
now automatically disable Async Chunks as well as Minecrafts thread
pool.
It is never recommended to use async chunks on a single CPU as context
switching will be slower than just running it all on main.
This also bumps the number of server worker threads by default too.
Mojang does not utilize the workers in an effecient manner, resulting
in them using barely any sustained CPU.
So give it more workers so more chunks can be processed concurrently
This change also improves urgent chunk loading, so players flying into
unloaded chunks will hurt a little bit less (but still hurt)
Ping #3395#3363 (Not marking as closed, we need to make prevent moving work)
