Commit Graph

4748 Commits

Author SHA1 Message Date
Aikar
ff4ca31f0d
Fix premature unloading of neighbor chunks for world gen
We must check the level tracker as ticket levels add "virtual"
tickets to neighbors.

Also added neighbor tracking during generation to be extra safe.

Fixes #3465
Fixes #3451
Fixes #3459
2020-05-28 22:59:40 -04:00
Aikar
fde81356bb
Fix Chunks unloading immediately for priority access - Fixes #3461
Adding a priority ticket was apparently causing the level tracker
to immediately unload the chunk.

That code is scary
2020-05-28 22:57:27 -04:00
Aikar
9980f427db
Fix some logic on villager poi max attempts
Hopefully will help #3462
2020-05-28 20:45:36 -04:00
Aikar
919ab67a13
Fix concurrency issue in light engine (Vanilla bug)
Mojang implemented a cache like chunks have, but this cache
is accessed by multiple threads and is totally not safe.

So just remove it

Fixes #3466

Also missed a pooled nibble release, so slid that in there too.
2020-05-28 20:37:40 -04:00
Spottedleaf
0d5ec65372
Cleanup async chunks (#3456)
Co-authored-by: Spottedleaf <Spottedleaf@users.noreply.github.com>
2020-05-28 19:34:58 -04:00
Shane Freeder
c3c005c947
Updated Upstream (Bukkit/CraftBukkit) (Fixes #3443)
Upstream has released updates that appears to apply and compile correctly.
This update has not been tested by PaperMC and as with ANY update, please do your own testing

Bukkit Changes:
d6db2839 #497: Add Material#getCraftingRemainingItem() to get item left behind after crafting
1c2eaf96 SPIGOT-5748: Add instant effect potion break to the Effect enum

CraftBukkit Changes:
aae46f82 #665: Add test for Material#getCraftingRemainingItem()
cfeef75c SPIGOT-5749: ItemMeta serializing to YAML not saving black colour code
eb1b19d9 SPIGOT-5748: Add instant effect potion break to the Effect enum
2020-05-28 12:22:50 +01:00
Aikar
b0b54e4ef3
Optimize Villager Pathfinding - Massive Villager Improvement
This change reimplements the entire BehaviorFindPosition method to
get rid of all of the streams, and implement the logic in a more sane way.

We keep vanilla behavior 100% the same with this change, just wrote more
optimal, as we can abort iterating POI's as soon as we find a match....

One slight change is that Minecraft adds a random delay before a POI is
attempted again. I've increased the amount of that delay based on the distance
to said POI, so farther POI's will not be attempted as often.

Additionally, we spiral out, so we favor local POI's before we ever favor farther POI's.

We also try to pathfind 1 POI at a time instead of collecting multiple POI's then tossing them
all to the pathfinder, so that once we get a match we can return before even looking at other
POI's.

This benefits us in that ideally, a villager will constantly find the near POI's and
not even try to pathfind to the farther POI. Trying to pathfind to distant POI's is
what causes significant lag.

Other improvements here is to stop spamming the POI manager with empty nullables.
Vanilla used them to represent if they needed to load POI data off disk or not.

Well, we load POI data async on chunk load, so we have it, and we surely do not ever
want to load POI data sync either for unloaded chunks!

So this massively reduces object count in the POI hashmaps, resulting in less hash collions,
and also less memory use.

Additionally, unemployed villagers were using significant time due to major ineffeciency in
the code rebuilding data that is static every single invocation for every POI type...

So we cache that and only rebuild it if professions change, which should be never unless
a plugin manipulates and adds custom professions, which it will handle by rebuilding.
2020-05-27 01:35:36 -04:00
Aikar
33a2134b65
Workaround for really evil plugins breaking sounds
Some plugins are doing really really bad things to worlds breaking the
ability to send sounds to some users.

So creating another reference to the player chunk map that plugins wont be breaking, and
print a stack trace at world creation if we ever get an expected world state to identify
who is doing it!

If we encounter this illegal state, we fall back to the old method of sending sounds, so
sending sounds will still work, just less effecient.
2020-05-27 01:00:43 -04:00
Aikar
b88ee11f5a
Don't unregister the primary dimensions on world unload 2020-05-27 00:15:57 -04:00
Aikar
21f9efa2ca
Improve implementation of chunk leak clean to not run as often 2020-05-25 21:42:53 -04:00
Aikar
7ee711460f
Add missing null check for structure start chunk access
Spigot made structure start not load chunks, but forgot to null check
the result...

This likely never blew up before due to the chunk leak issue, but now
that leaky chunks are cleaned up, it was identified.
2020-05-25 14:19:11 -04:00
Aikar
b12177fb04
One more to seal the deal to improve unloading inactive chunks
While last was mostly there, still had some slight risk of unloading
before it was fully finished.

So just going to bump the delay to 3 minutes to be safe. Better than
forever at least.

Was really hoping we could unload them as soon as they were done to
any memory prematurely promoting to old generation, but guess we can't.
2020-05-25 13:09:56 -04:00
Aikar
78ee3b4e88
Restore a condition I accidently removed in last build 2020-05-25 12:53:40 -04:00
Aikar
88a6882930
Fix unloading inaccessible chunks too fast regressing gen speed
A chunk was loaded but not yet finished in use and was unloaded too early.

This caused it to be reloaded again or caused crashes.

Now also check if the chunk pops out of the unload queue that it also
doesn't now have a ticket either.
2020-05-25 12:36:19 -04:00
Aikar
210a32f26f
Unload leaked Cached Chunks
Due to some complexity in mojangs complicated chain of juggling
whether or not a chunk should be unloaded when the last ticket is
removed, many chunks are remaining around in the cache.

These chunks are never being targetted for unload because they are
vastly out of view distance range and have no reason to be looked at.

This is a huge issue for performance because we have to iterate these
chunks EVERY TICK... This is what's been leading to high SELF time in
Ticking Chunks timings/profiler results.

We will now detect these chunks in that iteration, and automatically
add it to the unload queue when the chunk is found without any tickets.
2020-05-25 11:12:22 -04:00
Aikar
c9795e9221
Fix Spigot bug with chunk unloading
Spigot inserted their Slack Activity Accountant in the wrong location
resulting in a chunk being removed from the unload queue, inserted into
the unload map, but never calling the function to finish the removal....

This caused the chunk to become stuck in the unload map if ever hit, because
the unload map was meant to be a TEMPORARY location while it was saving.

Fix this by abort iteration AFTER the current chunk is finisehd processing

Also, improve how aggressive we are at unloading chunks, targetting 10% per tick instead.
These saves are asynchronous so there should be less of a hit here.
2020-05-25 11:09:44 -04:00
Aikar
b6cf80ee66
Preload important classes such as Logger and JLine
This is for 2 reasons:
1) Ensuring our log4j is mostly loaded at OUR version.
   I've seen stack traces with line numbers that do not match our version. This means that some
   plugin has shaded in log4j and their loaded version is mixing with ours....
   So by at least trying to load a bunch of log4j classes before we load plugins, we can be
   more sure mixed versions are not loading.

2) If the jar file is replaced while the server is runnimg class not found errors galore
   This will preloaod a bunch of classes commonly seen to error during shutdown due to this.

   The goal here is to help let the server shutdown gracefully as possible. Some plugins will
   still blow up here if they access a class that hadn't been loaded yet, but goal is to at least
   stop freezing the shutdown process as it does with JLine and Log4j errors requiring an external kill.

   Ideally you should not replace jars while the server is running, but it is something that happens in
   development for testing.

Updated test server to do a copy though to avoid this happening in Paper development.
2020-05-24 13:19:08 -04:00
Aikar
ea1a909c80
Fix netty version to 50 and restore epoll native support
Accidently used the snapshot, as well as needed to exclude the
existing old 25 build of libnetty so that we load the newer build of
epoll instead.

Thanks to 56738 for the catch.
2020-05-24 13:07:53 -04:00
Aikar
10502558e9
Workaround for some hacky environments that mess up things
2 people had issues where some plugin is doing some reallly insane NMS hackery
that created invalid worlds, which caused some errors...

Really don't understand what in the world they did, but putting in a dumb guard that
shouldn't even be necessary to just not send the sound effect rather than erroring.
2020-05-23 22:27:37 -04:00
Aikar
8b2a437b9e
Fix plugins calling getChunkAtAsync asynchronously
While this method has async in it's name, it's not actually meant
to be called asynchronously.... It just means IT will load the chunk
asynchronously without blocking main.

So fix this so that if a plugin calls it async, it forces the request back to main thread.
2020-05-23 18:38:29 -04:00
Aikar
c8055836e8
Update Netty to 4.1.51
Minecraft's Netty version was severely out of date. There has been
numerous security fixes, bug fixes, and likely performance fixes
since the version Minecraft uses (4.1.25).

This fixes some known issues with "Closed Channel" spam.

Fixes #3388
2020-05-23 17:58:45 -04:00
Aikar
64cfcf3e4b
Optimize sending packets to nearby locations (sounds/effects)
Instead of using the entire world or player list, use the distance
maps to only iterate players who are even seeing the chunk the packet
is originating from.

This will drastically cut down on packet sending cost for worlds with
lots of players in them.

Closes #3437
2020-05-23 17:17:14 -04:00
Aikar
cf82dad399
Fix Non Full Status Chunk NBT Memory Leak
Any full status chunk that was requested for any status less than full
would hold onto their entire nbt tree and every variable in that function.

This was due to use of a lambda that persists on the Chunk object
until that chunk reaches FULL status.

With introduction of no tick, we greatly increased the number of non
full chunks so this was really starting to hurt.

We further improve it by making a copy of the nbt tag with only the memory
it needs, so that we dont have to hold a copy to the entire compound.
2020-05-23 01:40:27 -04:00
Aikar
5a12515a42
Update Log4J Configuration file to stop truncating stack traces
This should help greatly (as long as this change works...) in
understanding an exception when it doesn't get truncated with
"... and 14 more" at a vital point of the stack trace.
2020-05-22 23:52:56 -04:00
Aikar
7c001d64a5
More Improvements to Chunks
Fixed issues where urgent and prioritized chunks didn't actually
always get their priority boosted correctly....

Properly deprioritize non ticking chunks.
Limit recursion on watchdog prints to stop flooding as much
Remove neighbor priorities from watchdog to reduce information
reduce synchronization duration so that watch dog won't block main should main actually wake up
probably fixed a deadlock risk in watchdog printing also that was leading to crashes
fixed chunk holder enqueues not being processed correctly
added async catchers in some locations that should not be ran async

Fixed upstream bug where VITAL callbacks that must run on main actually could
sometimes run on the server thread pool causing alot of these nasty bugs we've seen lately!

This build will provide massive improvements to stability as well as even faster
sync chunk load/gens now that priority is correctly set.

Fixes #3435
2020-05-22 23:52:56 -04:00
Aikar
e1c451968c
Fix high memory use of non ticking chunks
The nibble pooling for NBT Tags was 'semi' leaked from loaded chunks
as we store the NBT Tag of Tile Entities in a Chunk, but don't process
them and remove them until chunk reaches Entity Ticking status....

This caused some phantom references to persist causing high memory use
of these chunks.

So I just got rid of pooling from NBT deserialization and we'll have to
take the hit on memory allocations there because too many cascading concerns
with anyone using NBT Tag Byte Arrays.

Fixes #3431
2020-05-22 23:52:51 -04:00
Aikar
ee9f0d513f
Fix another case of breaking blocks causing sync chunk loads 2020-05-22 18:57:48 -04:00
Aikar
6009ba8f44
Drop AABB limit patch until it can be tested more
Fixes #3428
2020-05-22 02:21:32 -04:00
Aikar
0e9c24e595
Fix log spam about Hanging entities bounding boxes
Nothing really "Broke" them, just a lot of log spam. This will clean that up
2020-05-22 02:04:00 -04:00
Aikar
83fadad7da
Fix conversion for deserializing raw nbt itemstacks - Fixes #3424 2020-05-22 01:08:18 -04:00
Aikar
4d38ee111f
Many fixes and improvements to chunk prioritization
I believe this brings us back to stable. A lot of complexity was
learned about juggling priorities.

We were essentially promoting more chunks to urgent than really
needed to be urgent.

So this commit adds a lot more logic to juggle neighbor priorities
and demote their priority once they meet the requirements needed of
them.

This greatly improves the performance of "urgent" chunks".

Fixes #3410
Fixes #3426
Fixes #3425
Fixes #3416
2020-05-22 01:03:42 -04:00
Aikar
281181c7c7
Use saner Entity bounding box limits
CB only protected from > 64 but there's no reason an entity should ever
be more than 2x its width or 1x height as the BB is supposed to represent
the entity size.

BB is / 2 to calculate position.
2020-05-22 00:40:56 -04:00
Aikar
edd6b6a2ba
Protect the visible chunk map from plugins touching it, trim Timing Errors
Blow up if a plugin tries to mutate visibleChunks directly and prevent them
from doing so.

Also provide a safe get call if any plugins directly call get on it so
that it uses the special logic to check pending.

Also restores ABI for the visibleChunks field back to what it was too.

Additionally, remove the stack trace from Timings Stack Corruption for any
error thrown on Minecraft Timings, and tell them to get the error ABOVE this
instead, so people stop giving us useless error reports.

Also fixes a memory leak when the source map down sizes but dest map didn't,
which resulted in lingering references to old chunk holders.

Fixes #3414
2020-05-22 00:39:16 -04:00
Aikar
18c686576b
Optimize performance of object pool
synchronized arraydeque ends up still being way faster.
Kinda shocked how much that strategy was using, it wasn't really
that complicated... but oh well, this is even simpler and not
seeing blocked threads show up at all in profiling because
the lock is held for such a short amount of time.

also because most uses are on either server thread pool or chunk load pool.

Also optimize the pooling of nibbles to not register Cleaner's
for Light Engine directed usages, as we know we are properly
controlling clean up there, so we don't need to rely on GC.

This will return them to the pool manually, saving a lot of Cleaners.

Closes #3417
2020-05-20 21:45:43 -04:00
Aikar
7e1525ea2d
Many improvements to chunk prioritization and bug fixes
Fixed a few bugs, and made numerous improvements.
Fixed issue where a sync chunk load could have its ticket removed and the
priority ticket could expire...
Still not perfect there but better than before.

Also fixed few other misc issues such as watchdog cpu usage, chunk queue update
had risk of double enqueue due to it no longer being a set.

Added much more information about chunk state to watchdog prints.

I see some more room for improvement even, but this is much better than before.

Fixes #3407
Fixes #3411
Fixes #3395
Fixes #3389
2020-05-20 05:13:01 -04:00
Aikar
c82b292ab0
Fix pooled buffer leak resulting in dynmap black spots - Fixes #3386
Dynmap accessed the raw bytes because it utilized NBT locally, but the
NBTTagcompound was garbage collected while the bytes were still being used.

This will return getBytes() back to being safe, and add a new PoolSafe method
that will prevent the additional allocations for general chunk loading.

Also fixed applyPatches for people with paths in their working directory
if they have mcdev sources built.
2020-05-20 00:55:30 -04:00
stonar96
6327447259
Fix ./paper edit continue for Windows 2020-05-19 23:14:34 -04:00
stonar96
eb5a3058dd
Fix path in CONTRIBUTING.md (#3406) 2020-05-19 23:08:37 -04:00
syeyoung
f6ed326dc1
Fix a small error in CONTRIBUTING.md (#3403) 2020-05-19 23:07:34 -04:00
Aikar
614a664bd3
Implement Chunk Priority / Urgency System for Chunks
Mark chunks that are blocking main thread for world generation as urgent

Implements a general priority system so that chunks that are sorted in
the generator queues can prioritize certain chunks over another.

Urgent chunks will jump to the front of the line, ensuring that a
sync chunk load on an ungenerated chunk does not lag the server for
a long period of time if the servers generator queues are filled with
lots of chunks already.

This massively reduces the lag spikes from sync chunk gens.

Then we further prioritize loading order so nearby chunks have higher
priority than distant chunks, reducing the pressure a high no tick
view distance holds on you.

Chunks in front of the player have higher priority, to help with
fast traveling players keep up with their movement.

This commit also improves single core cpu scenarios in that we will
now automatically disable Async Chunks as well as Minecrafts thread
pool.

It is never recommended to use async chunks on a single CPU as context
switching will be slower than just running it all on main.

This also bumps the number of server worker threads by default too.
Mojang does not utilize the workers in an effecient manner, resulting
in them using barely any sustained CPU.

So give it more workers so more chunks can be processed concurrently

This change also improves urgent chunk loading, so players flying into
unloaded chunks will hurt a little bit less (but still hurt)

Ping #3395 #3363 (Not marking as closed, we need to make prevent moving work)
2020-05-19 04:09:37 -04:00
Mariell
c8028d1c76
Fix data version check for ItemStack serialization (#3394)
The expected version should be equal to or newer than the one stored.

Although Aikar claims he did this on accident (and NOT my ligatures!), I
claim this is all a big conspiracy by followers of the Taco cult.
2020-05-18 12:24:06 -05:00
Aikar
9254a80a1b
Fix race condition reintroduced in Prioritize class loader patch 2020-05-18 01:18:44 -04:00
Mariell Hoversholm
6f196fe701
Add Raw Byte ItemStack Serialization
Serializes using NBT which is safer for server data migrations than bukkits format.
2020-05-17 00:52:24 -04:00
Spottedleaf
df43f82838
Allow server startup for those poor people running <1G Xmx
Clamp logic for poolsize so we dont pass zero
2020-05-17 00:32:37 -04:00
Spottedleaf
3c9b65ef9d
Fix cases where no-tick < tick view distance 2020-05-17 00:24:49 -04:00
MeFisto94
72f89a076a
Workaround for Client Lag Spikes (MC-162253)
When crossing certain chunk boundaries, the client needlessly
calculates light maps for chunk neighbours. In some specific map
configurations, these calculations cause a 500ms+ freeze on the Client.

This patch basically serves as a workaround by sending light maps
to the client, so that it doesn't attempt to calculate them.
This mitigates the frametime impact to a minimum (but it's still there).
2020-05-17 00:21:42 -04:00
Mariell Hoversholm
3f941c0cce
Add option for console having all permissions
Overrides permissions set to default false (which op does not get)
2020-05-16 23:55:54 -04:00
Mariell Hoversholm
d2ae4658ec
Add permission for command blocks 2020-05-16 23:48:54 -04:00
Mariell Hoversholm
9f8ae5cb61
Prioritise own classes where possible
This adds the server property `Paper.DisableClassPrioritization` to disable
prioritization of own classes for plugins' classloaders.

This value is by default not present, and this will therefore break any
plugins which abuse behaviour related to not using their own classes
while still loading their own. This is often an issue with failing to
relocate or shade properly, such as when shading plugin APIs like Vault.

A plugin's classloader will first look in the same jar as it is loading
in for a requested class, then load it. It does not re-use other
plugins' classes if it has the chance to avoid doing so.

If a class is not found in the same jar as it is loading for and it does
find it elsewhere, it will still choose the class elsewhere. This is
intended behaviour, as it will only prioritise classes it has in its own
jar, no other plugins' classes will be prioritised in any other order
than the one they were registered in.

The patch in general terms just loads the class in the plugin's jar
before it starts looking elsewhere for it.
2020-05-16 23:33:47 -04:00
kickash32
74466412ee
Check portal restrictions when entering end gateways 2020-05-16 23:28:59 -04:00