Commit Graph

26 Commits

Author SHA1 Message Date
Aurora Lahtela 776fa3327f Fixed issues with peak player count query optimization 2023-02-13 18:23:38 +02:00
Aurora Lahtela 226bc20e3f Speed up /v1/network/servers by 66%
Benchmark: endpoint (fetchPeakPlayerCount is the highest contributor)
- Before: 10 seconds (1s * server count)
- Before: 3 seconds (332ms * server count)
2023-02-12 10:37:52 +02:00
Aurora Lahtela 38785a9505 Added Untrusted-annotation to be more careful around user given data
- Fixed SQL-injection vulnerability in an endpoint
- Fixed XSS on Whitelist deny 403 page
- Fixed XSS on Internal Error 500 page if untrusted data ends up in exception message
2023-01-14 23:25:35 +02:00
Aurora Lahtela 0a14d17d7a Fixed issues related to MySQL ONLY_FULL_GROUP_BY
MySQL breaks GROUP BY syntax of standard SQL, and allows arbitrary columns with aggregate functions.
The ONLY_FULL_GROUP_BY stops this, instead raising an error. Since the SQL was originally designed
with this mode on, restricting the SQL broke the code in a few places.

Adding the extra group by clauses solves the issue without effecting query results.

These issues will be caught by MySQLTest in the future, since the issues could be reproduced by
enabling ONLY_FULL_GROUP_BY mode.

Affects issues:
- Fixed #2619
2022-09-25 10:30:13 +03:00
Aurora Lahtela ead1b401bc Created a benchmark utility 2022-09-07 15:49:53 +03:00
Aurora Lahtela 92910d647a Revert API jar to Java 8 and update to Java 11 APIs 2022-06-18 11:31:53 +03:00
Aurora Lahtela b74e338721
Query optimizations (#2320)
* Optimized network ping table query
* Removed icon id selection subqueries

- Take the icon ID into memory when the icons are stored

* Fix typos in the optimized ping table query
* Optimize server ping table query
* Attempt to optimize /v1/servers tps data query
* Optimize ping and geolocations tables uuid -> user_id foreign key
* Prevent Plan from crashing if patching takes too long

- HikariCP auto commit was true for some reason even though all transactions have commit mechanism built-in.
- The setting was reset during connection recreation and that could cause an index out of bounds error.

* Reduce try-nesting in ExecStatement
* Use user_id and server_id instead of uuid for plan_world_times table
* Use user_id and server_id instead of uuid for plan_sessions and plan_user_info table
* Fix more issues and test queries used by Query Filters
* Use deferRender for data tables to load data into the table faster
* Swap uuids to user ids for query page filters

Fixes an issue where SQL is too big to execute

Affects issues:
- #2196
2022-04-09 20:52:29 +03:00
Risto Lahtela 4b24cf1cdd Order by different date in start date query
Affects issues:
- Possibly fixed #2080
2021-09-18 21:16:23 +03:00
Risto Lahtela 84fa3ad4cd
Added server uptime (#2012)
* Wrote a query and utility for getting server uptime
* Added current_uptime to json endpoints
* Load current uptime on the website
* Moved nukkit repo to nukkit module
* Added equals and hashcode to QueryStatement and QueryAPIQuery
* Remove dependency on codemc repository

Affects issues:
- Close #1845
2021-07-18 10:43:38 +03:00
Risto Lahtela 13823c044a
Network performance tab (#2009)
* Fixed disk medium threshold not showing color
* Added 'serverName' and 'serverUUID' to optimizedPerformance endpoint
* Added /v1/network/listServers endpoint
* Added /v1/network/performanceOverview?servers endpoint
* Hide negative values from performance graphs
* Allow json cache bypass by not providing timestamp parameter in URIQuery
* Ignore negative values in low tps spike count
* Added (Unavailable with Export) to exported network html performance tab title

Affects issues:
- Close #1693
2021-07-17 12:19:33 +03:00
Risto Lahtela 03782f18d5 Wrote an access control test for bypass checks
One bypass was discovered for detecting who has played on a server, where
403 is given for level 2 when user has played and 400 when they have not.
This was fixed.

One 500 error was discovered when the network has no proxy server, /v1/network/servers
ran into NullPointerException in a query.
2021-03-22 07:13:27 +02:00
Risto Lahtela 8f85c23c2f Refactoring: ActiveSession, FinishedSession and ServerUUID
- Split Session into ActiveSession and FinishedSession, replaced their usage
- Replaced UUID with ServerUUID when the data type

Affects issues:
- Close #1746
2021-03-12 18:03:12 +02:00
Rsl1122 94c9dd3f95 Username change Rsl1122 -> AuroraLS3 2021-02-13 15:16:03 +02:00
Risto Lahtela db0a8359b9 Fixed optimized performance graph issues with gap on
Because of the new resolution reduction the gap algorithm was not
advanced enough to differentiate between lower resolution and missing data.

This was fixed by adding 3 different Gap strategies for the different
resolutions
2021-02-03 19:38:53 +02:00
Risto Lahtela 7480aa9d63 Rename things for better understanding
fetchQueryPreviewPlayersOnline -> fetchViewPreviewGraphData
FilterQuery -> SpecifiedFilterInformation
ExtensionQueryPlayerDataTableQuery -> ExtensionQueryResultTableDataQuery
ExtensionServerPlayerDataTableQuery -> ExtensionServerTableDataQuery
2021-01-31 10:55:08 +02:00
Risto Lahtela 1c71922b04 Solved some new code smells
- Return empty set instead of null FilterQuery#getSetParameters
- DateRangeFilter constructor is protected
- String constant JSONFileStorage
- String constant VersionChecker
- String constants FLOOR(, MIN(, MAX( TPSQueries
2021-01-30 19:17:59 +02:00
Risto Lahtela 4b4aa2d7d9 Added a players online graph to Query page to help selecting view 2021-01-29 11:58:06 +02:00
Risto Lahtela e26a272e60 Fixed TPS Storage test 2021-01-25 11:26:30 +02:00
Risto Lahtela ccd492c052 Reduced the performance graph size
Reduced resolution of data:
- Last 30 days: Full resolution (1 per minute)
- Last 60 - 30 days: 1 per 5 minutes
- 60+ days old: 1 per 20 minutes
Effect:
- Reduced /v1/graphs?type=performance size from 21 MB to 9.15 MB (126k rows in database)

Added new endpoint /v1/graphs?type=optimizedPerformance that doesn't parse series separately
- Sends a single array of arrays instead of one array for each series
  - Added a parseDataSeries to graphs.js that translates the data
Effect:
- Reduced from 9.15 MB to 3.35 MB
- Moved some workload to the browser

Affects issues:
- Fixed #1622
2021-01-24 17:40:30 +02:00
Risto Lahtela 154cfc4a2b Added config setting to disable disk gathering
'Data_gathering.Disk_space' default 'true'

Also fixed all performance average calculations where -1
might be present

Affects issues:
- Close #1360
2020-03-13 10:28:47 +02:00
Rsl1122 a3cd3adb32 Cleaning up some code:
- Some extension API implementation things refactored
- getOrDefault+put calls replaced with computeIfAbsent
  using Maps#create, Lists#create
- stream().map(mapper).collect(toList/toSet) optimized
  using Lists#map, #mapUnique
- stream().filter(by).collect(toList) optimized
  using Lists#filter
2019-12-19 00:09:29 +02:00
Rsl1122 a50576e6e9 Fixed wrong usage of the word 'parse'
Reason: Parse means extracting information - In many cases the word was being
        used wrong (In Finnish 'parsia' means 'to patch together', which caused
        the wrong use)

The word 'parse' replaced with 'build', 'create' or 'generate' where appropriate
2019-12-05 20:25:42 +02:00
Rsl1122 1511162f5b Fixed a bunch of IntelliJ inspections
- Fixed possible null issues with NicknameCache
- Removed bunch of unused code, such as:
  - Point reduction algorithm implementations
  - HighCharts data String parsing methods
  - Unused Mutators
  - Unused AnalysisKeys
  - Leftovers from ConnectionSystem (Response codes)
  - Unused queries (Leftovers from Server box and Players table queries)
  - rendering.html.icon.Icons
- Made bunch of fields final

Note that old deprecated API classes do not have signature changes.
2019-11-02 16:28:43 +02:00
Rsl1122 549cad4da5 Use getDouble with SQL function AVG()
MySQL that restricts float being queried with getLong caused an
SQLException

All usages of getLong with AVG function replaced with
getDouble cast to long.

Affects issues:
- Fixed #1181
2019-10-19 12:42:25 +03:00
Rsl1122 e6696abffa PlaceholderAPI support (Thanks to aidn5) 2019-09-29 13:00:55 +03:00
Risto Lahtela 66a19d7de1 Removed system. from packages (#1147)
* system.commands -> commands

* system.delivery -> delivery

* system.gathering -> gathering

* system.identification -> identification

* system.processing -> processing

* system.settings -> settings

* system.storage -> storage

* system.version -> version

* system -> com.djrapitops.plan
2019-09-03 09:32:30 +03:00