Upstream
/
dynmap
mirror of https://github.com/webbukkit/dynmap.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Created AWS S3 Storage (markdown)

mikeprimm 2022-02-23 21:01:56 -06:00
parent 3f6a3cfa9c
commit 434dc0061f
1 changed files with 78 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

78
AWS-S3-Storage.md Normal file

@ -0,0 +1,78 @@
This option offers both an alternative storage solution, and an external web serving solution, allowing the map data to be published to, and then served from, an AWS S3 'bucket', providing a convenient option for servers on hosting environments where the option for large local storage and/or the need for an additional TCP/IP port for the internal web server is a problem.
To set up this solution, it is required for the server administrator to have an AWS (Amazon Web Services) account.
**Note: while there is a 'free tier' on AWS for new accounts, and this will allow up to about 5GB to be hosted in S3 for free for the first 12 months of an account, it IS very important to realize that it will be quite possible to run up non-trivial monetary charges when using S3. We will offer some advice on how to optimize this, but understand that you can and should monitor any expenses incurred, and not assume this is a free or risk free option.**
To set up the AWS S3 storage option, do the following:
1. If needed, set up an AWS account for yourself [(see article here)](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/)
2. Use your account to set up a new S3 bucket, and configure it to act as a static web site [(see article here)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/WebsiteHosting.html). It is important to select an appropriate region (this controls which part of the world your S3 bucket is hosted - closer to the bulk of your community is probably best) and record the ID of this region (e.g. us-east-1, eu-west-1) [(see article about regions)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-regions). Also, remember the name you've selected for your S3 bucket - this name must be unique among all S3 buckets in the region. When setting up the Static Web Site settings, use 'index.html' for the **Index document** setting, and 'images/blank.png' for the **Error document** setting (adjust this setting appropriately if you decide to include a prefix path on your published web - see below). Also, DO NOT ENABLE BUCKET VERSIONING - this can result in unbounded growth in storage use in the bucket, given how often tiles and status data are updated!
<img width="810" alt="Screen Shot 2022-02-23 at 7 32 04 PM" src="https://user-images.githubusercontent.com/654487/155440338-4e534ec5-1aff-4af3-9177-0ab2d33143e6.png">
3. Take note of the 'Bucket website endpoint' URL on the Properties tab for the now-configured S3 bucket. This is the default URL for your web site - in order to make this URL more 'friendly', you can configure a CNAME record on your DNS service to direct your desired DNS name to this bucket (DNS configuration is very DNS provider specific process, so you're on your own with sorting that out - every different DNS hoster has documentation on how to do this).
<img width="1112" alt="Screen Shot 2022-02-23 at 7 35 22 PM" src="https://user-images.githubusercontent.com/654487/155440652-02a56171-9e5a-426c-9130-be985b7a779f.png">
4. While not necessary, it is VERY strongly suggested that you create a dedicated service IAM user ID under your account, to be used by Dynmap on your server, and to give this user ID the minimum privilege needed to publish data to your S3 bucket. Do NOT use your root account - anyone who gets access to your Minecraft server could use the account to run up arbitrary amounts of charges on your account! To do this process, follow the following procedure [(see article here)](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console) with these choices suggested:
* For **Select AWS credential type**, select **Programmatic access**
* Select **Attach existing policies directly** on **Set permissions**, and then select **Create Policy**. Select JSON, and apply the following JSON (with your S3 bucket name substituted for YOURBUCKETNAME):
`
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:DeleteObject",
"s3:GetObject",
"s3:ListBucket",
"s3:PutObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::YOURBUCKETNAME",
"arn:aws:s3:::YOURBUCKETNAME/*"
]
}
]
}
`
* This will give your IAM account just the privileges needed to support access to your web site bucket. When you see 'Review Policy', give the access policy an appropriate name (e.g. 'DynmapBucket'). Then, return to the tab where the user is being created, refresh the Policy list, and type the name in the Search to find the new Policy.
<img width="1112" alt="Screen Shot 2022-02-23 at 7 57 34 PM" src="https://user-images.githubusercontent.com/654487/155442799-7f1e5801-54a4-40da-8b19-ee1be055d34e.png">
Be sure to check the checkbox to grant the user this policy.
* Once you finish the creation, be careful to record the Access Key ID and Secret Access Key value - the secret key will never be shown again, and you'll need to reset the keys if you lose them. This values will be used for the **aws_access_key_id** and **aws_secret_access_key** settings, respectively.
5. Now, set up the storage option in **configuration.txt**:
<img width="719" alt="Screen Shot 2022-02-23 at 8 38 38 PM" src="https://user-images.githubusercontent.com/654487/155446985-499e0522-fd2c-4e96-9f93-fff60d9bc5e2.png">
Make sure other storage settings are commented out.
6. The following additional settings changes should also be done:
* Uncomment the JsonFileClientUpdateComponent section, and comment or delete the InternalClientUpdateComponent. Also, set **allowwebchat** to **false** (it is unsupported), and it is suggested to set **writeinterval** to 15 (or higher) - this will reduce how often chat, player positions, and other data are written to the S3 bucket (which can reduce API call charges (below)).
<img width="815" alt="Screen Shot 2022-02-23 at 8 25 35 PM" src="https://user-images.githubusercontent.com/654487/155445528-03d3e975-dabd-446c-a434-d8cac17feafa.png">
* It is strongly suggested that the new **defaulttilescale** setting be used to reduce the number of tiles posted to S3, while increasing the size of the tiles (a setting of *2* is recommended). This will reduce the number of GET calls needed by the users to load the tiles on their screen, while still generating about the same bandwidth. It is also suggested to use the default **image-format** setting (jpg-q90) or the webp format, in order to minimize file sizes (png is NOT recommended, as this results in significantly larger files, and associated storage and bandwidth costs).
* The internal web server must be disabled by setting the *disable-webserver* setting to *true*
* If you wish to publish multiple servers to the same S3 bucket, the *prefix* setting can be used to put the files for the server under a specific path in the bucket: this will correspond to the map for the server being under that path relative to the Bucket Website Endpoint (e.g. **prefix: test123/test** would result in a URL akin to **http://dynmap.s3-website-us-east-1.amazonaws.com/test123/test/**). If set, the **Error document** setting for the bucket should correspond to one of the prefix values (so that the **images/blank.png** file from one of the servers will be used for all 40x errors across the servers in the bucket).
7. That should be it - restart the server and see if it is able to access the bucket. The server will automatically publish the static web site files from Dynmap to the bucket, marker images, and start using the bucket for publishing files.
<img width="1052" alt="Screen Shot 2022-02-23 at 8 49 53 PM" src="https://user-images.githubusercontent.com/654487/155448283-226fd54c-fef1-4ea1-ba4c-d4a073d62afc.png">
## AWS Cost Considerations
At the present time, the following costs are representative for use of AWS in us-east-1 in the United States - prices in other countries, or in the future, may vary [(see pricing here)](https://aws.amazon.com/s3/pricing/).
* $0.023 per GB per month for storing data in the S3 bucket
* $0.005 per 1000 requests to PUT, COPY, POST, LIST (these are done for updating tiles, zoom tiles, and reporting updates, chat messages and the like)
* $0.0004 per 1000 requests for GET (each tile loaded by each user browser is one of these calls, as are polling for updates)
* $0.09 per GB transferred from the bucket (tile reads, update reads, etc. Browser caching will help this). In general, data transfer IN to the bucket has no charges (other than the PUT API call, above).