Skip to content
GitLab

CDN Hosting at Google Cloud Storage · Changes

Page history
general update. authored by Peter Krautzberger's avatar Peter Krautzberger
Hide whitespace changes
Inline Side-by-side
CDN-Hosting-at-Google-Cloud-Storage.md
View page @ 2b50052f
......@@ -3,67 +3,75 @@
### Preliminaries
* The documentation home is https://developers.google.com/storage/index.
* The main tool is `gsutil`.
* The main tool is `gsutil`; can be installed using `pip`
* Note: due to rapid development a lot of information on the web uses outdated syntax (but is still useful, e.g., https://gist.github.com/mhulse/4362104).
### Setting a default ACL (access control list)
### Copying
Notes:
* https://developers.google.com/storage/docs/gsutil/commands/defacl
Notes:
**This is a one-time task**
* https://developers.google.com/storage/docs/gsutil/commands/cp
* gsutil option: -m multithreaded (higher load+cost but faster)
* cp options: recursively (-R), compressed (-z), verbose (-v)
* compression is necessary to get gzipped delivery. Google Cloud storage does not compress on the fly but should decompress on the fly for old browsers that can't accept decompressed files. We could probably skip it since CloudFlare will zip it if it's not.
The GCS default for access control is private. GCS does not allow for ACLs to persist when files are overwritten.
The basic recursive operation is
Since we update files in `\latest` with every releases and files in `beta` during beta runs, we need to prevent GCS from marking those new files as private (breaking public access to the CDN).
gsutil -m cp -Rzv my-folder gs://our-bucket/
The solution is to set the default ACL for the bucket once following the instructions in the `gsutil` documentation.
**Note: not all metadata is preserved by default**, especially when updating a bucket from local files . The previous sections take care of public access, caching, CORS; see below for mime-types.
To set this either use
This also works within GCS, i.e., bucket to bucket. If you copy within GCS from bucket to bucket, you can use the `-p` option to preserve metadata (*note*: this incurs higher costs).
$ gsutil defacl set public-read gs://our-bucket
gsutil -m cp -Rp gs://bucket1/file1 gs://bucket2/
Or create a suitable XML file containing
(You'll do need to set the CORS once for each bucket though; see the One-time-task section.)
<Entry>
<Scope type="AllUsers"/>
<Permission>READ</Permission>
</Entry>
### MIME-type headers
Notes:
### Copying
* https://developers.google.com/storage/docs/gsutil/commands/setmeta
* the `**` wildcard includes subdirectories, https://developers.google.com/storage/docs/gsutil/addlhelp/WildcardNames
Notes:
EOT and SVG `content-type` headers are detected correctly during upload to GCS.
* gsutil option: -m multithreaded (higher load+cost but faster)
* cp options: recursively (-R), compressed (-z), verbose (-v) https://developers.google.com/storage/docs/gsutil/commands/cp
* Note: compressed is necessary to get gzipped delivery. Google Cloud storage does not compress on the fly but should decompress on the fly for old browsers that can't accept decompressed files.
* For otf:
gsutil -m cp -Rzv mathjax gs://our-bucket/
gsutil setmeta -r -h "Content-Type:font/opentype" gs://our-bucket/mathjax/**.otf
* For woff:
This also works within GCS, i.e., bucket to bucket.
gsutil setmeta -r -h "Content-Type:application/font-woff" gs://our-bucket/mathjax/**.woff
**Note that metadata isn't preserved by default** (public access, caching, custom headers etc), especially when updating a bucket from local files (**TODO** there must be some way to specicy metadata?).
WOFF is set to the old mime type `x-font/woff`; that works but we decided to go with the future-proof `font-woff`. Either change it as below or [see advice here](https://github.com/GoogleCloudPlatform/gsutil/issues/183#issuecomment-35741417) to do this automatically on upload.
If you copy within GCS from bucket to bucket, you can use the `-p` option to preserve metadata (*warning*: this incurs higher costs).
`gsutil -m cp -Rp gs://bucket1/file1 gs://bucket2/`
## Testing
(You'll still need to set the CORS for each bucket though; see below.)
* CORS allow-origin [vital]
* A simple test: switch `www.mathjax.org` over to the GCS origin (bucket.storage.googlapis.com) and use saucelabs to check rendering across IE, FF, Chrome, Safari.
* Since CORS is set on bucket level, we should not have to worry about it.
* font mimetypes [non-vital]
* **TODO** write script to `curl -I` all woff and otf files, compare header.
### make files public
### One-time tasks
Notes:
#### Setting a default ACL (access control list)
* https://developers.google.com/storage/docs/gsutil/commands/acl#set
Notes:
* https://developers.google.com/storage/docs/gsutil/commands/defacl
The GCS default for access control is `project-private`. GCS does not allow for ACLs to persist when files are overwritten.
This has to be done whenever files are uploaded (**TODO**: automate!)
Since we update files in `\latest` with every releases and files in `beta` during beta runs, we need to prevent GCS from marking those new files as private (breaking public access to the CDN).
The solution is to set the default ACL for the bucket once following the instructions in the `gsutil` documentation.
gsutil -m acl set -R -a public-read gs://our-bucket/path/to/file_or_wildcard
To set this just use
See also [Caching](#caching) below
$ gsutil defacl set public-read gs://our-bucket
### set CORS
#### set CORS headers
Notes:
......@@ -86,7 +94,7 @@ Create an XML file with
<ResponseHeaders>
<ResponseHeader>*</ResponseHeader>
</ResponseHeaders>
<MaxAgeSec>86400</MaxAgeSec>
<MaxAgeSec>3600</MaxAgeSec>
</Cors>
</CorsConfig>
......@@ -94,45 +102,26 @@ Save as `cors.xml` and run
gsutil cors set cors.xml gs://our-bucket
### mime-type headers
#### Set up caching
Notes:
* https://developers.google.com/storage/docs/gsutil/commands/setmeta
* the `**` wildcard includes subdirectories, https://developers.google.com/storage/docs/gsutil/addlhelp/WildcardNames
* EOT and SVG `content-type` headers are detected correctly during upload to GCS.
* WOFF is set to the old mime type `x-font/woff`; that works but we decided to go with the future-proof `font-woff`. Either change it as below or [see advice here](https://github.com/GoogleCloudPlatform/gsutil/issues/183#issuecomment-35741417) to do this automatically on upload.
* For otf:
* https://developers.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata#cache-control
gsutil setmeta -r -h "Content-Type:font/opentype" gs://our-bucket/mathjax/**.otf
* For otf:
The default caching time for public files is 1h. That's good enough for us since caching really happens on CloudFlare.
gsutil setmeta -r -h "Content-Type:application/font-woff" gs://our-bucket/mathjax/**.woff
### Caching
**Note** Since we are combining GCS with CloudFlare's caching, we will have short caching times on GCS.
This has to be repeated whenever files get uploaded from local.
If you need to change it use something like
$ gsutil -m setmeta -r -h "Cache-Control:public, max-age=3600" gs://our-bucket/mathjax/
### CNAME
#### CNAME
To use a CNAME for a bucket, the bucket needs to be named like the CNAME, see https://developers.google.com/storage/docs/reference-uris#cname.
You also need to verify domain ownership; see https://support.google.com/webmasters/answer/35179?hl=en.
## Testing
* CORS allow-origin [vital]
* A simple test: switch mathjax.org over to the GCS origin (bucket.storage.googlapis.com) and use saucelabs to check rendering across IE, FF, Chrome, Safari.
* Since CORS is set on bucket level, we should not have to worry about it.
* font mimetypes [non-vital]
* **TODO** write script to `curl -I` all woff and otf files, compare header.
### Logging
#### Logging
We don't log on GCS anymore since we combine it with CloudFlare (and get their stats).
......