Difficulties Hosting in an AWS S3 Bucket

I had recently finished up the site for my wife’s business Toast Coffee & Wine Bar, which we hosted at Dreamhost on shared hosting. It was trivially easy and have no complaints. I decided to try it out as a site hosted on an S3 bucket as that seems like a pretty good model. I could manage it in the same place and with the same knowledge base as more complex cloud applications, and could run it for virtually no cost. I ran through the wizard, it took about 5 minutes, and it was working. Because it wasn’t worth messing with something that was already working, that was as far as I took it. I should have at least clicked somewhere other than the homepage.

When it came time to host this site, I decided to go the S3 Bucket route. I figured the hosting aspect would be a trivial effort, so I ran through the wizard. It registered the domain with AWS Route 53, set up a Cloudfront CDN distribution, and I thought I was in business. Then I tried browsing past the homepage, and ran into my first issue. Permission denied.

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId></RequestId>
<HostId></HostId>
</Error>

My first thought was that’s a pretty obvious problem, I just need to add permissions, and that it seems odd the wizard didn’t do that correctly. Reading the documentation does indicate that GetObject permissions needs to be added like so:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<bucket-name>/*"
        }
    ]
}

But sure enough, the wizard did add that correctly. On to the next thought.

Researching a bit, I found the way I was generating links and serving the html pages maybe didn’t fit with the s3 website model. I was generating links like /about and files like /about.html and relying on the server to serve the .html file. There are 2 solutions:

• generate pages/links as folders with a index.html file in them: about.md -> /about/index.html. Then configure s3 to serve an index.html file by default.

• remove the .html from files before uploading to s3, and in s3 set the content-type to text/html. This is a deployment/environment dependency I don’t like at all. The closer I can keep my relevant environments, the fewer deployment surprises I’ll have. Deployment surprises are among my least favorite surprises.

I went with solution 1, but it still didn’t solve it. In retrospect that makes sense. If that had been the problem, it would have manifested as a 404, like so:

    Code: NoSuchKey
    Message: The specified key does not exist.
    Key: about/index.html
    RequestId: 
    HostId: 

Different (potential) problem solved, but still no dice. On to the next thought.

Troubleshooting is often isolating variables and making small changes from a known good state, so I set about to simplify my problem space. The wizard sets up a CDN with Cloudfront by default, so I set up to use just the s3 bucket domain, not through my custom domain or Cloudfront. Everything was fine there. Links are fine, no permission issues. Next step, through Cloudfront, no custom domain. Now I see the same permissions issue again, so it seems the problem is in the Cloudfront setup. Looking into the settings there, we see a setting like:

origin-domain-name: <bucket-name>.s3.amazonaws.com

The actual domain is:

<bucket-name>.s3-website-us-east-1.amazonaws.com

And that was it. Add the region to the Cloudfront setup and everything is great.

All in all an interesting troubleshooting exercise, highlighting several ideas:

• Wizards can be nice, but they gloss over a lot. If it doesn’t work, we’re working with a lot of unknown and poorly understood variables.
• Reduce the problem space. It’s much easier to understand small steps from a known state than reason about an entire system at once.