TL;DR
This past Monday, our DNS provider, DNS Simple, experienced a distributed denial of service attack which took down their DNS resolution service.
You can find more information about the DNS outage at our provider here.
Our monitoring alerted us that there was a problem with domain resolution and we began investigating. Our DNS provider is both our registrar and our DNS provider, so there was, unfortunately, nothing that we could do during the outage.
During this time, some customers were unable to resolve our domain name packagecloud.io. Customers who had our DNS cached or added an entry to their /etc/hosts file were unaffected by the outage.
We’ve made some changes to help mitigate our DNS provider having an outage in the future.
More info
We were alerted by our monitoring services at 19:21 UTC on December 1, 2014 that DNS resolution was failing.
We immediately began investigating the issue and found that DNS Simple was experiencing a distributed denial of service attack.
You can find more information about the DNS outage at our provider here.
Our DNS provider is both our registrar and our DNS provider. Their service was down in its entirety, so we were unable to login to switch our namserver settings to an alternate provider during the outage.
Customers with our DNS cached on their systems were unaffected by the outage and we saw several customers downloading and uploading packages during the DNS outage.
Once the service at our DNS provider was restored, we made some changes to help mitigate potential outages like this in the future.
Changes
It’s possible to configure your DNS settings to use more than one provider to protect against a particular DNS provider having an outage.
In order to do this, you need two DNS providers which support DNS zone transfers.
We researched our options and selected two providers that support DNS zone transfers, migrated our DNS zones to the new providers, and updated our nameservers at our DNS registrar.
We sincerely apologize for the outage our customers experienced and hope that the changes we made to our infrastructure help protect customers against future outages of this nature.
If you have any questions, please feel free to email us at support@packagecloud.io.
