Finland "Carolina" server issue
Incident Report for Templ
Postmortem
  • The issues were caused by an attack with an extremely high frequency of incoming data packets targeting the IP address 35.228.132.196.
  • Our cloud provider, GCP (Google Cloud Platform), limits the packet frequency to 1.2 million per second, and as the frequency approached this limit, GCP began to 'drop' data packets, meaning that some packets didn't reach their destination.
  • This, in turn, resulted in disruptions to all network traffic through this IP address, becoming extremely slow at times.
  • This IP address is used by several sites, all of which were affected, becoming slow and occasionally unresponsive.

Unfortunately, we didn't receive a clear indication that an attack was ongoing, as none of our monitoring systems indicated higher-than-normal load. Since the attack consisted of extremely small but numerous data packets, it only affected our cloud provider's network, which we didn't directly monitor. We have now added additional monitoring to be notified of this type of attack on our cloud provider.

Unfortunately, network attacks are something that cannot be avoided. However, there are measures to minimize disruptions when they occur. Many attacks can be easily stopped if they originate from one or a few external addresses. In cases like this, with attacks from millions of addresses simultaneously, it's not always possible to avoid disruptions.

It's also the case that an attack usually has a specific target but can affect seemingly unrelated sites because some network resources and addresses are shared. Therefore, it's important to identify the attack's target and remove it. We have now identified the target of the attack and removed it.

To counteract similar disruptions in the future, we have also developed better solutions for quickly changing the IP addresses for affected sites, which bypasses the issues more rapidly.

However, to change IP addresses, access to a site's A-records or proxy settings is required. In cases where we don't have this access, we will, if necessary, notify site owners as soon as possible about the actions they need to take to change their IP address.

We apologize for any inconveniences this has caused.

Posted Sep 12, 2023 - 12:40 UTC

Resolved
We're marking this incident as resolved. Affected customers will be contacted individually with more information.
Posted Sep 08, 2023 - 10:39 UTC
Monitoring
An update about the issues affecting the Carolina server, and websites hosted there.

The issues have been resolved, and all sites should now work as expected.

However, we are continuing our monitoring in case it returns.

The issues were caused by a sudden and unexpected network problem. We are still investigating the underlying reasons for this, and will have more to say once we have more info from our network provider.

We are continuing to work on mitigation and preparing for the eventuality that the problem returns.

Once again, we apologize for any inconveniences this has caused and please don’t hesitate to contact us should any issue remain.
Posted Sep 07, 2023 - 15:08 UTC
Update
While we are working on resolving the network issue, we are also working on a workaround fix in the form a proxy. It will require you to update the A-records of your domain and point them to a new IP.

More details will be provided.
Posted Sep 07, 2023 - 12:07 UTC
Update
After further investigation, we have narrowed down the issue. The problem is NOT a server issue, as we suspected earlier, but rather a network issue that is only affecting the "Carolina" server. Investigation is still ongoing and updates will be provided.
Posted Sep 07, 2023 - 11:48 UTC
Update
The issue is still ongoing and we are investigating further.
Posted Sep 07, 2023 - 09:58 UTC
Identified
We've identified the issue. There is an "attack" on the server, we've applied fixes to remedy the attack and will continue monitoring the situation.
Posted Sep 07, 2023 - 08:19 UTC
Investigating
We are investigating issues with the server in Finland called "Carolina".

Sites hosted on this server may experience problems or interruptions .
Posted Sep 07, 2023 - 07:08 UTC
This incident affected: Finland data center.