News DDOS attack
On 18th/19th Nov 2015 we suffered a significant distributed denial of service attack that disrupted all of our services.
The attack first started around 17:45 on Wednesday 18th and lasted several minutes. We were able to tell this was an unusual attack as it appeared to be targeting several IP addresses. The attack did not last long. However, around 20:40 the attack resumed and continued until 11:20 on 19th. This is very unusual for any sort of attack.
The actual impact on service was quite significant for around 2 hours on the 18th, and several small periods of disruption on 19th while we played whack-a-mole. Some customers in the same IP blocks as the target were out of service for a longer period.
The attack appeared to be a mixture of DNS reflection attacks (i.e. DNS requests from spoofed IP addresses) causing massive amounts of replies in to our network from all over the world; direct TCP port 80 connections, from all over the world; and some odd protocol 17 packets as well.
Oddly the target of the attack appeared to be an entire /24 (256 addresses) which is unusual. We have a variety of systems in place, some automated and some manual, to handle attacks on individual IPs, but we do not normally have to deal with wider attacks such as this.
We had moved customers from the affected block, and then found, as you may expect, that the attack moved (extended) to cover a new /24 at around 08:30 on the 19th. This move helped us track down the actual target customer.
We do now have some ideas why this may have happened, but we are unable to say any more on the matter at present as the police are involved.
Obviously we have no idea who started the attack, but we do now have a good idea of exactly who was the target. We are unable to say any more on the matter at present as the police are involved.
During the attack we moved some people's IP addresses out of the impacted blocks - customers are welcome to ask us to move them back now.
During the attack we announced a lot of /32 routes to everyone with black-hole community tags - this disrupted some BGP links due to prefix limits. We are working on ways to avoid this in future unless you have specific arrangements for black-hole routing with us. In the mean time we'd suggest allowing at least 512 head room on our prefixes.
What are we doing about this in the future?
Sadly, I do not think it prudent to go in to too much detail. I am happy to discuss more with individual customers if you need. The main action we are taking is that we are making some changes to our network infrastructure to make it a lot easier for us to quarantine blocks of IP addresses quickly in future. We cannot, however, protect against all possible attacks and whilst we are taking steps which, in hindsight, would have helped in the case, we cannot say for sure how they will help next time.
Attacks of this nature affect all ISPs and hosting companies and some larger companies all of the time. Please be assured we are learning from this and improving the way we do things.
Obviously I do apologise for the inconvenience caused. I would also like to thank customers for their support and understanding.
Update 22nd Nov
We are rolling out updates to all of our routers now which allow us to manage quarantining sections of our network quickly in the event of a similar attack in future. The full roll out should be complete early this week.
Adrian Kennard, Director