DNS (Domain Name System) Servers are one of the fundamental services that allow the internet to run as we know it. This post is about some fascinating attacks on DNS servers and how I discovered them through a side project of mine -- but before we get into that, let’s go over a few basics. (If you’re already familiar with DNS and DDoS attacks, you can skip the next few paragraphs.)
As you probably already know, computers connected to a network are given an IP address. These addresses are often something like 192.168.1.1 for a computer on your home network, but can be anything from 0.0.0.0 to 255.255.255.255. This means that as a planet, we can have at most 4228250625 (4.2 billion) unique addresses on the global internet. Now, each of these public addresses may be the door to an entirely other private network, such as your home network, but any device connecting to the public internet must have a unique numeric address. The problem is that these addresses aren’t easy to remember. Tech people can often keep a handful in their heads at any given time, but imagine how annoying it would be if you had to look at your notes for the IP address for each website you wanted to visit? It wouldn’t be a very smooth browsing experience. That is why some very smart people came up with the Domain Name System, which is a system for linking the easy-to-remember text name for a site to its numerical IP address.
Basic DNS Systems
The way DNS works is actually very simple. When you open your web browser and type in google.ca, your browser will send a special message to your home router asking: “where can I find google.ca?”. Your router often won’t know the answer, but doesn’t want you to know that, so it will ask the internet service provider the same question. Most ISPs keep the addresses of common websites on hand, so in the case of Google the ISP would respond with the address to your router, who would then respond to your browser, who would then connect to Google and display the page. If you were asking for a lesser known site, your ISP would not know the address, and would have to ask a regional DNS server, who may have to ask a national DNS server, who could then ask a continental DNS server. These continental servers are connected and can ask each other, but if they don’t know the answer, then this means that the site hasn’t registered anywhere publicly. This kind of structure looks like a tree, and allows for most queries to be answered without going very high. When a DNS server has to ask another server for an address, it caches the result, allowing it to respond directly next time.
UDP vs TCP
Now that we are all on the same page about the basics of domain name resolution, there are two other things that are important. The first is the difference between a UDP and a TCP packet. TCP packets are how a large part of the internet communicates. It allows a user to bundle some data in a form that devices all around the world will be able to understand. As part of this communication, there is bidirectional communication between the server and the client. This means that if a user sends a TCP packet with the wrong sender address, the server will not be able to respond, and there will be no communication. This back and forth is important for managing things like connection speed, and packet loss. A UDP packet on the other hand, does not have this bidirectional communication, and is rather used as a “fire and forget” message. There is no way for the sender to know the packet has even arrived at the destination!
The second (and last bit) of background that’s important here is about Distributed Denial Of Service (DDoS) attacks. If you wanted to disrupt a server, one method of doing so is to force a large group of networks to fire off as many messages as possible to the target all at the same time. This is the computer equivalent to a group of people yelling in your ear at the same time, while you try to have a regular conversation with someone. It quickly becomes impossible to communicate, and your only option is to wait until the yelling stops. DDoS attacks are one of the most common types of attacks because they require no real knowledge of the victim, and you simply need enough people yelling. To make matters worse, there are groups that have written malware to infect internet devices, creating what is known as a botnet, and they make these botnets available for hire, so then all you really need for an attack is some bitcoin and a target. I should also mention that a DDoS attack is a Denial of Service (DoS) attack that is spread (distributed) across many clients. In the late 90s, when this method of attack was discovered, infrastructure wasn’t nearly what it is today, and you could do some real damage with just a single system. Now, with the massive bandwidths and computing hardware available to anyone with $5 to spend monthly at any number of cloud providers (such as Amazon), it is much more difficult to cause a disruption, so attackers have relied on leveraging massive networks to achieve their goals.
On a more technical level, there are two main strategies: stress the server, or flood the server. The goal of flooding is to send enough messages that it dilutes the valid traffic to the point that the server becomes unusable. Stressing can be done in a number of ways with varying levels of effect, depending on the victim. For example you could be aware of a specific request that will trigger a CPU-heavy database query. By bombarding the server with these requests you may be able to cause the system to run out of resources and crash, and ‘denying service’ to users (hence the name).
DNS Amplification Attacks
Whew, ok, now for the good stuff! What is a DNS Amplification attack? Very simply, using recursive DNS servers, an attacker can amplify their effective bandwidth to flood (and thus deny service to) a target system with DNS responses . A single DNS query is typically under 512 bytes (or 0.000512 Megabytes), however the response from the DNS server can be up to 65535 bytes. This means the ratio of what you receive to what you send can be almost 130x! But it isn’t very helpful to just send and receive data from a DNS server when you are trying to attack someone else. This is where the “fire and forget” nature of UDP packets comes in. It is possible to craft a packet to request a large DNS response, but with a sender address of the target system (rather than yourself). Doing this will cause the DNS server to send the requested lookup to your victim, not back to you. This means that, instead of sending the victim 512 bytes directly using the same bandwidth, you can have the victim receive many times that, effectively ‘amplifying’ your bandwidth. Clever...
The ‘flood’ type of Amplification attack is quite difficult to deal with. From the DNS server’s point of view, it is getting legitimate requests from a client it cannot verify. So what to do? A few ideas that may come to your mind are:
Simply blocking UDP requests. This isn’t a great thing to do because it will restrict the legitimate users of your system.
Banning the IP addresses sending the UDP packets. This isn’t very useful either because the packets aren’t coming from the IP they claim to be anyway. Maintaining a blacklist of addresses part of the botnet can be logistically complicated, plus you’re loading a lot more work to your firewall, which could cause other issues.
There are, however, a few things that can be helpful:
- Rate limiting can be useful. It will reject responses from clients making above the set threshold of requests per time window. The trouble is, some legitimate clients may have a fairly high burst rate, and could get blocked. Another issue is, with a clever botnet of sufficient size, each client may have to make only a few requests, but spread the attack over a very large number of clients.
- Blacklisting the domains used in the lookup request. This is quite problematic because if the lookup domain is anything common (such as google.ca), you block it from legitimate clients. However, I have found that in many cases the malicious traffic is using only a few domains for the lookup, so this is manageable.
The best way to stop an amplification attack can’t be done at the DNS level but rather at the router/gateway or even Internet Service Provider (ISP) level. A gateway is aware of what address range it supports, and therefore knows what addresses are not within its network. So if the network range is 220.127.116.11 to 18.104.22.168, but then receives a packet from one of its clients with a spoofed address claiming to be from 255.1.1.1, it knows that can’t be a valid address and could easily block the packet. This would restrict amplification attacks to being on the same network as the victim, which is restrictive enough to effectively make them useless (in most cases).
Running a DNS Server
So, why am I talking about DNS amplification attacks anyway? Well, as a side project I have been running a DNS server for approximately 3 months, as a way to collect data on both legitimate, and malicious traffic. I keep it totally open, and monitor it regularly for attacks. My method for stopping amplification attacks is doing both rate limiting, and domain blocking. As explained above, domain blocking isn’t a very good strategy, but so far the attacks have been using a very small handful of uncommon sites, and there isn’t very much legitimate traffic to worry about. I’ve collected information on each lookup requested, and have started to tie in additional information about the clients. As of writing this I am only doing a geoip lookup of the client IP address, and tracking stats on what domains are requested the most, what countries have the most lookups, etc. To date the server as seen 19,142,413 requests. This includes requests made after I’ve blacklisted domains used in amplification attacks.
Figure 1. Plots of DNS lookup counts. Top plot is total counts per 10 minute interval for the last 24 hours, bottom plot is daily counts over the last 30 days. Note the large spikes in daily traffic, these often (including the cases shown here) indicate the beginning of an amplification attack.
Figure 2: Unique IP addresses per country.
Figure 3: Domains with the highest number of queries.
Discussion of Data
A few things are important to note. First of all, I live in Canada, and I use this DNS server myself, so most of the legitimate traffic is from me. Interestingly, however, by far the most traffic is IP addresses in the USA, followed by Brazil. Canada doesn’t rank until #5! Secondly, of the top domains, my traffic doesn’t show up until my home IoT devices such as belkin.com (philips lights), and august.com (door lock). I was actually quite surprised how often these lookups are done. It really doesn’t seem necessary for my lock to lookup the address of its own server once a second, but that is a rant for a different blog post... Ignoring the malicious traffic such as fema.gov, and ‘.’, most of the lookups for legitimate traffic are done by services such as google drive, or by advertising/tracking services such as blzddist1-a.akamaihd.net.
At first glance of the unique IPs per country plot, I was a little surprised. I didn’t expect the US to be a head and shoulders ahead of every other nation. After thinking about it for a moment, this surprise wasn’t not very logical. It is important to remember that these demographics are of the victims, not the attackers. The US has many of the world's largest tech companies, and a huge population with ubiquitous access to internet connected devices. Therefore it makes perfect sense to me that they would be primary targets of these attacks. One thing that still is a little confusing is why Brazil of all places is so highly ranked. I don’t have much of an idea for that one. In the coming weeks I will be working to build an automated scanning tool to collect more information on these victims. Hopefully that data will shed some light on the reasoning.
|IP Address||Operating System||Company or Domain||Country||Open Ports|
|22.214.171.124||Windows||China||80,88,135,139,445, 593,1025,1720,3389,4444, 5060,5800,5900|
|126.96.36.199||incapdns.net||USA||53,390,443,444, 445,1002,1111,1443,1494, 2000,2068,2100,2200,3030, 3333,4343,4443,5225,5226, 5987,5988,5989,5998,5999, 6100,7004,7070,7443,8200, 8443,9081,9100,9101,9102, 9103,15002,18040,60443|
Table 1: This table displays some simple nmap scan info collected on some of the client IP addresses which have the highest number of lookups on the server.
Table 1 shows some preliminary data on the victims of these amplification attacks my server has been made a part of (I am blocking the traffic remember, so this server isn’t actually part of the attack). These clients are some of the most frequently attacked based on the traffic I’ve seen. I am both surprised and not surprised Twitter is near the very top. On the one hand it is a huge target I’m sure many people would love to disrupt, but on the other hand, the level of effort required to disrupt such a monster would be enormous with a simple flood attack. The nmap output when scanning the Windows computer apparently in China was very confusing. One client that was quite interesting was the incapdns.net which is a company called incapsula. From what I can tell this server is running proxies on hundreds of ports (I omitted them from the table), and offer a service where you can speed up traffic to your site by having clients proxy through incapsula.
To date I have had approximately 205k unique IP addresses ‘request’ a lookup. Of these 205k I approximately 99.425306999341% of them are malicious lookups. I got this number by counting how many IP addresses have made multiple lookups to only a single domain, and compared that number to the total number of unique IP addresses in the raw dataset. I’ve also been tracking IP addresses that make only a single query, in the hopes that I can begin to identify the scanners finding my server, but even if that works out there isn’t a great way to tell which ones are from other security researchers, malicious tools, or just curious people seeing what is out in ipv4 space.
I thought I understood DNS servers, but it wasn’t until I began running my own that I really learned about them. It actually has been a very enjoyable side project (project name ideas, anyone??!). These plots are being generated by a basic web app I built to visualize the data. I plan to release this information publicly once I have the web app completed. I will update this blog with the address when I have it.
This post actually got delayed a few weeks because when I began writing this I wanted to have a line that says “there are approximately %d active dns servers doing open recursive queries at the time of writing this”. I wanted to measure the number so I began building a ‘simple’ tool to scan all of ipv4 space looking for such servers. I got fairly obsessed with building this tool and it has turned into a more general purpose scanning tool capable of scanning all ipv4 space (on one port) in less than a day. I have lots of plans for how I’ll use this to support this dns work, but alas - I will leave these details for another post.
Thanks for reading!
— Jason Hopper