ABSTRACT |
|
In May 2003, the University of Wisconsin - Madison found that it was the recipient of a continuous large scale flood of inbound Internet traffic destined for one of the campus' public Network Time Protocol (NTP) servers. The flood traffic rate was hundreds-of-thousands of packets-per-second, and hundreds of megabits-per-second. Subsequently, we have determined the sources of this flooding to be literally hundreds of thousands of real Internet hosts throughout the world. However, rather than having originated as a malicious distributed denial-of-service (DDoS) attack, the root cause is actually a serious flaw in the design of hundreds of thousands of one vendor's low-cost Internet products targeted for residential use. The unexpected behavior of these products presents a significant operational problem for UW-Madison for years to come. This document includes the initial public disclosure of details of these products' serious design flaw. Furthermore, it discusses our ongoing, multifaceted approach toward the solution which involves the University, the products' manufacturer, the relevant Internet standards (RFCs), and the public Internet service and user communities. |
Table of Contents
Flawed Routers Flood University of Wisconsin Internet Time Server Netgear Cooperating with University on a Resolution The Initial Flood Figure 1. The Initial Flood Blocking the Flood Background: Simple Network Time Protocol (SNTP) Figure 2. A SNTP Request Packet Figure 3. A Unicast SNTP Reply Packet The Flood Continues Figure 4. The Flood Continues: One Month Later Investigation Contacting Source Networks Figure 5. Email Notification to Peer Institution Gathering Background Information Examining the Netgear Code Contacting Netgear Figure 6. Email to Netgear Support Figure 7. Email from Netgear Support The Review Process The Flawed SNTP Client Impact to Netgear Customers Code Upgrades for Affected Netgear Products Figure 8. Affected Netgear Products Flawed Product Counts Figure 8a. Netgear SNTP Clients Per Day Suggested Fixes The Initial Fix: "Instant" Code Network Operational Options: To Serve or To Sever? Endgame A: UW-Madison Netgear Anycast Time Service Figure 10. A WiscNet BGP-based Anycast Time Service Endgame B: Attempt to Suppress the Requests Figure 11. Using the Global BGP Routing Table to Squelch Requests Endgame B: IP Resources Required Figure 12. IP Resources Required for BGP-based Suppression Inform the Internet Community Clarify Internet Best Current Practice and Protocol Standards Status, August 21, 2003 Figure 13. The Most Recent Flood Afterthoughts Acknowledgements Analysis Tools References / Further Reading Frequently Asked Questions What is Netgear's liability for causing (however inadvertently) this denial of service for your network? Have you considered putting up a server which sends back fake answers to netgear clients, to cause people to upgrade? What is the expected life-time for these products? In figure 13, could that "shark fin" spike have anything to do with last week's power grid failure (Blackout 2003), and subsequent "rolling" restoration? Are there other devices than those mentioned which also suffer from the flaw which causes inadvertent flooding of your network? What was the effect of this article being slashdotted? Why is a traditional manufacturer recall/defect solution not a possibility? I'm with [the IT press], do you have some time to speak with me? How has this story been covered in the press?
Figure 1 is a graph of inbound traffic to our campus over a 48 hour period, tuesday through thursday, May 13-15, 2003.
The first half of the graph shows typical traffic levels for our campus, with peak inbound packet rates of about 40,000 packets-per-second. However, as you can see, our inbound packet-per-second rate increased dramatically starting May 14 at about 8AM localtime, primarily from our commodity Internet Service Provider, WiscNet. At about 9:40AM this additional traffic began to cause problems with our measurement infrastructure and some of our legacy intra-campus routers. By 11AM we had identified the inbound flood traffic by protocol and port numbers. It was destined for our public time server and we blocked the incoming traffic upstream, at WiscNet's border routers, which alleviated the problem for the time being. This is a typical action for network operators to take in reaction to malicious Denial-of-Service flood attacks, of which we assumed this was one.
The traffic in question appeared to be Network Time Protocol (NTP) queries in that they consisted of 76-byte IP packets destined for UDP port number 123 (NTP). However, these packets had an unusual characteristic: although they appeared to come from many sources, they all had the same source port number: 23457. Therefore, it was possible to configure our routers to block just a subset of inbound queries to our NTP server, and continue to service the other legitimate requests normally. We just blocked all UDP traffic sourced from port 23457 and destined for port 123 (NTP) of the NTP server in question. (Note that the number 23457 seems hand-picked, as the number subsequent to 23456.) At this point we simply chalked it up to naivete on the part of the "attacker", which we presumed was forging many random source addresses, and left it at that, presuming that the flood would subside within hours as "script kiddie"-launched flood attacks often do.
Paraphrased from RFC2030 by Dave Mills:
The Simple Network Time Protocol (SNTP) is an adaptation of the Network Time Protocol (NTP) used to synchronize computer clocks in the Internet. It is a simple, stateless remote-procedure call (RPC) system with accuracy and reliability expectations similar to the UDP/TIME protocol described in RFC-868. SNTP can be used when the ultimate performance of the full NTP implementation is not necessary.
Note that SNTP uses the same packet format as NTP. In this way, SNTP clients can utilize NTP servers, even though they do not implement the complexities of the full peer-to-peer NTP protocol.
SNTP conversations typically follow these steps:
A client that would like to know the time sends a UDP packet containing the SNTP request to the well-known NTP port number 123 of an NTP server, and awaits a reply.
Figure 2. A SNTP Request Packet |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |LI | VN |Mode | Stratum | Poll | Precision | | =0|= 1-4|= 3 | = 0 | = 0 | = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Root Delay | | = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Root Dispersion | | = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reference Identifier | | = 0 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Reference Timestamp (64 bits) | | = 0 | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Originate Timestamp (64 bits) | | = 0 | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Receive Timestamp (64 bits) | | = 0 | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Transmit Timestamp (64 bits) | | = n | | (some number: zero, or the time of request sent by client) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
The server responds with a UDP packet containing the SNTP reply from the well-known NTP port number 123 to the SNTP client.
Figure 3. A Unicast SNTP Reply Packet |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |LI=| VN |Mode | Stratum | Poll | Precision | |0-2|=req.|= 4 | = 1 - 14 | (ignore) | (ignore) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Root Delay | | (ignore) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Root Dispersion | | (ignore) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reference Identifier | | (ignore) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Reference Timestamp (64 bits) | | (ignore) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Originate Timestamp (64 bits) | | (copied from request Transmit Timestamp) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Receive Timestamp (64 bits) | | (time request was received by server) | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | Transmit Timestamp (64 bits) | | (time of reply sent by server) | | = n | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
Upon receiving the response, the client optionally uses the Originate Timestamp from the reply to validate the response, attempting to assure that it is indeed a response to this client's request. (If the reply were spoofed from another source, it would be unlikely to contain the correct value as the Originate Timestamp). Then it plucks the value from the "Transmit Timestamp", perhaps modifying it slightly to account for the estimated one-way end-to-end delay, and uses the result as the current time to set its local clock.
Now, back to our story...
One month later, we discovered that the flood of inbound NTP traffic persisted at an even more incredibly high rate, as evidenced by Figure 4 which plots our router's discarded packet rates beginning in early June 2003, for the traffic in question.
In Figure 4, note that: (1) there are slight daily fluctuations in rate (perhaps due to diurnal user behavior), (2) generally, the rate stays at a value over 250,000 packets-per-second (and over 150 megabits-per-second), and (3) that the traffic rate increases throughout the time shown. The sharp drops in traffic rate in this figure are not due to the flood subsiding but rather were due to network maintenance and a temporary block upstream from the observation point.
Once we found that this flood was continuing and was still increasing in rate, we investigated further. By carefully removing the block on some ingress interfaces, we allowed a trickle of traffic through to the server and captured the packets including their payload. We learned that these packets appeared to be legitimate, well-formed Simple Network Time Protocol (SNTP) version 1 queries, albeit at an inexplicably high rate from each client host. For instance, during one trace, many clients produced about one query per second. This would be highly unusual for a properly constructed SNTP client, since an application which uses SNTP is merely interested in setting its own clock relatively accurately so that its host has some reasonable notion of the current time. One query per second is ridiculous, and is far from best practice for NTP client behavior.
Also, we discovered that many of the IP addresses could be resolved to DNS names and furthermore that the IP addresses all appeared to be valid sources for the given ingress interface from which we removed the block. This indicated that it was quite possible that the source addresses were not forged but instead were real Internet hosts running some very unusual SNTP client.
Alas, none of the client source hosts were within our local campus network. This meant we would need to recruit the help of staff at remote sites to aid in the investigation.
Of the top talker source IP addresses from the aforementioned packet trace, I selected two client hosts from other universities with talented network staff who would be familiar with responding to such incidents.
The following is an email I sent to the Incident Response Team at one of those institutions to which one of the client host addresses belonged. (To maintain a modicum of anonymity, I have replaced the real SNTP client's IP address with 10.42.69.10 and have also removed the email domain names.)
Figure 5. Email Notification to Peer Institution |
Date: Sat, 14 Jun 2003 04:34:11 -0500
From: Dave Plonka <plonka@localdomain>
To: abuse@[remotedomain]
Subject: sntp/ntp query flood from 10.42.69.10 to ntp1.cs.wisc.edu
[Organization] network abuse folks,
Since May 14, 2003 ~0800 central time, one of our campus' NTP
servers "ntp1.cs.wisc.edu" (128.105.39.11) has been the recipient of a
large-scale flood of Simple Network Time Protocol (SNTP) requests -
much more than it can service. This dramatic increase in inbound SNTP
requests inexplicably continues even now. To mitigate this flood we
are currently blocking over over 250K pkts/sec, exceeding 150
megabits/sec, and it has been continuing for weeks.
We are in the process of trying to determine if this flood is potentially
malicious or if it is an SNTP client misconfiguration or bug.
This traffic primarily consists of 76-byte UDP packets that are SNTP
version 1 queries from very many source host addresses directed to
ntp1.cs.wisc.edu port 123 (NTP). Unusually, these requests all have a
UDP source port of 23457. We have identified the host address
10.42.69.10 as just one of the sources. (However, there are at
least tens of thousands of source host addresses.)
I have attached a timestamped log of a packet capture from the
afternoon of June 13, 2003 (Friday) evidencing the SNTP query packets
from the host 10.42.69.10 at an unusually high rate of about one
per second. A packet decomposition (by tethereal) and hex dump of the
last packet (frame 998) in the log is included as well, which shows
them to be valid SNTP v1 queries as described in RFC 1361,
http://www.ietf.org/rfc/rfc1361.txt.
Could you assist us ASAP with this investigation by identifying that
host's operating system and what SNTP client code may be running on
that host? It would be interesting to know if a process on
10.42.69.10 currently has UDP port 23457 bound and what code that
process is running.
Thanks,
Dave
P.S. Our investigation so far has shown that Windows systems such as
2000 and XP have an "Internet Time" feature which is usually configured
to send SNTP requests to the Microsoft server "time.windows.com", but
this server can be changed. I have yet to identify any SNTP client
that regularly uses UDP port 23457 as its source port. (Note that
port number seems hand-picked, as the number subsequent to 23456.)
----------------------------------------------------------------------
1 2003-06-13 16:32:24.8808 10.42.69.10 -> 128.105.39.11 NTP NTP
7 2003-06-13 16:32:25.9611 10.42.69.10 -> 128.105.39.11 NTP NTP
14 2003-06-13 16:32:27.0412 10.42.69.10 -> 128.105.39.11 NTP NTP
21 2003-06-13 16:32:28.1215 10.42.69.10 -> 128.105.39.11 NTP NTP
27 2003-06-13 16:32:29.2020 10.42.69.10 -> 128.105.39.11 NTP NTP
33 2003-06-13 16:32:30.2821 10.42.69.10 -> 128.105.39.11 NTP NTP
39 2003-06-13 16:32:31.3624 10.42.69.10 -> 128.105.39.11 NTP NTP
45 2003-06-13 16:32:32.4427 10.42.69.10 -> 128.105.39.11 NTP NTP
51 2003-06-13 16:32:33.5232 10.42.69.10 -> 128.105.39.11 NTP NTP
56 2003-06-13 16:32:34.6049 10.42.69.10 -> 128.105.39.11 NTP NTP
68 2003-06-13 16:32:36.7638 10.42.69.10 -> 128.105.39.11 NTP NTP
74 2003-06-13 16:32:37.8441 10.42.69.10 -> 128.105.39.11 NTP NTP
78 2003-06-13 16:32:38.9242 10.42.69.10 -> 128.105.39.11 NTP NTP
84 2003-06-13 16:32:40.0050 10.42.69.10 -> 128.105.39.11 NTP NTP
90 2003-06-13 16:32:41.0846 10.42.69.10 -> 128.105.39.11 NTP NTP
96 2003-06-13 16:32:42.1647 10.42.69.10 -> 128.105.39.11 NTP NTP
<snip>
998 2003-06-13 16:35:13.3789 10.42.69.10 -> 128.105.39.11 NTP NTP
Frame 998 (90 on wire, 90 captured)
Arrival Time: Jun 13, 2003 16:35:13.378978000
Time delta from previous packet: 0.524605000 seconds
Time relative to first packet: 168.498125000 seconds
Frame Number: 998
Packet Length: 90 bytes
Capture Length: 90 bytes
Ethernet II
Destination: 00:0a:41:db:58:00 (00:0a:41:db:58:00)
Source: 00:0a:8b:bf:70:7c (00:0a:8b:bf:70:7c)
Type: IP (0x0800)
Internet Protocol, Src Addr: 10.42.69.10 (10.42.69.10), Dst Addr: 128.105.39.11 (128.105.39.11)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 76
Identification: 0x2cc7
Flags: 0x00
.0.. = Don't fragment: Not set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 243
Protocol: UDP (0x11)
Header checksum: 0xb335 (correct)
Source: 10.42.69.10 (10.42.69.10)
Destination: 128.105.39.11 (128.105.39.11)
User Datagram Protocol, Src Port: 23457 (23457), Dst Port: 123 (123)
Source port: 23457 (23457)
Destination port: 123 (123)
Length: 56
Checksum: 0xb0bd (correct)
Network Time Protocol
Flags: 0x0b
00.. .... = Leap Indicator: no warning (0)
..00 1... = Version number: reserved (1)
.... .011 = Mode: client (3)
Peer Clock Stratum: unspecified or unavailable (0)
Peer Polling Interval: invalid (0)
Peer Clock Precision: 1.000000 sec
Root Delay: 0.0000 sec
Clock Dispersion: 0.0000 sec
Reference Clock ID: Unindentified reference source ''
Reference Clock Update Time: NULL
Originate Time Stamp: NULL
Receive Time Stamp: NULL
Transmit Time Stamp: NULL
0000 00 0a 41 db 58 00 00 0a 8b bf 70 7c 08 00 45 00
0010 00 4c 2c c7 00 00 f3 11 b3 35 0a 2a 45 0a 80 69
0020 27 0b 5b a1 00 7b 00 38 b0 bd 0b 00 00 00 00 00
0030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0050 00 00 00 00 00 00 00 00 00 00
|
Network staff from two universities investigated two source hosts which I reported as being sources of these anomalous SNTP queries. Both reported that a Netgear brand router was the source of the traffic. (Specifically, one was identified as model MR814.)
Now things started to make some sense. Many source hosts all using the same source port number could be explained by an embedded SNTP client in which the programmer hard-coded the source port number (23457).
While searching the web for background information on Netgear products' acclaimed NTP support, I came across the following quote (from ICSA Labs Firewall Lab Report on the NETGEAR FR114P):
The Netgear FR114P relied on a separate NTP-based time source to set the current date and time, as it did not have an internal battery and clock. The product is hard-coded with specific NTP time sources that are accessible through the public Internet. Even after configuring the product to access a specific NTP server, the product still attempted to access its hard-coded NTP time sources, while simultaneously accessing the time source specifiedNote that Netgear reports that the FR114P does not contain the specific SNTP flaws described here. I just found it interesting, since that was the only NTP-related flaw about which I found information on the web.Conclusion
The Candidate Firewall Product met all the criteria elements in the Baseline and Residential modules and therefore has attained ICSA Labs Firewall Certification.
In order to verify our hypothesis that the source of the flood of SNTP queries are the Netgear Platinum family products and to properly characterize this problem to the vendor, the Netgear code for a number of their products was downloaded and investigated.
Simply by using the Unix "strings" command, I was able to verify that indeed the Netgear code seems to contain the magic number 23457 (as a port number):
$ strings RP614_4_12.bin |grep 23457 on 23457 port. $ strings MR814_4_11.bin |grep 23457 on 23457 port.
Using a similar technique, I found the following IP addresses embedded as ASCII strings in RP614_4_12.bin:
(Note that I added the DNS name comments for clarity; those strings did not occur in the binary file.)128.105.39.11 # ntp1.cs.wisc.edu (a.k.a. "caesar.cs.wisc.edu") 192.168.1.101 66.37.215.43 12.234.94.14 192.168.0.1