This morning, a colleague of mine discovered a rather strange behaviour of our University's network when trying to download a file with a size of about 80 MB using HTTP. Later that day, we identified a strange behavior of the network itself. We do not know whether it is a router bug or a filter. But it is surely not neutral. +UPDATE There is a bonus section following the original conclusion. +UPDATE 14.11.2013,10:58 -- Seemingly the problem has been resolved; we are unable to reproduce the stalled downloads. +UPDATE 29.11.2013 "Transparent" IPS within the ITMZ Infrastructure kind of confirmed.

The symptoms

When Martin (http://running-sheep.com/members/leo) tried to download one of the following files

an error occurred: The transfer was interrupted, i.e. stuck. The connection was alive but no data reached his machine. The initial assumption was a size limit applied to msi-files, probably due to a local firewall. Involving colleagues, the problem could be confirmed, e.g. for which was subject to follow-up-testing. A colleague confirmed the problem for the wireless campus network. The result was always the same: Stuck connections. In ten tries to download the FreeBSD-Image, we retrieved 7844739 Bytes with a standard deviation of 352 bytes (test on catalina).

Speculations on probable causes

Of course, we started wild speculations about probable causes for the observed phenomenon: They covered problems resulting from obnoxious TCP stack implementations getting confused by each other resulting in deadlock, virus filtering gone wild, firewall misconfigurations including deep packet inspection with filtering as well as server problems.
This article shall not continue our speculation. It shall be kept to the observation. Hence, this section is over now.

Researching the problem

Before digging deeper in, here is a rough map of our network.

| - - - - - - U n i v e r s i t y   o f   R o s t o c k   C a m p u s   N e t w o r k - - - | - D F N - | - - DTAG - - | - - LRNC - - - |


| - - - Network of the Institute for CS - - - |


                                  some routing
                                  and firewalling                  Wireless
                                                                   Campus
 catalina.informatik.uni-rostock.de -----.                         Network
                                          \                           |
 ekas.informatik.uni-rostock.de -----------+--.                       |
                                               +-- [ University of Rostock's Data Center ] --+-- [DFN] --+-- [DTAG] --+--[Testing target]
 sashimi.informatik.uni-rostock.de --------+--`                       |
                                          /                           |
 tango.informatik.uni-rostock.de --------`                   unix.uni-rostock.de

Then we got systematic.
We ensured that all components (TCP stack, route, web server) were identical during the tests described below. We placed the FreeBSD-ISO on a well-known server in a known environment in a data center located in Frankfurt/Main (Germany). The routing is as sketched above: After passing the University's network, the networks of DFN and DTAG are to be passed before entering the server's network. Early tries were made using the workstations sashimi and tango as well as the login servers ekas and catalina inside the Institute's network. Final tests were made using unix.uni-rostock.de, ruling out the possibility for local firewalls to cause the observed phenomena, since they could be reproduced using this very machine which is not subject to the Institute's firewalling. We stuck there for further testing. Web server in further tests was Apache 2.2 running on FreeBSD 9.1.

Incomplete files

While the FreeBSD image was downloaded to the target's web directory, we started a first test download, still on catalina. I went on, jumping the magic 7.4MB hurdle flawless. Once the ISO on the testing server was complete, it stuck as described above. Unfortunately, this experience could not be reproduced.

We can shift it

Martin came up withe the idea of adding some random noise preceeding the ISO data itself. Thus, we took some bytes from /dev/urandom and created a file as sketched:

  randmom data from      FreeBSD 9.2
   /dev/urandom           ISO Image file
      n Bytes
[ randomrandomrandom | iso iso iso iso iso ]
The result was confusing: We were able to shift the position of the download to stuck: Adding 1MByte of noise, the download stuck at 8.4MB and so on.

The malicious sequence

If we can raise the amount of data transferred before the magic sequence blocks the transmission, can we also reduce the amount of data before the transmission gets stuck? In short: Yes, we can. We started dividing the ISO into smaller portions, using good old split.
We split the file into 5 MB blocks. The transfer stopps in block 2.
We split block 2 into 1 MB blocks. The transfer stopps in block 3.
We split block 3 into 250 kB blocks. The transfer stopps in block 2.
We split block 2 into 5 kB blocks. The transfer stopps in block 49.
We split block 49 into 1 kB blocks. The transfer stopps at block 5. No, it does not really stop. wget on unix.uni-rostock.de does not even receive a response.
We started toying with the 1 kB of fun.
Omitting the first 100 Bytes does not change behaviour, the transfer stopps. Omitting the first 200 Bytes changes the behavior -- the transfer continues. Ok, long story short, we tracked the problem down to the sequence provided below the article. It starts at offset 114 of the funny kilobyte and has a lenght of 871 bytes.
Martin tried to upload this seqeunce to an external server using FTP. It failed.

We did not provide absolute offsets within the FreeBSD image file to make this more fun for the reader.

Ruling out HTTP magic

We then tried to transfer the magic sequence of doom using simple TCP. One Netcat acting as server, one as client. And -- surprise, the sequence could be transmitted. Stepping back, the whole KB block could not be transferred. Trying to transmit the whole funny kilobyte lead again to no data transfer at all, so there had to be a sequence in it which needed identification. It started at offset 105 and has a lenght of 876 bytes. It's also attached below.

Talking UDP

What, if it is a pure TCP problem? Let's switch to UDP. Using netcat, we can use it as follows to have a semi-bidirectional data exchange.
Inside the university, we initiated a connection-look-alike to our testing target. We were able to exchange simple text typed into netcat's STDIN. Then we sent the mysterious TCP killer sequence on the testing server's side. It did not appear inside the University. Interestingly, we were not able to exchange any further packets. So yes, there is also something going on with UDP packets.

Third parties

We tried to download the sequence-containing file from other network locations. As from now, we can only exclude the routers at LRNC since tested routes both working and stopping cover them.

Conclusion with a little claim

As the workday came to an end, so did our investigation. Our semi-conclusion (which is far from being complete) is that we can say:

We can also fall back to wild speculations on causes: Finally, there is our little claim: If you have an answer to this behaviour or if you want to pay us for this work (just in case we found a bug in some recent firmware or did your work) -- get in touch. :)

Update 29.11.2013 Altough neither Martin or myself got any reaction from persons in charge of the University's infrastructure, other persons did. Clemens H. Cap http://wwwiuk.informatik.uni-rostock.de/wer_sind_wir/mitarbeiter/clemens_cap/ got in touch with the University's data centre. They admitted (well, could perhaps not deny knowledge of) the existence of an intrusion protection system which is intended to be invisible to the regular user. This IPS was given an update which went wrong.
So, yes, it was an IDS (ok, IPS) gone wild. And yes, we did someone elses work.

The bonus chapter: Approaching the enemy

Martin continued research from home. His hypothesis: When the reaction on receiving the evil sequence changes a router's behaviour, e.g. the way it reacts on receiving an unexpected UDP packet, we can utilize a traceroute having the malicious sequence as payload (compared to a traceroute with a random payload or no payload at all). Hence, he used hping3 in UDP mode. This is what he got.
Test 1, Sending UDP packets with a harmless pure-ASCII config-file as payload at all to unix.uni-rostock.de.

tree:~ leo$ sudo hping3 --traceroute -V --udp unix.uni-rostock.de -p 17322 -E config.ini -d 831
using en1, addr: 192.168.3.43, MTU: 1500
HPING unix.uni-rostock.de (en1 139.30.11.11): udp mode set, 28 headers + 831 data bytes
[main] memlockall(): Resource busy
Warning: can't disable memory paging!
[... omitting first 7 hops to preserve Martin's privacy ...]
hop=8 TTL 0 during transit from ip=80.81.192.222 name=zr-fra1-be1.x-win.dfn.de
hop=8 hoprtt=49.6 ms
hop=9 TTL 0 during transit from ip=188.1.144.142 name=cr-han1-hundredgige0-1-0-0-7.x-win.dfn.de
hop=9 hoprtt=54.1 ms
hop=10 TTL 0 during transit from ip=188.1.146.146 name=xr-ham1-te3-2.x-win.dfn.de
hop=10 hoprtt=61.5 ms
hop=11 TTL 0 during transit from ip=188.1.146.189 name=xr-ros2-te1-2.x-win.dfn.de
hop=11 hoprtt=59.0 ms
hop=12 TTL 0 during transit from ip=188.1.233.102 name=kr-uni-rostock.x-win.dfn.de
hop=12 hoprtt=57.7 ms
hop=13 TTL 0 during transit from ip=139.30.0.20 name=nx1-gate.uni-rostock.de
hop=13 hoprtt=60.9 ms
A ``classic'' ICMP-trace would reveal that unix.uni-rostock.de would be hop 14. It does not reply in this test case as it does not react to this unsolicited UDP packet, or does it reply with a ICMP TTL exceeded message (since the TTL was not exceeded before reaching the target). Well, we can live with it. Next, Martin used the evil packet as payload. It changed the result notably:
tree:~ leo$ sudo hping3 --traceroute -V --udp unix.uni-rostock.de -p 17322 -E Downloads/xae -d 1000
using en1, addr: 192.168.3.43, MTU: 1500
HPING unix.uni-rostock.de (en1 139.30.11.11): udp mode set, 28 headers + 1000 data bytes
[main] memlockall(): Resource busy
Warning: can't disable memory paging!
[... omitting first 7 hops to preserve Martin's privacy ...]
hop=8 TTL 0 during transit from ip=80.81.192.222 name=zr-fra1-be1.x-win.dfn.de
hop=8 hoprtt=54.3 ms
hop=9 TTL 0 during transit from ip=188.1.144.142 name=cr-han1-hundredgige0-1-0-0-7.x-win.dfn.de
hop=9 hoprtt=58.3 ms
hop=10 TTL 0 during transit from ip=188.1.146.146 name=xr-ham1-te3-2.x-win.dfn.de
hop=10 hoprtt=57.6 ms
hop=11 TTL 0 during transit from ip=188.1.146.189 name=xr-ros2-te1-2.x-win.dfn.de
hop=11 hoprtt=60.8 ms
hop=12 TTL 0 during transit from ip=188.1.233.102 name=kr-uni-rostock.x-win.dfn.de
hop=12 hoprtt=60.8 ms
Sweet! There is a difference, we do not reach hop 13. Obviously only depending on the probe packet's payload. Just for fun, Martin repeated the test using 1k random noise as payload:
tree:~ leo$ sudo hping3 --traceroute -V --udp unix.uni-rostock.de -p 17322 -E Downloads/xrand -d 1000
using en1, addr: 192.168.3.43, MTU: 1500
HPING unix.uni-rostock.de (en1 139.30.11.11): udp mode set, 28 headers + 1000 data bytes
[main] memlockall(): Undefined error: 0
Warning: can't disable memory paging!
[... omitting first 7 hops to preserve Martin's privacy ...]
hop=8 TTL 0 during transit from ip=80.81.192.222 name=zr-fra1-be1.x-win.dfn.de
hop=8 hoprtt=51.0 ms
hop=9 TTL 0 during transit from ip=188.1.144.142 name=cr-han1-hundredgige0-1-0-0-7.x-win.dfn.de
hop=9 hoprtt=57.3 ms
hop=10 TTL 0 during transit from ip=188.1.146.146 name=xr-ham1-te3-2.x-win.dfn.de
hop=10 hoprtt=69.7 ms
hop=11 TTL 0 during transit from ip=188.1.146.189 name=xr-ros2-te1-2.x-win.dfn.de
hop=11 hoprtt=60.5 ms
hop=12 TTL 0 during transit from ip=188.1.233.102 name=kr-uni-rostock.x-win.dfn.de
hop=12 hoprtt=61.4 ms
hop=13 TTL 0 during transit from ip=139.30.0.20 name=nx1-gate.uni-rostock.de
hop=13 hoprtt=61.5 ms
... and again, we reach hop 13. So there seems to ... something ... (a bug, a problem -- deliberate denial of downloads?) within the University's network. Sad to say, it makes this article a whole lot less interesting for the outside world. Whatever.

Currently, the problem persists as the initial thee URLs were still leading to stopping downloads (tests performed on ekas, n=1):

The numbers sure look funny. Feel free to start wild speculations. We're done on this one.

Appendix A. The malicious sequence in HTTP case as hexdump

0000000 0000 95dc 0000 0a02 0000 95e8 0000 0a02
0000010 0000 95fe 0000 0a02 0000 960a 0000 0a02
0000020 0000 9629 0000 0a02 0000 9646 0000 0a02
0000030 0000 9654 0000 0a02 0000 9666 0000 0a02
0000040 0000 9672 0000 0a02 0000 9690 0000 0a02
0000050 0000 96ad 0000 0a02 0000 96b9 0000 0a02
0000060 0000 96cf 0000 0a02 0000 96db 0000 0a02
0000070 0000 9e6d 0000 0a02 0000 9e8a 0000 0a02
0000080 0000 9e96 0000 0a02 0000 9eaa 0000 0a02
0000090 0000 9eb6 0000 0a02 0000 9fbc 0000 0a02
00000a0 0000 9fd9 0000 0a02 0000 9fe5 0000 0a02
00000b0 0000 9ff8 0000 0a02 0000 a004 0000 0a02
00000c0 0000 a047 0000 0a02 0000 a064 0000 0a02
00000d0 0000 a070 0000 0a02 0000 a084 0000 0a02
00000e0 0000 a090 0000 0a02 0000 a0cd 0000 0a02
00000f0 0000 a0ea 0000 0a02 0000 a0f6 0000 0a02
0000100 0000 a106 0000 0a02 0000 a112 0000 0a02
0000110 0000 a1c9 0000 0a02 0000 a1e6 0000 0a02
0000120 0000 a1f2 0000 0a02 0000 a204 0000 0a02
0000130 0000 a210 0000 0a02 0000 a22d 0000 0a02
0000140 0000 a24b 0000 0a02 0000 a257 0000 0a02
0000150 0000 a269 0000 0a02 0000 a275 0000 0a02
0000160 0000 a32f 0000 0a02 0000 a34c 0000 0a02
0000170 0000 a358 0000 0a02 0000 a36e 0000 0a02
0000180 0000 a37a 0000 0a02 0000 a4e3 0000 0a02
0000190 0000 a501 0000 0a02 0000 a50d 0000 0a02
00001a0 0000 a520 0000 0a02 0000 a52c 0000 0a02
00001b0 0000 a5a6 0000 0a02 0000 a5c3 0000 0a02
00001c0 0000 a5cf 0000 0a02 0000 a5df 0000 0a02
00001d0 0000 a5eb 0000 0a02 0000 afcf 0000 0a02
00001e0 0000 afed 0000 0a02 0000 aff9 0000 0a02
00001f0 0000 b00f 0000 0a02 0000 b01b 0000 0a02
0000200 0000 b0d7 0000 0a02 0000 b0f5 0000 0a02
0000210 0000 b101 0000 0a02 0000 b117 0000 0a02
0000220 0000 b123 0000 0a02 0000 b597 0000 0a02
0000230 0000 b5b4 0000 0a02 0000 b5c0 0000 0a02
0000240 0000 b5d6 0000 0a02 0000 b5e2 0000 0a02
0000250 0000 ba61 0000 0a02 0000 ba7f 0000 0a02
0000260 0000 ba8b 0000 0a02 0000 baa7 0000 0a02
0000270 0000 bab3 0000 0a02 0000 c12f 0000 0a02
0000280 0000 c14d 0000 0a02 0000 c159 0000 0a02
0000290 0000 c175 0000 0a02 0000 c181 0000 0a02
00002a0 0000 c22c 0000 0a02 0000 c24a 0000 0a02
00002b0 0000 c256 0000 0a02 0000 c26e 0000 0a02
00002c0 0000 c27a 0000 0a02 0000 c5e3 0000 0a02
00002d0 0000 c600 0000 0a02 0000 c60c 0000 0a02
00002e0 0000 c62e 0000 0a02 0000 c63a 0000 0a02
00002f0 0000 caf8 0000 0a02 0000 cb15 0000 0a02
0000300 0000 cb21 0000 0a02 0000 cb43 0000 0a02
0000310 0000 cb4f 0000 0a02 0000 cdd8 0000 0a02
0000320 0000 cdf5 0000 0a02 0000 ce01 0000 0a02
0000330 0000 ce23 0000 0a02 0000 ce2f 0000 0a02
0000340 0000 cf41 0000 0a02 0000 cf5e 0000 0a02
0000350 0000 cf6a 0000 0a02 0000 cf82 0000 0a02
0000360 0000 cf8e 0000 0002
0000367

Appendix B. The malicious sequence in plain TCP case as hexdump

0000000 000a be00 0095 0200 000a dc00 0095 0200
0000010 000a e800 0095 0200 000a fe00 0095 0200
0000020 000a 0a00 0096 0200 000a 2900 0096 0200
0000030 000a 4600 0096 0200 000a 5400 0096 0200
0000040 000a 6600 0096 0200 000a 7200 0096 0200
0000050 000a 9000 0096 0200 000a ad00 0096 0200
0000060 000a b900 0096 0200 000a cf00 0096 0200
0000070 000a db00 0096 0200 000a 6d00 009e 0200
0000080 000a 8a00 009e 0200 000a 9600 009e 0200
0000090 000a aa00 009e 0200 000a b600 009e 0200
00000a0 000a bc00 009f 0200 000a d900 009f 0200
00000b0 000a e500 009f 0200 000a f800 009f 0200
00000c0 000a 0400 00a0 0200 000a 4700 00a0 0200
00000d0 000a 6400 00a0 0200 000a 7000 00a0 0200
00000e0 000a 8400 00a0 0200 000a 9000 00a0 0200
00000f0 000a cd00 00a0 0200 000a ea00 00a0 0200
0000100 000a f600 00a0 0200 000a 0600 00a1 0200
0000110 000a 1200 00a1 0200 000a c900 00a1 0200
0000120 000a e600 00a1 0200 000a f200 00a1 0200
0000130 000a 0400 00a2 0200 000a 1000 00a2 0200
0000140 000a 2d00 00a2 0200 000a 4b00 00a2 0200
0000150 000a 5700 00a2 0200 000a 6900 00a2 0200
0000160 000a 7500 00a2 0200 000a 2f00 00a3 0200
0000170 000a 4c00 00a3 0200 000a 5800 00a3 0200
0000180 000a 6e00 00a3 0200 000a 7a00 00a3 0200
0000190 000a e300 00a4 0200 000a 0100 00a5 0200
00001a0 000a 0d00 00a5 0200 000a 2000 00a5 0200
00001b0 000a 2c00 00a5 0200 000a a600 00a5 0200
00001c0 000a c300 00a5 0200 000a cf00 00a5 0200
00001d0 000a df00 00a5 0200 000a eb00 00a5 0200
00001e0 000a cf00 00af 0200 000a ed00 00af 0200
00001f0 000a f900 00af 0200 000a 0f00 00b0 0200
0000200 000a 1b00 00b0 0200 000a d700 00b0 0200
0000210 000a f500 00b0 0200 000a 0100 00b1 0200
0000220 000a 1700 00b1 0200 000a 2300 00b1 0200
0000230 000a 9700 00b5 0200 000a b400 00b5 0200
0000240 000a c000 00b5 0200 000a d600 00b5 0200
0000250 000a e200 00b5 0200 000a 6100 00ba 0200
0000260 000a 7f00 00ba 0200 000a 8b00 00ba 0200
0000270 000a a700 00ba 0200 000a b300 00ba 0200
0000280 000a 2f00 00c1 0200 000a 4d00 00c1 0200
0000290 000a 5900 00c1 0200 000a 7500 00c1 0200
00002a0 000a 8100 00c1 0200 000a 2c00 00c2 0200
00002b0 000a 4a00 00c2 0200 000a 5600 00c2 0200
00002c0 000a 6e00 00c2 0200 000a 7a00 00c2 0200
00002d0 000a e300 00c5 0200 000a 0000 00c6 0200
00002e0 000a 0c00 00c6 0200 000a 2e00 00c6 0200
00002f0 000a 3a00 00c6 0200 000a f800 00ca 0200
0000300 000a 1500 00cb 0200 000a 2100 00cb 0200
0000310 000a 4300 00cb 0200 000a 4f00 00cb 0200
0000320 000a d800 00cd 0200 000a f500 00cd 0200
0000330 000a 0100 00ce 0200 000a 2300 00ce 0200
0000340 000a 2f00 00ce 0200 000a 4100 00cf 0200
0000350 000a 5e00 00cf 0200 000a 6a00 00cf 0200
0000360 000a 8200 00cf 0200 000a 8e00
000036c

Stichworte:


Impressum