Killing Floor 2 (or any Unreal Engine 3) dedicated server on RedHat/CentOS/Rocky/Alma Linux DDoS defense with the help of iptables/firewalld

An important update to this post from 7/3/2024 is that it’s here mostly for historical reasons, because the DDoS had ceased in the end of February 2024. None of my servers or other KF2 server hosters I keep in touch with experience it anymore. My speculation is that a good share of the server hosters defended themselves enough to reduce the amplification factor from 1:800 on average to some lower number that is no longer profitable for the malefactors and they dropped this game in favor of more lucrative surfaces from the amplification perspective. Never before there was this big of a downtime in DDoS as this, which is 4 months by now. This likely has been achieved solely through the community efforts, despite all Tripwire Interactive censorhip and bans.

This page is a home to the KF2 community DDoS defense initiative. The bulk of this post was written in 2021 on the official Tripwire Interactive’s forums, but I can’t support it there anymore. Now I’m back from a temporary ban and have an opportunity to move and update the content here.

A quick rundown of the DDoS longread

• Quick actions now and read later (ready to use firewall rules, 5 minutes and your server is protected);
• The nature of this DDoS (has nothing to do with KF2 community);
• The IP “fail2ban”-style defense basics and scripts;
• The rate limiting firewall rule;
• The TTL filtering firewall rule;
• KF2 running inside of the docker containers;
• Other approaches;
• The Tripwire Interactive’s stance on the situation (hint: it’s bad, even reputation ruining);
• Relation to the Tamari & FMX stolen mods drama (another chunk of lies);
• Conclusion.

Quick actions now and read later

If your Killing Floor 2 (or any UE3 based game server) dedicated game server behaves erratically, lags a lot, crashes suddenly multiple times a day or hangs up just minutes or hours after a restart, you press the “Join” button in the in-game server browser on your server while it’s being abused and nothing happens, you can’t join the server despite it being visible in the server browser with the “connection lost” error, grows in RAM consumption very quickly multiple times over its initial size (0.5GB ‘res’ column of top command=>1.5 GB), the window title (Windows only) shows 0 players, but thousands of connections, takes 100% of the CPU or close to that with no players present, or your hoster/ISP shuts your VM down because of “suspicious activity”, check the server log (default file name “Launch.log”) for presence of lines that look like these:

[10331.40] NetComeGo: Open TheWorld 192.119.8.93
[10331.40] DevNet: NotifyAcceptingConnection: Server TheWorld accept
[10331.40] NetComeGo: Open TheWorld 192.119.8.93
[10331.40] DevNet: NotifyAcceptingConnection: Server TheWorld accept

and so on for thousands of lines.

These are the signs of the reflected amplification UDP L3/L4 volumetric DDoS attack where your servers are abused to amplify the volume of traffic and your servers suffer from collateral damage, they aren’t the final target of this. The best working approach currently available is to put this firewall rule that filters out all the traffic with TTL values that aren’t typically seen from legitimate KF2 game clients. It’s very effective for now, however, the malefactors may adapt to this technique by simply mimicking the TTL values of the KF2 game clients. When this happens, this section will return to the combination of rate limiting and IP banning. These rules assume that your KF2 server runs on the same Linux device where you plan to filter out the malicious traffic. If your Linux device is a dedicated router instead, the FORWARD chain should be modified instead of the INPUT, just replace this word in one of the rules below. This also implies that your KF2 game server uses the default game ports, or, at the very least, they are in the UDP 7700-8700 range. Change them yourself if it’s not the case.

If you use iptables (be aware, that by default iptables rules don’t survive a reboot):

iptables -I INPUT -p UDP --dport 7700:8700 -m ttl --ttl-gt 200 --jump DROP

If you use Firewalld:

firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -p UDP --dport 7700:8700 -m ttl --ttl-gt 200 --jump DROP && firewall-cmd --reload

An important note: if you run your KF2 servers on Windows, there’s no good long-term solution for you, for a short term one check the “other approaches” section for the game port changing technique. While the IP blocking solution described below, as well as the GenZmey’s solution, written in Go, can be used on Windows, it doesn’t help well against the DDoS that adds new individual IP addresses frequently — and this is exactly what started happening in late 2022-early 2023. Unfortunately, you have to rely on a Linux or otherwise advanced router with a firewall to proxy all the KF2 traffic for it, in which case all the solutions in this article apply, but you have to replace the INPUT chain in all the firewall rules with the FORWARD chain. Or, you may decide that the inability of Windows to do the advanced firewalling, such as rate limiting or TTL analysis, is the last straw and you want to migrate your KF2 servers to Linux, which you totally should. There are success stories both ways: some moved their servers to Linux, the SPB-GS owner put a Mikrotik router in front of his Windows, on which his KF2 servers run, and relies on the FORWARD chain filtering.

You can also check the “Other approaches” section below, which talks about the server game port changes, which helps only temporarily, and hiding the server from Steam by stopping the server advertisement. At the very least, you may not be able to host your Windows-based KF2 server for random KF2 players due to the DDoS attacks, but you totally can have it hidden from the browser and play with your friends only, who will have to rely on the KF2 console “open 1.2.3.4:7777” command (change the IP and port to whatever you use).

The symptoms of this DDoS attack for servers running on Windows are exactly the same, from the appropriate logs entries in the Launch.log, through increased memory and CPU consumption, to server lags and eventually crashes, if the attack is intense enough.

The way you know it works is by checking the firewall rule counters by issuing either the nft or iptables command depending on if your setup relies on nftables or iptables. In linux, each firewall rule has a counter for it which counts all the IP datagrams the rule has triggered on, so if the firewall rule catches anything its counters should be steadily increasing. To the contrary, if the firewall rule doesn’t catch anything (because there’s no matching traffic or the rule is broken) the counters are always at zero.

If you use iptables:

iptables -xnvL --line-number

If you use nftables (remember to replace the chain INPUT with FORWARD if you filter the traffic on a dedicated linux router/VM):

nft list chain ip filter INPUT

Here’s how it looks like in both cases when the rules do indeed work as expected and drop the malicious DDoS traffic:

If you do not have admin rights over the OS in which your KF2 runs (some game hosting companies, e.g. “gameservers” do that) you may not be in position to add some or any of the firewall rules. Your only options are the IP banning script described below (in which case you may benefit from reading the logs as frequently as every minute and banning very wide IP ranges up to /8), if IP block firewall rules can be allowed, the game port change, which provides only temporary relief, and you can also attempt to prevent your server from advertising itself to Steam.

If your Linux is Debian or its flavors such as Ubuntu, there are reports of successful implementations of these anti-DDoS measures on these systems, but my support for these scenarios is limited as I don’t run them anywhere.

The nature of this DDoS

DDoS stands for distributed denial of service attacks. These are the cyber attacks perpetrated by the cybercriminals against the various Internet infrastructure. Specifically, this type of DDoS attacks are called amplification reflection-based volumetric L3/L4 distributed denial-of-service attacks.

They are “amplification”, because they involve a step in which their power is amplified, through the fact that an intermediary service is abused to amplify the volume of malicious traffic. In the case of KF2 servers, the volume of traffic is amplified 800 times.

They are “reflection-based”, because the intermediary service abused for amplification is used as a reflective surface that reflects whatever is projected onto it. To bring up a familiar analogy, imagine there was a mirror specifically engineered for that, which, if a flashlight beam landed on it, could amplify the intensity of the reflected light multiple times over to burn everything the reflected beam falls onto.

They are “volumetric”, because their goal is to just fully saturate the internet channel bandwidth of the attacked target with the sheer volume of traffic.

They are L3/L4, because they operate on layers 3 and 4 of the “Internet Protocol stack”, most of them rely on raw IP (which is a layer 3 protocol) or, as in our case, on UDP (which is a layer 4 protocol) protocols. The choice of protocols is determined by the fact that these protocols do not support handshakes by themselves, thus, making it possible to abuse intermediaries for traffic amplification, which is inadvertently sent not to the real sources of it, but to the attacked targets.

They are “distributed” because multiple sources or intermediaries are used to attack a single target.

They are “denial” because their goal is to break the availability of the attacked internet service, by rendering the service unavailable to its users.

And, finally, the “service” being attacked here are NOT Killing Floor 2 servers. Killing Floor 2 servers are the intermediaries abused to amplify the malicious traffic 800 times. The services being attacked are various businesses, typically, cloud providers, all around the world.

The approach used in these attacks isn’t much different from the regular snail mail spoofing. One can put a different address into a “from:” field on the envelope and send to some place one knows to be prone with responding with a lot of useless junk. The mail is going to reach its destination, because when the mail carrier routes the mail, they pay attention only to its destination displayed in the “to:” section of the envelope, as the “from:” section is of no use to them. Because this one place is going to use the “from:” field contents to send the mail back, which is spoofed, some unexpecting person or organization is going to receive a lot of unsolicited mail, wasting their time and resources.

These DDoS attacks have a clear purpose which lies outside of the KF2 community and has nothing to do with it. Their motives are clear, it’s either taking down the competitors in their business or holding the attacked companies or individuals for ransom, threatening with never stopping the attacks if the ransom isn’t paid. The majority of their targets are businesses, mostly various cloud services, including Google, Microsoft, OVH and other cloud providers. One of their biggest targets is Roblox, which is some kind of cloud service for building/running simplistic games. KF2 servers here just suffer from collateral damage that stems from abusing the fact that they 1) aren’t in position to verify the authenticity of the source IP address of the incoming datagrams 2) respond with 800 more voluminous responses and therefore are good for traffic amplification.

Here’s a sample of the spoofed IP addresses, so you can find out a typical set of targets yourself.

One can easily reproduce the phenomenon that is the root cause of this problem by using ncat/netcat between 2 Linux servers one of which runs the KF2 server:

1. Set up a KF2 server as described on the official TWI page. It’s free and doesn’t require registration. For the time being you can use my server at 5.161.149.88:7779, but it may not be around at this address by the time of reading;
2. Launch the KF2 server, ensure its service is available to the Internet;
3. Start collecting the traffic on that server: tcpdump -n -w test.pcap;
4. On another server install ncat or netcat and issue this command: ncat -u 1.2.3.4 7777 (change the IP and port of your KF2 server);
5. Type any letter and hit enter so it gets sent to the KF2 server and observe the screen populating with the server response for about 180 seconds (the default server connection timeout value);
6. Stop the capture from 3, load it to the wireshark program, analyze it with whatever tools available.

Here’s how a conversation analysis stats look like:

A typical response contents:

“1” is what I’ve sent to the KF2 server via ncat and the rest is the server’s response. It’s always more or less the same and the server probably would have kept responding with this garbage, but the “Duration” field hints that that 180 seconds server connection timeout kicks in:

Of course, in this example I did NOT spoof the source address of the IP datagrams, so all that garbage spewed by the server got back to ncat on the calling server. However, if I did put someone else’s IP into the source, say, Google’s DNS 8.8.8.8 IP address, all that garbage would have went Google’s way and I would have amplified my traffic 800 times over the course of 180 seconds, all multiplied again by the number of UE3 servers participating in the attack.

So far nobody has managed to provide a good explanation of what this nonsense is or what could be the purpose of all these voluminous server responses and is there a way to cut it down. But, even if there was such a way, it would likely not solve the problem, as long as the default settings stay as is. As long as a good share of the servers are configured such that the servers respond with these voluminous responses, the incentive remains for the malefactors to keep sending the spoofed traffic in hopes it is amplified. Only if the default settings are changed such that the server doesn’t respond with that much garbage towards unauthenticated IPs and most servers are updated this situation may come to an end. Obviously, this can only be done by the owner of the product, which is Tripwire Interactive, LLC.

Theoretically, a server owner can reduce these timeout counters and through that reduce the amount of traffic KF2 servers respond with. However, through testing we’ve found that significantly smaller numbers (e.g. 60 seconds or 120 seconds) cause some legitimate KF2 game clients to disconnect, especially those whose games are installed on HDD, during the map change times, so this 180 seconds number seems more or less reasonable and I don’t recommend changing it at this point. It still could be that these numbers could use some reductions, but probably rather mild and further testing is needed to establish the exact numbers.

It’s reasonable to suspect that the malefactors either have read the topic on the official TWI forums or just anticipated such a defense measure, but they adapt by changing the spoofed source IP addresses frequently, so the IP banning solution isn’t as effective as it was in the earlier days in 2021. The initial IP banning solution was banning the individual /32 IP addresses. But in order to completely saturate the attacked target’s internet connection bandwidth one can choose any IP address that is routed through this connection, which allowed the malefactors to switch the IPs quickly, as long as the spoofed IPs still belong to the same victim. To counter this, the IP banning script now suggests expanding the spoofed IP addresses into /20 subnets, because by doing it the script is much likelier to ban the whole range of IP addresses belonging to the currently DDoSed victim. For example, it makes sense to ban the whole range of IP addresses belonging to the constantly attacked Roblox Corporation’s cloud, instead of picking up each individual attacked IP from the logs and banning them one by one. The /20 subnet mask is somewhat conservative, some people opted to ban much broader /8 ranges with reportedly good results. Feel free to choose your own mask in between these 2 extremes. Note, that you don’t have to understand how IP addresses and CIDR notation network masks work. All you need is a general understanding that the lower the CIDR notation number is, the bigger the bunch of IPs the script is going to ban. For example, the /20 range bans 4096 addresses in the contiguous IP space. The /8 range bans whopping ~16 million addresses in the contiguous IP space.

Given that the malefactors likely have read the topic on the TWI forums and took some steps to bypass some of the defenses the community came up with, it’s reasonable to expect them to read this article as well and adjust the TTL values of their datagrams. When/if this happens, I’m going to change the top recommendations to fall back to the rate limiting firewall rule + IP range banning script, which may be the next best thing at this point.

But for now all their malicious datagrams have abnormally high TTL values:

There are a number of things that are wrong with the malicious datagrams that allow for good fall back techniques to be used to reliably distinguish between the good and the bad, if the malefactors decide to adjust their TTL values, but I’m not giving them clues here, as this information may prove very useful when the TTL adjustment happens.

Here’s how the rolling nature of this attack looks like:

These commands compare the banned /20 IP range lists from 2 adjacent days. You can note from here that there are hundreds of different /20 IP ranges get targeted in any given day and they have a rather small, about 10-15%, intersection between them, which makes the accumulation of the IPs in the ban list of marginal value, because yesterday’s spoofed IPs aren’t likely to be attacked tomorrow.

There’s limited evidence that other dedicated servers for games based on Unreal Engine 3 behave in a similar way. Here’s an article in Portuguese, written by people working for some kind of information security company in Brazil, that describes a very similar situation with other UE3-based game servers that host “America’s Army: Proving Grounds”. The article is dated in 2018. According the the screencaps provided, the AA:PG’s amplification factor was rather mild, at most 1:21 bytes sent:bytes amplified. This is not surprising, as one would imagine, the aforementioned server timeouts can be configured differently for different games, as appropriate development studios find reasonable for their games. For example, it’s conceivable to suggest that TWI, the game studio behind KF2, did some tests and put those 180 second timeouts based on the results, which are determined by the overall game performance at specific times, such as the map loading screen between the matches. It could very well be the case that AA:PG uses much lower values here, because the expected wait times are much shorter, thus, resulting in the much lower amplification factor.

To step back to a broader picture, this whole thing with the spoofed IP addresses and the related DDoS attacks exists on the Internet for decades. A solution was proposed long time ago to filter the ingress traffic on the last line ISP side. Say, there’s an ISP named “zgularity”, they have residential customers in the 1.2.3.0/24 range. They totally should configure their routers such, that if their customers send datagrams with source IP addresses outside of this range, they drop it. This initiative is knowns as BCP 38, it’s taking decades by now to be fully adopted and a good enough share of ISPs still don’t do it which allows the malefactors to abuse the situation.

Another parallel effort should be directed to network engineers who design network protocols. Under no circumstances your protocol should produce responses drastically disparate in their ratio to the requests coming from untrusted sources.

And just to make sure, this reflection-based amplification DDoS has nothing to do with the DoS against the KF2 web admin consoles I’ve described 2 posts ago. The differences are:

• This is DDoS, which is distributed. That is DoS, which typically has just 1 source;
• The goal of this DDoS is clear, which is the target’s bandwidth saturation, it has nothing to do with KF2 servers, they are just abused and suffer from collateral damage, while that DoS targets KF2 servers to shut them down;
• This attack isn’t unique and resembles a lot of other similar attacks, e.g. the NTP DDoS, while that DoS is pretty unique and very rarely used;
• This DDoS attack’s traffic abuses the regular KF2 UDP 7777 game ports, while that DoS traffic abuses the web server’s TCP 8080 ports;
• This DDoS is relatively harmless and, if not abused too much, doesn’t even interfere with KF2 gaming, while that DoS definitely interferes with KF2 gaming and definitely maliciously crashes the KF2 servers.

The IP “fail2ban”-style defense basics and scripts

As long as this DDoS attack stays relatively mild and doesn’t reach the intensity which causes the KF2 server to leak memory enormously, lag and eventually crash, one can use the following scripts to retrieve the fake source IP addresses from the KF2 server logs and ban them on the firewall, to prevent KF2 servers from receiving and responding to this traffic altogether. This approach has its pros and contras, but the good thing is it can be combined with other methods described in this article.

First, you need to create a set that can be populated with IP addresses by issuing these two commands:

firewall-cmd --permanent --zone=drop --add-source=ipset:networkblock
firewall-cmd --reload

The first command creates a firewall IP set named “networkblock” and associates this IP set with the “drop” zone. The second command applies the pending firewall changes.

Then you need to instruct your server to ban all the spoofed IP addresses of this DDoS by adding them to the just created IP set. For that, you need to put these lines into your crontabs (command “crontab -e”), but change the path to your log files and logfile names (I use custom logfile names, such as ServerVanillaFifth.log). This assumes that the “crontab” software is already installed and operational, which should be true for major Linux distros:

*/10 * * * * tail -5000 /home/kf2-admin/kf2-server/KFGame/Logs/Launch.log|grep -F -A2 'Connection timed out after'|awk -F" |:" '/Close/ {a[$7]++} END {for (b in a) {if (a[b]>10) {print b}}}'|uniq|while read ip; do firewall-cmd --permanent --ipset=networkblock --add-entry=$ip/20;done && firewall-cmd --reload >/dev/null 2>&1

Now let me break down what all this means.

“*/10 * * * *” is a regular “crontab” syntax that instructs your Linux scheduler to launch something periodically. This particular line instructs it to launch the following one-liner script every 10 minutes of every hour, of every day of the month, of every month, every day of the week. Note, that the 10 minute number is rather arbitrary and depends on the intensity of the attack. If the attack is intense you can launch the banning script as frequently as every minute by specifying the “* * * * *” prefix. The script is rather lightweight and shouldn’t stress the system even if invoked every minute. Note that crontab scripts cannot be invoked more frequently than once a minute.

“tail” is a Linux program that reads the last number of lines from any text file, in this case it’s 5000 lines. Note that this number is arbitrary and should be increased significantly if the attack is intense, e.g. at one point I was running 50,000 lines. Also note that KF2 populates the logs at different rates depending on your “suppress/unsuppress” settings in the ini files or other mods you may run. This number assumes a regular vanilla log rate and low DDoS intensity.

“/home/kf2-admin/kf2-server/KFGame/Logs/Launch.log” this is a path to the KF2 log file, this path is necessarily different for all KF2 server installations, find your log file and replace this section with your path.

“|” is a “pipe” special symbol, that instructs the Linux bash interpreter to accept the results of the previous command (which is 5000 lines of the KF2 server log file) and pass it on to the next command, which is

“grep -F -A2 ‘Connection timed out after'” which, basically, looks for the string in single quotes and extracts a line of text that follows this string in 2 lines after. This is the line that contains one of the spoofed source IP addresses participating in this attack.

“awk -F” |:” ‘/Close/ {a[$7]++} END {for (b in a) {if (a[b]>10) {print b}}}'” that’s a very complex Linux style text parser that basically breaks down all the lines collected in the previous steps, extracts values that reside at a certain position in a line (in our case this is IP addresses), sums them up, and, if the number of unique values is higher than 10, then it prints these IP addresses and passes on to the next command.

“uniq” this just filters out the previous list of IP addresses to ensure it has no duplicates.

“while read ip; do firewall-cmd –permanent –ipset=networkblock –add-entry=$ip/20;done && firewall-cmd –reload >/dev/null 2>&1” this line reads the list of IP addresses from the previous step and passes it to the firewall-cmd to add to the set of IP addresses which is associated with a “drop” zone. Note that the individual host IP addresses collected at this step are expanded to the whole /20 CIDR (“Classless Inter-Domain Routing”) network addresses. This is because we have found that while attacking a certain business, the malefactors use different IP addresses assigned to this business by IANA, so it totally makes sense and saves a lot of hassle and resources to ban them all at once. The /20 CIDR notation tells the firewall to ban 4096 IP addresses at once. This is pretty wide, and, theoretically, this can result in your server banning some legitimate players, who reside at IP addresses adjacent to these poor attacked businesses that you have to ban to ensure your KF2 server can continue operating. But, in practice, it’s very unlikely because these attacks are targeting commercial entities, typically their IPs are not adjacent to the residential IP ranges, from which typically KF2 players come from, as most of them play from the comfort of their homes. Basically, you are banning businesses here, not individuals.

Overall, this line just analyzes the KF2 server log for the signs of this DDoS attack going on, extracts the IP addresses and adds them to the block list of firewalld.

Understand here, that the script bans IP addresses, which denies any communication with these addresses. While the source IP addresses of the malicious datagrams are spoofed to look like they are coming from the businesses that are being attacked, the IP ban doesn’t distinguish communications with these IPs if they are malicious or not. Theoretically, your server can end up in a situation when it bans the remote service it actually needs to function. The not so far-fetched example is if your server relies on Google’s DNS 8.8.8.8 for DNS name resolution services. The malefactors behind this DDoS are known to attack Google’s DNS periodically, so this IP can easily end up in the ban list, which is going to prohibit your server from talking to Google’s DNS for name resolution.

This issue is addressable, a script can be amended such that it bans the IP addresses only for incoming traffic and only towards the UDP ports on which KF2 servers listen on. However, this significantly increases the complexity of the solution and puts additional stress on CPU. Given how rarely such situations happen (my servers don’t rely on Google DNS and so far it was the only issue experienced by a few server hosters) I don’t think it’s warranted for the script to burn the cycles doing the work that is rarely needed. However, if you struggle with this, feel free to reach out as I have some half-baked approach that takes this into account.

If you are concerned that the script may ban some IP ranges that it should not, like the Google DNS in the example above, you can set up another crontab entry to unban everything that is banned periodically. As it was shown above, the nature of these attacks is rolling, they keep adding new IP ranges and removing old ones daily such that there’s not much intersection between even 2 adjacent days in terms of IPs attacked. It doesn’t make much sense to accumulate these IPs as most of the IPs from a week ago aren’t likely to be attacked on any given day. The following example unbans everything once in three days at 6:00 in the morning:

0 6 */3 * * firewall-cmd --ipset=networkblock --get-entries|while read ip; do firewall-cmd --permanent --ipset=networkblock --remove-entry=$ip;done >/dev/null 2>&1

You can tell that this script works by checking periodically if the IP set gets populated with new banned IP ranges by issuing this command:

firewall-cmd --ipset=networkblock --get-entries

The main drawback of this method is it takes time. As the crontab entries hint, it launches only once in 10 minutes. Whatever happens during the next 10 minutes is uncontrolled for while it lasts and will be picked up only later. Theoretically, you can change the “10” to any other value to speed up or slow down the process of reading the logs and collecting the IPs, but you can’t have it running more frequently than once a minute. But it turns out that even this may be enough to ruin the games on a KF2 server as on occasion the intensity of this DDoS can reach 5000 connection attempts per second, so by the time the script needs to extract the IPs from the log the server may be already crashed.

Another minor drawback is it’s hard to distinguish between the DDoS and legitimate KF2 players whose KF2 crashes soon after joining an online game, because their session is going to timeout in a similar way the malicious connection attempts time out. This is why the script above doesn’t ban the IP addresses, unless 10 failed connection attempts were made, so if someone’s KF2 crashes, it has to crash 10 times per a rather short period of time, which is quite unlikely.

On the other hand it works wonders for the low intensity DDoS and can sit in the background for long times without any issues. If the attack’s intensity is too high for this script to handle, the rate limiting firewall rule should be implemented, read on this below.

Here’s how it looks like in action on the ntop traffic diagram:

The long green spikes downwards are the cases when the malefactors add new IP addresses to the attack. The attack traffic, which is blue on the diagram, because it’s incoming towards the KF2 servers, causes the KF2 server to respond in 800 more traffic, which produces these voluminous green responses. However, they get cut off abruptly, because the IP banning script gets invoked periodically, analyzes the logs, extracts the IP addresses, expands them to IP ranges and bans them in bulk. If not for the IP banning script, the whole graph would be filled in this disgusting green goo and would likely cause the hoster to shut down the VM.

The reason why any of these methods work is a firewall operates on the IP layer of the TCP/IP stack and is in position to drop the malicious traffic (which still is going to keep reaching the KF2 servers) with minimal resources spent on it. This is because the application layer, on which KF2 server software operates, gets stressed by this malicious traffic enormously higher than the firewall does. Not only the stress, the fact that the incoming malicious traffic gets dropped, ensures that the KF2 server never has a chance to respond with its garbage voluminous responses and amplify the malicious traffic 800 times.

There’s a ready and easy to use solution written in Go by GenZmey that does pretty much the same thing as the IP banning script above, however, I cannot support it if you have any questions or run into any issues. It can be found here:

https://github.com/GenZmeY/KF2-AntiDDoS/tree/master

This IP banning scripting work was also included into the Noobient’s “Killlinuxfloor” server management scripting solution. I also cannot support it, but here it is for your reference:

https://github.com/noobient/killinuxfloor

Another solution from baz that relies on the fact that legitimate users discover the servers first before joining them:

https://forums.tripwireinteractive.com/index.php?threads/kf2-or-any-unreal-engine-3-server-on-redhat-centos-rocky-alma-linux-ddos-defense-with-the-help-of-firewalld.2337631/page-6#post-2355626

The amplification vulnerability testing solution developed in Perl by own3mall:

https://forums.tripwireinteractive.com/index.php?threads/kf2-or-any-unreal-engine-3-server-on-redhat-centos-rocky-alma-linux-ddos-defense-with-the-help-of-firewalld.2337631/post-2355830

It’s worth noting that Tripwire Interactive, after ignoring this issue for >3 years, have finally implemented their own fail2ban style solution that does pretty much the same thing that is described in this chapter and is now available on their official KF2 dedicated server page. While, similarly to the fail2ban scripts described in this chapter, it’s not perfect, it should work at mitigating the DDoS attack to a degree. I still favor the TTL drop first. If you want to rely on the official TWI solution, I still recommend to complement it with the rate limiting, because otherwise it’s likely to not address the game lags in full.

The rate limiting firewall rule

If the DDoS attack’s intensity is too high for the IP banning script to take action before the KF2 server crumbles, the rate limiting firewall rule should be put in place. It is also okay to employ this firewall rule preemptively as it has a low chance to interfere with legitimate KF2 gaming. It is 100% compatible with the IP banning script and they both can work together very well. One can also use the rate limiting script alone, on its own, to successfully mitigate the DDoS attack to some manageable level, however, in this case the KF2 server will always keep being abused and keep participating in the DDoS attacks against the various targets on the Internet, just at a rather slow rate. This was first developed and tested by a KF2 player and KF2 server hoster who goes by the name “baz”.

Here are the rules. You need to add only one, either firewalld or iptables, whatever firewall management tool you use. Be aware that iptables by default does not preserve the rules after a reboot and firewall-cmd needs a special “–reload” command (included below) to apply the changes, all applied with firewall-cmd rules survive a reboot:

firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -p udp --dport 7700:8700 -m connlimit --connlimit-above 5 --connlimit-mask 20 -j DROP
firewall-cmd --reload

This rule assumes that you host the KF2 server on the same host you firewall it on, this determines the use of the INPUT chain. If your firewall is on a separate host (say, a router), use the FORWARD chain instead. This also implies that your KF2 game server uses the default game ports, or, at the very least, they are in the UDP 7700-8700 range. If not, replace with the port range of your servers.

Let me break down all the options here.

“firewall-cmd” is just a command line tool to manage the firewalld firewall.

“–permanent” as it says, makes the rule permanent, survive a reboot, etc.

“–direct” is an option to allow an alternative iptables-like syntax, which is used for any advanced rules.

“–add-rule” adding a firewall rule.

“ipv4” IP version 4 (not 6).

“filter” use “filter” firewall table.

“INPUT 0” use INPUT chain and put the rule on the top of it, by specifying 0 priority.

“-p UDP” use UDP protocol for initial filtering. Basically, says that this rule applies only to UDP traffic and ignores anything else.

“–dport 7700:8700” use this destination port range for filtering. Applies only to this range and ignores anything else.

“-m connlimit” a “target” of the rule, a connection limit. We are looking to alter the limit of connections we can have of that type.

“–connlimit-above 5” configuring the condition for the connection limit. It says if the number of connections with previously mentioned conditions grows above 5 do the following.

“–connlimit-mask 20” that’s the scope of the following action, what do we apply this action for. It says that we are going to get the source IP of a datagram that matches our rule and define a scope for it to be a /20 CIDR IP mask, which, basically, expands the IP address into a whole 4096 IP address range network of which this source IP is a member of.

“-j DROP” defining the traffic destination, or “jump”. Basically, for whatever traffic satisfies our criteria above, we drop it.

“firewall-cmd –reload” apply the pending firewall rules.

Then, the iptables syntax:

iptables -I INPUT 1 -p udp --dport 7700:8700 -m connlimit --connlimit-above 5 --connlimit-mask 20 -j DROP

If the rate limiting firewall rule is used, it’s highly recommended to combine it with the IP banning script, to further stifle the malefactors from abusing the KF2 servers, because the rate limiting firewall by itself doesn’t stop the abuse in full, it just prevents the excessive abuse.

You can also rely on baz’s whitelisting solution, referred above, which also completely prevents the abuse, by relying on legitimate KF2 game client peculiarities to completely filter out the malicious traffic, although you may find it rather complex.

The TTL filtering firewall rule

The firewall rule filtering by TTL came up the last, because of the research efforts by a KF2 player and KF2 server hoster “Noodle”. He noticed that all the malicious datagrams come with rather unusually high TTL numbers in the range of 230+. TTL stands for “time to live” and counts either seconds or the number of routers the datagram travels through. The “seconds” is by now redundant as this whole thing was developed back in the day when the connections were slow, so only the number of routers traversed is used. Basically, routers drop the datagrams for which this value reaches zero, so for the authors of any traffic on the internet it makes sense to assign the initial TTL values higher than the expected number of routers in between the source and the destination. It’s notable, however, that many systems assign different default TTL values for IP datagrams generated with the help of their native TCP/IP stack drivers. For example, most Linux and Unix systems have the default TTL value of 64. Windows has the default value of 128. From my experience I also know that the load balancers (e.g. Citrix NetScaler or F5 BigIP) have unusually high TTL numbers, e.g. 254.

Because the vast majority of KF2 players play on Windows and a small notable minority play on Linux with the help of Proton, it’s safe to assume that legitimate KF2 clients never use anything higher than 128. And even that number is going to be lower realistically, as it’s necessary for an IP datagram to traverse at least 1 router before it reaches the KF2 server. Thus, it’s safe to drop all the traffic with TTLs higher than 128. For the sake of caution, however, “Noodle” suggests we drop anything higher than 230.

This method works perfectly fine and has no drawbacks at this point. It reliably distinguishes between the legitimate and malicious traffic. The only potential drawback is it’s relatively easy for the malefactors to adapt to and start crafting their malicious IP datagrams with more common TTL values, so they can better blend in with the legitimate traffic. When/if this happens, the proper course of action should be to fall back to the IP banning script and the rate limiting rule. When/if this happens I’m going to alter this article’s quick actions section to suggest the IP banning script & rate limiting rule instead, but for now TTL filtering is the best solution.

Iptables syntax:

iptables -I INPUT -p UDP --dport 7700:8700 -m ttl --ttl-gt 200 --jump DROP

Firewalld:

firewall-cmd --permanent --direct --add-rule ipv4 filter INPUT 0 -p UDP --dport 7700:8700 -m ttl --ttl-gt 200 --jump DROP && firewall-cmd --reload

Once again, be aware that by default iptables rules take effect immediately, but won’t survive a reboot. On the other hand, firewall-cmd commands don’t take effect immediately and need to be applied with the “– reload” option, but do survive a reboot once applied.

I keep monitoring the state of the DDoS daily. I have two firewall rules on one of the servers, the first one is the TTL dropping rule, which is followed by the rate limiting rule:

As you can see, as of now the top rule gets almost all of the counters on dropped packets and bytes, underscored by red. When/if the malefactors adapt to the TTL defense measure the top rule should start showing zeroes on its counters, while the second rule’s counters, underscored by green, should blow up. This will be the day when I amend this article.

KF2 running inside of the docker containers

If you have control over the host VM inside of which docker containers run, I recommend running all the containers with the –net=host option. This option avoids the network bridging and avoids docker containers bypassing the existing firewall rules on the VM which are typically put into the INPUT chain. This is mainly because docker doesn’t support neither firewalld nor ufw well and the official docker recommended approach to rely on the DOCKER-USER chain to create rules that work for every container simply does not work with any firewalld rules that rely on –direct syntax. But if you run your docker containers with the –net=host option, everything suggested in this article applies to docker containers traffic as if they were just regular host processes, everything just works no different from non-dockerized applications.

An example of the docker launch command I use:

docker run --net=host -d --restart always --name hoeplus1 -v /home/kf2-admin/kf2-docker/hoeplus:/home/kf2/hoeplus -v /home/kf2-admin/steamapps:/home/kf2-server/Binaries/Win64/steamapps -v /home/kf2-admin/Cache:/home/kf2-server/KFGame/Cache -v /home/kf2-admin/Logs:/home/kf2-server/KFGame/Logs -e HoePlus=hoeplus -e Launch=kf-nuked?AllowSeasonalSkins=0 kf2

Because the “–net=host” option is used, one doesn’t have to rely on -p syntax to expose the ports of the containers to the Internet, as by default all the ports of the containerized application are exposed, unless the host firewall rules are in place to prevent it. You should totally do firewall, at the very least, the admin Web Console, as it, in its turn, has another vulnerability that allows anyone on the Internet to crash the KF2 servers remotely without authentication, this vulnerability is described a few posts ago on this blog here: Tamari’s servers: the cyberattacks against competitors. That article also suggests appropriate firewall rules and discusses other defense countermeasures.

The entrypoint is a bash script that does various modding magic, but the important last KF2 server launch line that relies on the environment variables passed through the launch command above looks like this:

/home/kf2-server/Binaries/Win64/KFGameSteamServer.bin.x86_64 $Launch -ConfigSubDir=$HoePlus -log=$HoePlus.log

This setup relies on baking the steam client and KF2 vanilla installation into the docker image during the build process, while the workshop content and the logs get written into the attachable host volumes that are shared between all the KF2 docker containers running on the host VM in order to save space and optimize log analysis.

Other approaches

It’s worth noting other approaches that are either flawed or do not work well.

• You can change the game port to any random game port number, either through a config file or a command line option. This provides an immediate relief from the DDoS. This happens because by the nature of this attack the malefactors cannot monitor the attack, they have no idea does your particular server amplify the traffic or not, so they can’t tell when they need to update the attacked game port. Instead, what they likely do, is they discover the game ports through Steam API, similarly to how a server browser works. Every server by default advertises itself to Steam, so it can be found in the browser, so the malefactors learn about the server’s game port from the browser. Naturally, they have to do it periodically, so until the next discovery happens, your server is likely to be spared. If you choose to use this approach, I recommend a) choosing higher number ports for the game port, e.g. 30,000+ and b) choose a user defined steam query port, do not rely on the game to assign it automatically.
• You can remove your server from the Steam browser by prohibiting the server advertisement. Nobody would be able to join it using Quick Matching or using Steam browser or in-game browser. The only way to join would be to use the “open IP:port” console command. But this removes your server from the DDoS because there’s no easy way for the malefactors to learn the IP and the game port of your server. There’s a number of ways to remove your server from the Steam browser, although some of them are less researched, some of them are: a) configure your firewall to allow the game port, but disallow the Steam query port b) set the bShouldAdvertise variable to false (no easy way to do it) c) use Forrest Mark X’s “Unofficial KF2 Patch” which is discouraged for obvious reasons. You shouldn’t want to patronize and support mod thieves and suspected cybercriminals who DoS their competition, if you are a decent man of good moral character.
• Advanced firewalls, including relatively cheap ones, such as Mikrotik or pfSense, allow for complex filtering, such as by TTL and also allow for rate limiting. The IP banning script, obviously, still needs to run on a host where KF2 is installed, but, theoretically, if the firewall management can be scripted (e.g. via API) it should be doable to configure the firewall remotely by feeding it with the IP addresses/ranges to ban. However, most routers that are sold on the home router market are not advanced enough to do any filtering discussed in this article, e.g. Ubiquity, etc. The downsides of this approach are you are on your own at navigating the GUI of these firewalls and setting up the rules that resemble the ones provided in this article, as most such firewalls don’t allow for direct iptables/nftables manipulation. Plus, these firewalls are at the very least not cheap;
• Changing the game port to any 5-digit number (i.e. anything between 10,000 and 65,535) drastically reduces the amount of malicious traffic due to a bug in the DDoS software which causes it to truncate the port number to the first 4 digits. Read more about this discovery here. Given how popular this particular approach has been among the community KF2 server hosters, it was probably the #1 contributor to this DDoS not being lucrative enough anymore.

The Tripwire Interactive’s stance on the situation

Tripwire Interactive is a game studio that develops the Killing Floor 2 game and thus is responsible for its code. It’s reasonable to expect the vendor to at the very least acknowledge such a pervasive problem as we have with this DDoS, and take actions to correct it. Without this issue corrected a lot of commercial businesses on the internet have to suffer and lose profits because their businesses are affected by the huge amounts of traffic their equipment was not designed to handle. In some cases, the Internet Service Providers may drop the attacked businesses completely to save others when the situation becomes hard to handle for them, a situation known as “null-route”. KF2 players can’t have lag-free gameplay experience, unless they use community-hosted KF2 servers that use reliable DDoS defenses. KF2 community server hosters have to deal with their servers crashing, lagging, participating in the amplification of this DDoS, getting shut down by the cloud vendors and accused of running malware, etc.

For starters, I’m going to do my best to assess the scale of this DDoS problem. This DDoS has been going on for >2 years by now. Previously I assessed that about 200-400 KF2 servers (which is about 10-20% of the total KF2 worldwide server number) get simply shut down, because either their hosters aren’t willing to invest that much time into fighting off this DDoS, or they host their servers in places that shut them down automatically, here are a few descriptions from people who found time to communicate this:

But to assess just the number of servers, consider that there was a short period around June 2023, when, for whatever reason, the DDoS had ceased for about a month. Before that the server number was around 1,800, which in its turn went down from >2,000 at the beginning of 2023. Here’s how it looked like in February 2023:

But then, when the attack had ceased in June 2023, this screencap was taken:

The number of servers rebounded temporarily likely because all those server hosters, who shut down their servers because of not being able to defend them against the DDoS, brought them back online, after they heard that the DDoS is gone. My guess is a good share of them are Windows hosters, because they don’t have any good options, but to either introduce Linux into the traffic flow or shut down their servers. However, after the attack resumed in July 2023, the server number went down to ~1,800 once again:

It was argued before that TWI doesn’t care about what’s going on with the community servers and has been advising the playerbase to use their official servers instead. But this DDoS also seems to affect the official TWI KF2 servers. TWI is the #1 KF2 hoster and hosts several hundred servers, roughly about 1/3 or soon to be 1/2 of all KF2 servers in the world. There’s no evidence or rumors to confirm one way or another, do they defend their servers against this DDoS or just let them amplify the malicious traffic. It could very well be that they don’t defend and a good share of player complaints about the official servers lagging tremendously can be best explained by their servers participating in the DDoS:

 

 

 

To compare, here’s one of the community server owners describing the player experience when the server participates in the DDoS:

The way the DDoS manifests itself is not different from a regular Internet or server lag. It just feels like your pawn or your mates pawns or zeds are teleporting, the zeds are running into the walls and then suddenly everyone dies, etc. Of course, a good share of the bigger community KF2 server hosters set up proper defenses against this DDoS and one is better off never playing on the official TWI servers and playing on familiar and well-defended against this DDoS community servers instead.

Another way to measure the impact of this DDoS on the KF2 server hosting is to measure the popularity of the forum topic I’ve created on the TWI official forums, where I’ve offered my initial analysis of this DDoS and proposed a couple of solutions that started the community collaboration.

And here’s this topic surrounded by other typical everyday topics on TWI forums, compare the number of views and the number of replies:

As you can see a typical topic on the official TWI forums gets a couple of dozen views, rarely into hundreds and a few replies here and there. You have to go back probably years to find a comparable long living and most viewed topic. It’s not to brag about it, as there’s not much to brag here about, as I was just doing my regular netizen and KF2 hoster duty and serving the community as every other hoster should. It’s to show that if there’s such a demand in reading and contributing to this topic, especially as compared to other topics, including official TWI announcements, this means that the issue is probably significant enough and impacts a lot of people to warrant some reaction from the vendor.

It’s very reasonable to assume that EVERY KF2 server that is exposed to the Internet and advertises itself to Steam gets abused. Some new servers, or servers that changed the game port recently, may be spared for the duration of the unknown period between the server discoveries, but eventually this thing comes to everyone. It’s reasonable to assume that all KF2 servers that are currently online, whose owners play the game periodically in order to understand when their servers lag, have employed some defenses against this attack. TWI don’t play their own game on any consistent basis and it could very well be that they just don’t care that their servers lag and crash. A good share of known major hosters reached out to me one way or another for help with anti-DDoS defenses. There’s also necessarily a share of server hosters who didn’t reach out because they just relied on the anti-DDoS solution we’ve come up with on the TWI official forums and it worked for them without issues and no further help was needed. I’ve been helping so far everyone, except for Tamari supporters and stolen code enablers (e.g. the guy behind the Fluxiservers which are now offline because I didn’t help), which I think is perfectly reasonable and understandable. Why would I help people who support the mods thieves and alleged cybercriminals and participate in the smear campaign against the good names of people from whom the mods were stolen or whose servers were DoSed? I don’t care as much if they are ignoramuses who bought into Tamari’s lies and stubbornly defend him or support him despite knowing that he does, indeed, steal mods, lies and likely engages in cyberattacks against his perceived enemies.

That was the impact on the KF2 gaming, which, in the end, may not be the most profound impact on humanity this situation results in. After all, the damage inflicted on KF2 gaming is the collateral damage. The main reason these attacks are engaged in is to shut down various Internet infrastructure through saturating the Internet connections with voluminous UDP floods.

Let me now do brief back of the napkin calculations on how much garbage traffic these attacks may generate. In all these calculations it is assumed that 1 byte = 8 bits, 1 Kbit = 1024 bits and so on, all binary, no decimal SI garbage. Let me bring up this screencap of just 1 amplification test again:

The basic facts about this communication:

• The “on the wire” ethernet frame size for almost all frames (except 2) in this exchange amounts to 44 bytes. This includes the ethernet frame header, IP header, UDP header and 2 bytes of the actual payload. Because the payload of these frames/packets/datagrams is so small, the majority of it comes from the headers, which makes the number of packets more important than the total volume of data.
• The total server response, as measured by a total size of all ethernet frames, is 39,472 bytes, or 315,776 bits. There’s a slight discrepancy with the way Wireshark’s “conversations” analyzer calculates its values in a weird way, but Wireshark is slightly off here.
• The total time over which these bytes were sent is 180 seconds, which is determined by the timeout value configured on the server side, the 180 is the default one. Which means that 99% of the KF2 server hosters have it at that, as the default values are rarely changed if there’s no incentive to do it.
• The total rate of the relevant server traffic (i.e. egress only traffic heading towards the real DDoS target) is therefore 1,754.3(1) bits/second on the ethernet layer of the TCP/IP stack.

Here’s a helpful table with all these calculations:

Now let’s assume that 1 KF2 server is getting exactly 1,000 malicious requests to connect per second, which is a bit on the higher end of the typical intensity of this DDoS, but still multiple times lower than the top intensity ever registered. It’s easy to calculate that in this case the rate is going to reach 1,754,311.(1) bits/second.

Let’s assume that the attacked business has an internet connection of 1 Gbit/sec, which is rather typical. How many KF2 servers do we need to send requests to, assuming 1,000 requests per server rate, to fully saturate this internet connection and bring the business to a halt? Keep in mind, that typically 30-60% of the Internet connection bandwidth is consumed at any time by legitimate traffic, so, typically, to knock down a business, one needs to reach about 60% of the total bandwidth to render the service unavailable. So, in our case, this means 600 Mbits/second UDP flood. Again, it’s easy to calculate the number of KF2 servers needed to participate in such a DDoS attack by dividing 600 Mbits/sec targeted bandwidth by 1,754,311.(1) bits/sec generated UDP flood from 1 KF2 server assuming 1,000 requests per second. 629,145,600 bits/sec / 1,754,311.(1) bits/sec = 358.63 servers. Basically, by having 360 KF2 servers participating in the DDoS attack we knock out at least 1 business that relies on 1 Gbit/sec Internet connection, as counted on the ethernet layer.

After that was published I was approached by someone with the network background, who pointed out that the situation is even worse than in the calculations above. This is because if Ethernet is used as a layer 2 protocol, and that’s a vast majority of all internet communications, then the traffic is subject to three amendments: 1) the minimum size of an ethernet frame is 64 bytes and if the payload is less than that the rest of the frame is padded 2) there’s a “preamble” that precedes each frame and is always 8 bytes 3) each 2 frames get separated by an interframe gap that is always 12 bytes. None of these factors are taken into account by the Wireshark conversation analyzer. This makes the minimum size of the total ethernet packet on the wire 84 bytes, which is almost twice higher than used in the above calculations. I’m going to leave the calculations as is as this may apply to other layer 2 protocols, but be aware that for Ethernet-based communication the generated UDP flood takes roughly 2 times more volume than calculated above.

As of now we have a total of ~1,700 KF2 servers worldwide. There’s limited evidence that these DDoS attacks abuse other dedicated servers based on UE3 engine as well, e.g. America’s Army or Subsistence, but for our purposes it’s out of scope. But abusing just KF2 servers these malefactors behind this DDoS can effectively remove from the Internet several businesses at once, or force their hosters/providers to null-route all their traffic to save other customers.

There’s also a separate can of worms calculations that assess the stressing capacity based on packets per second, because all Internet routers are limited in pps capacity as well and can be brought on their knees through saturating this capacity or their bandwidth, whichever can be reached first. I may do these calculations one day as well.

If you research this DDoS on behalf of a company which incurred losses due to degraded or lost availability because of the resulting UDP flood, please notify the company that this thing exists because of the well documented negligence of Tripwire Interactive, LLC. Check with me if there’s a need to consult on if the issue can be brought to a court of law.

There’s evidence out there that people have been reaching out to TWI through their support portal as well as on their official discord server, not mentioning just creating topics in “Discussions” on the KF2 Steam Content Hub or KF subreddit, urging them to do something about it, supplying them with various evidence, screen captures, network captures and so on. Which is totally natural for the issue of this magnitude. At the very least TWI could have shared the evidence they have accumulated with the community to help at defending against this DDoS, even if they don’t use it themselves.

Yet, TWI never acknowledged this problem, spare coordinating the defense efforts or suggesting a solution or addressing the problem by providing a patch, as they should have. Mind you, TWI’s community “coordinators” find time to respond to ridiculously unimportant requests one would think can be safely ignored, example:

More than two years, hundreds of community KF2 servers lost, tons of cases of unbearable game lags, not mentioning the damage to the completely unrelated Internet infrastructure that was targeted by this DDoS. No doubt, this resulted in hundreds and hundreds of customers lost, players who gave up because of not being able to have clean gameplay without lags. This DDoS issue is arguably the biggest and most impactful issue in the KF2 history, both for KF2 gameplay and for the outside world. No reaction from TWI. Moreover, TWI by now has banned 2 out of 4 major contributors to the community DDoS defense initiative, namely “Noodle”, the author of the best solution to this DDoS as of now and myself. I’m currently out of a temporary ban on their forums I’ve received for “personal attacks” and about to get permabanned if they choose to escalate their response, which is a story for another day. Noodle received a game developer ban (so he can’t even play the game on almost any TWI and/or community hosted KF2 servers) for a harmless in-game prank.

TWI at the very least could have referred the KF2 hosters to the community developed solution on this web site or on their own official forums. While this solution is far from perfect, it works. Instead, they prohibit links to this website on their official forums by replacing the website name with asterisks, as if it was some kind of slur or curse and ban people who post the direct links to it. It can be argued that for whatever reason their KF2 support is understaffed, despite them being a 100+ employees size game studio that was recently estimated to be worth of hundreds of millions of dollars. Given the severity of both the vulnerabilities, the one that is exploited in this amplification DDoS and the one that gets exploited in the DoS against the admin web consoles, this seems to me like borderline criminal negligence and the lack of due care and due diligence. One would think that the studio would find resources to address the issue of such magnitude that exists in a game that is their #1 cash cow as of now. I can understand, but only to a degree, the lack of patching or communication for >2 years, but their active hindrance of the community self-defense efforts is inexcusable.

All that makes it hard to support this topic on the official TWI forums, so I’m moving this content here and will no longer support it over there. If you have questions or need help with the DDoS, feel free to reach out here in the comments or on Steam. At this point, the complete lack of reaction or acknowledgement from TWI, along with the bans, is a stance by itself. It’s just yet another way Tripwire keeps ruining their own game, just one item in a long list. Act accordingly.

For obvious reasons I couldn’t put this assessment on the official TWI forums, as TWI reacts disproportionately to far less intense criticisms, but now, since I’ve moved the content here, my hands are not tied anymore and I can freely write what I really think about TWI.

Update from 7/28/2024:

Tripwire Interactive, after ignoring this issue for >3 years, have finally implemented their own fail2ban style solution that does pretty much the same thing that is described in this chapter and is now available on their official KF2 dedicated server page. While, similarly to the fail2ban scripts described in this chapter, it’s not perfect, it should work at mitigating the DDoS attack to a degree. I still favor the TTL drop first. If you want to rely on the official TWI solution, I still recommend to complement it with the rate limiting, because otherwise it’s likely to not address the game lags in full.

Here’s a meme made by GenZmey about TWI’s reaction:

Relation to the Tamari & FMX stolen mods drama

One would think that the DDoS has nothing to do with the stolen KF2 mods drama, described in the previous posts on this website. And it doesn’t, in essence, except Tamari and FMX chose to make the following statements about it on the official TWI discord. It’s clickable if the font is too small to read otherwise.

Tamari claims here that “a guy who they had to kick out of his servers years ago”, and this guy is yours truly, alleges that Tamari or Forrest Mark X are responsible for these DDoS attacks. This is straight up another Tamari’s lie. Never, anywhere, publicly or in private, I alleged that either Forrest Mark X or Tamari are responsible for the DDoS attacks. I challenge every Tamari’s acolyte and Tamari or FMX themselves to show any evidence of what they claim I allege that they are responsible for the DDoS. As it’s usual with them, they never produce any evidence to supplement their lies (e.g. a screencap with the exact quote, like I do here) and I bet that this is going to end up as usual: an empty lie with no evidence to support it. Just for the sake of argument, do your best to find any evidence to supplement it and if it turns out to be a lie, ask yourself, what else he may be lying about. Challenge me on this. It will surely prove that I’m the bad person Tamari claims here I am, if you can find these allegations Tamari refers to, right? The story of the kick, by the way, is available on this humble web site a couple of posts ago.

Tamari here claims that I supposedly do spread these non-existent BS allegations in order to make him look bad. Not true, because there’s plenty of real evidence of him engaging in mods theft, various lies, ridiculous ban sprees resulting in a good share of all the CD players in the world banned from his servers, the smear campaign against the good names of myself and fanta this screencap is a good example of and so on. This is more than enough to make him look very, very bad (but still something that Tamari claims as “for no reason”). I do, however, blame them both in the DoS cyberattacks, which is a completely different thing.

It’s hard to argue that he confused the DoS and the DDoS, given how intimately familiar he is with both of them. A reminder, that he is the server hoster who had to deal with the amplification DDoS just like any other hoster had to and should now very well understand the nature of this attack. He actually shares his experiences with the DDoS in this screencap. Another reminder, he’s the server hoster who was the first to stop exposing the admin web consoles to the Internet, those same web consoles that were abused in targeted DoS attacks against fanta’s and mine servers later. Unless one wants to play an amnesia card or a complete literal schizo card, he HAS to know both these attacks in detail well enough to be capable of distinguishing between the two even if someone woke him up in the middle of the night and asked this question. Moreover, it’s rather likely that he right now relies on the work developed either by myself or with my help in defending his own servers from this DDoS.

But, on the other hand, who can confuse the DoS against the KF2 admin web consoles and the reflection-based amplification DDoS, is Tamari’s acolytes. They have no idea. Almost none of them are modders or map makers or server hosters. Some of them are, though, which makes them more complicit in Tamari’s wrongdoings. Instead, they are just regular clueless gamers. Unfortunately, instead of lifting them up by enlightening them, Tamari chooses to abuse their ignorance by spreading lies about what I say and who I am, to further tarnish my reputation and through that invalidate my message, which is that Tamari built his operation on the stolen mods profiteering and likely engaged in cyberattacks against his competitors. It’s just a continuation of his regular smear campaign that I’m a white supremacist or racist or misogynist and all other ridiculous bad names he could come up with.

Instead, this Tamari’s message is exactly what he accuses me of, it’s his own attempt to spread BS to make me look bad.

Note, that he made this statement behind the closed doors of the hidden channel of the official TWI discord, which is invitation only channel for the KF2 server hosters. Although it’s not private, I’m not a member of it, as it requires TWI’s community coordinator approvals, and they are highly unlikely to approve my request to join, despite me being one of the top server hosters in this game and arguably a hoster who contributed to the KF2 community server hosting the most through various activities, such as this DDoS defense, helping various hosters with MOTD image hosting and so on and on.

Why he doesn’t accuse me of anything publicly? Why do every accusation I make against Tamari I scrupulously publish and allow everyone with access to the Internet to challenge it? This is because I tell truth and the truth doesn’t mind to be challenged. This is why Tamari spreads this BS behind the closed doors, among his borderline-cultist acolytes, who never have courage to challenge him on any of his lies.

Another lie here is that “I do everything I can to spread this crap”. This is another Tamari’s projection. He spreads the lies about me in DMs and behind the closed doors, of which I have plenty examples of, the quoted screencap is just one of them. Right in this conversation Flux unhelpfully to their cause admits that Forrest Mark X and others do the same and reach out to people via DMs to lie about me. Again, never in public, all hush-hush sneaky accusations. I never promoted this web-site except for just a few times on my friends activity feed to give a note about the major milestones or a couple of times in my Steam group and as a website in my profile on the official TWI forums. Nowhere did I spam this. Just a measured and I would even say restrained “Just FYI” approach. Other, unaffiliated people, on the other hand, sometimes spread it, it’s true. I like to think that they do it because it’s truth and they learnt it, found it convincing, and wanted to genuinely share the truth with the community, which is a natural and commendable inclination of honest people to spread the truth.

Another screencap (again, clickable for a higher resolution image):

This message is full of lies and is self contradictory, but there’s not much I’m willing to do here, I’m tired by now of dissecting his lies and self contradictions, his acolytes should just get some courage and start confronting him on these contradictions themselves. It’s as simple as asking Tamari to provide a quote for anything he claims. E.g. Tamari says: “Carl blames me for DDoS”. And someone in the audience is like: “Yeah, but can you provide an exact quote of this”? This alone will signify the beginning of the end for him. All who “know him well” should also know well enough that he lies in almost every statement he makes and so far hasn’t made any statement publicly where either myself or other people who know the truth can challenge him. And the audacity of his lies: he blames here me, arguably one of the top contributors who helped at developing the solution to defend the community servers from this DDoS, that same solution this whole article is about, in hurting the community! He, who himself contributed exactly zero towards developing any part of this solution defames me, who contributed at the top. Moreover, it can be argued that it’s his reporting campaigns where they report me on different platforms for “harassment” that prevents me from effectively contributing to the DDoS defense on the platforms from where I was banned as a result of it.

Most of the top community KF2 server hosters have reached out to me by now, either in private or on the TWI forums, and I’ve helped almost all of them by giving tailored advice or having screen sharing troubleshooting sessions.

Another thing here that highlights his TWI’s lapdog status (the whole long read post is coming soon about it) is he is allowed to use “personal attacks”, by referring to me as “prick” and “idiot”. It’s another double standard for TWI, as they will swiftly ban you from any platform under their control if you ever allow yourself to refer to Tamari as any of these words.

The reason why no one else has to deal with “this idiot”, who is yours truly, is because nobody else steals the mods from at least 5 authors, profiteers from it, lies about it and likely engages in cyberattacks against their perceived enemies and competitors, but Tamari and Forrest Mark X do. Yeah, please be concerned about your safety, as all the criminals should.

But the best and most hilarious part here is his passage on blaming someone for DDoS being extremely ballsy, especially without rock solid evidence. While I don’t recall myself ever blaming any particular person for DDoS, I know someone else who did. Lo and behold, Forrest Mark X himself and Tamari’s hypocrisy at its finest:

What FMX is doing here? Is it “extremely ballsy of him to blame for DDoS” this Tristens dude (never heard of him)? Is FMX “trying to make a play on misinformation” here? Is “he trying to make a play on people’s lack of knowledge about the character” of this dude? Is he a “prick”? Is he an “idiot”? Is he trying to “hurt the entire game’s community, because it steers us all from the core issue at hand”? Tamari, quick! We need your misinformation expertise to establish if your own co-conspirator and co-profiteer Forrest Mark X does all that!

Of course, Tamari doesn’t care about it. It matters only when his competitors or those who blame him in stolen mods profiteering do it. If FMX does it — see, it’s a totally different story, not even worth mentioning.

As for FMX, I don’t tend to take his words here seriously, whoever this Tristens dude might be. As I already observed previously, the factual load of his statements is rather low and one is better off not taking them at face value. Here’s another questionable statement made by FMX on the TWI’s official discord, that is tangentially related to this article as well:

The update is long out by now (and multiple updates after it) and as we know it didn’t fix anything regarding the reflection attack exploit, which is the flaw in the KF2 server code that allows this DDoS to exist. There’s no evidence whatsoever that TWI either acknowledged the flaw in their KF2 server or worked on addressing it in the slightest. He was just talking BS here.

Conclusion

Use the TTL filtering approach for now, it’s the best thing that completely reduces this DDoS to zero. If it stops helping check back with me here, my sleeve is full of surprises.

Huge thank you to GenZmey, baz, Noodle, Noobient and everyone else I may have forgotten who researched and contributed to this anti-DDoS solution. Go ahead and write +rep thank you notes on their profiles, so they don’t feel like they put all those countless hours working on this in vain.

Shame on ISPs for not doing the ingress filtering on their own IP range according to BCP 38.

Shame on TWI for not working on a patch to address this DDoS, not even acknowledging the issue for >2 years, engaging in exactly zero public communication regarding this issue, banning the 50% of all the major contributors to the DDoS defense initiative, instead of encouraging the community members to work on it.

Shame on Tamari and FMX for profiteering from the stolen mods, banning the CD players from their servers, likely engaging in cyberattacks against their perceived enemies and competitors, and spreading lies about those who really contribute to the community DDoS defense.

Now that the anti-DDoS content is safely moved, I can put a reference to this web site in my post on the official TWI forums and get permabanned.

PS: as predicted:

OK, Tripwire. Let’s see how it develops.