Summary: 2FA token was only some digits, so can be brute forced. But they implemented rate limiting, based on IP. Unfortunately the application accepted the X-Forwarded-For header as if it were the real IP and by randomizing that header, you can do as many requests as you want.
There are no legitimate request forwarders. A botnet can use tens of thousands of IP addresses, which is cutting it very tight even with 6 digits, let alone if you allow everyone to play proxy. But there are a lot of things you can do:
- If you are not too large a service, an elevated failed login attempt rate (globally for your service) could trigger things like lowering the allowed attempts per IP from 4 to 2 and lengthen the recovery period per IP from 1 to 24 hours. Or perhaps you could ordinarily allow 2 requests from different X-Forwarded-For headers with a limit of 5 per IP, but with elevated attempt rates, you only allow 2 attempts per IP and ignore the XFF header.
- Similar with elevated failed login rates to a single account: one could choose to block the account depending on its sensitivity (are you a bank or are you hacker news?) or, again, lower how many attempts one may do from different origins.
- Alternatively to limiting all IPs that attempt to login to an account under attack, you could temporarily whitelist IPs whose geoIP is from the user's last 5 cities.
- You can make tokens longer for an account under stress. Or for IPs that you never saw before (or that are from unusual countries, or from countries where it is now 5AM, or...).
You don't need to do all of this: if you're not large enough to have to bother with all of this, the solution is much simpler: rate limit without looking at anything user-controlled (such as the XFF header). For high-security applications, you should implement some monitoring and decide what to restrict (which depends on what metric you are monitoring), but still in no event use user-supplied values. Not only is it unnecessary for small services (people won't be having IP collisions often enough), it is also very likely to go wrong at some point (one day a developer will make an honest mistake) if you allow any sort of user-controlled value to influence your security limits.
Note that "IP" refers to a full IPv4 address or a /48 v6 address (those are often subscriber-assigned, though a few penny-pinching ones assign /64 or even individual addresses).
