One of my co-workers started a packet capture during the attack and we found it indeed coming from a single IP address and the closer inspection of the SIP packets and some strong google-fu, his analysis indicated it was an attack by the the SIPVicious tool suite. An attack by this tool set is pretty easy to classify if the attacker has not modified the tool suite as identifies its' self as 'friendly scanner' in the SIP invites it sends.
Instead of waiting for fail2ban to catch the attack, we decided to just block the source IP on the PBX's firewall.
Really, we all expected the attack would have been halted at this point. We did the basics: confirmed where the attack was coming from, decided on the corrective measures, acted on the corrective measures immediately since they where low risk changes to our production systems but we did run into a problem. What we found was that even though SIP communications are done over UDP packets and that is a considered by many to be stateless, the firewall was still letting the attack continue!
After a bit more digging, I realized that the Linux kernel and iptables considered the connection 'established'. Most default firewall rules allow 'established' connections to pass through more or less unmolested. That left me with a bit of a problem.. how do I tell the Linux kernel to forget about any established connections with the attackers IP?
Turns out there is a tool for such a problem: conntrack
It's available in the default repositories so installing it is simple:
$ sudo apt-get install conntrack
Once you figured out the IP address the attack is coming from, either by a PCAP (packet capture) or by examining the application log (in this case /var/log/asterisk/full) you can get a list of all the connections the Linux kernel is tracking with the following command:
$ sudo conntrack -L --orig-src 220.127.116.11
The next step was to remove those established connections so that the firewall rules could block this IP permanently:
$ sudo conntrack -D --orig-src 18.104.22.168
The output of these command was similar to the first command to list all the connections, but the very last line gives a summary of how many connections where deleted.
Once that was done, the attack was stopped. It wasn't stopped completely of course, the attackers packets where still hitting the server but the firewall was dropping them before they could reach the application so the threat was neutralized but there was still some amount of bandwidth being consumed by the attacker. Unfortunately, there is little you can do about that on the server itself, you'd have to contact your ISP to block that IP completely. Some hosted services and co-location facilities do offer that type of service, so it may be worth pursuing that avenue.
Hopefully this information can help other people in stopping other types of DDOS and brute force attacks quickly. The methods we used can really apply to nearly any other application or operating system even if the tools are different. One of the key things is how well your team can communication during this type of event. We were all at the office at the time so that was incredibly simple as we were all near each other. Next important things to do is analyse the threat by checking the logs and even starting a packet capture either for immediate analysis or for full forensics later. Next is to decide how to neutralize the threat and decide if that can reasonably while in production. Finally, it's very important to assess if the attack was truly stopped and if the attacker actually got anything.