Webmail Incident Report

[#1]

The Webmail Incident Report is re-produced below. This will be distributed to customers via our newsletter soon.

Web mail Incident Report

Our customers will be aware of a recent serious security issue affecting our Webmail service. This document contains the results of our investigation and what actions we are now taking to minimise the resulting impact for our customers. We are extremely sorry for the inconvenience and upset this may have caused our customers. We hope this report will provide some reassurance to our customers about how seriously we are taking these events.

Contents

1. Incident Summary 2. Breakdown of events 3. Our response and plans going forward

1.Incident Summary

Further details, explanations and answers to questions we have been asked are below the summary section.

a.What happened?

A malicious attack was performed against vulnerability in our implementation of our Webmail software which allowed the attackers to: - Take a copy of our webmail database, which contained the email addresses of customers who have used Webmail. This also included email addresses contained in customers' online address books and addresses customers had sent to and received from, using our Webmail service. Additionally the database contained some email addresses from our previous Webmail system. - Change code in a web page, served by one of the six webmail servers, to present a pop-up page which, for a period of time, carried a Trojan payload. Only customers without up-to-date Windows software and inadequate anti-virus software would have the Trojan affect their PCs.

b.When did it happen?

Our investigations have shown that the exploit was initiated at around 17:30 on the evening of Friday the 4^th May, 2007. Customers started receiving spam on the evening of Sunday, 13^th May 2007.

c.Why did it happen?

A vulnerability within our implementation of Webmail code in our portal was discovered and used by malicious attackers. Our subsequent investigations found a number of vulnerabilities with our implementation of the Atmail application, including the vulnerability which had been exploited. This led to the decision we took to stop using the software entirely.

d. Who is suspected of this attack?

This has been the subject of a criminal investigation, which means we are not in a position to share all of the details which we are aware of. However, the timing of the attack and the sophistication of the exploit indicates a considerable amount of planning and expertise. The code was written in Russian and was of high quality.

e.Was this preventable?

With commercial hacking of this nature it just isn't possible to be 100% protected. However, from this incident we have identified a number of processes and procedures that need improving, so that we can minimise the risk of future incidents and reduce the impact of any attacks. We have also increased the frequency of our scanning process to help us identify attempted intrusions in the future.

f. What have we done to resolve the problem and prevent something similar in the future?

Since the issue came to light the entire team at PlusNet has focused on the security of our network and customer data, and email system improvements. In order to resolve the issue and limit the impact on our customers we have: - Undertaken a complete external security audit and rebuilt aspects of our platform that we felt didn't meet stringent security best practices - Created a dedicated PlusNet security team which is formally responsible for all aspects of data and software security on our platform - Pledged to continue to proactive identify and contact customers who exhibit signs of having a malware infection - Improved general security for all customers, such as the introduction of strong passwords, and additional safeguards against further attacks We are also: - Developing a method to allow customers to change from their current username-based email address. - Improving our anti-spam platform to allow more options to have mails which are identified as spam to be separated, reviewed via our website, and/or deleted without them being delivered into the main mailbox - Introducing more improvements to our help and support pages to help customers avoid spam - Researching longer term options to upgrade our Squirrelmail webmail service to a higher specification that integrates with our portal and other communication tools We'll be contacting all our customers over the coming weeks to let them know about the improvements. You can also discuss proposals in the customer forums e.g. http://portal.plus.net

2. Breakdown of events

This section of the document is structured into a timeline of events, split into four key periods: 4th May - 8th May, 9th May - 12th May, 13th May - 16th May and 17th May - 21^st May. We have included details of both what we knew at the time and what came to light afterwards. We very much hope this will answer everyone's questions in a direct and factual way, but would ask that customers help us, if they see areas which require further explanation, by commenting on the Blog version of this article. We will be happy to update these details on the basis of comments where we agree that this could be clearer or more informative.

a. Friday 4th May- Tuesday 8th May

Our first indication of an issue came from two tickets picked up on Sunday 6^th May by our support team. These were flagged on a handover report by a Customer Support Centre (CSC) shift manager, but within the CSC at that time we were unable to replicate the problem. Initially, as both customers were using the same anti-virus software, the issue was thought to be related to a recent McAfee update which was misreporting a problem. By Bank Holiday Monday (7^th May), we had received around 10 reports of the same issue but had still been unable to replicate the problem internally. The significance of the reports had not been recognised at this stage, and in our overall volume of calls and tickets these 10 reports were a tiny percentage of contacts received. It has since transpired that due to the nature of the load balancing system in use on our Webmail platform that connections were 'Persistent'. "This means that each computer is connected to the same server on each attempt, hence not being able to hit the affected server. This resulted in tests from within our CSC all hitting an unaffected server and as such, replicating this issue from our test machines proved to be very difficult. In the early hours of Wednesday 9^th May, we identified the real source of the reported problem and were able to begin taking responsive action.

b. Wednesday 9th May - Saturday 12th May

On the morning of Wednesday 9^th May we formed an incident response team made up of staff from Networks, Development and Customer Support and a priority problem (42694) was raised. A number of HTML files had been modified on the server 'WM04', which was one of our six public Webmail servers located in London. Each of these machines was a Linux server, operating the Apache Web server software and the Atmail Webmail code. The files which were changed had additional code added to them which generated an 'IFRAME' pop-up containing a link to a video on a Russian website which, when played, activated the Trojan. At this stage we were focussed on understanding the exploit which had resulted in customers potentially being exposed to a malicious webpage. We were not aware that customer email addresses had been obtained illegally at this point. As the significance of the problem was understood we decided to take our Webmail service off-line to check the other servers for the same compromise. A Service Status notice was posted and our Webmail servers were taken offline at 16h00 on Wednesday 9^th May 2007. Once we had removed the affected server, we did run a thorough check for the same problem as had affected WM04 across the platform and no compromise was found on the other servers. The underlying problem was resolved by taking the WM04 server out of action so we returned the Webmail platform to service. From Wednesday afternoon, we identified customers with the Trojan data signature on their line; blocked the Trojans from getting config updates and we also looked for changes in customer SMTP traffic patterns. Using this data our CSC began calling customers who exhibited symptoms of the Trojan to offer advice on remedial action.

c. Sunday 13th May - Tuesday 15th May

While we were continuing to address the Trojan issue, a new issue (42837) was raised on the evening of Sunday 13^th May. This followed numerous reports of spam to mailboxes which had never received spam before. This immediately took priority over the Trojan investigation due to the number of customers affected but the two issues were worked in parallel. On initial investigation by our on-call engineers we were unable to locate the specific cause of this and none of our monitoring showed a notable increase in overall mail volume on our platform (less than 2% difference to the same period the previous week). By this time the first influx of spam seemed to have stopped. At this stage we believed that we had mitigated the original Webmail security flaw and successfully dealt with the underlying issue. It however appeared likely those responsible for injecting the Trojan on WM04 were also behind the spam outbreak. We did think it would be unlikely for anyone to have run the type of query needed to extract this information without triggering certain alarm systems we have in place. (These are targeted at looking for unusual or large database queries). Following further investigation, at 11h00 on Monday 14^th May we managed to replicate a query that could have been used to do this without triggering our monitoring. As such we began cross-checking all of the affected email addresses in order to confirm that a Webmail breach could account for all of the spam sent to email addresses that was being reported. We did eventually find every email address that was reported as receiving these spams within the Webmail system. These were stored in a number of different areas such as address book contacts, account identities and the logs of received and sent mail which were stored within the Webmail system. We issued an announcement alerting customers of our findings and continued with our full scale investigation into the breach. At that stage we decided to leave Webmail running, but took the decision to lock down logins to Webmail and our website to UK only IP addresses to reduce the risk of speculative follow-up attacks. At this stage, we had engineers on site in London who were performing integrity checks and server forensics across the entire platform. We believed that one server (WM04) had been compromised using a vulnerability in part of the Atmail code, which was coupled with some security-related weaknesses on our own server configuration. Once the affected server had been recovered for further forensic investigation, we found a malicious file disguised to look like an image file. This image file contained PHP code that allowed an attacker to run commands on our webmail server. Using this code they were able to transmit tables from the Webmail database to a remote location, and make changes to other HTML files so that visitors were presented a pop-up page containing a Trojan. After a full sweep we also found other files of a similar nature dated from the 4th May 2007 on WM04. The attacker gained information from the Webmail database from several tables which included customer email addresses, entries in customer address books, email addresses for mail sent by customers via Webmail, email addresses for mail received by customers via Webmail. Additionally when we first implemented the Atmail solution in 2004, we imported customers email addresses contact details into the Webmail database from our main databases. We did this to make the solution more convenient for users by setting up their email account for them initially. Customers who joined us after this time only had data created in the Webmail database the first time they logged in. We advised Calacode, authors of Atmail of the incident and they provided excellent support and have since published hotfixes to the code. The incident was confined to the Webmail database and personal data including payment details were not obtained. Our Networks team worked through the night on Monday/Tuesday performing further security checks and server hardening where necessary. This included turning off Website Wizard and our WiFi portal applications, which were not running the latest version of PHP and for which a quick upgrade was not possible. Overnight we also performed penetration testing (Using Nessus and other tools) across our entire platform. In the afternoon of Tuesday 15^th May we emailed those broadband customers who were exhibiting signs of being infected by a Trojan virus. We sent an additional email on Wednesday 16^th May to dial-up customers who may have logged into WM04. Shortly after this we made a detailed Service Status to customers to explain the progress we had made in our investigation. At this time we also felt we had enough evidence of malicious intent to notify the Police, and an incident was raised.

d. Wednesday 16th May - 21st May

Our network engineers and software developers continued working throughout the night on Tuesday/Wednesday. At this stage we were planning to rebuild our Webmail service using brand new servers configured to use the PHP version of Atmail, rather than our original Perl-based version. At the time we believed this would prove more secure, and our priority was to restore a working Webmail service. As well as building servers, work continued to focus on network hardening throughout night. By Wednesday morning we had identified further potential vulnerabilities within the newest available versions of the Atmail Webmail system and decided to take our existing Webmail out of service and we commenced work on a replacement system using Open Source code. During this time we were continuing to harden every aspect of our platform security we could, and were working closely with the BT security team to ensure that our network was as secure as technically possible. This included work on areas such as the improvement of our scripts that detect abnormal sessions activity and the commencement of work to allow stronger customer passwords. By Thursday afternoon we had completed the majority of these improvements and our attention turned to restoring a working Webmail service, using alternative Webmail software. By Friday, a temporary Webmail solution based upon the SquirrelMail software had been built and we were in the final stages of security, penetration and functionality testing of the new servers and software. Our replacement Webmail service was released on Saturday evening. 3. Our Response and plans going forward Throughout the Webmail issue, we have been keen to listen to our customers, and we hope this document answers many of our customers' questions about the incident. As soon as we were aware of the severity of the situation we swung into action and we feel we can be proud of our response to the issue. Many members of our Networks, Development and Customer Support Teams gave up their social and family lives and gave their own time to work on implementing the things that we identified as needing to be done urgently. We are determined that out of these issues will come a change in our company culture We want to be regarded as an on-line business where security is placed at the forefront of everything we build. We are currently in negotiation with a number of potential partners who we will work closely with to ensure that our network and the software running on it always remains as secure as it is possible to be and that this is regularly audited by a third party. We are also reviewing processes to ensure that diagnosis, communication and resolution can be swifter in the future. We have had literally hundreds of suggestions and requests from customers regarding ways to reduce spam and we are grateful for them all. Listed below are the current items we are able to commit to at this stage, timelines for which will be provided as soon as possible. Other customer suggestions are currently being reviewed and prioritised. We have been grateful for the response and support expressed by so many of our customers and we want to repay that by ensuring that we deliver as much of what is requested as soon as we possibly can. From the suggestions we have received from customers so far we will implement the following: 1) The ability to change your username 2) Get a free .uk domain name 3) SSL encrypted connections for POP3 and IMAP email and FTP 4) Improvements to the 'Manage my mailbox' tool 5) The ability to blackhole username@username.plus.com 6) Publishing our spam detection rates on our website for comparison with other ISPs 7) Publishing our new Privacy Policy and Data Retention policies on our portal. We will contact all our customers with firm timescales for the above improvements. --- Again, apologies for the lateness of this one tonight - It's been another late one for us all. Regards, Ian Wild