At 15:41 we reported apparent routing issues to EntaNet. EntaNet have informed us of a power cut at Telehouse East - London. This will affect all DSL subscribers terminating into Telehouse East based LNSs. We have not been given an estimated time to fix. We are attempting to call all customers that are effected by this. We advise as soon as further information arrives.
On the 6th June between 00:00 and 06:00 2 x 10Gbps blades in one of the routers we connect through via a transit provider will be replaced. If our connection through that router drops we will automatically connect via a diverse route but there may be a short period of routing instability. We are assured that the work will be done in a controlled fashion, completing individual blades at a time. Users directly connected to these blades will suffer a small period of downtime whilst the work is completed, however this is not anticipated as being greater than 30 minutes.
We will contact all customers who are connected directly to this router.
Following on from recent issues with the DSL service it is necessary for our ISP to replace a core router that terminates users connected to Central Pipes 1 and 6. This work will impact users of our DSL network during the times specified above and may require a rebalance the Central pipes following on from the work.
We apologise for the problems and inconvenience that this work will cause.
We are currently aware of some issues with DSL circuits and our ISP is investigating. We have high call volumes coming into support so please bear with us. We will give more information as we have it.
We would like to make you aware of a scheduled maintenance window.
On Tuesday 22nd May at 1930 we will be moving our power systems at our Telford location over to a new central redundant UPS system. We have planned closely with our electrical contractors that the down time to the current UPS supplies will be short enough that no system power will be interrupted. However, please be advised that should the changeover take longer than anticipated due to any unforeseen issues it will be necessary to power down the systems.
We are advised by the electrical contractors that the downtime at worst case would be 1 hour but we are confident of no downtime.
If this time will cause any issues to you please do let us know as a matter of urgency.
Possibly affected services:
Hosted Exchange Platform
Managed Server Hosting
Unaffected Services:
DNS
SMTP email
POP3 Email
Web Hosting
At 09:38 Rockford NOC received notifications of ADSL and SDSL services have failed. This appears to be affecting only certain customers, connected to a particular BT Central.
We have liased with Entanet, they inform that engineers are currently investigating as a matter of urgency.
Our Primary service provider has recently become aware of a potential security risk that affects all their core routers. In order to implement the fix, it is necessary to upgrade all IOS software on the core routers. They will be undertaking this work this morning and will have it completed by midday. Each successful upgrade will require a minimum of 15 minutes of downtime as reboots will be necessary. In order to minimise the amount of time taken, multiple core routers will be upgraded simultaneously. Our expectation is that as many as 5 core routers may be down at any point in time.
Naturally we apologise for the inconvenience that this may cause.
Today we will be removing all old email scanners from service. The move to the new platform over the last two weeks has been sucessful. We have monitored the old scanners for any genuine email over the last week and there has been none so these servers will be shut down and removed this afternoon.
We will today be replacing fw2.dc3-telf with a higher specification device. This device protects 8 mail scanners. Expected down time will be arrroximatley 5 minutes. However, all mail will fall over to the replica mail servers during this time so there will be no loss of service.
At 16:20 power to central Telford was interrupted. The systems team took the decision to gracefully shutdown some clients servers hosted in our Data Centre which were felt to be high risk or under heavy load. Power was restored to the area at approximately 16:45.
At this point we began to power customer equipment back on.
We are now a matter of weeks from having our diesel generator installed to carry all customer equipment through such episodes.