Just over 4 years after my last SAP transition at Anglian Water I was back for another. This time it wasn’t just a simple transition but a load of os/db migrations comprising 60 SAP systems comprising nearly 200 servers.
SAP is central to Anglian Water’s business providing, amongst others, management of customers, billing, maintenance planning, financial reporting and payroll. It is also crucial for a large mobile work force of more than 4000 employees and also has a multitude of complex interfaces.
The project was a huge undertaking. It involved nearly 100 AWS Staff and more than 100 Capgemini employees from Apps and Infra business units, from the UK, India, Poland, and Romania; it was also supported by IBM hardware and migration consultants for some of the larger systems.
In ‘simple terms’ the project was a hardware refresh and os/db migration from Solaris/Oracle to AIX/DB2 for ECC, BW, Portal, Gateway, Web Dispatcher, PI, Content Server, Composite Environment, GRC, Adobe Forms, Solution Manager, Redwood Batch Scheduling Systems , and Click databases. All moving from traditional SAP central services to distributed highly available instances on fancy new IBM Power8 servers with NetApp storage.
If that wasn’t daunting enough the timescales were tight, described (accurately) by some as aggressive, less than 8 months to go live: 5 separate landscapes, necessitating a non standard highly stylised SAP delivery approach. In addition goalposts were moved several times, the weather on which AW’s SAP systems depend so much caused no end of issues including cancellations of critical test downtimes, and there were also several forced key personnel changes along the way requiring extensive re-planning of resources.
Anyway, we managed all that and got through to the pre-go live week in only just about a state of readiness. We flew in 3 IBM os/db migration experts from Germany together with a specialist IBM AIX Power8 expert, the cutover basis team arrived from India, and an incident management team from Romania, the project team all booked 2 weeks solid in local Huntingdon hotels.
Then it started. First saplogon.ini, as we all know, the smallest and most insignificant SAP component which had caused problems from the beginning of the project because of the way its packaged and deployed threatened to prevent the cutover proceeding. Transport strikes in India caused travel chaos to team members in India. Then the AW PM, our key and only connection with the business had a family emergency and for a time it was touch and go as to whether they would be able to participate in the cutover. The weather got worse and worse and the go-live decision was postponed on each successive day leading up to cutover. EE, not something you might consider important for a SAP cutover, changed some national settings which went wrong and effectively took out their entire UK business mobile telephone network and all AW remote engineers were unable to synch their mobile devices. Algae levels at a critical drinking water reservoir bloomed to unacceptably high levels affecting all AW personnel and all SAP systems.
The weather forecast for the cutover weekend went from bad to worse. On the morning of the cutover, with the Go-live decision still pending, a milestone review meeting had to be cancelled when the critical resource, based in the Kolkata office had to evacuate the building because of an earthquake.
Having negotiated all the issues and still reeling from all that another critical incident was raised, not technical but another major environmental issue: a water reservoir had sprang a leak and emptied, all attempts to stem the leak and refill the reservoir failed, and water tankers had to be drafted in to supply water. AW SAP systems were required to manage operations, so with the expectation of a No-Go decision imminent a rapid re-planning exercise was frantically undertaken in attempt to best make use of all the landed resources to build the remaining non-production systems.
Fortunately, even though issues remained, AW management took the decision to go ahead with the cutover as not to do so we have far bigger and wider reaching consequences. So at 18:00 on the Friday night with only 2 hours to go before the scheduled shutdown we finally got the green light to Go.
Earlier I had stated that ‘in simple terms’ the project was a hardware refresh with an os/db migration but this belies the complexity and magnitude of what we had to do in the 3 day cutover window. All proceeded well and pretty much to plan. But even as it did so the weather outside was getting worse and worse.
By Sunday night all the systems were built and handed over to the first business testers for their evaluation. Time to leave site and get a well earned curry. We left the office and stepped out into torrential rain. Not a problem for me in my Landrover, but as we got nearer and nearer to the curry house the rain got harder and harder, then literally as I pulled up to park we got an urgent call the testers couldn’t proceed, I dropped all but the key basis resource off and set off to return the short distance to his hotel.
I’ve never seen rain like it. I could barely see the bonnet in front of me: I was wading through deep puddles and weaving between abandoned cars, then with less than a mile to go he said its left here. I couldn’t see anything and had no idea where I was so blindly followed the instruction onto the A1 heading (slowly) south towards Cambridge and away from his hotel. The weather didn’t let up and it took nearly 2 hours to get to the next junction turn round and get back up to Huntingdon to drop him off.
Eventually, I arrived back at the curry house I had no idea if I parked on double yellow lines, I couldn’t see if there were any lines, I couldn’t even see the curb, so I left my car outside the curry house where I thought the edge of the road should be in what was by then a fully flowing river. After climbing over sandbags stacked at the door I finally got to a most unsatisfactory late curry.
Having said all that, the cutover, despite the non standard approach and often releasing key basis resources to ensure parallel Capgemini projects remained on track was due to detailed advanced preparation, effort and dedication of all teams delivered on time within budget. It was described by senior management as “a great example of a collaborative delivery, demonstrating Capgemini’s ability to use an innovative delivery approach to meet the client’s needs”. AW Management were equally enthusiastic and complimentary, realising that the hard work would enable them to reap significant benefits for years to come. They also published an article that featured prominently in their in-house newspaper (though I was away at SAP TechEd for the photo shoot.)
Anglian Water News featuring SAP migration
Of course there were some post go-live issues, there always is on projects of this size. Even the usual knee jerk reactions blaming SAP for every single issue, Citrix always the worst offender, couldn’t detract from a job well done delivering a more performant, stable platform. We have left AW with a completely new environment with lots of innovative features ready for AW to exploit its capabilities in next 5 years. They will realise a massive ROI and have made a massive saving in expensive Oracle licences. So we have delivered AW significant cost savings, performance benefits and hopefully many future opportunities for us to assist with when the opportunity arises. Many thanks to all those involved.