There was one remarkable legacy system when I was working in a bank sometime back.
This could be summarized as an internal AWS lambda equivalent. The offering was complete with full CI, CD, unit tests, code review, automatic deployment, etc…
You could write a python script (python is the only supported language) and get it reviewed and pushed to production in about 5 minutes. Ready for the world to use.
The thing was originally created in 2008 over a few weeks. The core remained mostly still since inception but there were a number of evolution at the periphery (code review, logging, performance, authentication).
Officially it wasn’t a “strategic” solution, it didn’t exist and wasn’t advertised.
Unofficially it was quite used, by multiple lines of business, and there continued to be new developments from time to time. It was just too great to abandon.
I recall summer interns in our department who’s got their small project almost completed after 3-4 weeks leveraging this thing (summer is supposed to be 2-3 months). While other groups of interns had not figured out how to ship an application in the strategic framework du jour.
The last count had more than 500 applications running there. More remarkably, production logs shown that 80% of these were in active use.
Sadly, every good thing must come to an end in the ever churning wold of software. The thing would eventually have to be reworked or thrown away: Python 2.7 was going end of life. Authentication libraries were abandoned. Middleware had not been recompiled or upgraded in a long long time. There were a few latent UI improvements begging to be done too.
Long story short, it was upgraded, by me, turned software archeologist. Upgraded to support both python 2.7 and python 3.7, to support OpenID Connect and Kerberos, to add a few messages and errors here and there to ease usage, update the developer documentation, etc…
There was one more remarkable thing that happened while staging the upgrade to the first batch of production scripts.
Two users reported issues at one hour interval. Investigated the issue and it was limited to internet explorer. Some things didn’t work perfectly in internet explorer. Meh.
Turns out, the bank had deprecated Internet Explorer some time ago, in favor of Google Chrome (#strategic). Employees firmwide (250k) were actively pushed away from using Internet Explorer. Trying to open some sites in IE got blocked, displaying a full error page saying to use Chrome instead. This had side effect on user navigation because everything is interconnected (imagine report.example.com linking to otherreport.example.com and calling internalapi.example.com with some of URLs blocked).
Good news, this wasn’t a regression due to the recent upgrade and this was easy to sort out. Blocked Internet Explorer and sent a notice to the 5000 developers that Internet Explorer is no longer supported. Problem solved!
[For reference: It’s best to hard block IE for user experience, IE users are used to retrying in another browser. Maybe one third of internal apps developed in the last years don’t work at all outside of Chrome, the page remains blank or has broken widgets all over the place.]
The upgrade continued as planned.
There was one last minor hiccup on the way. One unique script stopped working.
The script was originally created more than a decade ago, in the months after the framework was created. This was one of the very first applications to be shipped, maybe the 8th judging from the history (there are more than 500 now).
It was not actually code, but a template of sort. Unbeknownst to all. The framework had came with its own templating language and routines (note to developers in the XXI century: please use jinja for templating). There was nothing left using that templating capability, save for this ONE last thing, that had evaded logging somehow. Made some adjustments and there it goes, back in working order.
The upgrade was completed.
The only downside is the high startup time for lambda requests (5 seconds baseline). It’s fine for serving reports and API accessed occasionally. It’s not suitable for high performance API or caching, where other technology would be better indicated, highly recommend tornado for that as a web server and web framework.
6 thoughts on “The Most Remarkable Legacy System I Have Seen”
Really curious on its architecture, and how it is implemented.
I wonder if this relates to some ancient Greek goddes 🙂
I’d love to know about code execution/isolation if you can point me to anything. Tinkering with idea of programmable note / IDE at the moment and any information on how to safely run scripts code from third parties in systems like these are of interest.
Would love to learn the details of the system was built.
Would love to learn the details of how the system was built.
Ahh I’m glad to hear the greek goddess of wisdom lives on.