Medical data management system

A serious project that turned out to be a stressful learning experience.

More information

Imagine being called in to help with a project, which needs to be migrated to a more modern and secure runtime. Imagine this project not having automated CI/CD for a variety of factors and now needing to gradually migrate everything over.

Not only that, but imagine newer versions that were developed before your involvement in the project now getting deployed and eventually surfacing underlying problems with the technologies themselves - resulting in outages that need to be mitigated ASAP.

What I did

While I did plenty of development on this project, as well as fixing bugs, I'd say that the most important thing I did was solve a production outage. It turned out that the project was in some ways "rotten to its core" because the underlying database pooling solution had issues that eventually caused threads to hang, eventually leading to a total failure of the service to work.

It's one thing to encounter such an issue and to be able to roll back to an earlier version. It's another thing to realize that such an issue had been present for a while and has surfaced only now due to the system doing more and getting used more. I actually got to go on site, to help and figure out why these failures happened and how to fix them.

Eventually I managed to avert the entire disaster, by tracking the cause of the issues down to the database thread pooling solution, ripping it out altogether and replacing it with a more stable and well tested option.

What I learnt

If it is not your project, a project that you've developed from day 1 and know, you're losing. If the project doesn't have CI/CD set up and if you don't have proper analytics and APM, you are still losing. It takes a lot to keep a project running satisfactorily. It takes very little to make a piece of software fail in a plethora of ways. Then again, you don't always get to choose which projects you work on - I think that some of the code comments were in Lithuanian or something.