Wednesday, July 27, 2011

Have disciplines

For 2 years I was working on a project called Resource Certification that is meant to make the internet routing safer by certifying internet resources. If you remember the YouTube hijacking you know what I'm talking about.
This project is based on Public Key Infrastructure (PKI) therefore it involves quite some cryptography. We had drafts and standards (actually we were and still are creating them while providing a reference implementation) and we were strictly following them. What happens under the hood is quite complicated and it's hard to imagine how much logic is involved by a simple operation you make on the UI. It's like an iceberg: the strength is beneath the surface and would be hard to examine by naked eyes. We developed it with Test-Driven Development therefore everything was tested. We have a lot of tests. We were disciplined enough to resist the temptation of cutting corners. We were confident that if our continuous integration server showed green boxes the system worked well. I can recall only one occasion when the system failed despite all tests passing. It was because of the way our Hardware Signing Module (HSM) works and of course, we don't use HSMs for unit testing. The bug was caught on the test machine (that does have an HSM) in the next test phase and never made it into production.
During those 2 years I got used to this confidence that if I saw green boxes the system was in good shape. We deployed with confidence and our discipline never let us down. The system is really maintainable - we even got a 5-star certificate as the result of an external audit. The stars are assigned for high quality and low maintenance cost.
A few weeks ago I was assigned to another project. I am happy to change every now and then and I was eager to do something different for a while.
How different is my new project? Here it's much more transparent what's happening. There's a shiny User Interface, no cryptography. Just by looking at the UI you can guess what might happen under the hood and your guess will most likely be right.
What I started missing here on the first day is the confidence. I had no trust in the green boxes for this project anymore. The (missing) tests let me down. There's some duplication, too. I change things and the system breaks.
Can this be fixed? Sure, we are on top of it. Our unit test coverage finally went above 80% and removed a lot of duplicated code. We have automated integration (Selenium) test for the UI. We are doing static code analysis. It's all getting back to where it should have been. I am gaining more and more confidence. While I was afraid to rename a simple field (for a reason) I can do bigger surgeries now. After a few weeks the tests we created already give me confidence for changing things.
But why were those tests not created for the first place? Why the duplication? And why was this tolerated?
In my opinion it can all be traced back to the origin of the project. The goal was not quite clear at the beginning. Neither was the way to achieve them. It all started with some spike. Then the result of the spike was not thrown away but turned into production code but not production quality code. There was a temptation to move on with what the team had at that point. It was working after all, though not thoroughly tested. The technical debt kept accumulating and at some point it all started falling back on the team.
What's the moral of this story? No matter how simple the project or your class seems to be - test it. And I mean automated tests. Defend your processes and follow your disciplines at all times. If you work TDD - keep doing that. Don't ignore tests or stop creating them. If you have a definition of done check if it's really done. Technical debt will bite you later.
And one more thing: don't cut corners even if pressure arises. Your disciplines should be followed even in the depths of a crisis. In the end it all pays off.

No comments:

Post a Comment