Notes about Clean Agile

These are my highlights while reading Robert C. Martin's book Clean Agile: Back to Basics.

Preface

Uncle Bob starts with a disclaimer. This book is not a work of research. Think of it as a memoir. I knew this upfront and I was looking forward to see him grumbling about what the world has done to his Agile.

Big things don't get done by big teams. Big things get done by many small teams collaborating while doing many small things. This is what programmers in 1950s and 1960s new instinctively and this is what got lost in 1970s. Couple of folks reinvented this idea and called it 'Agile'. It has been nearly 20 years from the start of this movement and Martin thinks it is already time for yet another reboot.

Acknowledgments

I usually don't read Acknowledgments, but these ones have exceptional list of names. It is basically a to-do list for following and reading the mentioned people. Ward Cunningham, Kent Beck, Martin Fowler, Ken Schwaber, Mary Poppendick, Ron Jeffries and Mike Beedle.

Chapter 1 - Introduction to Agile

While there are many movements that die out (aspect-oriented programming, logic programming, CRC cards), some become a mainstream - object-oriented programming or Agile. Agile had a competing methodology - a Scientific Management. In 1970 Winston Royce wrote a paper where he had a waterfall-like image with phases of software development process on a prominent page of the paper. Although it was not what Royce recommended, people took only this image from his paper and so the Waterfall was born and it was a dominant way to develop software for the next 30 years. It actually sounds quite reasonable. How could thoroughly analyzing and carefully designing a software solution upfront fail so spectacularly most of the time?

In 1999 there was an ongoing rumor that OO does not work well with Waterfall. Actually there was nothing special on OO that earned it this rumor, but this is just the way how rumors work. Martin was an OO consultant with C++ specialization. Martin stumbled upon Kent Beck's writings on Extreme Programming (XP). He fled to Beck's home in Oregon and there he tasted the TDD for the first time and was hooked. In summer of 2000 Beck organized a meeting about the future of XP. Martin was for creating a non-profit organization around it but many folks were against it, because of bad memories of what happened with the Design Patterns. Martin was pissed but Fowler suggested to meet again in fall. There they created an invitation for Light Weight Process Summit and invited couple of folks, including Alistair Cockburn, who had a similar idea and a list of more attendees to invite. 

And so the meeting in Snowbird was scheduled. There were 17 white middle-aged men attending. Their background was in 5 different lightweight processes. Martin was part of the XP group. Only Fowler was skeptical of any branded methodology and supported them all. At some point either Cunningham or Fowler wrote on the whiteboard the Agile Manifesto. They all agreed and then invented a name for it, the 'Agile'. After Snowbird they returned to their normal lives and most of them thought this was the end of the story. 

The iron cross of project management. Project manager can pick any 3 of the 4 qualities of the software development project, but he cannot have the fourth: good, fast, cheap and done. Good manager tries to make the project good enough, fast enough, cheap enough and done enough. Agile can help with this, but of course it is not guaranteed to succeed with it.

The way Agile helps with iron cross is by providing the data. With team velocity graph and burn-down chart the management knows what can be done in time. Agile development is first and foremost a rapid feedback-driven approach.

The first thing you know in a project is the date, of course. The requirements will constantly change, yet the date will remain, usually because of some important business reasons. 

The Waterfall way here is to divide the remaining time into time for analysis, design and implementation. The analysis and design are not hard deliverables, so the Management and Business will know something is wrong only in the last phase, the implementation, so too late. Ironical lesson learned for the next time is to do more analysis and more design upfront. 

The Agile is a different approach. It divides the remaining time into small iterations or sprints. The size of iteration is typically two weeks or a week. In Agile we never stop analyzing and designing. It is misleading to think of iteration as a mini-waterfall because there is no hand-off of analysis or design to the next phase guys. Rather all the activities continue through all the project. At the end of the iteration there are couple of stories done. This gives us the real data what could be done during an iteration. This data can be used to compute the more precise delivery date. This loss of hope-driven management is a major goal of Agile. Agile was never about going fast. Agile is about knowing as soon as possible just how screwed we are. 

What can we do to mitigate the project being late? Usually, the date cannot be postponed. But it can happen sometimes. New staff could be added, but more staff on a late project only makes it later. The new people only suck life from the old people on the project at first. Skipping quality is not an option. Low quality will always make you slower, even in the short term. This is the hard-earned lesson from being a programmer for 20-30 years. The most common way is to change the scope. Maybe, just maybe, there are some features that are just nice-to-haves. That's why we ask at the beginning of each iteration the business people what to do next. We are the developers, we can handle the dependencies between the stories (which should be minimal anyway).

So this is Agile from 20,000 feet high. It is an iterative process, where every iteration produces a data about the schedule. Features are implemented in the order of business value. Quality is kept as high as possible. The schedule is primarily managed by manipulating scope. That's Agile.

Chapter 2 - The Reasons for Agile

What drew Martin into Agile was the high commitment to discipline over ceremony. He thinks our industry need to improve on professionalism. He hopes that agile disciplines will be the first steps to turn computer programming into a honorable profession. 

What is expected of us? We should ship quality products. The system should be technically deployable after each iteration. Management and customers don't expect us to slow down as the project grows. Big redesigns are a complete failure. They expect the system can be easily changed and that the cost of the changes will be small and proportionate. They expect continuous and steady improvement. They expect we do not fear to clean mess we find in the product. QA should find nothing. If they find anything, the development team should react in horror and find out why and fix the root cause so it never happens again. The manual tests are expensive and unnecessary so they are always target for reduction. 

Moreover Martin expects us to cover for each other. He expects us to give relative estimates. He expects us to say "no" when no solution can be found. As our imaginary "CTO" he expects us to always keep learning. He expects us to teach. 

Kent Beck said that the goal of Agile was to heal the divide between the Business and the Development. He wrote the bill of rights for both parties.

Customers have the right to an overall plan and to know what can be accomplished when and at what cost. Either scopes or dates must be soft. Customers have the right to get the most possible value from every iteration. Customers have the right to see progress in a running system, proven to work by passing repeatable tests that they specify. Customers have the right to change their minds, to substitute functionality, and to change priorities without paying exorbitant costs. Customer have the right to be informed of schedule and estimate changes in time to choose how to alter the scope to meet the required date. Customers may cancel at any time and be left with a useful working system reflecting investment to date. The customers have the right to know that the date is in jeopardy so that it can be mitigated.

Developers have the right to know what is needed with clear declarations of priority. This right only applies within the context of iteration. Developers have the right to product high-quality work at all times. Nobody has right to tell developers to cut corners. Developers have the right to ask for and receive help from peers, managers and customers. Developers have the right to make and update their own estimates. Developers have the right to accept their responsibilities instead of having them assigned.

Agile is a framework of disciplines that support professional software development.

Chapter 3 - Business Practices

The business practices are the way to heal the divide between development and business.

Planning

Trivariate estimation works for large tasks. You estimate the best-case, nominal-case and worst-case. This technique is however too imprecise for day-to-day management that we need within the project. So we create user stories. Think of them as index cards. The rejection of the detail is a discipline. And it is hard. Everyone has to resist the urge to capture all the discussed details. Stories are placeholders, not requirements.

Then we assign the stories story points. We assign points to stories, so we can compare them relatively by complexity. The simpler workflows for estimation tend to work better than complex ones. The estimation should be roughly linear. The story with 2 story points should be done 2x faster than story with 4 story points. It seems imprecise, but according to law of the big numbers, it will eventually end up OK.

The iteration begins with the Iteration Planning Meeting. It should take one twentieth of the iteration. For two-week iterations it should take half day. The team estimates its velocity at first, i.e. how many story points it can complete in an iteration. The velocity is not a commitment. It is a guess. 

Then the business guys will play a Return of Investment calculation naturally. They will pick up the most important stories which do not cost too much story points.

At the midpoint of an iteration, many stories should be done. The plan for the iteration can be adjusted, stories can be removed from the sprint. 

Iterations never fail. The purpose of the iteration is to generate data for the management. It would be nice if each iteration generated code, but even if it does not, it still has to generate data. The best predictor for today's weather is yesterday's weather. This is how we use data from the iterations. Management can know quite precisely, how well the project is going.

The project is not over when all the stories are implemented. It is over when there are no more stories worth implementing. 

Stories are independent of each other. They are negotiable between business and the development. The story must have clear and quantifiable value to the business. It must be concrete enough to estimate. It should be small enough than two developers could do it in the iteration. The business should be able to articulate tests that will prove that the story has been completed.

Story estimation is typically done by Flying Fingers game. Everybody hides their hand and estimates the story on their fingers (up to 3 or 5 depending on the team). Then somebody counts 3, 2, 1 and everyone shows their estimate. Then they discuss until consensus is made.

Merging stories is simple, you just merge them. Splitting stories is more interesting, because you have to maintain their qualities, namely e.g. independency. A spike is a meta story, a story for estimating a story. It has to be planned. 

During the iteration is is far better to get the 80% stories done than get each story 80% done. Focus on driving the stories to completion. When Planning meeting ends, the developers should choose stories for which they will each individually be responsible. 

Acceptance tests writing should go quickly. We expect them all to be written before the midpoint of the iteration. If it is not the case, some of the developers should stop programming and start working on acceptance tests. If QA continues to miss the midpoint of the iteration milestone, than the ratio of QA and developers is likely wrong. The definition of "done" is when all acceptance tests pass. 

The demo should include showing that all acceptance tests run - including all previous acceptance tests and all unit tests. It should also show off the newly added features.

If we see the velocity raising, it usually does not mean the team is going faster. The developers are inflating the story points. The velocity is a measurement not an objective. It's control theory 101 - don't put pressure on the thing you are measuring. Remember, the only failing iteration is the iteration which fails to produce data.

If we see the velocity falling, then the most likely cause is the quality of the code. As the velocity falls, the pressure on the team grows, which can result in story points inflation, hiding the falling velocity.

One way to avoid story points inflation is the golden story trick. You have one story to compare others with. If the Fix Spelling Error in Menu Item has an estimate of 10 and the golden story Login had an estimate of 3, we know we have a story points inflation at work.

Small Releases

The development team should release their work as often as possible. The new goal is Continuous Delivery, i.e. to release the code to production after every change. We want to shorted every cycle to be able to do so. E.g. git helps here because it reduces the checkout time to almost zero.

Acceptance Tests

The practice of Acceptance Tests is one of the least understood, least used and most confused of all the Agile practices. This is strange because the underlying idea is remarkably simple: Requirements should be specified by the business. The problem is that the business people are wary of formal languages. 

The business analyst and the QA write the formal requirements and developers automate them and integrate those tests into continuous build. Business analyst specified the happy paths. QA's role is to write the unhappy paths. Developers give feedback to analysts and QA to make tests technically possible. 

When QA is the last role in the iteration, all the pressure falls on them. So QA does not run the tests. It's the programmer's job to run the tests.

Whole Team

The practice of Whole Team was initially called On-Site customer. The efficiency of teams increases simply by co-location. Remote work can never quite reach this effectivity, but it can come close. 

Chapter 4 - Team Practices

Metaphor Ubiquitous Language

For a team to communicate effectively, it needs a shared understanding of common concepts and terms. A metaphor can provide a vocabulary that allows a team to communicate effectively. On the other hand, some metaphors are silly to the point of being offensive to the customer. The more correct name for this practice should have been the Ubiquitous Language from Eric Evans' DDD.

Sustainable Pace

Software project is a marathon, not a sprint, nor a sequence of sprints. In order to win, you have to pace yourself. You must run at a Sustainable Pace. There are very rare circumstances when overtime is justified. We should be very aware that the cost of the overtime will likely be greater than the time saved on the schedule. BTW we should get plenty of sleep.

Collective Ownership

The team owns the code collectively. Knowledge becomes distributed across the team. Each member gains a better understanding of the boundaries between modules and of the overall way that the system works. This improves the ability of the team to communicate and make decisions.

Continuous Integration

In the early days of Agile, this practice meant that each developer should check in their work every couple of hours. When deployed features should not be used yet, we use feature toggles. Continuous build (e.g. in Jenkins) should never break. A broken build is a Big Deal. All programmers should stop all work and fix the build.  

Standups should take 10 minutes even for large teams. Everybody has 30 seconds.

Chapter 5 - Technical Practices

Without Technical Practices Agile becomes flaccid shell of what was intended to be. They are so essential that any effort without them is doomed to fail.

Test-Driven Development

TDD is equivalent to double-entry bookkeeping. TDD developers are not skilled at operating the debugger. Tests are the perfect kind of documentation for programmers: code. When you write the tests first according to the Three Rules, it's fun. Coverage of high 90s is likely all that is required. Martin suggests not to fail the build based on insufficient coverage. I think he meant the overall test coverage. What works for me is to enforce test coverage of the changed code (changed based on git diff). Testability is a synonym of decoupling, so writing the tests first helps you design your system correctly.  When you have a complete test suite, you loose the fear of changing the code.

Refactoring

The Red/Green/Refactor cycle means we create a failing test, make it past and then explicitly clean the code. Refactoring is a separate step. The Refactoring should never appear on a schedule. Refactoring is a minute-by-minute approach to writing software. Even when we need a big refactoring, we do it incrementally in very small steps, while still working on new features. 

Simple Design

We write only the code that is required with a structure that keeps it simple, small and most expressive. Kent Beck's steps: 

1. Pass all the tests. 
2. Reveal the intent. This is where we implement simple and more cosmetic refactorings. 
3. Remove duplication. These refactorings are usually more complicated.
4. Decrease elements. We remove unnecessary classes, functions, variables, etc.

Pair Programming

Pairing is optional. The amount of pairing should be at 50% or so. It could be as low as 30% up to 80%. The roles can be driver and navigator. Or the one writing a test and the other to making it work (Ping-Pong). With pairing we share knowledge, we may even replace code reviews. We should never ask for permission to pair, or test, or refactor. We are the experts, so we decide.

Chapter 6 - Becoming Agile

Kent Beck named 4 values of Agile - courage, communication, feedback and simplicity. There is a difference between courage and recklessness. A team that sits together and communicates frequently can work miracles. All Agile is geared toward rapid feedback for the management. Simplicity is directness in code, communication and behavior.

In the end, whichever method we choose, we will end up tweaking the method to our needs. Martin strongly suggests to adopt the full Circle of Life, especially including the technical practices. There is a term Flaccid Scrum which suffers for not adopting any technical practices and failing because of the code rot.

The barrier of the agile transformation is the middle management. These folks were hired to not take risks, to avoid directness, to follow and enforce the chain of command with a minimum of communication. We can "fake it" - do agile under the covers. It is not uncommon to see small organizations to transform to agile completely, as they have no middle management and the goals of executives and workers are aligned. Martin predicts that we will see larger companies create new divisions that use Agile for their software development.

In general, Agile coaches are not trainers. Their role is to defend the process within the team. In the heat of the development, the developers might be tempted to go off the process. The coach always reminds the team of the promises they made to themselves and the values they agreed to hold. The role typically rotates between team members. In scrum, the coach is called a Scrum Master.

The agile certifications that exist are a complete joke and an utter absurdity. There is nothing wrong with the programs, but it is foolish to train just one person in the team.

Agile movement was quickly recognized as means for organizing a small team of 4-12 developers. But how do you build "Pyramids"? Large teams are a solved problem. The problem not solved was the problem of small teams. The Agile solved the small teams software problem. There is no such thing as Agile in large. The problem of large teams is the problem of getting many diverse kinds of teams to cooperate.

Agile likes to use many tools, like whiteboards, tape, index cards, markers and various sizes of sticky notes. Agile Lifecycle Management Tools (ALMs) utterly fail at being great. Instead use a physical tools, or something lightweight like Trello. 

Damon Poole suggests an alternative view on coaching and agile in large. He suggests that instead of adopting Scrum, Kanban, XP or one of the scaling frameworks, we should consider which single practice from his list is most relevant to use, try it and repeat:

• Kanban practices - make work visible, limit work in progress, pull work through the system.
• XP practices, ...

Chapter 7 - Craftsmanship

This chapter was written by Sandro Mancuso. He warns again about implementing the highest-priority item from the backlog without any strategy or design. The result is accumulated technical debt of course. The result is many developers are moving apart from agile (or vice versa). 

The Software Craftsmanship manifesto describes an ideology, a mindset. Well-crafted software means the code is well designed and well tested. Steadily adding value means we should always be committed to continuously provide increasing value to our clients and employers. A community of professionals means that we should share and learn with each other, raising the bar of our industry. Productive partnerships means that we have a professional relationship with our clients and employers.

Methodologies and practices are like bike training wheels. They get us started. The Software Craftsmanship considers XP the best set of Agile development practices available. They criticize the Agile movement for focus only on the process and lack of focus on the engineering.

The Agile movement started to heal the divide between development and business. Unfortunately as managers flooded into it, the developers started distrusting Agile and formed Software Craftsmanship movement. The Agile and Software Craftsmanship  movements should not be separate and let's hope one day they come together again.