Thursday, November 12, 2009

Scrum and Politics

Here's why so many people hate the Agile: consultants. Yes, many of them are nice people who truly believe in what they're teaching. Unfortunately, this doesn't change the fact that, at the end of the day, they get to walk away and you're stuck with whatever they did or failed to do.

Think about the process consultants for a moment: their job is to come to someone else's workplace and make them change the way they do things. They don't come uninvited, though; you have to invite them in, just like vampires. Once they're inside, they have to make things work the new way. Invariably, they will encounter resistance from people who are comfortable doing things the old way. They have to know how to filter out legitimate idiosyncrasies of a particular workplace from loads of bullshit coming their way. In other words, they have to be well-versed in workplace politics.

Above all, though, a process consultant has to demonstrate success, because that's what he's selling. The project has to be successful or the consultant's company loses face and money. And that's the bottom line: the project will be perceived as successful, no matter what. That's also the top reason why a consultant needs to be a good "office politician", because he has to turn casualties into "collateral damage".

Don't get me wrong. Like I said, a lot of those people are essentially nice; they don't do this stuff out of malice, but out of belief that they're helping. What I'm trying to explain here is that the road to hell really is paved with good intentions.

Why am I singling out Agile, though? Mainly because of the Scrum. When it comes to project management, Agile consultants will always reach for Scrum and that's the source of more than half of the trouble they bring. I hereby postulate that Scrum is especially vulnerable to office politics.

The Secret of Scrum
“If you can't get rid of the skeleton in your closet,
you'd best teach it to dance.”
George Bernard Shaw

A typical sales pitch for Scrum is based on the comparison with the "traditional waterfall" process. Don't let the consultants fool you: incremental and iterative development is not unique to Scrum. Neither is the idea that you should ship often. What really sets Scrum apart is a very simple idea: free the developers.

The whole "chicken and pig" philosophy, the existence of a Scrum Master who focuses on "removing impediments", the emphasis on self-organizing teams -- all of that aims to clear all the typical obstacles that a team faces when trying to develop a product. The theory is that, once these obstacles are out of the way, the team will be free to develop a great product. That's why they have all these people at their disposal, right? The team focuses on building great stuff and everyone else focuses on giving the team what they need. And then they all live happily ever after...

The most fascinating thing about it is that it really can and does work. I've seen a few teams that pulled it off. I've also seen a few that could have pulled it off, but the management wasn't inclined to let them try. The one thing that the consultants won't tell you or your bosses, however, is that not all teams can pull it off. That's Scrum's dirty, dark little secret. If you have a process that requires a self-organizing team, then it should be obvious that your team needs to be capable of organizing itself. It's surprising to see how many people neglect this issue.

Growing Up
“A person's maturity consists in having found again
the seriousness one had as a child, at play”

Friedrich Nietzsche

Let me spell it out clearly: self organization is a very difficult skill to obtain and maintain. The typical response to this concern is that maturity can be achieved in time and that this "only" implies a ramp-up period. While I agree with that sentiment, this is usually the point where consultants screw things up: typically the ramp-up period is one sprint or so and there's not enough support for the team. You can't just dump it on a team and expect it to "ramp up" to it. Sure, every good team wants to be free, but that doesn't mean they know how to be free. It's an ironic, but not unusual situation to obtain the freedom you've always wanted, only to realize that you don't know what to do with it.

Maturity, both for teams and their members, means many things. It's not just about adapting to the fact that you won't have a manager telling you what to do. It's also about having enough context. It's amazing how little context a typical software developer needs, as a minimum, in order to do his or her bit and live happily oblivious of the bigger picture. Many developers out there know a lot about the data model of the underlying systems or about this application server or that legacy code, but don't really understand what the products really do and what makes them successful. If you put such people on an important project and tell them to organize themselves, they'll be like fish out of water. They know how to attack technical challenges, but they don't know how to make the project succeed. This issue is also easily addressed, at least in theory: ensure that the Product Owner guides the team. In practice, however, the person who should be the Product Owner is often too busy to participate properly. In such cases, you either have an absentee Product Owner or a surrogate whose contributions are of dubious quality.

Control Chaos, But Avoid Anarchy
“It is best to do things systematically, since we are only humans,
and disorder is our worst enemy”


Yet another critical mistake some consultants will make is to take the idea of cross functional teams to an extreme. Personally, I think this mistake might be inspired by the name "scrum", which comes from rugby. The whole approach was initially inspired by a paper published by Hirotaka Takeuchi and Ikujiro Nonaka, called "The New New Product Development Game"[pdf]. In the paper, the waterfall approach is compared to a relay race and the new approach is compared to rugby, "where a team tries to go the distance as a unit, passing the ball back and forth." Other people later named it Scrum to emphasize the idea that, just like in rugby scrum, all team members must "push" together to overcome the challenges.

Ironically, people who put too much emphasis on this aspect tend to forget the bit about "passing the ball back and forth". Team members all have roles and they were put in those roles for a reason. Working together in a cross-functional team means that the team is composed of different functions that cooperate smoothly, not that every team member should be called to perform any function at any moment, even if they are capable of doing so. There's a reason you hired someone to be a QA analyst in your company. It doesn't matter if they happen to have the skills to do some other job too; unless they've been doing that other job -- in your company -- long enough to feel comfortable in it, they can't do that other job on as well as a person whose job it really is. On top of that, the money you're paying someone must have at least something to do with the job they're doing and with their contractual responsibilities. It's bad form to piss all over that in the name of "cross-functional teamwork".

Wagging The Dog
“Those who say religion has nothing to do with politics
do not know what religion is.”

Mahatma Gandhi

What does all this have to do with my initial claim that Scrum is especially vulnerable to office politics? Think about it: all of these issues can be, and often are, highly political. Saying that your team lacks maturity can be seen as an admission of failure. Alternately, it can single you out as the "negative thinker" or "that guy who opposes the change". And let's not forget that we're talking about a group of people here and that even if some of them are ready to admit the lack of maturity, others might take offense at this. All these are great incentives to sweep the problem under the carpet.

Maturity isn't the only political issue here, either. Take the Product Owner, for example. If the only person who knows enough to be the Product Owner is too high in the food chain, this means that not only he or she won't be able to participate as required, it also guarantees that nobody will want to ask them for guidance as often as necessary.

And heaven help you if you run into a consultant who's a "cross-functional extremist". Since Agile is all about moving away from "heavyweight" model, if you dare to pipe up about this issue you'll be accused of "insisting on ceremony" and "placing too much importance on titles". Do you detect the odor of politics yet?

Make It Work
“Organizing is what you do before you do something,
so that when you do it, it's not all mixed up.”

A. A. Milne

Okay, so Scrum can be thoroughly screwed up by office politics. So what else is new? It's easy to point out the problem, let's hear some solutions. As always, there's no silver bullet. There are, however, ways to avoid some common pitfalls.

First and foremost: help the team reach the necessary maturity. This is not a trivial task and it should not be underestimated. Remember all that time you're supposed to save by having a lightweight process with minimal overhead? Well, that doesn't come for free. Initially, you might have to invest a hell of a lot of time and effort into getting the team up to the point where it can truly be self-organized. Make sure that the right people are in charge of this effort. The person doing this should ideally be a project manager, hopefully in the Scrum Master role. Whatever you do, don't just dump this responsibility into the lap of whoever seems to be convenient. For example, your architects might seem like the good choice of being "team leads" who can take care of this, but you should remember that there's a reason you hired them as architects: they'll be busy making sure everything fits in the "big picture" on the technical side of things; yes, this does include guiding other team members, but not organizing and managing them.

Second, don't mistake teamwork and cross-functional approach for anarchy. The team needs to work in an organized way, which means that roles are there for a reason. Self-organization isn't a democracy, either. Everyone should be free to contribute, but if the key decisions about your product are made by "the majority", you get Design By Committee and we all know how well that usually turns out. In good teams, decisions are made by people who are the best fit for making them and there are always conflict-resolution mechanisms in place.

Third, always maintain a measure of stability. Agile is all about adapting to changes quickly, but "change" here has a specific meaning; it refers to evolving requirements, not just any kind of change. You're already pushing a big change by switching to Scrum and you also know that your project requirements are bound to evolve over time, so that means your team will be facing changes on two fronts. Try to keep the rest of their environment as stable as possible. If you really have to introduce changes, do so gradually. Changes in team structure -- such as dividing the team into smaller teams to avoid crowding the daily scrum or merging two smaller teams into a larger one -- can be very disruptive. The lack of stability might easily demoralize your team, so that you end up losing more than what you hoped to gain by restructuring.

So far, I've talked about measures that could be taken by anyone with enough authority to see them through. The last piece of advice I'd like to offer applies to consultants exclusively. It's something that should be completely unnecessary to say, yet it seems that some consultants might need to be told this: be there. Yes, I know that you're just one consultant and you're bringing change to the whole company and you have to go scurrying hither and yon to assess all the risks and ensure success and whatnot. But if you're not there to support your team all they need, they -- and their project -- are going to suffer. They're under pressure from all sides: they have an important project which must succeed and they're expected to make this happen in a new way, using a new process. Be there for your team, listen carefully to their needs and do whatever you can to help them. You're the one who understands how things should be done, so make sure that the team can rely on you. It's not enough to tell them you're there to help them; actions speak louder than words and if you're not there, you'll lose their trust quickly. And it all goes downhill from there.

Sounds hard? Welcome to the real world, where there ain't no such thing as a free lunch.