Old School or Just Old?

I’ve been looking for a diplomatic way to start this rant. After several false starts and a couple of glasses of wine, I realized that beating around the bush wouldn’t do it. Instead, I'll dispense with the bullshit and jump straight into what’s bothering me:

I’m tired of being immersed in a culture of mediocrity.

There, I’ve said it. Now I have to spend the rest of the post explaining what the hell I meant by that. Don’t expect me to beat around the bush, either. If you disagree with what I say, fine, but I’m not going to sugar-coat what I think, because too much sugar-coating is part of the problem.

I want to make one thing clear right from the start: I have a good job. There’s a lot to love about it. For one thing - the most important one, really - the work I do is interesting, challenging, ambitious and with plenty of opportunities to learn new things. It’s the kind of work that I often wished I could do and the fact that my wish came true is something to be happy about.

Plus, the compensation is good enough to allow me and my family to live comfortably. To top it off, my boss is a nice dude, trusts me to do good work, helps me out when I need it and doesn’t make a fuss when I need to telecommute.

Why, then, do I describe all that as a "good job", instead of, say, "dream job"?

I guess I could blame it all on my parents. They helped me discover computers and programming when I was six and it has been my passion since then. I’m almost thirty-three now and it has been a wonderful journey so far. Most of the stuff I can do, I discovered by trying to do it and looking for information on how it can be done. Not that I mean to imply I can do or learn anything. On the contrary, there have been things I tried to do or learn and gave up. The important thing, to me, is the fact that I acknowledged the fact that I gave up: for whatever reason, I chose not to expend the necessary effort.

On that journey of discovery, I’ve had the privilege to meet quite a few brilliant people. The first time was in 1993. Having graduated as top of my class in the ground school, I sailed into my high school like Titanic and was promptly sunk by the realization that almost everyone was more brilliant than I was. It was one of the best times in my life.

Another was when I went to work for Synopsys. It had taken me a bit more than seven years and almost as many jobs to find them. Once again, I was surrounded by brilliant people, all sharing their knowledge and pooling their skills to overcome nearly impossible challenges and improve themselves and the software they were developing.

Like I said, I can blame it all on my parents. My dad used to say, "Don’t go to the army if you don’t intend to be a general." By that he didn’t mean I should be unreasonably ambitious; rather, he meant I should always do my best and try to get better. The idea is to aim as high as you can, work hard and see where that gets you.

That is why I get so pissed off by people who refuse to do their job without someone holding their hand. That is why I can’t fathom how people think they can change the code without understanding what it does or what they’re doing. That is why I can’t stand people who don’t even know the basic stuff anyone with their job description is required to know, but are quite happy to choose their own solution even after you patiently explain why they should do it differently.

The worst thing, however, is that you’re not allowed to say "No, I won’t have a half-hour NetMeeting session with you. You didn’t even try to understand what it is you’re doing. You just blindly tried to apply a completely arbitrary set of changes and when that didn’t work, you gave up and decided you need someone to do it with you. And no, I won’t do it for you, either."

Don’t get me wrong, I’m not ranting about the lack of knowledge. I’ve yet to see someone who was magically born knowing how to write code. I have no problem helping people who need a hand, either. What I’m railing against is the bad attitude, the kind you get in people who don’t even try. You can spot it right away, because they don’t even ask you questions. There’s a vast difference between:

I tried doing X, but it didn’t work. Then I looked at Y and I think it does Z, but I’m not sure why P and Q aren’t there. Am I supposed to do A and B before X?

and:

I tried doing X. It didn’t work and I don’t know why. Can we have a NetMeeting session so you can help me out?

To make things worse, managers often nurture that kind of attitude. For example, suppose you’re writing some code that has to do X and there’s a robust, mature open source library Y that helps you do X without reinventing the wheel. If you decide to use Y, then you’ve shackled yourself with maintenance of that bit of code forever. Whenever someone gets the task of doing something with that code, all they have to do is complain that they don’t know how to use Y and the task will most likely get reassigned to you. It doesn’t matter that you didn’t know how to use Y, either, before you decided to use it to do X. Apparently, you’re the only one who can learn how to use an open source library whose documentation is freely available out there.

That’s what I was referring to by "culture of mediocrity". It’s not just the belief that you don’t even have to try to learn something or expend some additional effort, it’s the belief you shouldn’t have to try. It’s the belief that being given a job is the proof that you know enough and if you don’t, it’s the fault of whoever gave you the job. It’s the belief that nobody has the right to expect you to go an inch beyond that. In extreme cases, it’s the belief that going beyond is only going to get you saddled with more work, so you’re a fool to do it.

By all means, don’t take this as a criticism of my workplace, specifically. It’s not endemic to the company I work for. Rather, it’s widespread among the BigCo type of companies. I’m not sure why that happens and why there are so few non-startup companies, like Google or Synopsys, that don’t allow that to happen on a grand scale. Frankly, I’m not in the mood to speculate about it, either.

Things aren’t likely to change, no matter what I say here. For my part, I refuse to adapt to that situation. I’ll take pleasure in working with the few brilliant people scattered around in the sea of mediocrity that surrounds us. I’ll grumble and groan about the rest, until the work stops being interesting, the pay falls below the comfort threshold or the nice boss gets replaced by someone not so nice; in short, until it's time to move on.

I like to think of myself as "old school", since I’ve gone from writing spaghetti code in ZX Spectrum’s BASIC and even worse Z80A assembler code to writing my own DSLs and compiling them to JVM.  But maybe that’s not it. After all, I can often name the song playing in an elevator or a mall. I remember when MTV actually had music videos playing all the time. One of my favorite bands broke up after thirty years of playing together. I used to have a BBS that was a point in the FidoNet. Maybe I’m not "old school", but simply old.

Whatever. If you can’t be bothered to even try to learn new things, then get off my lawn.

Scrum and Politics

Here's why so many people hate the Agile: consultants. Yes, many of them are nice people who truly believe in what they're teaching. Unfortunately, this doesn't change the fact that, at the end of the day, they get to walk away and you're stuck with whatever they did or failed to do.

Think about the process consultants for a moment: their job is to come to someone else's workplace and make them change the way they do things. They don't come uninvited, though; you have to invite them in, just like vampires. Once they're inside, they have to make things work the new way. Invariably, they will encounter resistance from people who are comfortable doing things the old way. They have to know how to filter out legitimate idiosyncrasies of a particular workplace from loads of bullshit coming their way. In other words, they have to be well-versed in workplace politics.

Above all, though, a process consultant has to demonstrate success, because that's what he's selling. The project has to be successful or the consultant's company loses face and money. And that's the bottom line: the project will be perceived as successful, no matter what. That's also the top reason why a consultant needs to be a good "office politician", because he has to turn casualties into "collateral damage".

Don't get me wrong. Like I said, a lot of those people are essentially nice; they don't do this stuff out of malice, but out of belief that they're helping. What I'm trying to explain here is that the road to hell really is paved with good intentions.

Why am I singling out Agile, though? Mainly because of the Scrum. When it comes to project management, Agile consultants will always reach for Scrum and that's the source of more than half of the trouble they bring. I hereby postulate that Scrum is especially vulnerable to office politics.

The Secret of Scrum

“If you can't get rid of the skeleton in your closet,
you'd best teach it to dance.”
George Bernard Shaw

A typical sales pitch for Scrum is based on the comparison with the "traditional waterfall" process. Don't let the consultants fool you: incremental and iterative development is not unique to Scrum. Neither is the idea that you should ship often. What really sets Scrum apart is a very simple idea: free the developers.

The whole "chicken and pig" philosophy, the existence of a Scrum Master who focuses on "removing impediments", the emphasis on self-organizing teams -- all of that aims to clear all the typical obstacles that a team faces when trying to develop a product. The theory is that, once these obstacles are out of the way, the team will be free to develop a great product. That's why they have all these people at their disposal, right? The team focuses on building great stuff and everyone else focuses on giving the team what they need. And then they all live happily ever after...

The most fascinating thing about it is that it really can and does work. I've seen a few teams that pulled it off. I've also seen a few that could have pulled it off, but the management wasn't inclined to let them try. The one thing that the consultants won't tell you or your bosses, however, is that not all teams can pull it off. That's Scrum's dirty, dark little secret. If you have a process that requires a self-organizing team, then it should be obvious that your team needs to be capable of organizing itself. It's surprising to see how many people neglect this issue.

Growing Up

“A person's maturity consists in having found again
the seriousness one had as a child, at play”
Friedrich Nietzsche

Let me spell it out clearly: self organization is a very difficult skill to obtain and maintain. The typical response to this concern is that maturity can be achieved in time and that this "only" implies a ramp-up period. While I agree with that sentiment, this is usually the point where consultants screw things up: typically the ramp-up period is one sprint or so and there's not enough support for the team. You can't just dump it on a team and expect it to "ramp up" to it. Sure, every good team wants to be free, but that doesn't mean they know how to be free. It's an ironic, but not unusual situation to obtain the freedom you've always wanted, only to realize that you don't know what to do with it.


Maturity, both for teams and their members, means many things. It's not just about adapting to the fact that you won't have a manager telling you what to do. It's also about having enough context. It's amazing how little context a typical software developer needs, as a minimum, in order to do his or her bit and live happily oblivious of the bigger picture. Many developers out there know a lot about the data model of the underlying systems or about this application server or that legacy code, but don't really understand what the products really do and what makes them successful. If you put such people on an important project and tell them to organize themselves, they'll be like fish out of water. They know how to attack technical challenges, but they don't know how to make the project succeed. This issue is also easily addressed, at least in theory: ensure that the Product Owner guides the team. In practice, however, the person who should be the Product Owner is often too busy to participate properly. In such cases, you either have an absentee Product Owner or a surrogate whose contributions are of dubious quality.

Control Chaos, But Avoid Anarchy

“It is best to do things systematically, since we are only humans,
and disorder is our worst enemy”
Hesiod

Yet another critical mistake some consultants will make is to take the idea of cross functional teams to an extreme. Personally, I think this mistake might be inspired by the name "scrum", which comes from rugby. The whole approach was initially inspired by a paper published by Hirotaka Takeuchi and Ikujiro Nonaka, called "The New New Product Development Game"[pdf]. In the paper, the waterfall approach is compared to a relay race and the new approach is compared to rugby, "where a team tries to go the distance as a unit, passing the ball back and forth." Other people later named it Scrum to emphasize the idea that, just like in rugby scrum, all team members must "push" together to overcome the challenges.

Ironically, people who put too much emphasis on this aspect tend to forget the bit about "passing the ball back and forth". Team members all have roles and they were put in those roles for a reason. Working together in a cross-functional team means that the team is composed of different functions that cooperate smoothly, not that every team member should be called to perform any function at any moment, even if they are capable of doing so. There's a reason you hired someone to be a QA analyst in your company. It doesn't matter if they happen to have the skills to do some other job too; unless they've been doing that other job -- in your company -- long enough to feel comfortable in it, they can't do that other job on as well as a person whose job it really is. On top of that, the money you're paying someone must have at least something to do with the job they're doing and with their contractual responsibilities. It's bad form to piss all over that in the name of "cross-functional teamwork".

Wagging The Dog

“Those who say religion has nothing to do with politics
do not know what religion is.”
Mahatma Gandhi

What does all this have to do with my initial claim that Scrum is especially vulnerable to office politics? Think about it: all of these issues can be, and often are, highly political. Saying that your team lacks maturity can be seen as an admission of failure. Alternately, it can single you out as the "negative thinker" or "that guy who opposes the change". And let's not forget that we're talking about a group of people here and that even if some of them are ready to admit the lack of maturity, others might take offense at this. All these are great incentives to sweep the problem under the carpet.

Maturity isn't the only political issue here, either. Take the Product Owner, for example. If the only person who knows enough to be the Product Owner is too high in the food chain, this means that not only he or she won't be able to participate as required, it also guarantees that nobody will want to ask them for guidance as often as necessary.

And heaven help you if you run into a consultant who's a "cross-functional extremist". Since Agile is all about moving away from "heavyweight" model, if you dare to pipe up about this issue you'll be accused of "insisting on ceremony" and "placing too much importance on titles". Do you detect the odor of politics yet?

Make It Work

“Organizing is what you do before you do something,
so that when you do it, it's not all mixed up.”
A. A. Milne

Okay, so Scrum can be thoroughly screwed up by office politics. So what else is new? It's easy to point out the problem, let's hear some solutions. As always, there's no silver bullet. There are, however, ways to avoid some common pitfalls.

First and foremost: help the team reach the necessary maturity. This is not a trivial task and it should not be underestimated. Remember all that time you're supposed to save by having a lightweight process with minimal overhead? Well, that doesn't come for free. Initially, you might have to invest a hell of a lot of time and effort into getting the team up to the point where it can truly be self-organized. Make sure that the right people are in charge of this effort. The person doing this should ideally be a project manager, hopefully in the Scrum Master role. Whatever you do, don't just dump this responsibility into the lap of whoever seems to be convenient. For example, your architects might seem like the good choice of being "team leads" who can take care of this, but you should remember that there's a reason you hired them as architects: they'll be busy making sure everything fits in the "big picture" on the technical side of things; yes, this does include guiding other team members, but not organizing and managing them.

Second, don't mistake teamwork and cross-functional approach for anarchy. The team needs to work in an organized way, which means that roles are there for a reason. Self-organization isn't a democracy, either. Everyone should be free to contribute, but if the key decisions about your product are made by "the majority", you get Design By Committee and we all know how well that usually turns out. In good teams, decisions are made by people who are the best fit for making them and there are always conflict-resolution mechanisms in place.

Third, always maintain a measure of stability. Agile is all about adapting to changes quickly, but "change" here has a specific meaning; it refers to evolving requirements, not just any kind of change. You're already pushing a big change by switching to Scrum and you also know that your project requirements are bound to evolve over time, so that means your team will be facing changes on two fronts. Try to keep the rest of their environment as stable as possible. If you really have to introduce changes, do so gradually. Changes in team structure -- such as dividing the team into smaller teams to avoid crowding the daily scrum or merging two smaller teams into a larger one -- can be very disruptive. The lack of stability might easily demoralize your team, so that you end up losing more than what you hoped to gain by restructuring.

So far, I've talked about measures that could be taken by anyone with enough authority to see them through. The last piece of advice I'd like to offer applies to consultants exclusively. It's something that should be completely unnecessary to say, yet it seems that some consultants might need to be told this: be there. Yes, I know that you're just one consultant and you're bringing change to the whole company and you have to go scurrying hither and yon to assess all the risks and ensure success and whatnot. But if you're not there to support your team all they need, they -- and their project -- are going to suffer. They're under pressure from all sides: they have an important project which must succeed and they're expected to make this happen in a new way, using a new process. Be there for your team, listen carefully to their needs and do whatever you can to help them. You're the one who understands how things should be done, so make sure that the team can rely on you. It's not enough to tell them you're there to help them; actions speak louder than words and if you're not there, you'll lose their trust quickly. And it all goes downhill from there.

Sounds hard? Welcome to the real world, where there ain't no such thing as a free lunch.

Reflection Recipes

Have you ever seen that cartoon in which Speedy Gonzales guides Daffy Duck across a minefield?

BOOM! There's one! BOOM! There's another! What do you mean you don't know where they are? You haven't missed one yet!

I've been plumbing the depths of .NET reflection lately and it felt just like Daffy's trip: I knew where I was and where I wanted to be, but I hit every landmine on my way there. It's not Microsoft's fault, really. The .NET reflection is a powerful beast and covering every little "gotcha" is a heroic work, one that the guys in charge of MSDN almost pulled off. Nearly everything you'll need to survive is somewhere in the library. If you're trying to do something simple, you'll be fine; if you're trying to pull off something advanced, you'll need a map of the minefield. Here are some of the mines I hit so far.

Attributes: Not What It Looks Like, Honey

It's easy to think of attributes as something that gets attached to other elements of your code when it's compiled. Unfortunately, it's also incorrect. Consider the following code:

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = true)]
public class UniqueIDAttribute : Attribute
{
    private static int nextID = 1;

    public int ID { get; private set; }

    public UniqueIDAttribute()
    {
        ID = nextID++;
    }
}

public class Base
{
    [UniqueID]
    public virtual void MyMethod() { }
}

public class Derived : Base
{
    public override void MyMethod()
    {
        base.MyMethod();
    }
}

class Program
{
    static void Main(string[] args)
    {
        UniqueIDAttribute attr;
        attr = (UniqueIDAttribute) Attribute.GetCustomAttribute(
        typeof(Derived).GetMethod("MyMethod"),
        typeof(UniqueIDAttribute));
        Console.Write(attr.ID);
        Console.Write(" ");
        attr = (UniqueIDAttribute) Attribute.GetCustomAttribute(
        typeof(Base).GetMethod("MyMethod"),
        typeof(UniqueIDAttribute));
        Console.WriteLine(attr.ID);
    }
}

I was honestly surprised that the output of this was "1 2" instead of "1 1" I expected. I thought that placing a custom attribute on a method will attach an instance of that attribute to that method. This is not feasible for a variety of good reasons; for example, how do you handle a case of an attribute placed on a member of a generic class? Most of these problems could be worked around with lots of effort, but Microsoft decided to do things the easy way and invest that effort somewhere else.

Here's what happens when you place an attribute on a code element: the compiler takes all the information necessary to construct that attribute and emits it together with that element. When you query the attribute at run time, the reflection code pulls out this information and uses it to construct the instance of the attribute at that moment. As an example, consider the following code:

[AttributeUsage(AttributeTargets.Method)]
public class SampleAttribute : Attribute
{
    public int Value { get; protected set; }
    public string Comment { get; set; }

    public SampleAttribute(int value)
    {
        Value = value;
    }
}

public class SomeClass
{
    [Sample(17, Comment = "Some text")]
    public void SomeMethod() { }
}

Now take a look at the IL for SomeMethod:

.method public hidebysig instance void  SomeMethod() cil managed
{
.custom instance void Sandbox.CSharp.SampleAttribute::.ctor(int32) = ( 
               01 00 11 00 00 00 01 00 54 0E 07 43 6F 6D 6D 65   // ........T..Comme
               6E 74 09 53 6F 6D 65 20 74 65 78 74 )             // nt.Some text
// Code size       2 (0x2)
.maxstack  8
IL_0000:  nop
IL_0001:  ret
} // end of method Base::SomeMethod

As you can see, the method metadata contains custom attribute information that tells the runtime which constructor to call, what parameter to supply to it and which property to set to which value.

Having understood all that, it's no problem to understand why the CustomAttributeBuilder class exists or how to use it.

Non-Virtual Interface: The Dark Secret

Have you ever wondered how it is that you can implement an interface method in a class, without marking it as virtual? It had been bothering me since I learned C#, but I never really had a compelling reason to solve that particular mystery. It was enough to know that it Just Worked. At least, it was until I tried to dynamically create a class and make it implement an interface too.

I'll stop here to make a quick aside for readers who know only C# (and/or Java) and might wonder why I think that interface methods should necessarily be virtual. A virtual method is late-bound: you don't know at compile time which implementation will be invoked when someone calls the method. Because of that, there's a layer of indirection between the method declaration and the method body; the class that defines the body of a virtual method -- either for the first time or by overriding it -- "configures" this layer of indirection to "point" to that particular body. An interface is nothing but a bunch of abstract (and therefore virtual) methods. That's why it seems counter-intuitive to implement an interface method without specifying the "virtual" keyword.

If you try to run the following code, you'll get an interesting exception:

public interface IMyInterface
{
    void MyMethod();
}

class Program
{
    static void Main(string[] args)
    {
        AssemblyBuilder assembly = AppDomain.CurrentDomain.DefineDynamicAssembly(
            new AssemblyName() { Name = "MyAssembly" },
            AssemblyBuilderAccess.Run);
        ModuleBuilder module = assembly.DefineDynamicModule("MyModule");
        TypeBuilder myClass = module.DefineType(
            "MyClass",
            TypeAttributes.Class | TypeAttributes.Public,
            typeof(Object),
            new Type[] { typeof(IMyInterface) });
        MethodBuilder myMethod = myClass.DefineMethod(
            "MyMethod",
            MethodAttributes.Public | MethodAttributes.HideBySig,
            typeof(void),
            Type.EmptyTypes);
        ILGenerator il = myMethod.GetILGenerator();
        il.Emit(OpCodes.Ret);
        Type myClassType = myClass.CreateType();
    }
}

The exception you'll get is TypeLoadException and it'll be complaining about MyClass not implementing MyMethod. If you add the MethodAttributes.Virtual flag to the method attributes, the problem will disappear.

But what happens if you don't want MyMethod to be virtual? That's not really an option. For the reasons I explained above, the method must be virtual. What you might want is to make sure it cannot be overridden. To do that, you have to also add the MethodAttributes.Final flag to the method attributes. That will give you the equivalent of writing:

public class MyClass : IMyInterface
{
    public void MyMethod() { }
}

The funny thing is that you can't declare a method as virtual sealed, yet that's exactly what the compiler does under the hood. I guess it's just one of those trade-offs: either you use a slightly leaky abstraction or you risk antagonizing people by making them learn the underlying concepts thoroughly.

Naming Your Type: Basic Hygiene

Isn't it funny how something as useful as the string type can also cause so much trouble? Sooner or later everyone has to learn to sanitize their strings or risk having their software let the little Bobby Tables wreak havoc on their data:

I must admit, though, that I didn't expect to have to sanitize the name when creating a type at run time. I found out, quite unexpectedly, that TypeBuilder seems to be allergic to types whose names contain a comma. Not that it blows up right away. You can use a name riddled with commas and still be fine when you create the type. What you can't do is use that type as a parent type. If you do, TypeBuilder will blow up with a rather unhelpful COMException saying "Record not found on lookup."

If you google for it, you'll eventually find out that there's a bug in reflection, supposedly triggered by strings that contain any of the characters []*&+,\ in the type name. My experiments indicate that the only problematic character seems to be the comma, but I'm getting rid of all of those characters anyway.

It's a nasty little bug and it's not documented in MSDN. I figured I should warn people about it, since it's such a comma mistake after all.

Metadata Tokens: Remembering Past Lives

There are two stages in the life of a dynamic class: before and after the call to CreateType. In the first stage, you're using the TypeBuilder to define type features, such as fields, properties, methods, nested types, etc. When you're done defining your type, you call the CreateType method and the magic reflection fairy turns your TypeBuilder into a real live Type. But your little Pinocchio can either be a marionette or a real boy, and never the twain shall meet.

In practice, this means that a FieldBuilder you get from calling DefineField on your TypeBuilder can only be used to define the field. Even though the FieldBuilder inherits from FieldInfo, you cannot use it to get or set the field value on the instances of the type obtained by calling CreateType on the TypeBuilder. If you try to do so, you'll get a NotSupportedException, informing you that the "invoked member is not supported in a dynamic module". Same thing applies to other builder classes.

Looking around the documentation, you'll see that the usual workaround seems to be to look up the member by its name in the newly created type. That can get ugly, though. For example, consider what you would have to do to look up a one of the several methods with the same name. The infuriating thing is that you already have the MethodBuilder for it, which is the equivalent of a MethodInfo in its TypeBuilder, so there should be a relatively painless way of getting its counterpart in a live Type.

It turns out that there is a rather painless way of doing it and it's not even a dirty hack. Every type and every member in has a metadata token that identifies it uniquely within its module. If you have the metadata token for a method, you can resolve it into a MethodBase by calling ResolveMethod on the method's Module. The usual way of obtaining a metadata token is via the MetadataToken property defined in the MemberInfo class. That won't work on a builder class, though. You'll get an InvalidOperationException complaining that the "operation is not valid due to the current state of the object".

In order to get the metadata token from a builder class, you have to use a corresponding method in the ModuleBuilder class. For example, if you have a MethodBuilder and want to get its metadata token, you have to call GetMethodToken on its ModuleBuilder. When you put it all together, converting a MethodBuilder for a TypeBuilder into its counterpart MethodInfo for the resulting live Type can be done like this:

public static MethodInfo GetLiveMethod(MethodBuilder method)
{
    return (MethodInfo) method.Module.ResolveMethod(
        ((ModuleBuilder) method.Module).GetMethodToken(method).Token);
}

Clean, concise and probably more efficient.

Generic Parameters: Working Around The Taint

When I was checking out Ruby, I ran into this neat concept of taint. It stuck with me, probably because I read about it in The Pragmatic Programmer's Guide, whose style is rather memorable. And, of course, because the concept is so neat.

What, then, is taint? Besides being "the opposite of tis", 'tis a way of keeping little Bobby Tables out of mischief. The idea is simple: anything coming from the outside user is considered tainted and anything derived from a tainted value is also considered tainted; if taint checking is active, the tainted values cannot be used for anything dangerous, such as SQL queries or host operating system commands.

What does this have to do with .NET reflection? The generic parameters for dynamic types and dynamic methods behave in a similar fashion. Specifically, if you call an otherwise innocent method MakeGenericType on an existing generic Type, you'll wind up with a Type with limited capabilities. Calls like GetField will fail on such a Type.

The workaround is actually documented in MSDN, which is why I saved this particular landmine for the end. If you've already read this far, chances are you're going to read and remember this solution before you actually run into the problem.

Here's what you have to do instead of calling GetField on the "tainted" Type. First get the FieldInfo from the generic type itself, as in:

FieldInfo myField = typeof(MyGenericType<>).GetField("MyField");

Then, get the "tainted" type:

Type constructedType = typeof(MyGenericType<>).MakeGenericType(myGenericParamBuilder);

Finally, do the workaround magic:

myField = TypeBuilder.GetField(constructedType, myField);

Rinse and repeat until you run out of "taint".

Further KABOOM Avoidance

The .NET reflection is really powerful and flexible, but it can be a real pain in the backside if you're not careful. Now that C# 3 is here, you can opt to use the Expression<TDelegate> instead of old DynamicMethod, but the old-style TypeBuilder is not likely to go anywhere soon. I hope that this map will help you get across the minefield without getting blown up often.

Interfacing Outside the Box

Happy New Year, everyone! Another hiatus is over and I hope to make it the last one. I've never been the one for New Year's resolutions before, but I guess there's a first time for everything: this year I resolve to post at least once a month.

I gave my self a head start by accumulating a number of topics I want to write about. After some careful consideration, I decided to start off with a bang: by introducing a new design pattern!

Hacking the Language

A while ago, I wrote about design patterns and how they're often a result of having to work your way around language limitations. No language is perfect, so you're bound to run into an annoying limitation sooner or later, even if you're not spending your days coding in Blub. If you're lucky, there's already a design pattern that allows you to hack your way around it. If not, you'll have to invent a solution yourself.

One of such limitations in C# is the boxing that occurs when you cast a value type to an interface. I've wrote about a related topic before, back when I was just learning about value types and boxing in C#. You could probably tell, because I made a spectacular mistake in my speculations.

Safe Deposit Box

In the course of devising a solution for the problem described in Enum Conundrum, I erroneously suggested that a possible solution might have been to have my enum implement the IEquatable interface, had the language allowed it. That, of course, wouldn't have helped. The enum would still have been boxed, although for a different reason.

Why do value types get boxed when cast to an interface? The reason is simple enough: for safety. For example, if you have a local value type variable, it resides on the stack and gets destroyed after the call ends. Without boxing, if you stored an interface reference to that value type instance in an object field, that field would wind up with an invalid reference as soon as the original call is done and the variable goes out of scope.

Of course, you might not care about boxing. After all, boxing was invented precisely so that people could use value types and reference types on equal footing: the alternative to it is all that nasty wrapping code that was so widespread in Java before it incorporated boxing. So boxing is neat, isn't it?

Get Back in Line

When you define a value type, you (should) do it for a good reason. Most of the time, it's because you need a type with value semantics; in other words, you want a type that behaves in such a way that it's not possible to affect one variable of that type by performing operations on another variable of the same type.

The way value semantics is enforced is through memory allocation. Value types are allocated in-line. This means that the memory allocated for a value type instance is a part of the memory allocated for whatever contains that instance. If it's a local variable, the instance is allocated directly on the stack. If it's a field, the instance is contained in the memory allocated for the field's object. Only when it gets boxed does a value type wind up on the heap independently, all by itself.

At times, this is precisely the reason why you'll choose a value type: not so much for its semantics, as for its memory allocation. For example, you might be writing a game in XNA and you don't want the garbage collector kicking in and ruining your frame rate.

Whatever your reasons for wanting to keep them in line, the fact remains that you cannot cast your value types to interface without them getting boxed. What, then, is the alternative?

Generic Static

Fun fact: a static field in a generic class is allocated separately for each closed constructed type of that generic class. What does this mean? Take a look at the following code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace BeardsEye
{
    class GenericStatic<T>
    {
        public static int SomeVal;
    }

    class Program
    {
        static void Main(string[] args)
        {
               GenericStatic<int>.SomeVal = 1;
               GenericStatic<string>.SomeVal = 2;

               Console.WriteLine(GenericStatic<int>.SomeVal);
               Console.WriteLine(GenericStatic<string>.SomeVal);
        }
    }
}

If you run it, it will print out 1 and 2. Of course, such behavior stands to reason; if you're not sure why, just change the type of SomeVal to T.

Why am I suddenly talking about this, anyway? Because, in some cases, you can use this trick as a substitute for what would otherwise be a dictionary. Specifically, for some uses, you could replace a Dictionary<Type, Something> with:

public static class MyTypeDictionary<TType>
{
    public static Something Value;
}

Of course, this is a far cry from a fully functional dictionary. You don't have the Count property, there's no way to remove values, iterate over them or find out whether a type has an associated value or not. And you can't have an arbitrary number of these "dictionaries".

On the other hand, when you need to associate something with a type and that "something" changes from type to type, this technique is just perfect.

Fun With Functions

Now we finally get to the solution itself. Suppose you have an interface IConfusticatable, for all types that can be confusticated to a certain degree:

public interface IConfusticatable
{
    double Confusticate(int degree);
}

You want to be able to confusticate classes that implement this interface, but you would also like to confusticate value types, without boxing them. You could use the following trick:

public static class Confusticator<T>
{
    public static Func<T, int, double> Confusticate = ((what, degree) => ((IConfusticatable) what).Confusticate(degree));
}

What do you have to do in your value type? Not much:

public struct MyStruct
{
    static MyStruct()
    {
        Confusticator<MyStruct>.Confusticate = ((me, degree) => me.Confusticate(degree));
    }

    public MyStruct(double val)
    {
        Val = val;
    }

    public double Val;

    public double Confusticate(int degree)
    {
        return Val + degree;
    }
}

How do you use this when you want to confusticate something? Like this:

MyStruct myVar = new MyStruct(17);
Console.WriteLine(Confusticator<MyStruct>.Confusticate(myVar, 3));

I'm sure that, at this point, you're thinking "So what good is this? If I already know that myVar is a MyStruct, I can just call its own Confusticate method directly."

This, of course, is perfectly true, unless you're writing generic code. In that case you don't know beforehand what your type parameter will be.

Warning: May Contain .NUTS

A few words of caution here. The proposed implementation of MyStruct relies on a static constructor, which is supposed to "register" the confustication lambda with the Confusticator. The problem with this is that it's a bit tricky to make sure that a value type static constructor is invoked.

If you dig around the C# Annotated Standard, you'll find some fascinating incompatibilities between different standards and their respective implementations.

According to the C# Standard, a static constructor for a value type is executed only when the first of the following occurs:

• An instance member of the struct is referenced.
• A static member of the struct is referenced.
• An explicitly declared constructor of the struct is called.

On the other hand, the CLI Standard requires execution only when the first of the following occurs:

• A static member of the struct is referenced.
• An explicitly declared constructor of the struct is called.

To further complicate the matters, the CLI Standard does not allow execution when an instance member of the struct is referenced. You'll note that this is in direct conflict with the C# Standard.

If you actually test what happens when you run a C# program on CLR, you'll see that the static constructor is called as soon as the first of the following occurs:

• An instance method of the struct is referenced.
• An instance property of the struct is referenced.
• A static member of the struct is referenced.
• An explicitly declared constructor of the struct is called.

However, the static constructor is not called if you reference an instance field of the struct. Not only does the CLR implementation fail to conform to either standard, it does so in a way that seems spectacularly arbitrary. I'm sure there are some perfectly valid -- if arcane and obscure -- reasons for this.

All in all, it would be a lot more reliable to stick the "registration" code in some initialization method that you know will be called. That's what you would have to do anyway, if you wanted to register a confustication function for an enum, for example.

Finishing Touches

What if your IConfusticatable interface has more than one method? There are several possible solutions, but the simplest one would be to give your Confusticator<T> one static delegate field per IConfusticatable method.

Speaking of Confusticator<T>, if you find it too ugly to write Confusticator<MyStruct>.Confusticate(myVar, 3), you can use type inference to introduce some syntactic sugar:

public static class Confusticator<T>
{
    public static Func<T, int, double> Confusticate = ((what, degree) => ((IConfusticatable) what).Confusticate(degree));
}

public static class Confusticator
{
    public static double Confusticate<T>(T what, int degree)
    {
        return Confusticator<T>.Confusticate(what, degree);
    }
}

With that, you would be able to simply write Confusticator.Confusticate(myVar, 3) and have the compiler infer the type automagically.

Share and Enjoy

Well, that's all for now. For those of you who have a legitimate need to avoid boxing your value types, I hope this proves useful. For the rest, I hope that you found this an interesting trick.

I'm planning to write more recipes for getting stuff that C# doesn't have, such as multiple dispatch, so stay tuned!

"Encryption Chip" Will Not End Piracy

Nolan Bushnell is full of it. There, I finally got that off my chest. It's arguably acerbic and rather rude, but it needed to be said. You have no idea how hard I've tried to avoid saying it. After all, he's the founder of Atari and a historical figure unto his own. He deserves certain respect for that.

The first time Nolan Bushnell claimed that the "encryption chip" will end piracy, I exercised due restraint. His statement reverberated all over the Internet, causing reactions that ranged from mild skepticism on one end of the spectrum to derision and disgust on the other.

So why am I writing now, more than two months later, if nobody believed him in the first place? In other words, why am I beating a dead horse? Partly, it's because he did it again and it pisses me off. Mostly, though, it's because I'm rather interested in copy protections and security; it's sort of a hobby of mine.

The most important lessons you learn in those two fields is that no protection is perfect and every solution spawns a new class of problems. This means that there will never be one final (technical) solution to the issue of piracy; there is no silver bullet. The experts from both fields are locked in an arms race with their adversaries. Once you've learned that, you'll have no problem recognizing that Nolan Bushnell is really just flogging his merch.

However, the issue runs deeper than that.

Copy Protection and Security

When I referred to copy protections and security I said "two fields", even though one can be considered a subset of the other; after all, copy protections are supposed to prevent the unauthorized use of software. Even though this is technically true, there are some drastic differences between the two.

An important difference is the level of cooperation from the users. When it comes to information security, the users actively cooperate with the protection systems, because it's in their best interest. You don't give access to your bank account to all your friends, do you?

On the other hand, copy protections often clash with the users' interests. Some of these interests are illegal, such as downloading a commercial game for free. But other interests are quite legal and legitimate. You added more memory to your computer? Odds are you might have to reactivate your Windows.

Another important difference is that a copy protection has to protect an application that lives on the user's computer. Unless we're talking about an MMOG, there's no server counterpart that executes a critical piece of code, without which the game can't work.

When you put those two things together, it becomes obvious why you can't make a perfect copy protection: you're relying on cooperation from a user that has complete control over his copy of your content or software. If that user doesn't want to cooperate, the best you can do is delay him. Even unbreakable ciphers won't help you, because sooner or later you'll have to decrypt the content and, when you do, the user will nab it.

But what if you could alter these conditions? You could make sure that there's a critical part of an application that executes somewhere where the user doesn't have control over it: that's what MMOGs do. The other option is to take away the control from the user.

Trust Controversy

Enter the "trusted computing". The first time I heard of it was back when Microsoft was touting Palladium. Back then, it sounded like a bad pun: a company found guilty in an antitrust lawsuit is proposing to build a "trusted computing platform" for its users. The irony was not lost on anyone and it provoked some enlightening responses from security experts.

Then, since nothing really seemed to happen and we didn't all suddenly wake up in some digital equivalent of 1984, I lost track of this topic for a while. I forgot about it until Nolan Bushnell started his TPM hype. A quick search engine query revealed that TPM stands for "Trusted Platform Module" and that it's the central component of "trusted computing".

What, then, is the so-called "trusted computing"? It's a technology that encompasses the following concepts:

1. Endorsement key is a cryptographic key pair unique to one computer. The chief use for it is to prove the computer's identity.
2. Secure I/O makes sure that the communication between the user and their software is secure and cannot be intercepted or altered.
3. Memory curtaining protects those parts of memory that contain sensitive data (such as cryptographic keys) from unauthorized access, even by the operating system itself.
4. Sealed storage binds data to the specific platform -- both hardware and software -- so that you cannot access it from any other platform.
5. Remote attestation allows authorized parties to detect changes to the platform configuration in order to make sure that they meet the expected parameters; in other words, to prove that nobody tampered with the platform.
   

That's just a brief summary, to give you an idea of what we're talking about here. If you want more information, I recommend that you start at Wikipedia and then go on directly to the Trusted Computing Group site.

So, the core idea is to make computers more secure, by ensuring that no "untrusted" code has access to your stuff. At least, that's supposed to be the core idea. Unfortunately, there has been a great deal of confusion about the word "trust" in "trusted computing". Specifically, who is supposed to trust whom?

If you read Bruce Schneier's essay on "trusted computing", you'll notice that there's a good deal of controversy and confusion surrounding the issue. As one commenter so aptly put it, the only one not trusted seems to be the owner of the computer.

All Your Base

Each of the five concepts of "trusted computing" addresses a real security problem:

1. Endorsement keys would be used to mitigate spoofing concerns in secure transactions by establishing the identity of each party involved.
2. Secure I/O is supposed to avoid security breaches through techniques such as keylogging.
3. Memory curtaining would make sure that sensitive information, such as cryptographic keys, is not allowed to "leak" somewhere where it could be extracted by malicious parties.
4. Sealed storage would do a similar thing for sensitive information in non-volatile storage.
5. Remote attestation could help network administrators easily detect intrusions and attacks on their machines.

Yet, after a closer look at them, it becomes evident that there's plenty of room for abuse. Imagine, for example, a system that enforces specific usage policies on your data:

• It would use sealed storage to bind that data to a particular application or set of applications that you're allowed to use on that data.
• It would employ memory curtaining to make sure you cannot extract that data directly from memory.
• It would use secure I/O to make sure you cannot intercept it on its way somewhere else.
• It would use remote attestation to report if you tamper with any part of the system.
• And it would clearly identify you as a "culprit" to whoever is interested in enforcing those policies, if it possessed both your personal information and your endorsement key.

Is there any kind of usage policy that springs immediately to mind? There are two, actually: DRM and vendor lock-in. Ross Anderson describes several ways to abuse TC in his FAQ. Richard Stallman dedicates a whole chapter to this topic, in his book "Free Software, Free Society"; although slightly reminiscent of Book of Revelations in tone, it offers some very interesting insights.

Another interesting aspect of "trusted computing" is that it actually raises the stakes when it comes to information security: imagine a worm that successfully exploits a bug in the supposedly secure OS code to install a "trusted" rootkit? Talk about irony.

Pirates vs. Ninjas

Getting back to the original topic, does this mean that Nolan Bushnell is right? Is his "stealth encryption chip" really going to send all the pirates to the Davy Jones's Locker? Not by a long shot! Remember, if the software in question does not have some critical code running on some computer under control of some "authority", you can eventually break its copy protection.

When it comes to policy enforcement, the most important part of the "trusted computing" is the remote attestation. This is the way to ensure you won't tamper with the policy enforcement code. Incidentally, it requires you to be online. Now back up a couple of months and remember what happened when BioWare tried to pull that trick on its players.

If you believe that pirates can't hide behind this forever, think again. There are numerous valid reasons to resist the attempts to introduce an artificial dependency on Internet connection into software and all those reasons boil down to one: the connection is not always available, yet the artificial nature of the dependency means that the software doesn't actually need it to work properly.

Besides, people are not yet convinced that "trusted computing" will actually make things better. Plus, there are all sorts of concerns about privacy and also about practicality of the whole approach. Still, the Trusted Computing Group has been formed, the commercial motivation is there and "trusted computing" will keep rolling, until all doubts and concerns have been dealt with, one way or another.

What, then, is the worst blow "trusted computing" could deal to pirates? Indulge me a bit, as I let my imagination run wild and explore a "what-if" future.

Crack Dealers

Back when I was a little kid, learning what makes the cute, little Spectrum 48 tick, pirates were selling games on audio tapes. Today, pirated games are free. They are cracked for free, by enthusiasts; they are uploaded for free, on sites that survive on advertising or donations; and they are downloaded for free. I can still see some pirates in the streets, selling CDs and DVDs, but I'm sure they won't be buying any Ferraris with that money.

Fast forward to the time when "trusted computing" is in its full swing. To crack protections, pirates need very specialized software, maybe even some hardware, and a lot more effort than before. More than ever, piracy is something that only the select few can do.

However, it is also more lucrative than ever. As the usage policies are enforced more rigorously, the multitudes who used to obtain their entertainment for free now have to go and buy it. The big companies take advantage of that and the prices are even higher than before. You can buy an overpriced game directly from its publisher; or you can take a chance and go buy yourself a pirated copy from a local "software crack dealer". It's illegal, sure, but it's a lot cheaper and you can afford to buy a lot more.

Suddenly, pirates are not your everyday enthusiasts anymore; instead, they're rich criminals. They have bodyguards with guns. They have shady lawyers. They have money laundering enterprises and fake fronts and lots of connections. They know powerful people. In your efforts to eradicate a problem, you managed to make it mutate into something worse.

If this seems improbable and exaggerated, that's okay: I don't believe it's likely to happen. My point is that you should always be on the lookout for unintended consequences. It would be nice if, just for once, we asked ourselves where we're going before we get there.

Update: Firefox 3 Works After All

Consider this a deep breath before I plunge into some more serious blogging. It turns out that Firefox 3 works rather well, after all. Not that I doubted it, but I couldn't really try it out until all my favorite add-ons were updated and relatively stable.

The biggest difference I've noticed so far is that it doesn't do its famous hog-the-CPU trick on pages such as Google Spreadsheets. Now that is a big relief for me; I absolutely hated that particular bug.

Make no mistake: I didn't change my mind about my first experience with Firefox 3. I still believe the developers should have allowed their users to do one of the following:

1. check the add-on compatibility during the installation
2. revert to Firefox 2
   
3. run Firefox 2 and Firefox 3 side by side

But, hey, at least it works now. And I've only confirmed that I'm a late adopter by nature. As if I really needed to confirm that: I've lived in Chile for nine years before I decided to try Carménère.

Firefox 3: A Five Act Tragedy

Act I: The Hype

I first heard of the Firefox Download Day at the office, from a co-worker. It wasn't a dark and stormy night, however much we all wished it to be -- the A/C was on the blink yet again.

Naturally, my curiosity was piqued. I've been using Firefox for years now and I've grown used to it and a bunch of add-ons for it. Sure, it had a couple of annoying bugs, but I could live with those. Of course, a new and improved version was likely to get rid of them, which would have been incentive enough to check it out.

So I went to the "Spread Firefox!" site to find out what the fuss was all about. When I found that Mozilla not only planned to give us all a new version of my browser of choice, but also establish a Guiness World Record, I was hooked. I pledged right away and also scheduled a reminder in my Google Calendar, to make sure I wouldn't forget.

Act II: The Outage

It seems childish in retrospect, but I got really excited. Not only was I going to get my hands on the hottest piece of software on the Web, but I was also going to be a part of the effort to set a Guiness World Record. And I wasn't the only one around: a few other co-workers were as excited about this as I was.

When the big day finally came, I made sure I was busy working to keep my mind off the wait. The time passed quickly enough and when the clock was finally done ticking its way to 1 pm, I went back to the Firefox site. That's when I got my first surprise: the site was down.

It's not unusual for a site to be brought down to its knees by overwhelming interest, but I would have expected Mozilla to be prepared. After all, they were the ones who did their best to attract all that attention. Why make a fuss if you can't cope with the results?

After a couple of hours of doing a Wile E. Coyote impression, Mozilla finally got its stuff together and I was finally able to download my copy. Once it was on my disk, I wasted no more time and clicked through the install impatiently. At last, the moment of truth was at hand: I launched Firefox 3!

Act III: The Incompatibility

The first thing my new Firefox did was to check my add-ons. It then proceeded to notify me that less than half of them had updates compatible with version 3 of Firefox. That left more than half of my add-ons disabled.

Being a coder means that an essential part of your job is to imagine worst cases for your code, so
I generally try to be an optimist about everything else, just to balance things out. Therefore I surmised that the Mozilla site was still experiencing difficulties. Did I mention that my optimism is rarely founded?

I spent some time fumbling with the new user interface, only to find out that the developers didn't stop at adding the add-on search into Firefox itself. They also decided to remove the link that took the users directly to the add-on page. I ended up checking manually and the add-on page was working quite well, thank you.

That's where I felt Mozilla managed to set a record, although it wasn't the one they were hoping for: I went from enthusiastic to disappointed in less than 5 minutes since I installed their software. What were they thinking?

Pretend you're in charge of developing a rather successful software product. One of the key advantages it has over the competition is its extensibility. There's a huge variety of add-ons for it and they do lots of different things, ranging from silly to extremely useful. Even though your users might be dissatisfied with some aspects of your software, they would still be reluctant to change to a product that doesn't have the add-ons they use. So what do you do when you decide to develop a shiny new version of your product?

The fashionable answer seems to be "break the add-ons that worked nicely with the older versions", even though the history teaches us the opposite.

Okay, I understand that you can't always maintain compatibility. Sometimes, preserving compatibility is in direct conflict with a higher-priority design goal and you just have to go ahead and break stuff. What you do in those situations is to try to minimize the damage, which is what Mozilla tried to do. But what do you do about the stuff the remains broken?

Here's what you do: you make your installer check the compatibility and report to user before installing the new version and breaking your user's stuff.

Act IV: The Retreat

Once again life conspired to remind me why I'm a late adopter. After a minute or two of staring at the list of things I wouldn't be able to use with Firefox 3, I decided to go back to version 2. A routine check of my download folder revealed that I apparently got rid of the installer at some point in time. No big deal, I can download it from the Firefox site, right?

Wrong! There is no link to Firefox 2 anywhere on the whole page. No matter how hard you try, you won't be offered a chance to turn back. And I tried very hard, trust me. Among other things, I tried typing "firefox 2" into the search box. That particular attempt rewarded me with zero results. I didn't even get a link to a press release or release notes for Firefox 2. Zilch. Firefox 2 has been exterminated.

Despite all its pictures of birds and balloons and a smiling Sun, the message Mozilla is sending its users is loud and clear: "Assume the position!"

In the end, I found the download link on www.firefox2.com, a site that features a prominent notice that it isn't associated with Mozilla in any way. It worked for me, but it probably won't work for you. As I'm writing, I'm also looking at the new content of the site, where every occurence of "Firefox 2" has been replaced with "Firefox 3", except the site URL, of course.

Just in case you really need to go back, the solution is to grab the Firefox 3 download link and change it manually to request the version 2.0.0.14. For example, the correct link to download the English (US) version of Firefox 2.0.0.14 for Windows is:

http://download.mozilla.org/?product=firefox-2.0.0.14&os=win&lang=en-US

Act V: The Repose

Having successfully overcome the unexpected difficulties, I decided it was time to wrap things up and go back to work. I installed Firefox 2 over its younger sibling and launched it. My optimism was proven unfounded yet again: Firefox neatly crashed when it tried to load my saved session.

I tried a few things before I stopped fighting the inevitable and uninstalled Firefox. Hoping fervently that the problem would be solved this time, I reinstalled it and launched it.

By this time I was more than a mite miffed or a tad testy: I was positively pissed off. In the spirit of their decision that I won't need version 2 ever again, Mozilla obviously didn't bother to test the downgrade process.

Fortunately, it turned out that no drastic measures were necessary. I was back to Firefox 2 and all my add-ons worked again.

Epilogue

While writing this post, I stumbled upon this article that explains how to make older add-ons work with new Firefox. In short, it describes a quick hack that makes Firefox skip the step where it checks for add-on compatibility. The article was about version 3.0rc1, so I don't know whether this will work in the final release. I didn't try it and I don't intend to. I prefer to wait until all the add-ons I use are updated. Having a new browser that's shiny and sleek is cool, but being able to use it my way is way cooler.