Is Software Design Literature Dead?

Sometimes, my clients ask me what to read about software design. Most often than not, they don't want a list of books – many already have the classics covered. They would like to read papers, perhaps recent works, to further advance their understanding of software design, or perhaps something inspirational, to get new ideas and concepts. I have to confess I'm often at loss for suggestions.

The sense of discomfort gets even worse when they ask me what happened to the kind of publications they used to read in the late '90s. Stuff like the C++ Report, the Journal of Object Oriented Programming, Java Report, Object Expert, etc. Most don't even know about the Journal of Object Technology, but honestly, the JOT has taken a strong academic slant lately, and I'm not sure they would find it all that interesting. I usually suggest they keep current on design patterns: for instance, the complete EuroPLoP 2009 proceedings are available online, for free. However, mentioning patterns sometimes furthers just another question: what happened to the pattern movement? Keep that conversation going for a while, and you get the final question: so, is software design literature dead?

Is it?
Note that the question is not about software design - it's about software design literature. Interestingly, Martin Fowler wrote about the other side of the story (Is Design Dead?) back in 2004. He argued that design wasn't dead, but its nature had changed (I could argue that if you change the nature of something, then it's perhaps inappropriate to keep using the same name :-), but ok). So perhaps software design literature isn't dead either, and has just changed nature: after all, looking for "software design" on Google yields 3,240,000 results.
Trying to classify what I usually find under the "software design" chapter, I came up with this list:

- Literature on software design philosophy, hopefully with some practical applications. Early works on Information Hiding were as much about a philosophy of software design as about practical ways to structure our software. There were a lot of philosophical papers on OOP and AOP as well. Usually, you get this kind of literature when some new idea is being proposed (like my Physics of Software :-). It's natural to see less and less philosophy in a maturing discipline, so perhaps a dearth of philosophical literature is not a bad sign.

- Literature on notations, like UML or SysML. This is not really software design literature, unless the rationale for the notation is discussed.

- Academic literature on metrics and the like. This would be interesting if those metrics addressed real design questions, but in practice, everything is always observed through the rather narrow perspective of correlation with defects or cost or something appealing for manager$. In most cases, this literature is not really about software design, and is definitely not coming from people with a strong software design background (of course, they would disagree on that :-)

- Literature on software design principles and heuristics, or refactoring techniques. We still see some of this, but is mostly a rehashing of the same old stuff from the late '90s. The SOLID principles, the Law of Demeter, etc. In most cases, papers say nothing new, are based on toy problems, and are here just because many programmers won't read something that has been published more than a few months ago. If this is what's keeping software design literature alive, let's pull the plug.

- Literature on methods, like Design by Contract, TDD, Domain-Driven Design, etc. Here you find the occasional must-read work (usually a book from those who actually invented the approach), but just like philosophy, you see less and less of this literature in a maturing discipline. Then you find tons of advocacy on methods (or against methods), which would be more interesting if they involved real experiments (like the ones performed by Simula labs) and not just another toy projects in the capable hands of graduate students. Besides, experiments on design methods should be evaluated [mostly] by cost of change in the next few months/years, not exclusively by design principles. Advocacy may seem to keep literature alive, but it's just noise.

- Literature on platform-specific architectures. There is no dearth of that. From early EJB treatises to JBoss tutorials, you can find tons of papers on "enterprise architecture". Even Microsoft has some architectural literature on application blocks and stuff like that. Honestly, in most cases it looks more like Markitecture (marketing architecture as defined by Hohmann), promoting canned solutions and proposing that you adapt your problem to the architecture, which is sort of the opposite of real software design. The best works usually fall under the "design patterns" chapter (see below).

- Literature for architecture astronauts. This is a nice venue for both academics and vendors. You usually see heavily layered architectures using any possible standards and still proposing a few more, with all the right (and wrong) acronyms inside, and after a while you learn to turn quickly to the next page. It's not unusual to find papers appealing to money-saving managers who don't "get" software, proposing yet another combination of blueprint architectures and MDA tools so that you can "generate all your code" with a mouse click. Yeah, sure, whatever.

- Literature on design patterns. Most likely, this is what's keeping software design literature alive. A well-written pattern is about a real problem, the forces shaping the context, an effective solution, its consequences, etc. This is what software design is about, in practice. On the other hand, a couple of conferences every year can't keep design literature in perfect health.

- Literature on design in the context of agility (mostly about TDD). There is a lot of this on the net. Unfortunately, it's mostly about trivial problems, and even more unfortunately, there is rarely any discussion about the quality of the design itself (as if having tests was enough to declare the design "good"). The largest issue here is that it's basically impossible to talk about the design of anything non-trivial strictly from a code-based perspective. Note: I'm not saying that you can't design using only code as your material. I'm saying that when the problem scales slightly beyond the toy level, the amount of code that you would have to write and show to talk about design and design alternatives grows beyond the manageable. So, while code-centric practices are not killing design, they are nailing the coffin on design literature.

Geez, I am usually an optimist :-)), but this picture is bleak. I could almost paraphrase Richard Gabriel (of "Objects have failed" fame) and say that evidently, software design has failed to speak the truth, and therefore, software design narrative (literature) is dying. But that would not be me. I'm more like the opportunity guy. Perhaps we just need a different kind of literature on software design.

Something's missing
If you walk through the aisles of a large bookstore, and go to the "design & architecture" shelves, you'll all the literary kinds above, not about software, but about real-world stuff (from chairs to buildings). except on the design of physical objects too. Interestingly, you can also find a few morelike:

- Anthologies (presenting the work of several designers) and monographs on the work of famous designers. We are at loss here, because software design is not immediately visible, is not interesting for the general public, is often kept as a trade secret, etc.
I know only one attempt to come up with something similar for software: "Beautiful Architecture" by Diomidis Spinellis. It's a nice book, but it suffers from a lack of depth. I enjoyed reading it, but didn't come back with a new perspective on something, which is what I would be looking for in this kind of literature.
It would be interesting, for instance, to read more about the architecture of successful open-source projects, not from a fanboy perspective but through the eyes of experienced software designers. Any takers?

- Idea books. These may look like anthologies, but the focus is different. While an anthology is often associated with a discussion or critics of the designer's style, an idea book presents several different objects, often out of context, as a sort of creative stimulus. I don't know of anything similar for software design, though I've seen many similar book for "web design" (basically fancy web pages). In a sense, some literature on patterns comes close to being inspirational; at least, good domain-specific patterns sometimes do. But an idea book (or idea paper) would look different.

I guess anthologies and monographs are at odd with the software culture at large, with its focus on the whizz-bang technology of the day, little interest for the past, and often bordering on religious fervor about some products. But idea books (or most likely idea papers) could work, somehow.

Indeed, I would like to see more software design literature organized as follows:

- A real-world, or at least realistic problem is presented.

- Forces are discussed. Ideally, real-world forces.

- Possibly, a subset of the whole problem is selected. Real-world problems are too big to be dissected in a paper (or two, or three). You can't discuss the detailed design of a business system in a paper (lest you appeal only to architecture astronauts). You could, however, discuss a selected, challenging portion. Ideally, the problem (or the subset) or perhaps just the approach / solution, should be of some interest even outside the specific domain. Just because your problem arose in a deeply embedded environment doesn't mean I can't learn something useful for an enterprise web application (assuming I'm open minded, of course :-). Idea books / papers should trigger some lateral thinking on the reader, therefore unusual, provocative solutions would be great (unlike literature on patterns, where you are expected to report on well-known, sort of "traditional" solutions).

- Practical solutions are presented and scrutinized. I don't really care if you use code, UML, words, gestures, whatever. Still, I want some depth of scrutiny. I want to see more than one option discussed. I don't need working code. I'm probably working on a different language or platform, a different domain, with different constraints. I want fresh design ideas and new perspectives.

- In practice, a good design aims at keeping the cost of change low. This is why a real-world problem is preferable. Talking about the most likely changes and how they could be addressed in different scenarios beats babbling about tests and SOLID and the like. However, "most likely changes" is meaningless if the problem is not real.

Funny enough, there would be no natural place to publish something like that, except your own personal page. Sure, maybe JOT, maybe not. Maybe IEEE Software, maybe not. Maybe some conference, maybe not. But we don't have a software design journal with a large readers pool. This is part of the problem, of course, but also a consequence. Wrong feedback loop :-).

Interestingly, I proposed something similar years ago, when I was part of the editorial board of a software magazine. It never made a dent. I remember that a well-known author argued against the idea, on the basis that people were not interested in all this talking about ins-and-outs, design alternatives and stuff. They would rather have a single, comprehensive design presented that they could immediately use, preferably with some source code. Well, that's entirely possible; indeed, I don't really know how many software practitioners would be interested in this kind of literature. Sure, I can bet a few of you guys would be, but overall, perhaps just a small minority is looking for inspiration, and most are just looking for canned solutions.

Or maybe something else...
On the other hand, perhaps this style is just too stuck in the '90s to be appealing in 2011. It's unidirectional (author to readers), it's not "social", it's not fun. Maybe a design challenge would be more appealing. Or perhaps the media are obsolete, and we should move away from text and graphics and toward (e.g.) videos. I actually tried to watch some code kata videos, but the guys thought they were like this, but to an experienced designer they looked more like this. (and not even that funny). Maybe the next generation of software designers will come up with a better narrative style or media.

Do something!
I'm usually the "let's do something" kind of guy, and I've entertained the idea of starting a Software Design Gallery, or a Software Design Idea Book, or something, but honestly, it's a damn lot of work, and I'm a bit skeptical about the market size (even for a free book/paper/website/whatever). As usual, any feedback is welcome, and any action on your part even more :-)

Notes on Software Design, Chapter 8. A simple property: Distance (part 1)

You're writing a function in your preferred language. At some point, you look at your code and you see that a portion just does not belong there. You move it outside, creating another function. Asked why, you may answer that by doing so you made it reusable; that you want to respect the Single Responsibility Principle; that you want to increase cohesion; that code just looks "cleaner" and "easier to read" that way; and so on. I guess it sounds rather familiar.

By moving the code outside the function, you also increased its distance with the code that is left inside. This is probably not so familiar: distance is not a textbook property of software. However, the notion of distance is extremely important. Fortunately, distance is a very simple concept. It has an immediate intuitive meaning, and although it's still rather informal, we can already use it to articulate some non-trivial reasoning.

I'm trying to keep this post reasonably short, so I'll cover only the artifact side of the story here. I'll talk about the run-time world next time.

A Concept of Distance
Consider this simple function (it's C#, but that's obviously irrelevant)

int sum( int[] a )
  {
  int s = 0;

  foreach( int v in a )
    s += v ;
  
  return s;
  }

I used two blank lines to split the function in three smaller portions: initialization - computation - return. I didn't use comments - the code is quite readable as it is, and you know the adage: if you see a comment, make a function. It would be unnatural (and with dubious benefits) to split that function into sub-functions. Still, I wanted to highlight the three tiny yet distinct procedural portions (centers) within that function, so I used empty lines. I guess most of you have done the same at one point or another, perhaps on a larger scale.

Said otherwise, I wanted to show that some statements were conceptually closer than others. They don't have to be procedural statements. I have seen people "grouping" variable declarations in the same way, to show that some variables sort of "lump together" without creating a structure or a class. I did that by increasing their physical distance in the artifact space.

A Measure of Distance
Given two pieces of information P1, P2, encoded in some artifacts A1, A2, we can define their distance D( P1, P2 ) using an ordinal scale, that is, a totally ordered set:

P1 and P2 appear in the same statement - that's minimum distance
P1 and P2 appear in the same group of consecutive statements
P1 and P2 appear in the same function
… etc

for the full scale, including the data counterpart, see my Summary at the Physics of Software website.

Note that Distance is a relative property. You cannot take a particular piece of information and identify its distance (as you could do, for instance, with mass). You need two.
Also, the ordinal scale is rather limiting: you can do no math with it. It would be nice to turn it into a meaningful interval or ratio scale, but I'm not there yet.

Is Distance useful?
As in most theories, individual concepts may seem rather moot, but once you have enough concepts you can solve interesting problems or gain better understanding of complex phenomena. Right now, I haven't introduced the concept of tangling yet, so Distance may seem rather moot on itself. Still, we can temporarily use a vague notion of coupling to explore the value of distance. It will get better in the [near] future, trust me :-).

Consider a few consecutive statements inside a function. It's ok if they share intimate knowledge. The three segments in sum are rather strongly coupled, to the point that it's ineffective to split them in subfunctions, but that doesn't bother me much. It's fine to be tightly coupled at small distance. As we'll see, it's more than fine: it's expected.

Functions within a class are still close together, but farther apart. Again, it's ok if they share some knowledge. Ideally, that knowledge is embodied in the class invariant, but private functions are commonly tied with calling functions in a rather strong way. They often assume to be called in specific states (that could be captured in elaborated preconditions), and the caller is responsible to guarantee such preconditions. Sequence of calls are also expected to happen in specific orders, so that preconditions are met. Again, that doesn't bother me much. That's why the class exists in the first place: to provide a place where I can group together "closely related" functions and data.

Distinct classes are even more distant. Ideally, they won't share much. In practice, classes inside the same component often end up having some acquaintance with each other. For instance, widgets inside a widget library may work well together, but may not work at all with widgets inside a different library. Still, they're distant enough to be used individually.

We expect components / services to be lightly coupled. They can share some high-level contract, but that should be all.

Applications shouldn't be coupled at all – any coupling should appear at a lower level (components).

The logical consequence here is that coupling must decrease as distance increases. There is more to this statement than is immediately obvious. The real meaning is:
a) large distance requires low coupling
b) small distance requires high coupling

When I explain the concept, most people immediately think of (a) and ignore (b). Yet (b) is very important, because it says:
1) if coupling with the surroundings is not strong enough, you should move that portion elsewhere.
2) the code should go where the coupling is stronger (that is, if code is attracted elsewhere, consider moving it elsewhere :-)). That's basically why feature envy is considered a bad smell – the code is in the wrong place.

Cohesion as an emergent property
Cohesion has always been a more elusive concept than coupling. Looking at literature, you'll find dozens of different definitions and metrics for cohesion (early works like Myers' Composite/Structured Design used to call it "strength"). I've struggled with the concept for a while, because it didn't fit too well with other parts of my theory, but then I realized that cohesion is not a property per se.

Cohesion is a byproduct of attraction and distance: an artifact is cohesive if its constituents are at the right distance, considering the forces of attraction and rejection acting upon that artifact. If the resulting attraction is too strong or too weak, parts of that artifact want to move either down or up in the distance hierarchy, or into another site at the same level.

Attraction is too weak: the forces keeping that code together are not strong enough to warrant the short distance at which we placed the code. For instance, a long function with well-identified segments sharing little data. We can take that sequence of statements and move it up in the hierarchy - forming a new function.

Attraction is too strong: for instance, we put code in different classes, but those classes are intimately connected. The easier thing is to demote one class to a set of functions (down in the hierarchy) and merge those functions with the other class. But perhaps the entire shape is wrong, at odd with the forcefield. Perhaps new abstractions (centers) must be found, and functions, or even statements, moved into new places.

This is closing the circle, so to speak. Good software is in a state of equilibrium: attraction and rejection are balanced with proper distance between elements.

Note: I'm talking about attraction and rejection, but I have yet to present most attractive / repulsive forces. Still, somehow I hope most of you can grasp the concepts anyway.

An Alexandrian look on the notion of distance
I've quoted Christopher Alexander several time in an early discussion on the concept of form. Now, you may know that Alexander's most recent theory is explained in 4 tomes (which I haven't deeply read yet) collectively known as "The Nature of Order". A few people have tried to relate some of his concepts with the software world, but so far the results have been rather unimpressive (I'm probably biased in my judgment :-).

On my side, I see a very strong connection between the concept of equilibrium as an interplay between distance and the artifact hierarchy and the Alexandrian concept of levels of scale: “A balanced range of sizes is pleasing and beautiful”.
Which is not to say that you should have long functions, average functions, small functions :-). I would translate that notion in the software world as: a balanced use of the artifact hierarchy is pleasing and beautiful. That is:
Don't use long function: use multiple functions in a class instead.
Don't use long classes: use multiple classes in a component instead.
Don't create huge components: use multiple components inside an [application/service/program] instead

This is routinely ignored (which, I think, contributes to the freescale nature of most source code) but it's also the very first reason why those concepts have been introduced in the first place! Actually, we are probably still missing a few levels in the hierarchy, as required for instance to describe systems of systems.

Gravity, efficiency, and the run-time distance
Remember gravity? Gravity (in the artifact world) provides a path of least resistance for the programmer: just add stuff where there is other vaguely related related stuff. Gravity works to minimize distance, but in a kind of piecemeal, local minimum way. It's easy to get trapped into local minimum. The minimum is local when we add code that is not tightly connected with the surroundings, so that other forces at play (not yet discussed) will reject it.

When you point out incoherent, long functions, quite a few programmers bring in "efficiency" as an excuse (the other most common excuse being that it's easier to follow your code when you can just read it sequentially, which is another way to say "I don't understand abstraction" :-).
Now, efficiency is a run-time concept, and I haven't explained the corresponding concept in my theory yet. Still, using again the informal notion of efficiency we all have, we can already see that efficiency [in the run-time world] tends to decrease as distance [in the artifact world] increases. For instance, moving lines into another function requires passing parameters around. This is a first-cut, rough explanation of the well-known trade-off between run-time efficiency and artifact quality (maintainability, readability, reusability).

Coming soon:
the concept of distance in the run-time world, and distance-preserving transformations
efficiency in the physics of software
tangling
not sure yet : ), but probably isolation and density

Bad Luck, or "fighting the forcefield"

In my previous post, I used the expression "fighting the forcefield". This might be a somewhat uncommon terminology, but I used it to describe a very familiar situation: actually, I see people fighting the forcefield all the time.

Look at any troubled project, and you'll see people who made some wrong decision early on, and then stood by it, digging and digging. Of course, any decision may turn out to be wrong. Software development is a knowledge acquisition process. We often take decisions without knowing all the details; if we didn't, we would never get anything done (see analysis paralysis for more). Experience should mitigate the number of wrong decisions, but there are going to be mistakes anyway; we should be able to recognize them quickly, backtrack, and take another way.

Experience should also bring us in closer contact with the forcefield. Experienced designers don't need to go through each and every excruciating detail before they can take a decision. As I said earlier, we can almost feel, or see the forcefield, and take decisions based on a relatively small number of prevailing forces (yes, I dare to consider myself an experienced designer :-).
This process is largely unconscious, and sometimes it's hard to rationalize all the internal reasoning; in many cases, people expect very argumentative explanations, while all we have to offer on the fly is aesthetics. Indeed, I'm often very informal when I design; I tend to use colorful expressions like "oh, that sucks", or "that brings bad luck" to indicate a flaw, and so on.

Recently, I've found myself saying that "bad luck" thing twice, while reviewing the design of two very different systems (a business system and a reactive system), for two different clients.
I noticed a pattern: in both cases, there was a single entity (a database table, a in-memory structure) storing data with very different timing/life requirements. In both cases, my clients were a little puzzled, as they thought those data belonged together (we can recognize gravity at play here).
Most naturally, they asked me why I would keep the data apart. Time to rationalize :-), once again.

Had they all been more familiar with my blog, I would have pointed to my recent post on multiplicity. After all, data with very different update frequency (like: the calibration data for a sensor, and the most recent sample) have a different fourth-dimensional multiplicity. Sure, at any given point in time, a sensor has one most recent sample and one set of calibration data; therefore, in a static view we'll have multiplicity 1 for both, suggesting we can keep the two of them together. But bring in the fourth dimension (time) and you'll see an entirely different picture: they have a completely different historical multiplicity.

Different update frequencies also hint at the fact that data is changing under different forces. By keeping together things that are influenced by more than one force, we expose them to both. More on this another time.

Hard-core programmers may want more than that. They may ask for more familiar reasons not to put data with different update frequencies in the same table or structure. Here are a few:

- In a multi-threaded software, in-memory structures requires locking. If your structure contains data that is seldom updated, that means it's being read more than written: if it's seldom read and seldom written, why keep it around at all?
Unfortunately, the high-frequency data is written quite often. Therefore, either we accept to slow down everything using a simple mutex, or we aim for higher performances through a more complex locking mechanism (reader/writer lock), which may or may not work, depending on the exact read/write pattern. Separate structures can adopt a simpler locking mechanism, as one is being mostly read, the other mostly written; even if you go with a R/W lock, here it's almost guaranteed to have good performance.

- Even on a database, high-frequency writes may stall low-frequency reads. You even risk a lock escalation from record to table. Then you either go with dirty reads (betting on your good luck) or you just move the data in another table, where it belongs.

- If you decide to cache database data to improve performances, you'll have to choose between a larger cache with the same structure of the database (with low frequency data too) or a smaller and more efficient cache with just the high-frequency data (therefore revealing once more that those data do not belong together).

- And so on: I encourage you to find more reasons!

In most cases, I tend to avoid this kind of problems instinctively: this is what I really call experience. Indeed, Donald Schön reminds us that good design is not for everyone, and that you have to develop your own sense of aesthetics (see "Reflective Conversation with Materials. An interview with Donald Schön by John Bennett", in Bringing Design To Software, Addison-Wesley, 1996). Aesthetics may not sound too technical, but consider it a shortcut for: you have to develop your own ability to perceive the forcefield, and instinctively know what is wrong (misaligned) and right (aligned).

Ok, next time I'll get back to the notion of multiplicity. Actually, although I've initially chosen "multiplicity" because of its familiarity, I'm beginning to think that the whole notion of fourth-dimensional multiplicity, which is indeed quite important, might be confusing for some. I'm therefore looking for a better term, which can clearly convey both the traditional ("static") and the extended (fourth-dimensional, historical, etc) meaning. Any good idea? Say it here, or drop me an email!

Lost

I’ve been facing some small, tough design problems lately: relatively simple cases where finding a good solution is surprisingly hard. As usual, it’s trivial to come up with something that “works”; it’s also quite simple to come up with a reasonably good solution. It’s hard to come up with a great solution, where all forces are properly balanced and something beautiful takes shape.

I like to think visually, and since standard notations weren’t particularly helpful, I tried to represent the problem using a richer, non-standard notation, somehow resembling Christopher Alexander’s sketches. I wish I could say it made a huge difference, but it didn’t. Still, it was quite helpful in highlighting some forces in the problem domain, like an unbalanced multiplicity between three main concepts, and a precious-yet-fragile information hiding barrier. The same forces are not so visible in (e.g.) a standard UML class diagram.

Alexander, even in his early works, strongly emphasized the role of sketches while documenting a pattern. Sketches should convey the problem, the process to generate or build a solution, and the solution itself. Software patterns are usually represented using a class diagram and/or a sequence diagram, which can’t really convey all that information at once.

Of course, I’m not the first to spend some time pondering on the issue of [generative] diagrams. Most notably, in the late ‘90s Jim Coplien wrote four visionary articles dealing with sketches, the geometrical properties of code, alternative notations for object diagrams, and some (truly) imponderable questions. Those papers appeared on the long-dead C++ Report, but they are now available online:

Space-The final frontier (March 1998)
Worth a thousand words (May 1998)
To Iterate is Human, To Recurse, Divine (July/August 1998)
The Geometry of C++ Objects (October 1998)

Now, good ol’ Cope has always been one of my favorite authors. I’ve learnt a lot from him, and I’m still reading most of his works. Yet, ten years ago, when I read that stuff, I couldn’t help thinking that he lost it. He was on a very difficult quest, trying to define what software is really about, what beauty in software is really about, trying to adapt theories firmly grounded in physical space to something that is not even physical. Zen and the Art of Motorcycle Maintenance all around, some madness included :-).

I re-read those papers recently. That weird feeling is still here. Lights and shadows, nice concepts and half-baked ideas, lot of code-centric reasoning, overall confusion, not a single strong point. Yeah, I still think he lost it, somehow :-), and as far as I know, the quest ended there.
Still, his questions, some of his intuitions, and even some of his most outrageous :-) ideas were too good to go wasted.

The idea of center, that he got from The Nature of Order (Alexander’s latest work) is particularly interesting. Here is a quote from Alexander:
Centers are those particular identified sets, or systems, which appear within the larger whole as distinct and noticeable parts. They appear because they have noticeable distinctness, which makes them separate out from their surroundings and makes them cohere, and it is from the arrangements of these coherent parts that other coherent parts appear.

Can we translate this concept into the software domain? Or, as Jim said, What kind of x is there that makes it true to say that every successful program is an x of x's?. I’ll let you read what Jim had to say about it. And then (am I losing it too? :-) I’ll tell you what I think that x is.

Note: guys, I know some of you already think I lost it :-), and would rather read something about (e.g.) using variadic templates in C++ (which are quite cool, actually :-) to implement SCOOP-like concurrency in a snap. Bear with me. There is more to software design than programming languages and new technologies. Sometimes, we gotta stretch our mind a little.

Anyway, once I get past the x of x thing, I’d like to talk about one of those wicked design problems. A bit simplified, down to the essential. After all, as Alexander says in the preface of “Notes on the Synthesis of Form”: I think it’s absurd to separate the study of designing from the practice of design. Practice, practice, practice. Reminds me of another book I read recently, an unconventional translation of the Analects of Confucius. But I’ll save that for another time :-).

Can AOP inform OOP (toward SOA, too? :-) [part 2]

Aspect-oriented programming is still largely code-centric. This is not surprising, as OOP went through the same process: early emphasis was on coding, and it took quite a few years before OOD was ready for prime time. The truth about OOA is that it never really got its share (guess use cases just killed it).

This is not to say that nobody is thinking about the so-called early aspects. A notable work is the Theme Approach (there is also a good book about Theme). Please stay away from the depressing idea that use cases are aspects; as I said a long time ago, it's just too lame.

My personal view on early aspects is quite simple: right now, I mostly look for cross-cutting business rules as candidate aspects. I guess it's quite obvious that the whole "friend gift" concept is a business rule cutting through User and Subscription, and therefore a candidate aspect. Although I'm not saying that all early aspects are cross-cutting business rules (or vice-versa), so far this simple guideline has served me well in a number of cases.

It is interesting to see how early aspects tend to be cross-cutting (that is, they hook into more than one class) but not pervasive. An example of pervasive concern is the ubiquitous logging. Early aspects tend to cut through a few selected classes, and tend to be non-reusable (while a logging aspect can be made highly reusable).
This seems at odd with the idea that "AOP is not for singleton", but I've already expressed my doubt on the validity of this suggestion a long time ago. It seems to me that AOP is still in its infancy when it comes to good principles.

Which brings me to obliviousness. Obliviousness is an interesting concept, but just as it happened with inheritance in the early OOP days, people tend to get carried over.
Remember when white-box inheritance was applied without understanding (for instance) the fragile base class problem?
People may view inheritance as a way to "patch" a base class and change its behaviour in unexpected ways. But truth is, a base class must be designed to be extended, and extension can take place only through well-defined extensions hot-points. It is not a rare occurrence to refactor a base class to make extension safer.

Aspects are not really different. People may view aspects as a way to "patch" existing code and change its behaviour in unexpected ways. But truth is, when you move outside the safe realm of spectators (see my post above for more), your code needs to be designed for interception.
Consider, for instance, the initial idea of patching the User class through aspects, adding a data member, and adding a corresponding data into the database. Can your persistence logic be patched through an aspect? Well, it depends!
Existing AOP languages can't advise any given line: there is a fixed grammar for pointcuts, like method call, data member access, and so on. So if your persistence code was (trivially)

class User
  {
  void Save()
    {
    // open a transaction
    // do your SQL stuff
    // close the transaction
    }
  }

there would be no way to participate to the transaction from an aspect. You would have to refactor your code, e.g. by moving the SQL part in a separate method, taking the transaction as a parameter. Can we still call this obliviousness? That's highly debatable! I may not know the details of the advice, but I damn sure know I'm being advised, as I refactored my code to be pointcut-friendly.

Is this really different from exposing a CreateSubscription event? Yeah, well, it's more code to write. But in many data-oriented applications, a well-administered dose of events in the CRUD methods can take you a long way toward a more flexible architecture.

A closing remark on the SOA part. SOA is still much of a buzzword, and many good [design] ideas are still in the decontextualized stage, where people are expected to blindly follow some rule without understanding the impact of what they're doing.

In my view, a crucial step toward SOA is modularity. Modularity has to take place at all levels, even (this will sound like an heresy to some) at the database level. Ideally (this is not a constraint to be forced, but a force to be considered) every service will own its own tables. No more huge SQL statements traversing every tidbit in the database.

Therefore, if you consider the "friend gift" as a separate service, it is only natural to avoid tangling the User class, the Subscription class, and the User table with information that just doesn't belong there. In a nutshell, separating a cross-cutting business rule into an aspect-like class will bring you to a more modular architecture, and modularity is one of the keys to true SOA.

Process as the company's homeostatic system

I was learning about EPOC (Excess Post-Exercise Oxygen Consumption) a few weeks ago, for no particular reason (as I often do :-). If you ever exercised at medium-to-high intensity, you've probably experienced EPOC: after you stop training, your oxygen intake stays higher than your usual resting intake for quite a while, declining over several hours.
The reason for EPOC "is the general disturbance to homeostasis brought on by exercise" (Brooks and Fahey, "Exercise physiology. Human bioenergetics and its applications", Macmillan Publishing Company, 1984).

In the next few days, inspired by different readings, I began thinking of process as the company's homeostatic system.
I've often claimed that companies have their own immune system, actively killing ideas, concepts and practices misaligned with the true nature of the company ("true" as opposed to, for instance, a theoretical statement of the company's values).
However, the homeostatic system has a different purpose, as it is basically designed to (wikipedia quote) "allow an organism to function effectively in a broad range of environmental conditions". Once you replace "organism" with "team", this becomes the best definition of a good development process that I've ever (or never :-) been able to formulate.

It is interesting to understand that the homeostatic system is very sophisticated. It works smoothly, but can leverage several different strategies to adapt to different external conditions. In fact, when good old Gerald Weinberg classified company's cultural patterns on the 1-5 scale, he called level-2 companies "Routine" (as they always use the same approach, without much care for context), level-3 "Steering" (dynamically picking a routine from a larger repertoire) and level-4 "Anticipating" (you figure :-). Of course, any company using the same process every time, no matter the process (Waterfall, RUP-inspired, XP, whatever), is at level 2.

For instance, if we find ourselves working with stakeholders with highly conflicting requirements and we don't apply a more "formal" process based on the initial assessment of viewpoints and concerns (see, for instance, my posts Value and Priorities and Requirements and Viewpoints for a few references) because "we usually gather users stories, implement, and get feedback quickly", then we're stuck in a Routine culture. Of course, if we always apply the viewpoints, we're stuck in a Routine culture too.

Well, there would be much more to say, especially about contingent methodologies and about congruency (another of Weinberg's favorite concepts) between company's process and company's strategy, but time is up again, so I'll leave you with a question. We all know that some physical activity is good for our health. Although high-intensity exercises bring disturbance to homeostasis, in the end we get a better homeostatic system. So the question is, if process is the company's homeostatic system, what is the equivalent of good training? How do we bring a dose of periodic, healthy disturbance to make our process stronger?

On the concept of Form (1)

I postponed writing on Form (vs. Function) for a while now, mostly because I was struggling to impose a rational structure on my thoughts. Form is a complex subject, barely discussed in the software design literature, with a few exception I'll quote when relevant, so I'm sort of charting new waters here.

The point is, doing so would take too much [free] time, delaying my writings for several more weeks or even months. This is exactly the opposite of what I want to do with this blog (see my post blogging as destructuring), so I'll commit the ultimate sin :-) and talk about form in a rather unstructured way. I'll start a random sequence of posting on form, writing down what I think is relevant, but without any particular order. I won't even define the concept of form in this first instalment.

Throughout these posts, I'll borrow extensively from a very interesting (and warmly suggested to any software designer) book from Christopher Alexander. Alexander is better known for his work on patterns, which ultimately inspired software designers and created the pattern movement. However, his most releavant work on form is Notes on the Synthesis of Form. When quoting, I'll try to remember page numbers, in which case, I'll refer to the paperback edition, which is easier to find.

Alexander defines two processes through which form can be obtained: the unselfconscious and the selfconscious process. I'll get back to these two concepts, but in a few words, the unselfconscious process is the way some ancient cultures proceeded, without an explicit set of principles, but relying instead on rigid tradition mediated by immediate, small scale adaptation upon failure. It's more complex than that, but let's keep it simple right now.

Tradition provides viscosity. Without tradition, and without explicit design principles, the byproducts of the unselfconscious process will quickly degenerate. However, merely repeating a form won't keep up with a changing environment. Yet change happens, maybe shortly after the form has been built. Here is where we need immediate action, correcting any failure using the materials at hand. Of course, small scale changes are sustainable only when the rate of change (or rate of failure) is slow.

Drawing parallels to software is easy, although subjective. Think of quick, continuous, small scale adaptation. The immediate software counterpart is, very naturally, refactoring. As soon as a bad smell emerge, you fix it. Refactoring is usually small scale, using the "materials at hand" (which I could roughly translate into "changing only a small fraction of code"). Refactoring, by definition, does not change the function of code. Therefore, it can only change its form.

Now, although some people in the XP/agile camp might disagree, refactoring is a viable solution only when the desired rate of change is slow, and only when the gap to fill is small. In other words, only when the overall architecture (or plain structure) is not challenged: maybe it's dictated by the J2EE way of doing things, or by the Company One True Way of doing things, or by the Model View Controller police, and so on. Truth is, without an overall architecture resisting change, a neverending sequence of small-scale refactoring may even have a negative large-scale impact.

I've recently said that we can't reasonably turn an old-style, client-server application into a modern web application by applying a sequence of small-scale changes. It would be, if not unfeasible, hardly economic, and the final architecture might be far from optimal. The gap is too big. We're expecting to complete a big change in a comparatively short time, hence the rate of change is too big. The viscosity of the previous solution will fight that change and prevent it from happening. We need to apply change at an higher granularity level, as the dynamics in the small are not the dynamics in the large.

Curiously enough (or maybe not :-), I'll be talking about refactoring the next two days. As usual, I'll try to strike a balance, and get often back to good design princples. After all, as we'll see, when the rate of change grows, and/or when the solution space grows, the unselfconscious process must be replaced by the selfconscious process.

Get the ball rolling, part 4 (of 4, told ya :-)

The past two weeks have been hectic to say the least, and I had no time to conclude this quadrilogy. Fear not :-), however, because here I am. Anyway,I hope you used the extra time to read comments and answers to the previous posts, especially Zibibbo's comments; his contribution has been extremely valuable.
Indeed, I've left quite a few interesting topics hanging. I'll deal with the small stuff first.

Code [and quality]:
I already mentioned that I'm not always so line-conscious when I write code. Among other things, in real life I would have used assertions more liberally.
For instance, there is an implicit contract between Frame and Game. Frame is assuming that Game won't call its Throw method if the frame IsComplete. Of course, Game is currently respecting the contract. Still, it would be good to make the contract explicit in code. An assertion would do just fine.
I remember someone saying that with TDD, you don't need assertions anymore. I find this concept naive. Assertions like the above would help to locate defects (not just detect them) during development, or to highlight maintenance changes that have violated some internal assumption of existing code. Test cases aren't always the most efficient way to document the system. We have several tools, we should use them wisely, not religiously.

When is visual modeling not helpful?
In my experience, there are times when a model just won't cut it.
The most frequent case is when you're dealing with untried (or even unstable) technology. In this context, the wise thing to do is write code first. Code that is intended to familiarize yourself with the technology, even to break it, so that you know what you can safely do, and what you shouldn't do at all.
I consider this learning phase part of the design itself, because the ultimate result (unless you're just hacking your way through it) is just abstract knowledge, not executable knowledge: the code is probably too crappy to be kept, at least by my standards.
Still, the knowledge you acquired will shape the ultimate design. Indeed, once you possess enough knowledge, you can start a more intentional design process. Here modeling may have a role, or not, depending on the scale of what you're doing.
There are certainly many other cases where starting with a UML model is not the right thing to do. For instance, if you're under severe resource constraints (memory, timing, etc), you better do your homework and understand what you need at the code level before you move up to abstract modeling. If you are unsure about how your system will scale under severe loading, you can still use some modeling techniques (see, for instance, my More on Quantitative Design Methods post), but you need some realistic code to start with, and I'm not talking about UML models anyway.
So, again, the process you choose must be a good fit for your problem, your knowledge, your people. Restraining yourself to a small set of code-centric practices doesn't look that smart.

What about a Bowling Framework?
This is not really something that can be discussed briefly, so I'll have to postpone some more elaborated thoughts for another post. Guess I'll merge them with all the "Form Vs. Function " stuff I've been hinting to all this time.
Leaving the Ball and Lane aside for a while, a small but significant improvement would be to use Factory Method on Game, basically making Game an abstract class. That would allow (e.g.) regular bowling to create 9 Frames and 1 FinalFrame, 3-6-9 bowling to change the 3rd, 6th and 9th frame to instances of a (new) StrikeFrame class, and so on.
Note that this is a simple refactoring of the existing code/design (in this sense, the existing design has been consciously underengineered, to the point where making it better is simple). Inded, there is an important, trivial and at the same time deep reason why this refactoring is easy. I'll cover that while talking about options.
If you try to implement 3-6-9 bowling, you might also find it useful to change
if( frames[ currentFrame ].IsComplete() )
   ++currentFrame;
into
while( frames[ currentFrame ].IsComplete() )
   ++currentFrame;
but this is not strictly necessary, depending on your specific implementation.
There are also a few more changes that would make the framework better: for instance, as I mentioned in my previous post, moving MaxBalls from a constant to a virtual function would allow the base class to instantiate the thrownBalls array even if the creational responsibility for frames were moved to the derived class (using Factory Method).
Finally, the good Zibibbo posted also a solution based on a state machine concept. Now, it's pretty obvious that bowling is indeed based on a state machine, but curiously enough, I wouldn't try to base the framework on a generalized state machine. I'll have to save this for another post (again, I suspect, Form Vs. Function), but shortly, in this specific case (and in quite a few more) I'd try to deal with variations by providing a structure where variations can be plugged in, instead of playing the functional abstraction card. More on this another time.

Procedural complexity Vs. Structural complexity
Several people have been (and still are) taught to program procedurally. In some disciplines, like scientific and engineering computing, this is often still considered the way to go. You get your matrix code right :-), and above that, it's naked algorithms all around.
When you program this way (been there, done that) you develop the ability to pour through long functions, calling other functions, calling other functions, and overall "see" what is going on. You learn to understand the dynamics of mutually recursive function. You learn to keep track of side effects on global variables. You learn to pay attention to side effects on the parameters you're passing around. And so on.
In short, you learn to deal with procedural complexity. Procedural complexity is best dealt with at the code level, as you need to master several tiny details that can only be faithfully represented in code.
People who have truly absorbed what objects are about tend to move some of this complexity on the structural side. They use shape to simplify procedures.
While the only shape you can give to procedural code is basically a tree (with the exception of [mutually] recursive functions, and ignoring function pointers), OO software is a different material, which can be shaped in a more complex collaboration graph.
This graph is best dealt with visually, because you're not interested in the tiny details of the small methods, but in getting the collaboration right, the shape right (where "right" is, again, highly contextual), even the holes right (that is, what is not in the diagram can be important as well).
Dealing with structural complexity requires a different set of skills and tools. Note that people can be good at dealing with procedural and with structural complexity; it's just a matter of learning.
As usual, good design is about balance between this two forms of complexity. More on this another time :-).

Balls, Real Options, and some Big Stuff.
Real Options are still cool in project management, and in the past few years software development has taken notice. Unfortunately, most papers on software and real options tend to fall in one of these two categories:
- the mathematically heavy with little practical advice.
- the agile advocacy with the rather narrow view that options are just about waiting.
Now, in a paper I've referenced elsewhere, Avi Kamara put it right in a few words. The problem with the old-fashioned economic theory is the assumption that the investment is an all or nothing, now or never, project. Real options challenge that view, by becoming aware of the costs and benefits of doing or not doing things, build flexibility and take advantage of opportunities over time (italics are quotes from Kamara).
Now, this seems to be like a recipe for agile development. And indeed it is, once we break the (wrong) equation agile = code centric, or agile = YAGNI. Let's look a little deeper.
Options don't come out of nowhere. They came either from the outside, or from the inside. Options coming from the outside are new market opportunities, emerging users's requests or feedback, new technologies, and so on. Of course, sticking to a plan (or a Big Upfront Design) and ignoring those options wouldn't be smart (it's not really a matter of being agile, but to be economically competent or not).
Options come also from the inside. If you build your software upon platform-specific technologies, you won't have an option to expand into a different platform. If you don't build an option for growth inside your software, you just won't have the option to grow. Indeed, it is widely acknowledged in the (good) literature that options have a cost (the option premium). You pay that cost because it provides you with an option. You don't want to make the full investment now (the "all" in "all or nothing") but if you just do nothing, you won't get the option either.
The key, of course, is that the option premium should be small. Also, the exercise cost should be reasonable as well (exercise price, or strike price, is what you pay to actually exercise your option). Note: for those interested in these ideas, the best book I've found so far on real option is "Real options analysis" by Johnathan Mun, Wiley finance series)

Now, we can begin to see the role of careful design in building the right options inside software, and why YAGNI is an oversimplified strategy.
In a previous post I mentioned how a class Lane could be a useful abstraction for a more realistic bowling scorer. The lane would have a sensor in the foul line and in the gutters. This could be useful for some variation of bowling, like Low Ball (look under "Special Games"). So, should I invest into a Lane class I don't really need right now, because it might be useful in the future? Wait, there is more :-).
If you consider 5 pin Bowling, you'll see that different pins are awarded different scores. Yeap. You can no longer assume "number of pins = score". If you think about it, there is just no natural place in the XP episode code to put this kind of knowledge. They went for then "nothing" side of the investment. Bad luck? Well guys, that's just YAGNI in the real world. Zero premium, but potentially high exercise cost.
Now, consider this: a well-placed, under-engineered class can be seen as an option with a very low premium and very low exercise cost. Consider my Ball class. As it stands now, it does precious nothing (therefore, the premium cost was small). However, the Ball class can easily be turned into a full-fledged calculator of the Ball score. It could easily talk to a Lane class if that's useful - the power of modularity allows the rest of my design to blissfully ignore the way Ball gets to know the score. I might even have a hierarchy of Ball classes for different games (well, most likely, I would).
Of course, there would be some small changes here and there. I would have to break the Game interface, that takes a score and builds a Ball. I used that trick to keep the interface compatible with the XP episode code and harvest their test cases, so I don't really mind :-). I would also have to turn the public hitPins member into something else, but here is the power of names: they make it easier to replace a concept, especially in a refactoring-aware IDE.
Bottom line: by introducing a seemingly useless Ball class, I paid a tiny premium cost to secure myself a very low exercise cost, in case I needed to support different kinds of bowling. I didn't go for the "all" investment (a full-blown bowling framework), but I didn't go for the "nothing" either. That's applied real option theory; no babbling about being agile will get you there. The right amount of under-engineering, just like the right amount of over-engineering, comes only from reasoning, not from blindly applying oversimplified concepts like YAGNI.
One more thing about options: in the Bowling Framework section, I mentioned that refactoring my design by applying Factory Method to Game was a trivial refactoring. The reason it was trivial, of course, is that Game is a named entity. You find it, refactor it, and it's done. Unnamed entities (like literals) are more troublesome. Here is the scoop: primitive types are almost like unnamed entities. If you keep your score into an integer, and you have integers everywhere (indexes, counters, whatever), you can't easily refactor your code, because not any integer is a score. That was the idea behind Information Hiding. Some people got it, some didn't. Choose your teacher wisely :-).

A few conclusions
As I said, there are definitely times when modeling makes little sense. There are also times when it's really useful. Along the same lines, there are times when requirements are relatively unknown, or when nobody can define what "success" really means before they see it. There are also times when requirements are largely well-known and (dare I say it!) largely stable.
If you were asked to implement an automatic scoring system for 10 pin bowling and 3-6-9 bowling and 9 pin bowling, would you really start coding the way they did in the XP episode, and then slowly change it to something better?
Would you go YAGNI, even though you perfectly know that you ARE gonna need extensibility? Or would you rather spend a few more minutes on the drawing board, and come up with something like I did?
Do you really believe you can release a working 3-6-9 and a working 9-pin faster using the XP episode code than mine? Is it smart to behave as if requirements were unknown when they're, in fact, largely known? Is it smart not to exploit knowledge you already have? What do you consider more agile, that is, adaptive to the environment?
Now, just to provoke a reaction from a friend of mine (who, most likely, is somewhere having fun and not reading my blog anyway :-). You might remember Jurassik Park, and how Malcom explained chaos theory to Ellie by dropping some water on her hand (if you don't, here is the script, look for "38" inside; and no, I'm not a fan :-).
The drops took completely different paths, because of small scale changes (although not really small scale for a drop, but anyway). Some people contend that requirements are just as chaotic, and that minor changes in the environment will have a gigantic impact on your code anyway, so why bother with upfront design.
Well, I could contend that my upfront design led to code better equipped to deal with change than the XP episode did, but I won't. Because you know, the drops did take a completely different path because of small scale changes. But a very predictable thing happened: they both went down. Gravity didn't behave chaotically.
Good designers learn to see the underlying order inside chaos, or more exactly, predominant forces that won't be affected by environmental conditions. It's not a matter of reading a crystal ball. It's a matter of learning, experience, and well, talent. Of course, I'm not saying that there are always predominant, stable forces; just that, in several cases, there are. Context is the key. Just as ignoring gravity ain't safe, ignoring context ain't agile.