I try to keep an eye on software design literature. I subscribe to relevant publications from IEEE and ACM; I get a copy of conference proceedings; I read blogs and, just like everybody else, I follow links and suggestions. Being aware of the potential risk of filtering out information that is not aligned with the way I think, I'm also following a few blogs that are definitely not aligned with my belief system, just to make sure I don't miss too much. I'm bound to miss something, anyway, but I can live with that :-).
A few weeks ago I realized that I wasn't following Kent Beck's blog. I don't know why: I'm not a fan of XP, but I'm rather fond of Kent. He's an experienced designer with many good ideas. So, I took a tour of his posts and discovered that I had completely missed the concept of Responsive Design. That's weird, because I'm reading/scanning quite a few agile-oriented blogs, and I've never seen any mention of it. Oh, well; time to catch up.
I did my homework and spent some time reading all posts in the Responsive Design category and watching Kent's interesting QCon presentation. Looking at blog dates, it seems like activity peaked in April 2009, but rapidly declined during 2009 to almost nothing in 2010. So, apparently, Responsive Design hasn't taken the world by storm yet. Well, design wasn't so popular in the pre-agile era, and it's not going to be popular in the post-agile era either :-).
Anyway, the presentation inspired me with a stream of reflections that I'd like to share with you guys. I would recommend that you watch the presentation first (I know, it takes about 1 hour, but I think it's worth it). What follows is not intended to be a critic. I'm mostly trying to relate Kent's perspective on "what software design is about" with mine. It's a rather long post, so I split it in three paragraphs. The first two are about concepts in the presentation itself. The third is about an interesting difference in perspective, and how it affects our thinking.
Reflections on the introductory part
The idea of taking notes as we design, to uncover our own strategies and tactics, sounded so familiar. I guess at some point some of us feel an urge to understand what we're really doing. Although I'm looking for something different from Kent, I share some of his worries: trivial or extremely complicated insights might be just around the corner :-)
The talk about the meaning of "responsive" around 0:13 is resonating very well with the concepts of forcefield, design as "shaping a material", and backtalk. From my perspective, Kent is mostly saying: am I going to look at the forcefield, and design accordingly, or am I going to force a preconceived set of design decisions upon the problem? (see also my empty cup post).
Steady flow of features. We all love that, of course, and in a sense it's the most agile, XP-ish concept in the entire talk. The part about "the right time to design" seems rather connected with the Least Responsible Moment, so I can't really spare you a link to my own little idea of Invariant Decisions.
Again, I have a feeling that there is never enough context when talking about timing. I've certainly designed systems with a large separation (in time) between design and implementation. In many cases, it worked out quite well (of course, we always considered design as something fluid, not cast in stone). I guess it has a lot to do with how much domain knowledge you can leverage. Not every project is about doing something nobody has done before. Many are "just" about doing something in a much better way. Context, context, context...
Starting with values: it's something I have to improve in my Physics of Software work. Although I've said several times that all new methods should start with a declaration of values and beliefs, so far I haven't been thorough on this side. For instance, I value honest, unbiased, free, creative communication and thinking about software design issues, where the best ideas, and not the best talkers, get to win (just because I'm a good talker doesn't mean I want to win that way :-)). That's why I'm trying to come up with a less ambiguous, wishy-washy way to think and talk about design.
"Most of the decisions I make while designing have nothing to do with the problem domain [… but ...] are shaped by the fact that I'm instructing a computer. Weird. This is not my experience. I surely reuse solutions across domains, but the problem domain is heavily shaping the forcefield. When you go down to small-scale decisions, yeah, it gets more domain-independent, but high-level design is heavily problem-dependent. For instance, while you can usually (but not always) implement a chosen data structure in a rather domain-independent way, choosing the right data structure is usually a domain-dependent choice.
Don't get me wrong: I know I can safely ignore some portions of the problem domain when I design - I do that all the time. My view is that I can ignore the (possibly very complex) functional issues that have little impact on the forcefield, and therefore on the form. This is a complex subject that would require an in-depth study.
Principles. I don't quite like the term "principle" because is so wide in meaning that you can use it for just about everything. So, while the "don't repeat yourself" is a commonly accepted design principle, Kent's examples of principles from the insurance world or from Dynamo looks much more like pre-made [meta] decisions to me.
"You should never to lose a write". Fine. It's a decision, you can basically consider that a requirement.
"We're there to aid human decision making". Fine. It's a meta-decision, meaning, it's not just a design decision: it will influence everything, from your value proposition down to functional requirements and so on. It's not really a design principle, not even a project-specific design principle: it's more akin to a metaphor or to a goal.
Aside from that distinction, I agree that sometimes constraints, whatever form they take, are actually helpful during design, because they prune out a lot of the decision space (again, see my post above on the empty cup). Of course, the wrong constraints can prevent you from finding the sweet spot, so timing is an issue here as well - some constraints should be considered fluid early on, because you may learn that they're not the right constraints for your project.
The short part about patterns and principles reminded me of my early work on SysOOD, some of which got published in IEEE Software exactly as Principles Vs. Patterns. So, while Kent wants to understand principles better, I've always been trying to eliminate principles altogether (universal principles, that is). Still, I'd like to see the "non-universal" or "project-specific" principles recast under some other name and further explored, because I agree that having a shared set of values / goals / constraints can help tremendously in any significant project. Oh, I still have the Byte smalltalk balloon issue somewhere : ).
Reflection on the 4 strategies
I see the "meat" of the talk as being about [meta] strategies to move in the decision space. Your software is at some point A in the multi-dimensional decision space; for instance, you decided early on that you would use filenames and not streams. Now you want to move your software to another point B in the decision space (where, for instance, streams are used instead of filenames). How do you go from A to B?
That's an interesting issue, where experienced designers/programmers routinely adopt different approaches compared to novices (see the empty cup post again), so it's both interesting and promising to see Kent articulate his reasoning. In a sense, it's completely orthogonal to both patterns/antipatterns (which may point you to B / away from A) and to my work (which is more concerned with understanding and articulating why B is a more desirable point than A, or to find B in the first place).
Actually, strictly speaking I'm not even sure this stuff is really about "design". In my view, design is more about finding B. The execution/implementation of design, however, is about moving to B. Therefore, Kent's strategies looks more like "design execution strategies" than "design strategies".
That said, there is a subtle, yet striking difference between Kent's perspective and mine. Kent is talking about / looking to software design in a "first-person perspective", while I'm looking through the "I'm shaping a material" metaphor. He says things like: "I'm here; how do I get there?", and he's literally mimicking small steps. I'm saying things like: "my software is here; how do I move it there?"; it may seem like a small, irrelevant distinction, but I think it's shaping our thoughts rather deeply. It surely shapes the names we use: I would have never used "stepping stone", because I'm not thinking of myself going anywhere : ). More on this on the third paragraph below.
Although this is probably the most “practical” part of the talk, concepts are a little fuzzy. Leap can be seen as a strategy, actually the simplest strategy of all: you just move there. Safe Steps is more like a meta-strategy to move inside the decision space (indeed, around 42:00 Kent talks about Safe/Small Steps as a "principle"). Once you choose Safe Steps, you have many options left, Parallel, Stepping Stones and Simplifications being 3 of them. Still, the granularity of those strategies is so coarse that it's rather surprising Kent found only 3 of them.
For instance, I probably wouldn't have used Parallel to deal with the filenames/streams issues. I can't really say without looking at the code, but assuming I wanted to take small steps, I could have created a new class, which IS-A stream and HAS-A filename, moved the existing code to use that class (a very simple and safe change, because both the stream and the filename are there), gradually moved all the filename-dependent code inside that class, removed the filename from the interface (as it's no longer used), moved the filename-dependent code in a factory (so that I could make a choice on the stream type). At that point I could have killed the temporary class, using a plain stream again. This strategy (I could call it wrap/extract/unwrap) is quite simple and effective when dealing with legacy code, and doesn't really fit in any of the strategies Kent is proposing.
Stepping Stone. While Leap and Parallel are more like strategies to implement a decision you have already taken, it seems to me that Stepping Stone is different. Although this might be my biased understanding of it, it's also the only interpretation I can give that makes Stepping Stone different from good old top-down decomposition (which Kent makes clear is different, at about 50:30).
I would say that Stepping Stone is about making progress where you can, and while doing so, shed some light on the surroundings as well. This is not very clear in the beginning, but as Kent moves forward, it's more and more obvious that he's thinking about novel situations, where you don't know exactly what you want to build.
Indeed, sometimes the forcefield is just too blurry. Still, we often can see a portion of it rather clearly, and we may have a sensible idea about the form we want to shape locally. Stepping Stone is a good strategy here: build (or design) what you understand, get feedback (or backtalk), get progress, etc. I usually keep exploring the dark areas while I'm building stepping stones. I'm also rather careful not to overcommit on stepping stones, as they may turn out not to fit perfectly with what's coming next. That's fine, inasmuch as we understand that design is a process, not a phase with a beginning and an end.
I can also build "stepping stones" when I know where I'm going to, but as I said, at that point it's hard to tell the difference between building stepping stones and executing top-down design. Oh, by the way, for the agile guys who hated me so much for saying that TDD is a limited design strategy: move around 50:20. Play. Rewind. Play again. Repeat till necessary :-). But back to Stepping Stones: I think the key phrase here is around 55:40, when Kent says that by standing on a stepping stone he can see where to go next, which I read like: a previously blurry portion of the forcefield is now visible and clear, and I can move further.
Simplification. This is more akin to a design strategy, not to an execution strategy. I could see Simplification as a way to create small steps (perhaps even stepping stones). Indeed, the difference between Stepping Stones and Simplification is not really obvious (around 1:02:00 a participant asks about the similarity and doesn't seem like Kent can explain the difference so well either). I guess simplification is about to solve one single problem by solving a simpler version of the same problem first, whereas Stepping Stones are about solving a complex problem by solving another problem first (one that, however, brings us closer to solving the initial problem). In this sense, Simplification can be used where you can see where you wanna go (but it's too big / risky / ambitious) but also where you can't see exactly where you want to go, but you can clearly see a smaller version of it. Or, from my "I'm shaping a material" perspective, you can't really see the form of the real thing, but you can see the form of a smaller version of it. As I said, this stuff is interesting but rather fuzzy, and honestly, I'm not really sold on those 4 strategies.
Overall, I'd like to see Responsive Design further developed. It would probably benefit from a better distinction between design, design execution, principles, values, goals, and so on, but it's interesting stuff in the almost flat landscape of contemporary software design literature.
Perspective Matters
As I said, Kent is looking at the decision space in first-person perspective, while I see myself moving other things around. There is an interesting implication: when you think about moving yourself, you mostly consider distance. You are who you are. You want to go elsewhere. Distance, and therefore Leaps, Small Steps, Stepping Stones, it's all that matters.
When you think about moving something else, you have to consider two more factors. That, in my opinion, allows better reasoning.
Sure, distance is an important factor. But, as I explained in my post about Inertia, another important factor is Mass. You cannot ignore mass, but it's unnatural to consider a varying mass from a first-person perspective.
A simple example: I use Visual Studio (yeah yeah I know, some of you don't like it : ). Sometimes, as I'm creating a new project, I click in the wrong place (yeah yeah I'm kinda dumb :-) and I get the wrong project type. I want a C# project and I get a VB.NET project (yikes!! :-)). That's quite a distance in the decision space. Yet there is a very simple, safe step (Leap): delete the project and start from scratch. The distance is not a problem, because mass is basically null. I can use Leap not because the distance is small, but because the mass is small. I wouldn't do that with an existing, large project (I may consider going through an IL -> C# transformation though :-).
There is more: when I discussed Inertia, I talked about software being in a state of rest or motion in the decision space. This is the third factor, and again, it's hard to think about it in first-person perspective. You want to go somewhere, but you're already going elsewhere. It's not truly natural. Deflecting a moving object, however, is quite ordinary (most real-life games involving a ball, for instance, involve deflecting a moving object).
Consider a large project. Maybe a team is busy moving a subsystem S1 to another place in the decision space. Now you want to move subsystem S2 from A to B. It's not enough to consider distance and mass. You have to consider whether or not S2 is already moving, perhaps by attraction from S1. So S2 may be small (in mass) and the distance A-B may be small too, but if S2 is already moving into a different direction your job is harder, more risky, and in need of a careful execution strategy.
A real-world example: you have a legacy C++/MFC application. Currently, the UI is talking with the application through a small, homebrew messaging component (MC), of which you are in charge. You just found a better messaging component (BMC) on the internet, say something that is more efficient, portable, and actively developed by a community. You want to move your subsystem to another place in the decision space, from MC to BMC. Part of BMC being "portable" is based on the reasonable assumption that messages are strictly C++ objects.
Meanwhile, another team is moving the UI toward .NET (that's a big mass - big distance issue, but it's their problem, not yours :-). Now, there is an attraction between your messaging component and the UI. A .NET UI may have some issues dealing with native C++ messages. So, like it or not, there is another force at play, trying to move your component to a different place (say, NMC). Now, the mass of the messaging component might be small. The distance between MC and BMC may be small too - perhaps something you can deal with by using an adapter. But probably BMC is in a very different direction than NMC, where the UI team is pushing your component to move. So it's not just a matter of distance. It's not even enough to consider mass. You need to consider distance + full-blown Inertia. Or, to cut it short, work
Of course, depending on your project, some factor may dominate over the others, and a simpler model might be enough, but usually, distance alone won't cut it.
There is more...
At the end of the presentation, Kent mentions power law distribution. This is something that has got me interested too in the past few months (I mentioned that in Facebook), but it's too early to say something about it, and I want to check a few ideas with the author of an interesting paper first, anyway.
See you guys soon with a post on Distance (not in the decision space, but in the artifact space!)
Tuesday, July 27, 2010
Thursday, June 24, 2010
Notes on Software Design, Chapter 7: a better Forcefield Diagram
In my previous post in the NOSD series, I mentioned how an improved forcefield diagram was needed to model the kind of reasoning I'm trying to bring in software design. I also discussed the artifact-run/time dualism, and how many concepts in language design were born out of the fundamental need to balance conflicting forces between these two worlds. I also mentioned the role of some patterns in resolving the same kind of conflict.
Here I'll show you a practical example, introducing the improved forcefield notation as we go. It's quite simple, and as usual, the reasoning is more important than the drawing. I'll use new colors and shapes, but it's all very informal, the notation is not cast in stone, and it will probably evolve and change over time.
A common problem
You have two classes (Class1, Class2); as we learnt in Chapter 6, that usually means you have two artifacts, and right now I feel like blue is a good color for artifacts, so I'll color them in blue.
Now, those classes have some commonality in behavior; for instance, they both represent a geometrical object, and can provide you with a bounding box.
Behavior is a run-time concept, and I'll color that information in pink.
Commonality in behavior is an attractive force; I haven't talked about this yet, but trust me : ), or just rely on your intuition that "things that do similar things are close to each other".
Commonality in behavior also attracts a natural desire in the artifact space: polymorphic/uniform access to such behavior. While writing calling code, I'd like to ask for a bounding box in the same way, perhaps polymorphically through a base class / interface. That would make my client (partially) unaware of the specific classes.
Unfortunately, Class1 and Class2 have been written with different conventions. They don't share a base class; they don't use the same naming for functions; they may not even use the same types for parameters. Different conventions are an artifact issue, so I'll color this in blue again. Of course, different conventions are keeping Class1 and Class2 apart, and actively rejecting polymorphic / uniform access. So here is the forcefield, representing our problem:
Note that there is nothing here about a solution. At this stage, the forcefield is a representation of the problem. Still, we have a conflict between forces, and somehow we have to deal with it. What if we don't? I'll keep that as last.
Enter decisions
A missing concept in my previous attempts at modeling the forcefield was the very important notion of decision. Although I'm trying to keep concepts to a bare minimum, the Decision Space is an important piece of the puzzle, and there can't be a Decision Space without Decisions. I'm using a yellow hexagon to represent a decision.
So, how can we deal with those conflicting forces? A simple, technically sound decision is to refactor Class1 and Class2 to a common base class (or interface, or a hierarchy of both). That decision has a strong impact on "Different Conventions", effectively removing it from the forcefield, including the rejection lines. I think the diagram speaks for itself:
So, decisions can alter the forcefield, for better or worse. Still, we may choose to keep the artifacts unchanged. Perhaps Class1 and Class2 come from third-party libraries, or perhaps they're shared with a lot of existing code, and refactoring may impact that code as well (remember Mass and Inertia). Again, it's useful to represent this decision explicitly, although it's just an intermediate step. I colored "Different Conventions" in green to say that we deliberately decided to keep it inside the forcefield.
Patterns
Design patterns are now mainstream, with dozens of books and hundreds, if not thousands, of papers describing the problem / context / solution triad.
Now, the solution is usually represented using a UML diagram and/or some source code. Problem and context are described using text, or an example in UML/code. That's because we don't (didn't :-) have any proper way to describe a problem (forcefield) and context (mostly, pre-made decisions).
Still, look at the picture above once again: that's (of course :-) the problem/context setting for a well-known pattern: adapter. So let's look at the adapter in action, or how using adapter will impact the forcefield:
Adapter "simply" breaks the rejection between Different Conventions and Polymorphic / Uniform Access, therefore allowing client code to ignore the specific interface of Class1 and Class2. It is very interesting to observe what the forcefield is telling us: we didn't completely remove conflict. There is still an attraction/rejection between Class1 and Class2. In this sense, Adapter is less effective than refactoring (which, however, requires us to change the artifacts).
That conflict will emerge over time; for instance, adding common functionality will take more time, possibly some duplication of code, etc. This kind of "consequence" is not very well documented in the GoF book.
Note: there is no redundancy with code here: we're talking about the problem space and the decision space. Contrast this with the usual redundancy between a UML diagram and code (with pros and cons, as usual). Also, isn't this diagram much better at communicating design problems, decisions, and impact than the largely ignored design rationale tools and notations?
What else?
There is also a different decision we can make; it's a particularly bad decision, therefore it's also very popular :-) wherever code quality is easily ignored. We can just forgo uniform access, and have clients deal with a non-uniform interface (through if, switch/case, whatever). This is what I meant by "not dealing with conflict" earlier. I called that decision "Tangled Clients" for reasons that will become clear in a future post.
Time out
Last time I said I would provide some examples of conflicting forces in the non-software world. I'll have to cut this post short, because the forcefield diagram I've come up with would take too much to explain. But I'll offer a few pointers for those of you with some time to spare:
- Very simple: the adapter is extremely frequent in the real world as well, as the well-known socket adapter. The forcefield is remarkably similar, but finding the exact translation of every concept is not necessarily trivial. Try this out :-).
- Harder, some engineering knowledge is required. Rotating pumps may leak (why? hint: some empty space is needed if you want something to rotate freely :-). Old pumps just used a seal, which wasn't really good at preventing leaks. At some point (I think around 1940) the end face mechanical seal has been adopted as a better seal for rotating pumps. However, the fluid can still leak, which ain't that good in a number of cases. An interesting solution here is the magnetic drive pump (look it up, guys :-). Draw the forcefield for the problem, and represent how using a magnetic field to transmit movement can compensate otherwise conflicting forces. By the way, this is the example I was thinking about when I wrote my previous post.
- Manufacturing is actually full of great examples of conflicting forces and different approaches to compensate or overcome conflict. Think about thermal grease, just to give a starting point. The list is endless. Trying to model some forcefield in detail is a fascinating exercise.
- If you want to stay on the software side, here is an interesting one. Take some programming language feature (for instance, if you use .NET, you may consider partial classes or attributes; if you use Java, annotations) and identify which tension between the run-time world and the artifact world they're meant to solve. Partial classes are a simple, but interesting exercise: most criticism I've heard about them is stemming from a very partial :-) understanding of the surrounding forcefield, just like the common abuse I've seen (split a large class in two files :-)).
As usual, there is much more to say about compensating forces, the artifact/run-time dual nature of software, and so on, including an unexpected intuition on economy of scale that I got just yesterday while running. See you soon, and drop me a line if you try this stuff out :-)
Here I'll show you a practical example, introducing the improved forcefield notation as we go. It's quite simple, and as usual, the reasoning is more important than the drawing. I'll use new colors and shapes, but it's all very informal, the notation is not cast in stone, and it will probably evolve and change over time.
A common problem
You have two classes (Class1, Class2); as we learnt in Chapter 6, that usually means you have two artifacts, and right now I feel like blue is a good color for artifacts, so I'll color them in blue.
Now, those classes have some commonality in behavior; for instance, they both represent a geometrical object, and can provide you with a bounding box.
Behavior is a run-time concept, and I'll color that information in pink.
Commonality in behavior is an attractive force; I haven't talked about this yet, but trust me : ), or just rely on your intuition that "things that do similar things are close to each other".
Commonality in behavior also attracts a natural desire in the artifact space: polymorphic/uniform access to such behavior. While writing calling code, I'd like to ask for a bounding box in the same way, perhaps polymorphically through a base class / interface. That would make my client (partially) unaware of the specific classes.
Unfortunately, Class1 and Class2 have been written with different conventions. They don't share a base class; they don't use the same naming for functions; they may not even use the same types for parameters. Different conventions are an artifact issue, so I'll color this in blue again. Of course, different conventions are keeping Class1 and Class2 apart, and actively rejecting polymorphic / uniform access. So here is the forcefield, representing our problem:
Note that there is nothing here about a solution. At this stage, the forcefield is a representation of the problem. Still, we have a conflict between forces, and somehow we have to deal with it. What if we don't? I'll keep that as last.
Enter decisions
A missing concept in my previous attempts at modeling the forcefield was the very important notion of decision. Although I'm trying to keep concepts to a bare minimum, the Decision Space is an important piece of the puzzle, and there can't be a Decision Space without Decisions. I'm using a yellow hexagon to represent a decision.
So, how can we deal with those conflicting forces? A simple, technically sound decision is to refactor Class1 and Class2 to a common base class (or interface, or a hierarchy of both). That decision has a strong impact on "Different Conventions", effectively removing it from the forcefield, including the rejection lines. I think the diagram speaks for itself:
So, decisions can alter the forcefield, for better or worse. Still, we may choose to keep the artifacts unchanged. Perhaps Class1 and Class2 come from third-party libraries, or perhaps they're shared with a lot of existing code, and refactoring may impact that code as well (remember Mass and Inertia). Again, it's useful to represent this decision explicitly, although it's just an intermediate step. I colored "Different Conventions" in green to say that we deliberately decided to keep it inside the forcefield.
Patterns
Design patterns are now mainstream, with dozens of books and hundreds, if not thousands, of papers describing the problem / context / solution triad.
Now, the solution is usually represented using a UML diagram and/or some source code. Problem and context are described using text, or an example in UML/code. That's because we don't (didn't :-) have any proper way to describe a problem (forcefield) and context (mostly, pre-made decisions).
Still, look at the picture above once again: that's (of course :-) the problem/context setting for a well-known pattern: adapter. So let's look at the adapter in action, or how using adapter will impact the forcefield:
Adapter "simply" breaks the rejection between Different Conventions and Polymorphic / Uniform Access, therefore allowing client code to ignore the specific interface of Class1 and Class2. It is very interesting to observe what the forcefield is telling us: we didn't completely remove conflict. There is still an attraction/rejection between Class1 and Class2. In this sense, Adapter is less effective than refactoring (which, however, requires us to change the artifacts).
That conflict will emerge over time; for instance, adding common functionality will take more time, possibly some duplication of code, etc. This kind of "consequence" is not very well documented in the GoF book.
Note: there is no redundancy with code here: we're talking about the problem space and the decision space. Contrast this with the usual redundancy between a UML diagram and code (with pros and cons, as usual). Also, isn't this diagram much better at communicating design problems, decisions, and impact than the largely ignored design rationale tools and notations?
What else?
There is also a different decision we can make; it's a particularly bad decision, therefore it's also very popular :-) wherever code quality is easily ignored. We can just forgo uniform access, and have clients deal with a non-uniform interface (through if, switch/case, whatever). This is what I meant by "not dealing with conflict" earlier. I called that decision "Tangled Clients" for reasons that will become clear in a future post.
Time out
Last time I said I would provide some examples of conflicting forces in the non-software world. I'll have to cut this post short, because the forcefield diagram I've come up with would take too much to explain. But I'll offer a few pointers for those of you with some time to spare:
- Very simple: the adapter is extremely frequent in the real world as well, as the well-known socket adapter. The forcefield is remarkably similar, but finding the exact translation of every concept is not necessarily trivial. Try this out :-).
- Harder, some engineering knowledge is required. Rotating pumps may leak (why? hint: some empty space is needed if you want something to rotate freely :-). Old pumps just used a seal, which wasn't really good at preventing leaks. At some point (I think around 1940) the end face mechanical seal has been adopted as a better seal for rotating pumps. However, the fluid can still leak, which ain't that good in a number of cases. An interesting solution here is the magnetic drive pump (look it up, guys :-). Draw the forcefield for the problem, and represent how using a magnetic field to transmit movement can compensate otherwise conflicting forces. By the way, this is the example I was thinking about when I wrote my previous post.
- Manufacturing is actually full of great examples of conflicting forces and different approaches to compensate or overcome conflict. Think about thermal grease, just to give a starting point. The list is endless. Trying to model some forcefield in detail is a fascinating exercise.
- If you want to stay on the software side, here is an interesting one. Take some programming language feature (for instance, if you use .NET, you may consider partial classes or attributes; if you use Java, annotations) and identify which tension between the run-time world and the artifact world they're meant to solve. Partial classes are a simple, but interesting exercise: most criticism I've heard about them is stemming from a very partial :-) understanding of the surrounding forcefield, just like the common abuse I've seen (split a large class in two files :-)).
As usual, there is much more to say about compensating forces, the artifact/run-time dual nature of software, and so on, including an unexpected intuition on economy of scale that I got just yesterday while running. See you soon, and drop me a line if you try this stuff out :-)
Sunday, June 06, 2010
Notes on Software Design, Chapter 6: Balancing two Worlds
Back in my time, computer science students were introduced to the distinction between syntax and semantics from day one. That distinction is central in understanding programming languages from the algebraic perspective.
When you are looking for a theory of software forces, however, you'll find another distinction more useful. We have artifacts, and we have run-time instances of artifacts.
Software is encoded knowledge, and as software developers we shape that knowledge. However, we can't act on knowledge itself: we must go through a representation. Source files are a representation, UML diagrams are a representation, and so on.
Source files represent information using a language-specific syntax; for instance, if you use Java or C#, you encode some procedural and structural knowledge about a class inside a text file. The text file is an artifact; it's not a run-time thing. Along the same lines, you can use UML and draw a class diagram, encoding structural knowledge about several classes. That's another artifact. Often, when we say "class C" or "function foo" what we mean is actually its representation inside an artifact (a sequence of LOCs in a source file, where we encode some knowledge in a specific language).
Artifacts can be transformed into other artifacts: without mentioning MDA, you can just compile a bunch of source files and get an executable file. The executable file is just another artifact.
At that point, however, you can actually run the artifact. Let's say that you get a new process: the process is a run-time instance of the executable file artifact. Of course, you may start several run-time instances of the same artifact.
Once the process is started, a wonderful thing happens :-). Your structural and procedural knowledge, formerly represented through LOCs inside artifacts, is now executable as well. Functions can be called. Classes can be instantiated. A function call is a run-time instance of a function. An object is a run-time instance of a class.
We never shape the run-time instances directly. We shape artifacts (like a class described through textual Java or C++) so as to influence the run-time shape of the instances. Sometimes we shape an artifact so that its run-time instances can shape other run-time instances (meta-object protocols, dynamically generated code, etc).
Interestingly, some forces apply at the artifact level, while other forces apply at the instance level. If we don't get this straight, we end up with the usual confusion and useless debates.
A few examples are probably needed. As I haven't introduced enough software forces so far, I'll base most of my examples on -ilities, because the same reasoning applies.
Consider reusability: when we say that we want to reuse "a class" what we usually mean is that we want to reuse an artifact (a component, a source file). Reusing a run-time instance is a form of object caching, which might be useful, but is not what is meant by reusability.
Consider efficiency: when we say "this function is more efficient" what we usually mean is that its run-time instances are faster than other comparable implementations.
Consider mass and gravity (the only software property/force I've introduced so far). Both are about the artifacts.
I guess it's all bread-and-butter so far, so why is this distinction important? In Chapter 1, I wrote: a Center is (in software) a locus of highly cohesive information. in order to create highly cohesive units, we must be able to separate things. so this is what software [design] is about: keeping some things near, some things distant.
Now here is something interesting: in many cases, we want to keep the artifacts apart, but have the run-time instances connected. This need to balance asymmetrical forces in two worlds has been one of the dominant forces in the evolution of programming languages and paradigms, although nobody (as far as I know) has properly recognized its role.
Jump on my time machine, and get back to the 70s. Now consider a sequence of lines, in some programming language, encoding some procedural knowledge. We want to reuse that procedural knowledge, so we extract the lines into a "reusable" function. Reusability of procedural knowledge requires physical separation between a caller and a (reusable) callee (that's basically why subroutines/procedures/functions had to be invented). So, we can informally think of reusability as a force in the artifact world, keeping things apart.
Now consider the specular world of run-time instances. Calling a function entails a cost: we have to save the instruction pointer, pass parameters, change the IP, execute the function's body, restore the IP and probably do some cleanup (details depend on language and CPU). Reusability on the artifact side just compromised efficiency on the run-time side. In most cases, we can ignore the side-effect in the run-time world. Sometimes, we can't, so we have two conflicting forces.
Enter the brave language designer: his job is to invent something to bring balance between these two forces, in two different worlds. His first invention is macro expansion through pre-processing (as in C or macro assemblers). C-style macros may solve the problem, but they bring in others. So the language designer gets a better idea: inline expansion of function calls. That works better. However is done, inlining is a way to keep things apart in the artifact world, and strictly together in the run-time world.
Move along time, and consider concepts like function pointers, functions as first-class objects, or polymorphism: again, they're all about keeping artifacts apart, but instances connected. I don't want the artifact defining function/class f to know about function/class g, but I definitely want this instance of f to call g. Indeed (as we'll explore later on) the entire concept of indirection is born out of this basic need: keep artifacts apart and instances connected.
Consider AOP-like techniques: once again, they are about separating artifacts, while having the run-time behavior composed.
Once you understand this simple need, you can look at several programming concepts and clearly see their role in balancing forces between the two worlds. Where languages didn't help, patterns did. More on this next time.
At this point, however, we can draw another conclusion: that a better forcefield diagram would allow us to model forces (in the two worlds) and how we intend to balance those forces (which is how we intend to shape our software), and keep these two aspects apart. After all, we may have different strategies to balance forces out. More on this another time (soon enough, I hope).
Interestingly, software is not alone in this need to balance forces of attraction and rejection. I'll explore this in my next post, together with a few more examples from the software world. I also have to investigate the relationship between the act of balancing forces and the decision space I've mentioned while talking about inertia. This is something I haven't had time to explore so far, so it's gonna take a little more time.
When you are looking for a theory of software forces, however, you'll find another distinction more useful. We have artifacts, and we have run-time instances of artifacts.
Software is encoded knowledge, and as software developers we shape that knowledge. However, we can't act on knowledge itself: we must go through a representation. Source files are a representation, UML diagrams are a representation, and so on.
Source files represent information using a language-specific syntax; for instance, if you use Java or C#, you encode some procedural and structural knowledge about a class inside a text file. The text file is an artifact; it's not a run-time thing. Along the same lines, you can use UML and draw a class diagram, encoding structural knowledge about several classes. That's another artifact. Often, when we say "class C" or "function foo" what we mean is actually its representation inside an artifact (a sequence of LOCs in a source file, where we encode some knowledge in a specific language).
Artifacts can be transformed into other artifacts: without mentioning MDA, you can just compile a bunch of source files and get an executable file. The executable file is just another artifact.
At that point, however, you can actually run the artifact. Let's say that you get a new process: the process is a run-time instance of the executable file artifact. Of course, you may start several run-time instances of the same artifact.
Once the process is started, a wonderful thing happens :-). Your structural and procedural knowledge, formerly represented through LOCs inside artifacts, is now executable as well. Functions can be called. Classes can be instantiated. A function call is a run-time instance of a function. An object is a run-time instance of a class.
We never shape the run-time instances directly. We shape artifacts (like a class described through textual Java or C++) so as to influence the run-time shape of the instances. Sometimes we shape an artifact so that its run-time instances can shape other run-time instances (meta-object protocols, dynamically generated code, etc).
Interestingly, some forces apply at the artifact level, while other forces apply at the instance level. If we don't get this straight, we end up with the usual confusion and useless debates.
A few examples are probably needed. As I haven't introduced enough software forces so far, I'll base most of my examples on -ilities, because the same reasoning applies.
Consider reusability: when we say that we want to reuse "a class" what we usually mean is that we want to reuse an artifact (a component, a source file). Reusing a run-time instance is a form of object caching, which might be useful, but is not what is meant by reusability.
Consider efficiency: when we say "this function is more efficient" what we usually mean is that its run-time instances are faster than other comparable implementations.
Consider mass and gravity (the only software property/force I've introduced so far). Both are about the artifacts.
I guess it's all bread-and-butter so far, so why is this distinction important? In Chapter 1, I wrote: a Center is (in software) a locus of highly cohesive information. in order to create highly cohesive units, we must be able to separate things. so this is what software [design] is about: keeping some things near, some things distant.
Now here is something interesting: in many cases, we want to keep the artifacts apart, but have the run-time instances connected. This need to balance asymmetrical forces in two worlds has been one of the dominant forces in the evolution of programming languages and paradigms, although nobody (as far as I know) has properly recognized its role.
Jump on my time machine, and get back to the 70s. Now consider a sequence of lines, in some programming language, encoding some procedural knowledge. We want to reuse that procedural knowledge, so we extract the lines into a "reusable" function. Reusability of procedural knowledge requires physical separation between a caller and a (reusable) callee (that's basically why subroutines/procedures/functions had to be invented). So, we can informally think of reusability as a force in the artifact world, keeping things apart.
Now consider the specular world of run-time instances. Calling a function entails a cost: we have to save the instruction pointer, pass parameters, change the IP, execute the function's body, restore the IP and probably do some cleanup (details depend on language and CPU). Reusability on the artifact side just compromised efficiency on the run-time side. In most cases, we can ignore the side-effect in the run-time world. Sometimes, we can't, so we have two conflicting forces.
Enter the brave language designer: his job is to invent something to bring balance between these two forces, in two different worlds. His first invention is macro expansion through pre-processing (as in C or macro assemblers). C-style macros may solve the problem, but they bring in others. So the language designer gets a better idea: inline expansion of function calls. That works better. However is done, inlining is a way to keep things apart in the artifact world, and strictly together in the run-time world.
Move along time, and consider concepts like function pointers, functions as first-class objects, or polymorphism: again, they're all about keeping artifacts apart, but instances connected. I don't want the artifact defining function/class f to know about function/class g, but I definitely want this instance of f to call g. Indeed (as we'll explore later on) the entire concept of indirection is born out of this basic need: keep artifacts apart and instances connected.
Consider AOP-like techniques: once again, they are about separating artifacts, while having the run-time behavior composed.
Once you understand this simple need, you can look at several programming concepts and clearly see their role in balancing forces between the two worlds. Where languages didn't help, patterns did. More on this next time.
At this point, however, we can draw another conclusion: that a better forcefield diagram would allow us to model forces (in the two worlds) and how we intend to balance those forces (which is how we intend to shape our software), and keep these two aspects apart. After all, we may have different strategies to balance forces out. More on this another time (soon enough, I hope).
Interestingly, software is not alone in this need to balance forces of attraction and rejection. I'll explore this in my next post, together with a few more examples from the software world. I also have to investigate the relationship between the act of balancing forces and the decision space I've mentioned while talking about inertia. This is something I haven't had time to explore so far, so it's gonna take a little more time.
Thursday, April 29, 2010
Notes on Software Design, Chapter 0: What?
As an author, I've learnt that the introduction is best written last, when text has unfolded, and you really know what your paper is about.
I'm far from that stage with my Notes on Software Design, but I see things clearly enough now that I can write a reasonable introduction, hence the Chapter 0, coming after Chapter 5.
It all starts with the realization that software is just a material, and software design (at any level) is an act meant to give shape (or form) to the material. Software, however, is not a physical material, so we can't borrow on traditional disciplines to seek guidance. Well, sort of.
What do we know about physical materials?
I'm not trying to write an essay on Materials science, so I'll focus on a few central issues here.
Materials have well defined properties. Examples of properties are Electrical conductivity, Coefficient of thermal expansion, Hygroscopy, and so on.
Properties enrich our language and make reasoning more effective (Donald Norman would say that properties "make us smart"). Arguments based on well-defined properties are more robust and don't lead to endless debates. "This material has high hygroscopy, so keep it away from moisture". Period. No debate.
Properties allow to focus: I need to sustain high compression. Therefore, I need high compressive strength. Etc.
Properties allow to select the best materials. Elasticity is required if you want your material to come back to its original shape after stress. Actually, if you know the stress, you can pick the best material, based on a set of criteria, defined by the target product. An important criterion, of course, would be cost.
Properties provide guidance while shaping the material. In fact, we even have manufacturing properties, like castability (see the wikipedia page above on material properties).
Now, properties are well-defined when they are based on replicable, experimental observations. Most often, they are based on both a quantitative measurement process and on a physical theory of matter.
The quantitative side is often based on forces. Compressibility is based on Pressure (and Volume). So we can link everything back to a few well-understood forces.
The theoretical side is usually based on a model of matter. Physical materials are subjected to the well-know laws, like newtonian physics (at the right scale). The model of matter itself (e.g. the notion of metals crystal structure) is helpful when we try to understand why materials behave in a certain way under some specific force.
Moving up (as I said, I'm not trying to write a comprehensive essay on physical materials), we have construction principles and patterns. This is encoded knowledge, prescribing what we should or should not do, or describing how to do something. The bright side is that, in the modern world, we can trace back most principles and patterns to a set of well-defined forces and properties.
Finally, we have tools. Finite-Element Analysis, for instance, can be used to investigate the mechanical, thermal, electromagnetic, [etc], properties of a large structure.
What do we know about software as a material?
Not much, I'm afraid. Our knowledge is basically articulated in:
- Principles
- Patterns and Blueprints
- Methods
- Metrics
- Ilities
There is no shortage of principles. This interesting page, for instance, lists quite a few. Many are ill-defined and redundant, but a working knowledge of most principles is considered necessary for any good programmer / designer.
We have patterns - I know I don't have to say more about this. We also have reference architectures, which are not quite the same as patterns or pattern languages, but close enough.
We have methods. Test Driven Design, for instance, is a method, not a principle.
We have metrics. Metrics seem to be very close to properties. Halstead defined a concept of "volume" for software, based on information theory. The well-known Chidamber and Kemerer suite defined a number of property-like concepts like Depth of Inheritance Tree, Number of Children, and so on. Metrics have the nice property of being easy to measure (most of the times). They have the dubious property of being very remotely connected with design reasoning. In fact, most literature on metrics is about proving they're not meaningless, mostly by showing some correlation with bug density or change density or something more connected with software development.
We have -ilities, like reliability, scalability, and so on. These are often ill-defined, hard to measure properties of the final product. More akin to defining a "safe car" than to defining the properties of an alloy.
Some authors have contributed more ility-like properties. Robert Martin, for instance, talks about Rigidity, Fragility, Viscosity and so on, but they are mostly based on metaphorical reasoning, not an a solid theory. They're also partially overlapping, and not precisely defined.
What do we ignore about software as a material?
We don't have a theory of forces. Lacking forces, it's basically impossible to come up with meaningful properties. Compressibility can't be defined when you don't even know about compression (pressure).
We don't have true properties. Properties should extend from language design to library design to application design. Properties should encompass paradigms. Properties should be based on a sensible theory of what software design is about, not on the mere fact that we can measure something or come up with a nice formula. Properties must be perceived as useful by software designers.
We don't have a way to model forces and properties, basically because we don't know jack about them.
Some perspective
I live in an old building (and I like it :-). When you look at the walls, however, you can almost hear the architect thinking "every problem can be solved by making some wall thicker". Modern buildings are designed with completely different techniques. The next-generation green buildings will make the most out of our knowledge of construction materials.
I also live in the software world. You probably know the adage "All problems in computer science can be solved by another level of indirection" (David Wheeler). Why is that? What is, exactly, a level of indirection? What is indirection, by the way? Why is it useful? What is the underlying theory, what is a reasonable measure? What is the real difference between indirection in data and in control flow? Pick your favorite book on software design, and look up the answer if you can find it :-).
So what?
Well, in the end, this is what I'm aiming for. A theory of software forces. A set of useful properties of software as a material. This is what all this stuff is about. I haven't worked out everything yet. Actually, I've changed my mind on several things I've written so far (hey, it's a blog, not a book :-). But it's slowly coming together.
Oh, by the way. I know quite a few people that won't feel good about the above. They want software to be an art. They want software to be "about humans".
Let me state this clearly: I'm not trying to pursue some sort of "deskilling", whereby any fool could put together great software by applying some sort of magic process. I don't believe in deskilling. Actually, I believe in upskilling. I also understand the idea of software development as a craft, and even as an art. I know the poetry of code, so to speak :-). I rely on intuition and tacit knowledge every single day.
Still, no amount of craftsmanship will prevent a cold, thin glass from breaking when force is applied. I want to know why, and how to shape it anyway, and I want a better way to say that, to teach that, I want to reach a deeper understanding upon which better, more ambitious systems can be built.
Too much? Most likely :-), but hey, what is life without a noble aspiration?
I'm far from that stage with my Notes on Software Design, but I see things clearly enough now that I can write a reasonable introduction, hence the Chapter 0, coming after Chapter 5.
It all starts with the realization that software is just a material, and software design (at any level) is an act meant to give shape (or form) to the material. Software, however, is not a physical material, so we can't borrow on traditional disciplines to seek guidance. Well, sort of.
What do we know about physical materials?
I'm not trying to write an essay on Materials science, so I'll focus on a few central issues here.
Materials have well defined properties. Examples of properties are Electrical conductivity, Coefficient of thermal expansion, Hygroscopy, and so on.
Properties enrich our language and make reasoning more effective (Donald Norman would say that properties "make us smart"). Arguments based on well-defined properties are more robust and don't lead to endless debates. "This material has high hygroscopy, so keep it away from moisture". Period. No debate.
Properties allow to focus: I need to sustain high compression. Therefore, I need high compressive strength. Etc.
Properties allow to select the best materials. Elasticity is required if you want your material to come back to its original shape after stress. Actually, if you know the stress, you can pick the best material, based on a set of criteria, defined by the target product. An important criterion, of course, would be cost.
Properties provide guidance while shaping the material. In fact, we even have manufacturing properties, like castability (see the wikipedia page above on material properties).
Now, properties are well-defined when they are based on replicable, experimental observations. Most often, they are based on both a quantitative measurement process and on a physical theory of matter.
The quantitative side is often based on forces. Compressibility is based on Pressure (and Volume). So we can link everything back to a few well-understood forces.
The theoretical side is usually based on a model of matter. Physical materials are subjected to the well-know laws, like newtonian physics (at the right scale). The model of matter itself (e.g. the notion of metals crystal structure) is helpful when we try to understand why materials behave in a certain way under some specific force.
Moving up (as I said, I'm not trying to write a comprehensive essay on physical materials), we have construction principles and patterns. This is encoded knowledge, prescribing what we should or should not do, or describing how to do something. The bright side is that, in the modern world, we can trace back most principles and patterns to a set of well-defined forces and properties.
Finally, we have tools. Finite-Element Analysis, for instance, can be used to investigate the mechanical, thermal, electromagnetic, [etc], properties of a large structure.
What do we know about software as a material?
Not much, I'm afraid. Our knowledge is basically articulated in:
- Principles
- Patterns and Blueprints
- Methods
- Metrics
- Ilities
There is no shortage of principles. This interesting page, for instance, lists quite a few. Many are ill-defined and redundant, but a working knowledge of most principles is considered necessary for any good programmer / designer.
We have patterns - I know I don't have to say more about this. We also have reference architectures, which are not quite the same as patterns or pattern languages, but close enough.
We have methods. Test Driven Design, for instance, is a method, not a principle.
We have metrics. Metrics seem to be very close to properties. Halstead defined a concept of "volume" for software, based on information theory. The well-known Chidamber and Kemerer suite defined a number of property-like concepts like Depth of Inheritance Tree, Number of Children, and so on. Metrics have the nice property of being easy to measure (most of the times). They have the dubious property of being very remotely connected with design reasoning. In fact, most literature on metrics is about proving they're not meaningless, mostly by showing some correlation with bug density or change density or something more connected with software development.
We have -ilities, like reliability, scalability, and so on. These are often ill-defined, hard to measure properties of the final product. More akin to defining a "safe car" than to defining the properties of an alloy.
Some authors have contributed more ility-like properties. Robert Martin, for instance, talks about Rigidity, Fragility, Viscosity and so on, but they are mostly based on metaphorical reasoning, not an a solid theory. They're also partially overlapping, and not precisely defined.
What do we ignore about software as a material?
We don't have a theory of forces. Lacking forces, it's basically impossible to come up with meaningful properties. Compressibility can't be defined when you don't even know about compression (pressure).
We don't have true properties. Properties should extend from language design to library design to application design. Properties should encompass paradigms. Properties should be based on a sensible theory of what software design is about, not on the mere fact that we can measure something or come up with a nice formula. Properties must be perceived as useful by software designers.
We don't have a way to model forces and properties, basically because we don't know jack about them.
Some perspective
I live in an old building (and I like it :-). When you look at the walls, however, you can almost hear the architect thinking "every problem can be solved by making some wall thicker". Modern buildings are designed with completely different techniques. The next-generation green buildings will make the most out of our knowledge of construction materials.
I also live in the software world. You probably know the adage "All problems in computer science can be solved by another level of indirection" (David Wheeler). Why is that? What is, exactly, a level of indirection? What is indirection, by the way? Why is it useful? What is the underlying theory, what is a reasonable measure? What is the real difference between indirection in data and in control flow? Pick your favorite book on software design, and look up the answer if you can find it :-).
So what?
Well, in the end, this is what I'm aiming for. A theory of software forces. A set of useful properties of software as a material. This is what all this stuff is about. I haven't worked out everything yet. Actually, I've changed my mind on several things I've written so far (hey, it's a blog, not a book :-). But it's slowly coming together.
Oh, by the way. I know quite a few people that won't feel good about the above. They want software to be an art. They want software to be "about humans".
Let me state this clearly: I'm not trying to pursue some sort of "deskilling", whereby any fool could put together great software by applying some sort of magic process. I don't believe in deskilling. Actually, I believe in upskilling. I also understand the idea of software development as a craft, and even as an art. I know the poetry of code, so to speak :-). I rely on intuition and tacit knowledge every single day.
Still, no amount of craftsmanship will prevent a cold, thin glass from breaking when force is applied. I want to know why, and how to shape it anyway, and I want a better way to say that, to teach that, I want to reach a deeper understanding upon which better, more ambitious systems can be built.
Too much? Most likely :-), but hey, what is life without a noble aspiration?
Friday, March 12, 2010
Where is your Knowledge?
Software development is a process of knowledge acquisition and knowledge encoding (see Phillip Armour, copiously quoted in this blog). Where, and how, do we store that knowledge? In several places, in several ways:
In source code: that's executable knowledge
In models: that's formal knowledge
In other kind of documents: that's written knowledge
In our brain, consciously: that's explicit knowledge
In our brain, unconsciously: that's tacit knowledge
Knowledge stored in source code has the extremely useful property of being executable, but we can't store the entire development knowledge in executable statements. Design Rationale, for instance, is not present in code (and not even in most UML diagrams, for that matter), and is basically stored at the conscious/unconscious level. My forcefield diagram is much better at formally capturing rationale.
Explicit knowledge is often passed by as oral tradition, while tacit knowledge is often passed by as "a way of doing things", just by working together. Pair programming, reviews, joint design sessions (and so on) help distribute both explicit and tacit knowledge.
Knowledge has value, but that value is not constant over time. In 1962, Fritz Machlup came up with the concept of Half-life of knowledge: the amount of time that has to elapse before half of the knowledge in a particular area is superseded or shown to be untrue.
Moreover, the initial value of a particular piece of knowledge can be very high, like a new algorithm that took you years to get right, or very small, like a trivial validation rule.
Recently, I began to think about the half-life of our knowledge repositories as well. With my usual boldness, I'll go ahead and define the Half-Life of a Knowledge Repository: the amount of time that has to elapse before half of the knowledge in a repository is unrecoverable or just too costly to recover. I could even define "too costly" as "higher than the discounted value of that knowledge when lookup is attempted".
The concept of recoverable knowledge is slightly deeper than it may seem. Sure, it does cover the obvious problems of losing knowledge for lack of backup procedures, or because it's stored in a proprietary format no longer supported, and so on. But it covers also several interesting cases:
- the knowledge is in the brain of an employee, who leaves the company
- the knowledge is in source code, but it's in an obsolete language
- the knowledge is in source code, but it's extremely hard to read
- etc.
I'll leave it up to you to define the half-life of source code, models, documents, brain (conscious and unconscious). Of course, more details are needed: niche languages, for instance, tend to have a shorter half-life.
Now, here is the real boon :-). We can combine the concept of Knowledge Half-Life, Knowledge Value, and Knowledge Repository Half-Life to map the risk of storing a particular piece of knowledge in a particular repository (only). Here is my first-cut map:
It's interesting to review one of the values in the Agile Manifesto (Working software over comprehensive documentation) under this perspective.
Let's say we have a piece of knowledge, and that knowledge can be indeed stored in code (as I said, you can't store everything in code).
If the half-life of knowledge is short, storing it in code only is probably the best economical choice. If the half-life of knowledge is long, we have to worry a little more. If we add relevant unit tests to that piece of code, we increase the repository half-life, as they make it easier to recover knowledge from code. If we use a mainstream language, we can also increase the repository half-life.
This may still not be enough. If you had to recover the entire knowledge stored in a non-trivial piece of code (say, an mp4 codec) having only the source code, and no (comprehensive) documentation on what that piece of code is doing, why, and how, it would take you far too much. The half-life of code is shorter than the half-life of code + documents.
Actually, depending on context, given the choice to have just the code and nothing else, or just comprehensive documentation and nothing else, we better be careful about what we choose (when knowledge half-life is long, of course).
Of course, the opposite is also true: if you store knowledge with short half-life outside code, you seriously risk wasting your time.
I've often been critic about teaching and applying principles and techniques without the necessary context. I hope that somehow, the table above and the underlying concepts can move our understanding of when to use what a little further.
In source code: that's executable knowledge
In models: that's formal knowledge
In other kind of documents: that's written knowledge
In our brain, consciously: that's explicit knowledge
In our brain, unconsciously: that's tacit knowledge
Knowledge stored in source code has the extremely useful property of being executable, but we can't store the entire development knowledge in executable statements. Design Rationale, for instance, is not present in code (and not even in most UML diagrams, for that matter), and is basically stored at the conscious/unconscious level. My forcefield diagram is much better at formally capturing rationale.
Explicit knowledge is often passed by as oral tradition, while tacit knowledge is often passed by as "a way of doing things", just by working together. Pair programming, reviews, joint design sessions (and so on) help distribute both explicit and tacit knowledge.
Knowledge has value, but that value is not constant over time. In 1962, Fritz Machlup came up with the concept of Half-life of knowledge: the amount of time that has to elapse before half of the knowledge in a particular area is superseded or shown to be untrue.
Moreover, the initial value of a particular piece of knowledge can be very high, like a new algorithm that took you years to get right, or very small, like a trivial validation rule.
Recently, I began to think about the half-life of our knowledge repositories as well. With my usual boldness, I'll go ahead and define the Half-Life of a Knowledge Repository: the amount of time that has to elapse before half of the knowledge in a repository is unrecoverable or just too costly to recover. I could even define "too costly" as "higher than the discounted value of that knowledge when lookup is attempted".
The concept of recoverable knowledge is slightly deeper than it may seem. Sure, it does cover the obvious problems of losing knowledge for lack of backup procedures, or because it's stored in a proprietary format no longer supported, and so on. But it covers also several interesting cases:
- the knowledge is in the brain of an employee, who leaves the company
- the knowledge is in source code, but it's in an obsolete language
- the knowledge is in source code, but it's extremely hard to read
- etc.
I'll leave it up to you to define the half-life of source code, models, documents, brain (conscious and unconscious). Of course, more details are needed: niche languages, for instance, tend to have a shorter half-life.
Now, here is the real boon :-). We can combine the concept of Knowledge Half-Life, Knowledge Value, and Knowledge Repository Half-Life to map the risk of storing a particular piece of knowledge in a particular repository (only). Here is my first-cut map:
| Knowledge Half-Life | Knowledge (initial) Value | Repository Half-Life | Result |
| Long | Long | Long | OK |
| Long | Long | Short | Risk |
| Long | Short | Long | Little Waste |
| Long | Short | Short | Little Risk |
| Short | Long | Long | Little Waste |
| Short | Long | Short | Little Risk |
| Short | Short | Long | Waste |
| Short | Short | Short | OK |
It's interesting to review one of the values in the Agile Manifesto (Working software over comprehensive documentation) under this perspective.
Let's say we have a piece of knowledge, and that knowledge can be indeed stored in code (as I said, you can't store everything in code).
If the half-life of knowledge is short, storing it in code only is probably the best economical choice. If the half-life of knowledge is long, we have to worry a little more. If we add relevant unit tests to that piece of code, we increase the repository half-life, as they make it easier to recover knowledge from code. If we use a mainstream language, we can also increase the repository half-life.
This may still not be enough. If you had to recover the entire knowledge stored in a non-trivial piece of code (say, an mp4 codec) having only the source code, and no (comprehensive) documentation on what that piece of code is doing, why, and how, it would take you far too much. The half-life of code is shorter than the half-life of code + documents.
Actually, depending on context, given the choice to have just the code and nothing else, or just comprehensive documentation and nothing else, we better be careful about what we choose (when knowledge half-life is long, of course).
Of course, the opposite is also true: if you store knowledge with short half-life outside code, you seriously risk wasting your time.
I've often been critic about teaching and applying principles and techniques without the necessary context. I hope that somehow, the table above and the underlying concepts can move our understanding of when to use what a little further.
Monday, March 08, 2010
Why you should learn AOP
A few days ago, I've spent some time reading a critic of AOP (The Paradoxical Success of Aspect-Oriented Programming by Friedrich Steimann). As often, I felt compelled to read some of the bibliographical references too, which took me a little more (week-end) time.
Overall, in the last few years I've devoted quite some time to learn, think, and even write a little about AOP. I'm well aware of the problems Steimann describes, and I share some skepticism about the viability of the AOP paradigm as we know it.
Too much literature, for instance, is focused on a small set of pervasive concerns like logging. I believe that as we move toward higher-level concerns, we must make a clear distinction between pervasive concerns and cross-cutting concerns. A concern can be cross-cutting without being pervasive, and in this sense, for instance, I don't really agree that AOP is not for singletons (see my old post Some notes on AOP).
Also, I wouldn't dismiss the distinction between spectators and assistants so easily, especially because many pervasive concerns can be modeled as spectators. Overall, the paradigm seems indeed a little immature when you look at the long-term maintenance effects of aspects as they're known today.
Still, I think the time I've spent pondering on AOP was truly well spent. Actually, I would suggest that you spend some time learning about AOP too, even if you're not planning to use AOP in the foreseeable future.
I don't really mean learning a specific language - unless you want/need to try out a few things. I mean learning the concepts, the AOP perspective, the AOP terminology, the effects and side-effects of an Aspect Oriented solution.
I'm suggesting that you learn all that despite the obvious (or perhaps not so obvious) deficiencies in the current approaches and languages, the excessive hype and the underdeveloped concepts. I'm suggesting that you learn all that because it will make you a better designer.
Why? Because it will expand your mind. It will add a new, alternative perspective through which you can look at your problems. New questions to ask. New concepts. New names. Sometimes, all we need is a name. A beacon in the brainstorm, and a steady hand.
As I've said many times now, as designers we're shaping software. We can choose many shapes, and ideally, we will find a shape that is in frictionless contact with the forcefield. Any given paradigm will suggest a set of privileged shapes, at macro and micro-level. Including the aspect-oriented paradigm in your thinking will expand the set of shapes you can apply and conceive.
Time for a short war story :-). In the past months I've been thinking a lot about some issues in a large CAD system. While shaping a solution, I'm constantly getting back to what I could call aspect-thinking. There are many cross-cutting concerns to be resolved. Not programming-level concerns (like the usual, boring logging stuff). Full-fledged application-domain concerns, that tend to cross-cut the principal decomposition.
Now, you see, even thinking "principal decomposition" and "cross-cutting" is making your first step into aspect-thinking. Then you can think about ways to bring those concerns inside the principal decomposition (if appropriate and/or possible and/or convenient) or think about the best way to keep them outside without code-level tangling. Tangling. Another interesting name, another interesting concept.
Sure, if you ain't using true AOP (for instance, we're using plain old C++), you'll have to give up some oblivousness (another name, another concept!), but it can be done, and it works fine (for a small scale example, see part 1 and part 2 of my "Can AOP inform OOP?")
So far, the candidate shape is causing some discomfort. That's reasonable. It's not a "traditional" solution. Which is fine, because so far, tradition didn't work so well :-). Somehow, I hope the team will get out of this experience with a new mindset. Nobody used to talk about "principal decomposition" or "cross-cutting concern" in the company. And you can't control what you can't name.
I hope they will gradually internalize the new concepts, as well as the tactics we can use inside traditional languages. That would be a major accomplishment. Much more important than the design we're creating, or the tons of code we'll be writing. Well, we'll see...
Overall, in the last few years I've devoted quite some time to learn, think, and even write a little about AOP. I'm well aware of the problems Steimann describes, and I share some skepticism about the viability of the AOP paradigm as we know it.
Too much literature, for instance, is focused on a small set of pervasive concerns like logging. I believe that as we move toward higher-level concerns, we must make a clear distinction between pervasive concerns and cross-cutting concerns. A concern can be cross-cutting without being pervasive, and in this sense, for instance, I don't really agree that AOP is not for singletons (see my old post Some notes on AOP).
Also, I wouldn't dismiss the distinction between spectators and assistants so easily, especially because many pervasive concerns can be modeled as spectators. Overall, the paradigm seems indeed a little immature when you look at the long-term maintenance effects of aspects as they're known today.
Still, I think the time I've spent pondering on AOP was truly well spent. Actually, I would suggest that you spend some time learning about AOP too, even if you're not planning to use AOP in the foreseeable future.
I don't really mean learning a specific language - unless you want/need to try out a few things. I mean learning the concepts, the AOP perspective, the AOP terminology, the effects and side-effects of an Aspect Oriented solution.
I'm suggesting that you learn all that despite the obvious (or perhaps not so obvious) deficiencies in the current approaches and languages, the excessive hype and the underdeveloped concepts. I'm suggesting that you learn all that because it will make you a better designer.
Why? Because it will expand your mind. It will add a new, alternative perspective through which you can look at your problems. New questions to ask. New concepts. New names. Sometimes, all we need is a name. A beacon in the brainstorm, and a steady hand.
As I've said many times now, as designers we're shaping software. We can choose many shapes, and ideally, we will find a shape that is in frictionless contact with the forcefield. Any given paradigm will suggest a set of privileged shapes, at macro and micro-level. Including the aspect-oriented paradigm in your thinking will expand the set of shapes you can apply and conceive.
Time for a short war story :-). In the past months I've been thinking a lot about some issues in a large CAD system. While shaping a solution, I'm constantly getting back to what I could call aspect-thinking. There are many cross-cutting concerns to be resolved. Not programming-level concerns (like the usual, boring logging stuff). Full-fledged application-domain concerns, that tend to cross-cut the principal decomposition.
Now, you see, even thinking "principal decomposition" and "cross-cutting" is making your first step into aspect-thinking. Then you can think about ways to bring those concerns inside the principal decomposition (if appropriate and/or possible and/or convenient) or think about the best way to keep them outside without code-level tangling. Tangling. Another interesting name, another interesting concept.
Sure, if you ain't using true AOP (for instance, we're using plain old C++), you'll have to give up some oblivousness (another name, another concept!), but it can be done, and it works fine (for a small scale example, see part 1 and part 2 of my "Can AOP inform OOP?")
So far, the candidate shape is causing some discomfort. That's reasonable. It's not a "traditional" solution. Which is fine, because so far, tradition didn't work so well :-). Somehow, I hope the team will get out of this experience with a new mindset. Nobody used to talk about "principal decomposition" or "cross-cutting concern" in the company. And you can't control what you can't name.
I hope they will gradually internalize the new concepts, as well as the tactics we can use inside traditional languages. That would be a major accomplishment. Much more important than the design we're creating, or the tons of code we'll be writing. Well, we'll see...
Friday, February 12, 2010
Form vs. Function: a Space Odyssey
I was teaching Object Oriented Design past week, and I mentioned the interplay between form and function (form follows function; function follows form). I'm rather cautious not to spend too much time on philosophy, although a little philosophy shouldn't hurt, and people who know me tend to expect a short philosophical digression every now and then.
Function follows form: that is to say, the shape of an object will suggest possible uses. Form follows function: the intended usage of an object will constrain, and therefore guide, its physical shape. This is true for software objects as well. It's just a different material, something we can't immediately see and shape with our hands.
Realizing that function follows form is a pivotal moment in the development of intelligence. You probably remember the opening of 2001: A Space Odyssey. The apes are touching the monolith and, shortly after, one of them is playing with a bone and bang! - it's not just a bone anymore: it's a tool. Function follows form. This chapter is known as "The Dawn of Man", and rightly so.
Watch a little more, and you'll see a doughnut-shaped space station. That's a very good example of form following function (exercise for the reader :-)
By the way, if you have never seen that apes stuff till now :-), here it is, at least until it gets removed from YouTube...
Function follows form: that is to say, the shape of an object will suggest possible uses. Form follows function: the intended usage of an object will constrain, and therefore guide, its physical shape. This is true for software objects as well. It's just a different material, something we can't immediately see and shape with our hands.
Realizing that function follows form is a pivotal moment in the development of intelligence. You probably remember the opening of 2001: A Space Odyssey. The apes are touching the monolith and, shortly after, one of them is playing with a bone and bang! - it's not just a bone anymore: it's a tool. Function follows form. This chapter is known as "The Dawn of Man", and rightly so.
Watch a little more, and you'll see a doughnut-shaped space station. That's a very good example of form following function (exercise for the reader :-)
By the way, if you have never seen that apes stuff till now :-), here it is, at least until it gets removed from YouTube...
Sunday, January 10, 2010
Delaying Decisions
Since microblogging is not my thing, I decided to start 2010 by writing my longer post ever :-). It will start with a light review of a well-known principle and end up with a new design concept. Fasten your seatbelt :-).
The Last Responsible Moment
When we develop a software product, we make decisions. We decide about individual features, we make design decisions, we make coding decisions, we even decide which bugs we really want to fix before going public. Some decisions are taken on the fly; some, at least in the old school, are somewhat planned.
A key principle of Lean Development is to delay decisions, so that:
a) decisions can be based on (yet-to-discover) facts, not on speculation
b) you exercise the wait option (more on this below) and avoid early commitment
The principle is often spelled as "Delay decisions until the last responsible moment", but a quick look at Mary Poppendieck's website (Mary co-created the Lean Development approach) shows a more interesting nuance: "Schedule Irreversible Decisions at the Last Responsible Moment".
Defining "Irreversible" and "Last Responsible" is not trivial. In a sense, there is nothing in software that is truly irreversible, because you can always start over. I haven't found a good definition for "irreversible decision" in literature, but I would define it as follows: if you make an irreversible decision at time T, undoing the decision at a later time will entail a complete (or almost complete) waste of everything that has been created after time T.
There are some documented definitions for "last responsible moment". A popular one is "The point when failing to decide eliminates an important option", which I found rather unsatisfactory. I've also seen some attempts to quantify that better, as in this funny story, except that in the real world you never have a problem which is that simple (very few ramifications in the decision graph) and that detailed (you know the schedule beforehand). I would probably define the Last Responsible Moment as follows: time T is the last responsible moment to make a decision D if, by postponing D, the probability of completing on schedule/budget (even when you factor-in the hypothetical learning effect of postponing) decreases below an acceptable threshold. That, of course, allows us to scrap everything and restart, if schedule and budget allows for it, and in this sense it's kinda coupled with the definition of irreversible.
Now, irreversibility is bad. We don't want to make irreversible decisions. We certainly don't want to make them too soon. Is there anything we can do? I've got a few important things to say about modularity vs. irreversibility and passive vs. proactive option thinking, but right now, it's useful to recap the major decision areas within a software project, so that we can clearly understand what we can actually delay, and what is usually suggested that we delay.
Major Decision Areas
I'll skip on a few very-high-level, strategic decisions here (scope, strategy, business model, etc). It's not that they can't be postponed, but I need to give some focus to this post :-). So I'll get down to the more ordinarily taken decisions.
People
Choosing the right people for the project is a well-known ingredient for success.
Approach/Process
Are we going XP, Waterfall, something in between? :-).
Feature Set
Are we going to include this feature or not?
Design
What is the internal shape (form) of our product?
Coding
Much like design, at a finer granularity level.
Now, "design" is an overly general concept. Too general to be useful. Therefore, I'll split it into a few major decisions.
Architectural Style
Is this going to be an embedded application, a rich client, a web application? This is a rather irreversible decision.
Platform
Goes somewhat in pair with Architectural Style. Are we going with an embedded application burnt into an FPGA? Do you want to target a PIC? Perhaps an embedded PC? Is the client a Windows machine, or you want to support Mac/Linux? A .NET server side, or maybe Java? It's all rather irreversible, although not completely irreversible.
3rd-Party Libraries/Components/Etc
Are we going to use some existing component (of various scale)? Unless you plan on wrapping everything (which may not even be possible), this often end up being an irreversible decision. For instance, once you commit yourself to using Hibernate for persistence, it's not trivial to move away.
Programming Language
This is the quintessential irreversible decision, unless you want to play with language converters. Note that this is not a coding decisions: coding decisions are made after the language has been chosen.
Structure / Shape / Form
This is what we usually call "design": the shape we want to impose to our material (or, if you live in the "emergent design" side, the shape that our material will take as the final result of several incremental decisions).
So, what are we going to delay? We can't delay all decisions, or we'll be stuck. Sure, we can delay something in each and every area, but truth is, every popular method has been focusing on just a few of them. Of course, different methods tried to delay different choices.
A Little Historical Perspective
Experience brings perspective; at least, true experience does :-). Perspective allows to look at something and see more than it's usually seen. For instance, perspective allows to look at the old, outdated, obsolete waterfall approach and see that it (too) was meant to delay decisions, just different decisions.
Waterfall was meant to delay people decisions, design decisions (which include platform, library, component decisions) and coding decisions. People decision was delayed by specialization: you only have to pick the analyst first, everyone else can be chosen later, when you know what you gotta do (it even makes sense -)). Design decision was delayed because platform, including languages, OS, etc, were way more balkanized than today. Also, architectural styles and patterns were much less understood, and it made sense to look at a larger picture before committing to an overall architecture.
Although this may seem rather ridiculous from the perspective of a 2010 programmer working on Java corporate web applications, most of this stuff is still relevant for (e.g.) mass-produced embedded systems, where choosing the right platform may radically change the total development and production cost, yet choosing the wrong platform may over-constrain the feature set.
Indeed, open systems (another legacy term from late '80s - early '90s) were born exactly to lighten up that choice. Choose the *nix world, and forget about it. Of course, the decision was still irreversible, but granted you some latitude in choosing the exact hw/sw. The entire multi-platform industry (from multi-OS libraries to Java) is basically built on the same foundations. Well, that's the bright side, of course :-).
Looking beyond platform independence, the entire concept of "standard" allows to delay some decision. TCP/IP, for instance, allows me to choose modularly (a concept I'll elaborate later). I can choose TCP/IP as the transport mechanism, and then delay the choice of (e.g.) the client side, and focus on the server side. Of course, a choice is still made (the client must have TCP/IP support), so let's say that widely adopted standards allow for some modularity in the decision process, and therefore to delay some decision, mostly design decisions, but perhaps some other as well (like people).
It's already going to be a long post, so I won't look at each and every method/principle/tool ever conceived, but if you do your homework, you'll find that a lot of what has been proposed in the last 40 years or so (from code generators to MDA, from spiral development to XP, from stepwise refinement to OOP) includes some magic ingredient that allows us to postpone some kind of decision.
It's 2010, guys
So, if you ain't agile, you are clumsy :-)) and c'mon, you don't wanna be clumsy :-). So, seriously, which kind of decisions are usually delayed in (e.g.) XP?
People? I must say I haven't seen much on this. Most literature on XP seems based on the concept that team members are mostly programmers with a wide set of skills, so there should be no particular reason to delay decision about who's gonna work on what. I may have missed some particularly relevant work, however.
Feature Set? Sure. Every incremental approach allows us to delay decisions about features. This can be very advantageous if we can play the learning game, which includes rapid/frequent delivery, or we won't learn enough to actually steer the feature set.
Of course, delaying some decisions on feature set can make some design options viable now, and totally bogus later. Here is where you really have to understand the concept of irreversible and last responsible moment. Of course, if you work on a settled platform, things get simpler, which is one more reason why people get religiously attached to a platform.
Design? Sure, but let's take a deeper look.
Architectural Style: not much. Quoting Booch, "agile projects often start out assuming a given platform and environmental context together with a set of proven design patterns for that domain, all of which represent architectural decisions in a very real sense". See my post Architecture as Tradition in the Unselfconscious Process for more.
Seriously, nobody ever expected to start with a monolithic client and end up with a three-tier web application built around a MVC pattern just by coding and refactoring. The architectural style is pretty much a given in many contemporary projects.
Platform: sorry guys, but if you want to start coding now, you gotta choose your platform now. Another irreversible decision made right at the beginning.
3rd-Party Libraries/Components/Etc: some delay is possible for modularized decisions. If you wanna use hibernate, you gotta choose pretty soon. If you wanna use Seam, you gotta choose pretty soon. Pervasive libraries are so entangled with architectural styles that it's relatively hard to delay some decisions here. Modularized components (e.g. the choice of a PDF rendering library) are simple to delay, and can be proactively delayed (see later).
Programming Language: no way guys, you have to choose right here, right now.
Structure / Shape / Form: of course!!! Here we are. This is it :-). You can delay a lot of detailed design choices. Of course, we always postpone some design decision, even when we design before coding. But let's say that this is where I see a lot of suggestions to delay decisions in the agile literature, often using the dreaded Big Upfront Design as a straw man argument. Of course, the emergent design (or accidental architecture) may or may not be good. If I had to compare the design and code coming out of the XP Episode with my own, I would say that a little upfront design can do wonders, but hey, you know me :-).
Practicing
OK guys, what follows may sound a little odd, but in the end it will prove useful. Have faith :-).
You can get better at everything by doing anything :-), so why not getting better at delaying decisions by playing Windows Solitaire? All you have to do is set the options in the hardest possible way:
now, play a little, until you have to make some decision, like here:
I could move the 9 of spades or the 9 of clubs over the 10 of hearts. It's an irreversible decision (well, not if you use the undo, but that's lame :-). There are some ramifications for both choices.
If I move the 9 of clubs, I can later move the king of clubs and uncover a new card. After that, it's all unknown, and no further speculation is possible. Here, learning requires an irreversible decision; this is very common in real-world projects, but seldom discussed in literature.
If I move the 9 of spades, I uncover the 6 of clubs, which I can move over the 7 of aces. Then, it's kinda unknown, meaning: if you're a serious player (I'm not) you'll remember the previous cards, which would allow you to speculate a little better. Otherwise, it's just as above, you have to make an irreversible decision to learn the outcome.
But wait: what about the last responsible moment? Maybe we can delay this decision! Now, if you delay the decision by clicking on the deck and moving further, you're not delaying the decision: you're wasting a chance. In order to delay this decision, there must be something else you can do.
Well, indeed, there is something you can do. You can move the 8 of aces above the 9 of clubs. This will uncover a new card (learning) without wasting any present opportunity (it could still waste a future opportunity; life it tough). Maybe you'll get a 10 of aces under that 8, at which point there won't be any choice to be made about the 9. Or you might get a black 7, at which point you'll have a different way to move the king of clubs, so moving the 9 of spades would be a more attractive option. So, delay the 9 and move the 8 :-). Add some luck, and it works:
and you get some money too (total at decision time Vs. total at the end)
Novice solitaire players are also known to make irreversible decision without necessity. For instance, in similar cases:
I've seen people eagerly moving the 6 of aces (actually, whatever they got) over the 7 of spades, because "that will free up a slot". Which is true, but irrelevant. This is a decision you can easily delay. Actually, it's a decision you must delay, because:
- if you happen to uncover a king, you can always move the 6. It's not the last responsible moment yet: if you do nothing now, nothing bad will happen.
- you may uncover a 6 of hearts before you uncover a king. And moving that 6 might be more advantageous than moving the 6 of aces. So, don't do it :-). If you want to look good, quote Option Theory, call this a Deferral Option and write a paper about it :-).
Proactive Option Thinking
I've recently read an interesting paper in IEEE TSE ("An Integrative Economic Optimization Approach to Systems Development Risk Management", by Michel Benaroch and James Goldstein). Although the real meat starts in chapter 4, chapters 1-3 are probably more interesting for the casual reader (including myself).
There, authors recap some literature about Real Options in Software Engineering, including the popular argument that delaying decisions is akin to a deferral option. They also make important distinctions, like the one between passive learning through deferral of decisions, and proactive learning, but also between responsiveness to change (a central theme in agility literature) and manipulation of change (relatively less explored), and so on. There is a a lot of food for thought in those 3 chapters, so if you can get a copy, I suggest that you spend a little time pondering over it.
Now, I'm a strong supporter of Proactive Option Thinking. Waiting for opportunities (and then react quickly) is not enough. I believe that options should be "implanted" in our project, and that can be done by applying the right design techniques. How? Keep reading : ).
The Invariant Decision
If you look back at those pictures of Solitaire, you'll see that I wasn't really delaying irreversible decisions. All decisions in solitaire are irreversible (real men don't use CTRL-Z). Many decisions in software development are irreversible as well, especially when you are in a tight budget/schedule, so starting over is not an option. Therefore, irreversibility can't really be the key here. Indeed, I was trying to delay Invariant Decisions. Decisions that I can take now, or I can take later, with little or no impact on the outcomes. The concept itself may seem like a minor change from "irreversible", but it allows me to do some magic:
- I can get rid of the "last responsible moment" part, which is poorly defined anyway. I can just say: delay invariant decisions. Period. You can delay them as much as you want, provided they are still invariant. No ambiguity here. That's much better.
- I can proactively make some decisions invariant. This is so important I'll have to say it again, this time in bold: I can proactively make some decisions invariant.
Invariance, Design, Modularity
If you go back to the Historical Perspective paragraph, you can now read it under a different... perspective :-). Several tools, techniques, methods can be adopted not just to delay some decision, but to create the option to delay the decision. How? Through careful design, of course!
Consider the strong modularity you get from service-oriented architecture, and the platform independence that comes through (well-designed) web services. This is a powerful weapon to delay a lot of decisions on one side or another (client or server).
Consider standard protocols: they are a way to make some decision invariant, and to modularize the impact of some choices.
Consider encapsulation, abstraction and interfaces: they allow you to delay quite a few low-level decisions, and to modularize the impact of change as well. If your choice turn out to be wrong, but it's highly localized (modularized) you may afford undoing your decision, therefore turning irreversible into reversible. A barebone example can be found in my old post (2005!) Builder [pattern] as an option.
Consider a very old OOA/OOD principle, now somehow resurrected under the "ubiquitous language" umbrella. It states that you should try to reflect the real-world entities that you're dealing with in your design, and then in your code. That includes avoiding primitive types like integer, and create meaningful classes instead. Of course, you have to understand what you're doing (that is, you gotta be a good designer) to avoid useless overengineering. See part 4 of my digression on the XP Episode for a discussion about adding a seemingly useless Ball class (that is: implanting a low cost - high premium option).
Names alter the forcefield. A named concept stands apart. My next post on the forcefield theme, by the way, will explore this issue in depth :-).
And so on. I could go on forever, but the point is: you can make many (but not all, of course!) decisions invariant, if you apply the right design techniques. Most of those techniques will also modularize the cost of rework if you make the wrong decision. And sure, you can try to do this on the fly as you code. Or you may want to to some upfront design. You know what I'm thinking.
OK guys, it took quite a while, but now we have a new concept to play with, so more on this will follow, randomly as usual. Stay tuned.
The Last Responsible Moment
When we develop a software product, we make decisions. We decide about individual features, we make design decisions, we make coding decisions, we even decide which bugs we really want to fix before going public. Some decisions are taken on the fly; some, at least in the old school, are somewhat planned.
A key principle of Lean Development is to delay decisions, so that:
a) decisions can be based on (yet-to-discover) facts, not on speculation
b) you exercise the wait option (more on this below) and avoid early commitment
The principle is often spelled as "Delay decisions until the last responsible moment", but a quick look at Mary Poppendieck's website (Mary co-created the Lean Development approach) shows a more interesting nuance: "Schedule Irreversible Decisions at the Last Responsible Moment".
Defining "Irreversible" and "Last Responsible" is not trivial. In a sense, there is nothing in software that is truly irreversible, because you can always start over. I haven't found a good definition for "irreversible decision" in literature, but I would define it as follows: if you make an irreversible decision at time T, undoing the decision at a later time will entail a complete (or almost complete) waste of everything that has been created after time T.
There are some documented definitions for "last responsible moment". A popular one is "The point when failing to decide eliminates an important option", which I found rather unsatisfactory. I've also seen some attempts to quantify that better, as in this funny story, except that in the real world you never have a problem which is that simple (very few ramifications in the decision graph) and that detailed (you know the schedule beforehand). I would probably define the Last Responsible Moment as follows: time T is the last responsible moment to make a decision D if, by postponing D, the probability of completing on schedule/budget (even when you factor-in the hypothetical learning effect of postponing) decreases below an acceptable threshold. That, of course, allows us to scrap everything and restart, if schedule and budget allows for it, and in this sense it's kinda coupled with the definition of irreversible.
Now, irreversibility is bad. We don't want to make irreversible decisions. We certainly don't want to make them too soon. Is there anything we can do? I've got a few important things to say about modularity vs. irreversibility and passive vs. proactive option thinking, but right now, it's useful to recap the major decision areas within a software project, so that we can clearly understand what we can actually delay, and what is usually suggested that we delay.
Major Decision Areas
I'll skip on a few very-high-level, strategic decisions here (scope, strategy, business model, etc). It's not that they can't be postponed, but I need to give some focus to this post :-). So I'll get down to the more ordinarily taken decisions.
People
Choosing the right people for the project is a well-known ingredient for success.
Approach/Process
Are we going XP, Waterfall, something in between? :-).
Feature Set
Are we going to include this feature or not?
Design
What is the internal shape (form) of our product?
Coding
Much like design, at a finer granularity level.
Now, "design" is an overly general concept. Too general to be useful. Therefore, I'll split it into a few major decisions.
Architectural Style
Is this going to be an embedded application, a rich client, a web application? This is a rather irreversible decision.
Platform
Goes somewhat in pair with Architectural Style. Are we going with an embedded application burnt into an FPGA? Do you want to target a PIC? Perhaps an embedded PC? Is the client a Windows machine, or you want to support Mac/Linux? A .NET server side, or maybe Java? It's all rather irreversible, although not completely irreversible.
3rd-Party Libraries/Components/Etc
Are we going to use some existing component (of various scale)? Unless you plan on wrapping everything (which may not even be possible), this often end up being an irreversible decision. For instance, once you commit yourself to using Hibernate for persistence, it's not trivial to move away.
Programming Language
This is the quintessential irreversible decision, unless you want to play with language converters. Note that this is not a coding decisions: coding decisions are made after the language has been chosen.
Structure / Shape / Form
This is what we usually call "design": the shape we want to impose to our material (or, if you live in the "emergent design" side, the shape that our material will take as the final result of several incremental decisions).
So, what are we going to delay? We can't delay all decisions, or we'll be stuck. Sure, we can delay something in each and every area, but truth is, every popular method has been focusing on just a few of them. Of course, different methods tried to delay different choices.
A Little Historical Perspective
Experience brings perspective; at least, true experience does :-). Perspective allows to look at something and see more than it's usually seen. For instance, perspective allows to look at the old, outdated, obsolete waterfall approach and see that it (too) was meant to delay decisions, just different decisions.
Waterfall was meant to delay people decisions, design decisions (which include platform, library, component decisions) and coding decisions. People decision was delayed by specialization: you only have to pick the analyst first, everyone else can be chosen later, when you know what you gotta do (it even makes sense -)). Design decision was delayed because platform, including languages, OS, etc, were way more balkanized than today. Also, architectural styles and patterns were much less understood, and it made sense to look at a larger picture before committing to an overall architecture.
Although this may seem rather ridiculous from the perspective of a 2010 programmer working on Java corporate web applications, most of this stuff is still relevant for (e.g.) mass-produced embedded systems, where choosing the right platform may radically change the total development and production cost, yet choosing the wrong platform may over-constrain the feature set.
Indeed, open systems (another legacy term from late '80s - early '90s) were born exactly to lighten up that choice. Choose the *nix world, and forget about it. Of course, the decision was still irreversible, but granted you some latitude in choosing the exact hw/sw. The entire multi-platform industry (from multi-OS libraries to Java) is basically built on the same foundations. Well, that's the bright side, of course :-).
Looking beyond platform independence, the entire concept of "standard" allows to delay some decision. TCP/IP, for instance, allows me to choose modularly (a concept I'll elaborate later). I can choose TCP/IP as the transport mechanism, and then delay the choice of (e.g.) the client side, and focus on the server side. Of course, a choice is still made (the client must have TCP/IP support), so let's say that widely adopted standards allow for some modularity in the decision process, and therefore to delay some decision, mostly design decisions, but perhaps some other as well (like people).
It's already going to be a long post, so I won't look at each and every method/principle/tool ever conceived, but if you do your homework, you'll find that a lot of what has been proposed in the last 40 years or so (from code generators to MDA, from spiral development to XP, from stepwise refinement to OOP) includes some magic ingredient that allows us to postpone some kind of decision.
It's 2010, guys
So, if you ain't agile, you are clumsy :-)) and c'mon, you don't wanna be clumsy :-). So, seriously, which kind of decisions are usually delayed in (e.g.) XP?
People? I must say I haven't seen much on this. Most literature on XP seems based on the concept that team members are mostly programmers with a wide set of skills, so there should be no particular reason to delay decision about who's gonna work on what. I may have missed some particularly relevant work, however.
Feature Set? Sure. Every incremental approach allows us to delay decisions about features. This can be very advantageous if we can play the learning game, which includes rapid/frequent delivery, or we won't learn enough to actually steer the feature set.
Of course, delaying some decisions on feature set can make some design options viable now, and totally bogus later. Here is where you really have to understand the concept of irreversible and last responsible moment. Of course, if you work on a settled platform, things get simpler, which is one more reason why people get religiously attached to a platform.
Design? Sure, but let's take a deeper look.
Architectural Style: not much. Quoting Booch, "agile projects often start out assuming a given platform and environmental context together with a set of proven design patterns for that domain, all of which represent architectural decisions in a very real sense". See my post Architecture as Tradition in the Unselfconscious Process for more.
Seriously, nobody ever expected to start with a monolithic client and end up with a three-tier web application built around a MVC pattern just by coding and refactoring. The architectural style is pretty much a given in many contemporary projects.
Platform: sorry guys, but if you want to start coding now, you gotta choose your platform now. Another irreversible decision made right at the beginning.
3rd-Party Libraries/Components/Etc: some delay is possible for modularized decisions. If you wanna use hibernate, you gotta choose pretty soon. If you wanna use Seam, you gotta choose pretty soon. Pervasive libraries are so entangled with architectural styles that it's relatively hard to delay some decisions here. Modularized components (e.g. the choice of a PDF rendering library) are simple to delay, and can be proactively delayed (see later).
Programming Language: no way guys, you have to choose right here, right now.
Structure / Shape / Form: of course!!! Here we are. This is it :-). You can delay a lot of detailed design choices. Of course, we always postpone some design decision, even when we design before coding. But let's say that this is where I see a lot of suggestions to delay decisions in the agile literature, often using the dreaded Big Upfront Design as a straw man argument. Of course, the emergent design (or accidental architecture) may or may not be good. If I had to compare the design and code coming out of the XP Episode with my own, I would say that a little upfront design can do wonders, but hey, you know me :-).
Practicing
OK guys, what follows may sound a little odd, but in the end it will prove useful. Have faith :-).
You can get better at everything by doing anything :-), so why not getting better at delaying decisions by playing Windows Solitaire? All you have to do is set the options in the hardest possible way:
now, play a little, until you have to make some decision, like here:
I could move the 9 of spades or the 9 of clubs over the 10 of hearts. It's an irreversible decision (well, not if you use the undo, but that's lame :-). There are some ramifications for both choices.
If I move the 9 of clubs, I can later move the king of clubs and uncover a new card. After that, it's all unknown, and no further speculation is possible. Here, learning requires an irreversible decision; this is very common in real-world projects, but seldom discussed in literature.
If I move the 9 of spades, I uncover the 6 of clubs, which I can move over the 7 of aces. Then, it's kinda unknown, meaning: if you're a serious player (I'm not) you'll remember the previous cards, which would allow you to speculate a little better. Otherwise, it's just as above, you have to make an irreversible decision to learn the outcome.
But wait: what about the last responsible moment? Maybe we can delay this decision! Now, if you delay the decision by clicking on the deck and moving further, you're not delaying the decision: you're wasting a chance. In order to delay this decision, there must be something else you can do.
Well, indeed, there is something you can do. You can move the 8 of aces above the 9 of clubs. This will uncover a new card (learning) without wasting any present opportunity (it could still waste a future opportunity; life it tough). Maybe you'll get a 10 of aces under that 8, at which point there won't be any choice to be made about the 9. Or you might get a black 7, at which point you'll have a different way to move the king of clubs, so moving the 9 of spades would be a more attractive option. So, delay the 9 and move the 8 :-). Add some luck, and it works:
and you get some money too (total at decision time Vs. total at the end)
Novice solitaire players are also known to make irreversible decision without necessity. For instance, in similar cases:
I've seen people eagerly moving the 6 of aces (actually, whatever they got) over the 7 of spades, because "that will free up a slot". Which is true, but irrelevant. This is a decision you can easily delay. Actually, it's a decision you must delay, because:
- if you happen to uncover a king, you can always move the 6. It's not the last responsible moment yet: if you do nothing now, nothing bad will happen.
- you may uncover a 6 of hearts before you uncover a king. And moving that 6 might be more advantageous than moving the 6 of aces. So, don't do it :-). If you want to look good, quote Option Theory, call this a Deferral Option and write a paper about it :-).
Proactive Option Thinking
I've recently read an interesting paper in IEEE TSE ("An Integrative Economic Optimization Approach to Systems Development Risk Management", by Michel Benaroch and James Goldstein). Although the real meat starts in chapter 4, chapters 1-3 are probably more interesting for the casual reader (including myself).
There, authors recap some literature about Real Options in Software Engineering, including the popular argument that delaying decisions is akin to a deferral option. They also make important distinctions, like the one between passive learning through deferral of decisions, and proactive learning, but also between responsiveness to change (a central theme in agility literature) and manipulation of change (relatively less explored), and so on. There is a a lot of food for thought in those 3 chapters, so if you can get a copy, I suggest that you spend a little time pondering over it.
Now, I'm a strong supporter of Proactive Option Thinking. Waiting for opportunities (and then react quickly) is not enough. I believe that options should be "implanted" in our project, and that can be done by applying the right design techniques. How? Keep reading : ).
The Invariant Decision
If you look back at those pictures of Solitaire, you'll see that I wasn't really delaying irreversible decisions. All decisions in solitaire are irreversible (real men don't use CTRL-Z). Many decisions in software development are irreversible as well, especially when you are in a tight budget/schedule, so starting over is not an option. Therefore, irreversibility can't really be the key here. Indeed, I was trying to delay Invariant Decisions. Decisions that I can take now, or I can take later, with little or no impact on the outcomes. The concept itself may seem like a minor change from "irreversible", but it allows me to do some magic:
- I can get rid of the "last responsible moment" part, which is poorly defined anyway. I can just say: delay invariant decisions. Period. You can delay them as much as you want, provided they are still invariant. No ambiguity here. That's much better.
- I can proactively make some decisions invariant. This is so important I'll have to say it again, this time in bold: I can proactively make some decisions invariant.
Invariance, Design, Modularity
If you go back to the Historical Perspective paragraph, you can now read it under a different... perspective :-). Several tools, techniques, methods can be adopted not just to delay some decision, but to create the option to delay the decision. How? Through careful design, of course!
Consider the strong modularity you get from service-oriented architecture, and the platform independence that comes through (well-designed) web services. This is a powerful weapon to delay a lot of decisions on one side or another (client or server).
Consider standard protocols: they are a way to make some decision invariant, and to modularize the impact of some choices.
Consider encapsulation, abstraction and interfaces: they allow you to delay quite a few low-level decisions, and to modularize the impact of change as well. If your choice turn out to be wrong, but it's highly localized (modularized) you may afford undoing your decision, therefore turning irreversible into reversible. A barebone example can be found in my old post (2005!) Builder [pattern] as an option.
Consider a very old OOA/OOD principle, now somehow resurrected under the "ubiquitous language" umbrella. It states that you should try to reflect the real-world entities that you're dealing with in your design, and then in your code. That includes avoiding primitive types like integer, and create meaningful classes instead. Of course, you have to understand what you're doing (that is, you gotta be a good designer) to avoid useless overengineering. See part 4 of my digression on the XP Episode for a discussion about adding a seemingly useless Ball class (that is: implanting a low cost - high premium option).
Names alter the forcefield. A named concept stands apart. My next post on the forcefield theme, by the way, will explore this issue in depth :-).
And so on. I could go on forever, but the point is: you can make many (but not all, of course!) decisions invariant, if you apply the right design techniques. Most of those techniques will also modularize the cost of rework if you make the wrong decision. And sure, you can try to do this on the fly as you code. Or you may want to to some upfront design. You know what I'm thinking.
OK guys, it took quite a while, but now we have a new concept to play with, so more on this will follow, randomly as usual. Stay tuned.
Monday, October 05, 2009
A ForceField Diagram
The Design Rationale Diagram I discussed in my previous post is hardly complete, and it could be vastly improved by asking slightly different questions, leading to different decision paths. Still, it's a reasonable first-cut attempt to model the decision process. It can be used to communicate the reasoning behind a specific decision, in a specific context.
That, however, is not the way I really think. Sure, I can rationalize things that way, but it's not the way I store, recall, organize information inside my head. It's not the way I see the decision space.
In the end, software design is about things going together and things staying apart, at all the granularity levels (see also my post on partitioning).
As I progress in my understanding of forces, I tend to form clusters. Clusters are born out of attraction and rejection inside the decision space. I've found that thinking this way helps me reach a better understanding of my design instinct, and to communicate my thoughts more clearly.
Now, although I've been thinking about this for long while (not full-time, lucky me :-), I can't say I have found the perfect representation. The decision space in inherently multi-dimensional, and I always end up needing more dimensions that I can fit either in 2D or 3D. Over time, I tried several notations, inventing things from scratch or borrowing from other domains. Most were dead ends. In the end, I've chosen (so far :-) a very simple representation, based on just 3 concepts (possibly 4 or 5).
- nodes
Nodes represent information, which is our material. Information has fractal nature, and I don't bother if I'm mixing up levels. Therefore, a node may represent a business goal, or the adoption of a tool or library, or a nonfunctional requirement, or a specific component, class, function. While most methods are based on a strict separation of concepts, I find that very limiting.
- an attraction relationship
Nodes can attract each other. For instance, a node labeled "reliable" may attract a node labeled "redundant" when reasoning about the large display problem. I just connect the two nodes using a thick line with little "hands" on the ends. I place attracted nodes close to each other.
- a rejection relationship
Nodes can reject each other. For instance, stateful most clearly reject stateless :-). Some technology might be at odd with another. A subsystem must not depend on another. And so on. Nodes that reject each other are placed at some distance.
It's all very simple and unsophisticated. Here is an example based on the large display problem, inspired by the discussion on design rationale:
and here are two diagrams I've used in real-world projects recently, scaled down to protect the innocent:
The relationship between a node, a cluster, and an Alexandrian center is better left for another time. Still, a node in one diagram may represent an entire cluster, or an entire diagram. Right now I'm tempted to use a slightly different symbol (which would be the fourth) to represent "expandable" nodes, although I'm really trying to keep symbols to a bare minimum. I'm also using colors, but so far in a very informal way.
As simple as it is, I've found this diagram to be very effective as a reasoning device, while too many diagrams end up being mere documentation devices. I can sit in front of my (large :-) screen, think and draw, and the drawing helps me focus. I can draw this on a whiteboard in a meeting, and everyone get up to speed very quickly on what I'm doing.
This, however, is just half the story. We can surely work with informal concepts and diagrams, and that's fine, but what I'm trying to do is to add precision to the diagram. Precision is often confused with details, like "a class diagram is more precise if you show all the parameters and types". I'm not looking for that kind of "precision". Actually, I don't want this diagram to be redundant with code at all; we already have many code-like diagrams, and they all get down the same roads (generate code from diagrams or generate diagrams from code). I want a reasoning device: when I want to code, I'm comfortable with code :-).
I mostly want to add precision about relationships. Why, for instance, is there an attraction between Slow Client and Stateful? Informally, because if we have a stateful system, the slow client can poll on its own terms, or alternatively, because the client may use a sophisticated subscription based on the previous state. Those options, by the way, could be represented on the forcefield diagram itself (adding more nodes, or a nested diagram); but that's still the "informal" reasoning. Can we make it any more formal, precise, grounded on sound principles?
This is where the ongoing work on concepts like gravity, frequency, and so on kicks in. Slow Client and Stateful are attracted because on a finer granularity (another, perhaps better, diagram) "Slow Client" means a publisher and a subscriber operating at different frequencies, and a stateful repository is a well-known strategy (a pattern!) to provide Isolation between systems operating at different frequencies (together with synchronization or transactions).
Now, I haven't introduced the concept of Isolation yet (though I mentioned something on my Facebook page :-), so this is sort of a spoiler :-)), but in the end I hope to come up with a simple reasoning system, where you can start with informal concepts and refine nodes and forces until you reach the "universal", fractal forces I'm discussing in the "Notes on Software Design" posts. That would give a solid ground to the entire diagram.
A final note on the forcefield diagram: at this stage, I'm just using Visio, or more exactly, I'm abusing some stencils in the Visio library. I wanted something relatively organic, mindmap-like. Maybe one day I'll move back to some 3D ideas (molecular structures come to mind), but I've yet to see how this scales to newer concepts, larger problems, and so on. If you want to play with it, I can send you the VSS file with the stencils.
Ok, I'll get back to Frequency (and Interference and Isolation and more :-) soon. Before that, however, I'd like to take a diversion on the Dependency Structure Matrix. See ya!
That, however, is not the way I really think. Sure, I can rationalize things that way, but it's not the way I store, recall, organize information inside my head. It's not the way I see the decision space.
In the end, software design is about things going together and things staying apart, at all the granularity levels (see also my post on partitioning).
As I progress in my understanding of forces, I tend to form clusters. Clusters are born out of attraction and rejection inside the decision space. I've found that thinking this way helps me reach a better understanding of my design instinct, and to communicate my thoughts more clearly.
Now, although I've been thinking about this for long while (not full-time, lucky me :-), I can't say I have found the perfect representation. The decision space in inherently multi-dimensional, and I always end up needing more dimensions that I can fit either in 2D or 3D. Over time, I tried several notations, inventing things from scratch or borrowing from other domains. Most were dead ends. In the end, I've chosen (so far :-) a very simple representation, based on just 3 concepts (possibly 4 or 5).
- nodes
Nodes represent information, which is our material. Information has fractal nature, and I don't bother if I'm mixing up levels. Therefore, a node may represent a business goal, or the adoption of a tool or library, or a nonfunctional requirement, or a specific component, class, function. While most methods are based on a strict separation of concepts, I find that very limiting.
- an attraction relationship
Nodes can attract each other. For instance, a node labeled "reliable" may attract a node labeled "redundant" when reasoning about the large display problem. I just connect the two nodes using a thick line with little "hands" on the ends. I place attracted nodes close to each other.
- a rejection relationship
Nodes can reject each other. For instance, stateful most clearly reject stateless :-). Some technology might be at odd with another. A subsystem must not depend on another. And so on. Nodes that reject each other are placed at some distance.
It's all very simple and unsophisticated. Here is an example based on the large display problem, inspired by the discussion on design rationale:
and here are two diagrams I've used in real-world projects recently, scaled down to protect the innocent:
The relationship between a node, a cluster, and an Alexandrian center is better left for another time. Still, a node in one diagram may represent an entire cluster, or an entire diagram. Right now I'm tempted to use a slightly different symbol (which would be the fourth) to represent "expandable" nodes, although I'm really trying to keep symbols to a bare minimum. I'm also using colors, but so far in a very informal way.
As simple as it is, I've found this diagram to be very effective as a reasoning device, while too many diagrams end up being mere documentation devices. I can sit in front of my (large :-) screen, think and draw, and the drawing helps me focus. I can draw this on a whiteboard in a meeting, and everyone get up to speed very quickly on what I'm doing.
This, however, is just half the story. We can surely work with informal concepts and diagrams, and that's fine, but what I'm trying to do is to add precision to the diagram. Precision is often confused with details, like "a class diagram is more precise if you show all the parameters and types". I'm not looking for that kind of "precision". Actually, I don't want this diagram to be redundant with code at all; we already have many code-like diagrams, and they all get down the same roads (generate code from diagrams or generate diagrams from code). I want a reasoning device: when I want to code, I'm comfortable with code :-).
I mostly want to add precision about relationships. Why, for instance, is there an attraction between Slow Client and Stateful? Informally, because if we have a stateful system, the slow client can poll on its own terms, or alternatively, because the client may use a sophisticated subscription based on the previous state. Those options, by the way, could be represented on the forcefield diagram itself (adding more nodes, or a nested diagram); but that's still the "informal" reasoning. Can we make it any more formal, precise, grounded on sound principles?
This is where the ongoing work on concepts like gravity, frequency, and so on kicks in. Slow Client and Stateful are attracted because on a finer granularity (another, perhaps better, diagram) "Slow Client" means a publisher and a subscriber operating at different frequencies, and a stateful repository is a well-known strategy (a pattern!) to provide Isolation between systems operating at different frequencies (together with synchronization or transactions).
Now, I haven't introduced the concept of Isolation yet (though I mentioned something on my Facebook page :-), so this is sort of a spoiler :-)), but in the end I hope to come up with a simple reasoning system, where you can start with informal concepts and refine nodes and forces until you reach the "universal", fractal forces I'm discussing in the "Notes on Software Design" posts. That would give a solid ground to the entire diagram.
A final note on the forcefield diagram: at this stage, I'm just using Visio, or more exactly, I'm abusing some stencils in the Visio library. I wanted something relatively organic, mindmap-like. Maybe one day I'll move back to some 3D ideas (molecular structures come to mind), but I've yet to see how this scales to newer concepts, larger problems, and so on. If you want to play with it, I can send you the VSS file with the stencils.
Ok, I'll get back to Frequency (and Interference and Isolation and more :-) soon. Before that, however, I'd like to take a diversion on the Dependency Structure Matrix. See ya!
Monday, August 31, 2009
Representing Design Rationale Inside Activity Diagrams
Design is about making choices. We often do so on the fly, leaning on experience and intuition, by talking about the problem with colleagues, or borrowing from literature (e.g. patterns). We also make some choice by habit, which is a different form of experience, one that has higher risk of becoming disconnected with the real problem.
Most of this process is tacit, and even when we discuss choices openly, it doesn't get recorded. Sometimes, a list of pros/cons is made when there is some disagreement about the best option.
This all works well when the problem is simple, but sometimes even experienced designers feel like they're not grasping the essential issues, that something has not yet been found, named, disentangled. This is when having yet one more tool can prove useful.
Now, I don't usually go through the effort to model and transcribe the rationale behind each and every design choice I make. It could be interesting, also from a pedagogical point of view, but it would take a lot of time and would probably disrupt my thought processes. However, when the issues are particularly thorny/unclear, or when there is a large disagreement on the best choice (or even on the goals and criteria), I've found that getting design rationale out of our individual heads and talk on a shared representation can move things a little forward.
Over the years, I've tried out a number of tools, approaches, and so on; lately, I've tried using Activity Diagrams in a rather unorthodox way, to represent my reasoning about design, not design itself. The idea is not to encode your decisions ex-post, but ex-ante, while you're thinking (that is, while they're still options, not decisions). Also, the diagram must be considered quite fluid, as it shows our current understanding, and we're building the diagram to improve our understanding.
Enough talk, let's see a realistic example. I'll refer to the Large Display problem I discussed a few months ago. Actually, I'll just cover the initial choice between using a real-time database or an IPC/messaging system. It's gonna be quite a mouthful anyway!
To start, I'll have to draw a line between a messaging system and a RTDB, and that in itself is not easy. I'll go for a very simple distinction, because my goal here is not really to talk about RTDBs, but about design rationale (the usual "look at the moon, not at the finger" concept).
So, consider a control system that reads some data from the field and then needs to publish those data for other processes. It could just send data through a messaging (publish/subscribe) system. Here I define a messaging system as stateless, meaning it simply keeps track of subscriptions, and sends everything that is published to the subscribers (according to some criteria, like message type or tag). It does not keep an history, or a snapshot of what has been last sent. Therefore, it cannot apply some filters, like "notify me only if the difference between the previous value and the current value is above a threshold" because the previous value is just not stored. Also, when a subscriber is started, it cannot get the current snapshot of the system, because it is not there: it will have to wait for messages to come, incrementally. Shortly stated, a RTDB will keep a snapshot of the system, and well, you can figure out the difference. Of course, a RTDB is also more complex.
So, how do we choose between a messaging system and a RTDB? We may write down a long list of pro/cons, but that's really unstructured, and that's not the way our brain works. To provide more structure, I use an Activity Diagram with orthogonal swimlanes (all the following pictures are taken from Star UML, a free tool that is rather fast and unobtrusive).
The vertical swimlanes are flexible: they represent the main concerns. The horizontal swimlanes are fixed: they provide structure.
For the Large Display problems, we could start with a few main concerns like performance, reliability, cost, and so on. We just drop the names on the vertical swimlanes. My template is then partitioned in 3 horizontal bands: the root question, the reasoning, the outcome. Everything inside is dynamic, and changes as we understand more: even the root questions may change, as we discover larger or smaller, independent problems. Sometimes, even the main concerns change, as we discover options or issues we didn't consider before.
We can focus on just one concern right now, let's say performance (don't we all like performance? :-). A first-cut, interesting top-level question could be: is the published data rate high or low? If the rate is low and we have no persistent state, when you turn on the large display you see nothing: you have to wait till some data gets published. On the other hand, if the rate is high, it may even overwhelm the display system: there is little need to refresh a value a thousand times per second. That actually depends on the display: if it's a real-time plot, you may want a high refresh rate too.
Ok, we could start modeling this part of our reasoning using the familiar activity diagram symbols. Actually, since most of the nodes here would be decision nodes, I just omit the diamond and use an activity node with multiple outgoing paths to show choices.
Note: The empty boxes are just placeholders for some later reasoning. It's just laziness on my side :-) and they wouldn't appear in a real diagram.
Now, this seems just like a decision tree, but it's slightly different. First, it's a decision graph: common choices between paths are shared, and this is a precious information because it shows crucial choices (more on this later). Second, it's a multifaceted graph: every vertical swimlane shows a facet of a more complex reasoning; for instance, what is good for performance might not be good for reliability or cost.
Let's try to move ahead a little. When the incoming data rate is higher than what [most] clients need, we have basically two choices:
1) smarter subscriptions; they could still be rather dumb, like "no more than 3 times per second" or much smarter like "when relative change is higher than 5%, but no more than 5 times per second". Note that the latter is more suited to a RTDB than to a stateless messaging system.
2) change paradigm and move to client-initiated polling. The clients will ask for data with their own timing. Of course, at this point we give up the possibility of not asking for data if the value has not changed. Anyway, this again requires some kind of stateful middleware; a messaging system won't do.
When data rate is low, but high startup time for clients is not an option, we can't wait for data to come: we have to poll, at least at startup. So, polling can solve two problems, of course at expense of bandwidth if it is the only available option.
While drawing this, we may come to the conclusion that we need to ask better questions: are we building a publisher-driven or a client-driven system? If it's client-driven, it cannot be stateless! What do we really know about clients? How many there will be? What about publishers? What is the typical data rate and configuration? What are we aiming for? Do we need to narrow the expectations? This might change the top question (client Vs. publisher driven) or even some concern. That's fine, it means the technique is working :-) and that it's helping us thinking.
Now, it would take quite a lot of time to explore all the facets of even a simple system like this. Actually, most people won't even do it in real life: they will fall in love with one idea, spend most of their time preaching and rationalizing about the virtues of their idea, and never really take the time to go through this kind of process. Still, trying to work out the "Reliability" swimlane would prove interesting. For instance, a common technique to achieve reliability is redundancy. Redundancy is much easier for a stateless system. Redundancy is easier when clients don't have to subscribe at all, but can simply poll. And so on. If you have some spare time, you may want to give it a try.
The notation I use is quite informal. I could improve that easily: UML is fairly flexible; so far I didn't, because people can grasp it anyway, even when I drop in the < < or > > to represent options or when I have just one arrow coming out, meaning that I've just decomposed a choice and a consequence. It's just a reasoning workflow, and I haven't felt the need to make it any more precise than that.
Back to the forcefield: the rationale is not the forcefield. The rationale, however, is talking about forces and centers. Outcomes (messaging and RTDB) are centers. Main choices, like "client driven" or "stateless", are again centers. Those centers are attracting or rejecting each other. This is the forcefield. This is closer to the way I think in the back of my mind, how I "see" the system, how I keep options open. Now, I just need a way to show this. That's for my next post :-).
Most of this process is tacit, and even when we discuss choices openly, it doesn't get recorded. Sometimes, a list of pros/cons is made when there is some disagreement about the best option.
This all works well when the problem is simple, but sometimes even experienced designers feel like they're not grasping the essential issues, that something has not yet been found, named, disentangled. This is when having yet one more tool can prove useful.
Now, I don't usually go through the effort to model and transcribe the rationale behind each and every design choice I make. It could be interesting, also from a pedagogical point of view, but it would take a lot of time and would probably disrupt my thought processes. However, when the issues are particularly thorny/unclear, or when there is a large disagreement on the best choice (or even on the goals and criteria), I've found that getting design rationale out of our individual heads and talk on a shared representation can move things a little forward.
Over the years, I've tried out a number of tools, approaches, and so on; lately, I've tried using Activity Diagrams in a rather unorthodox way, to represent my reasoning about design, not design itself. The idea is not to encode your decisions ex-post, but ex-ante, while you're thinking (that is, while they're still options, not decisions). Also, the diagram must be considered quite fluid, as it shows our current understanding, and we're building the diagram to improve our understanding.
Enough talk, let's see a realistic example. I'll refer to the Large Display problem I discussed a few months ago. Actually, I'll just cover the initial choice between using a real-time database or an IPC/messaging system. It's gonna be quite a mouthful anyway!
To start, I'll have to draw a line between a messaging system and a RTDB, and that in itself is not easy. I'll go for a very simple distinction, because my goal here is not really to talk about RTDBs, but about design rationale (the usual "look at the moon, not at the finger" concept).
So, consider a control system that reads some data from the field and then needs to publish those data for other processes. It could just send data through a messaging (publish/subscribe) system. Here I define a messaging system as stateless, meaning it simply keeps track of subscriptions, and sends everything that is published to the subscribers (according to some criteria, like message type or tag). It does not keep an history, or a snapshot of what has been last sent. Therefore, it cannot apply some filters, like "notify me only if the difference between the previous value and the current value is above a threshold" because the previous value is just not stored. Also, when a subscriber is started, it cannot get the current snapshot of the system, because it is not there: it will have to wait for messages to come, incrementally. Shortly stated, a RTDB will keep a snapshot of the system, and well, you can figure out the difference. Of course, a RTDB is also more complex.
So, how do we choose between a messaging system and a RTDB? We may write down a long list of pro/cons, but that's really unstructured, and that's not the way our brain works. To provide more structure, I use an Activity Diagram with orthogonal swimlanes (all the following pictures are taken from Star UML, a free tool that is rather fast and unobtrusive).
The vertical swimlanes are flexible: they represent the main concerns. The horizontal swimlanes are fixed: they provide structure.
For the Large Display problems, we could start with a few main concerns like performance, reliability, cost, and so on. We just drop the names on the vertical swimlanes. My template is then partitioned in 3 horizontal bands: the root question, the reasoning, the outcome. Everything inside is dynamic, and changes as we understand more: even the root questions may change, as we discover larger or smaller, independent problems. Sometimes, even the main concerns change, as we discover options or issues we didn't consider before.
We can focus on just one concern right now, let's say performance (don't we all like performance? :-). A first-cut, interesting top-level question could be: is the published data rate high or low? If the rate is low and we have no persistent state, when you turn on the large display you see nothing: you have to wait till some data gets published. On the other hand, if the rate is high, it may even overwhelm the display system: there is little need to refresh a value a thousand times per second. That actually depends on the display: if it's a real-time plot, you may want a high refresh rate too.
Ok, we could start modeling this part of our reasoning using the familiar activity diagram symbols. Actually, since most of the nodes here would be decision nodes, I just omit the diamond and use an activity node with multiple outgoing paths to show choices.
Note: The empty boxes are just placeholders for some later reasoning. It's just laziness on my side :-) and they wouldn't appear in a real diagram.
Now, this seems just like a decision tree, but it's slightly different. First, it's a decision graph: common choices between paths are shared, and this is a precious information because it shows crucial choices (more on this later). Second, it's a multifaceted graph: every vertical swimlane shows a facet of a more complex reasoning; for instance, what is good for performance might not be good for reliability or cost.
Let's try to move ahead a little. When the incoming data rate is higher than what [most] clients need, we have basically two choices:
1) smarter subscriptions; they could still be rather dumb, like "no more than 3 times per second" or much smarter like "when relative change is higher than 5%, but no more than 5 times per second". Note that the latter is more suited to a RTDB than to a stateless messaging system.
2) change paradigm and move to client-initiated polling. The clients will ask for data with their own timing. Of course, at this point we give up the possibility of not asking for data if the value has not changed. Anyway, this again requires some kind of stateful middleware; a messaging system won't do.
When data rate is low, but high startup time for clients is not an option, we can't wait for data to come: we have to poll, at least at startup. So, polling can solve two problems, of course at expense of bandwidth if it is the only available option.
While drawing this, we may come to the conclusion that we need to ask better questions: are we building a publisher-driven or a client-driven system? If it's client-driven, it cannot be stateless! What do we really know about clients? How many there will be? What about publishers? What is the typical data rate and configuration? What are we aiming for? Do we need to narrow the expectations? This might change the top question (client Vs. publisher driven) or even some concern. That's fine, it means the technique is working :-) and that it's helping us thinking.
Now, it would take quite a lot of time to explore all the facets of even a simple system like this. Actually, most people won't even do it in real life: they will fall in love with one idea, spend most of their time preaching and rationalizing about the virtues of their idea, and never really take the time to go through this kind of process. Still, trying to work out the "Reliability" swimlane would prove interesting. For instance, a common technique to achieve reliability is redundancy. Redundancy is much easier for a stateless system. Redundancy is easier when clients don't have to subscribe at all, but can simply poll. And so on. If you have some spare time, you may want to give it a try.
The notation I use is quite informal. I could improve that easily: UML is fairly flexible; so far I didn't, because people can grasp it anyway, even when I drop in the < < or > > to represent options or when I have just one arrow coming out, meaning that I've just decomposed a choice and a consequence. It's just a reasoning workflow, and I haven't felt the need to make it any more precise than that.
Back to the forcefield: the rationale is not the forcefield. The rationale, however, is talking about forces and centers. Outcomes (messaging and RTDB) are centers. Main choices, like "client driven" or "stateless", are again centers. Those centers are attracting or rejecting each other. This is the forcefield. This is closer to the way I think in the back of my mind, how I "see" the system, how I keep options open. Now, I just need a way to show this. That's for my next post :-).
Tuesday, June 09, 2009
Design Rationale
In the past few weeks I've taken a little time to write down more about the concept of frequency; while doing so, I realized I had to explore the concept of forcefield better, and while doing so (yeap :-)) I realized there was a rather large overlap between the notion of forcefield and the notion of design rationale.
Design rationale extends beyond software engineering, and aims to capture design decisions and the reasoning behind those decisions. Now, design decisions are (ideally) taken as trade-offs between several competing forces. Those forces creates the forcefield, hence the large overlap between the two subjects.
The concept of design rationale has been around for quite a few years, but I haven't seen much progress either in tools or notations. Most often, tools fall into the “rationalize after the fact” family, while I'm more interested in reasoning tools and notations, that would help me (as a designer) get a better picture about my own thoughts while I'm thinking. That resonates with the concept of reflection in action that I've discussed in Listen to Your Tools and Materials a few years ago.
So, as I was reading a recent issue of IEEE Software (March/April 2009), I found a list of recent (and not so recent) tools dealing with design rationale in a paper by Philippe Kruchten, Rafael Capilla, Juan Carlos Dueñas (The Decision View’s Role in Software Architecture Practice), and I decided to take a quick ride. Here is a very quick summary of what I've found.
Seurat
Seurat (see also the PDF tutorial on the same website) is based on a very powerful language / model, but the tool (as implemented) is very limiting. It's based on a tree structure, which makes for a nice todo list, but makes visual reasoning almost impossible. Actually, in the past I've investigated on using the tree format myself (and while doing so, I discovered others have done the same: see for instance the Reasoning Tree pattern), but restricting visualization to (hyperlinked) nodes in a tree just does not work when you're facing difficult problems.
Sysiphus
Sysiphus seems to have recently morphed into another tool (UniCase), but from the demo of UniCase it's hard to appreciate any special support for design rationale (so far).
AREL
(see also some papers from Antony Tang on the same page; Antony also had an excellent paper on AREL in the same issue of IEEE Software)
AREL is integrated with Enterprise Architect. Integration with existing case tools (either commercial or free) seems quite a good idea to me. AREL uses a class diagram (through a UML profile) to model design rationale, so it's not limited to a tree format. Still, I've found the results rather hard to read. It seems more like a tool to give structure to design knowledge than a tool to reason about design. As I go through the examples, I have to study the diagram; it doesn't just talk back to me. I have to click around and look at other artifacts. The reasoning is not in the diagram, it's only accessible through the diagram.
PAKME
Honestly, PAKME seems more like an exercise in building a web-based collaboration tool for software development than a serious attempt at providing a useful / usable tool to record design rationale. It does little more than organize artifacts, and it requires so many clicks / page refresh to get anything done that I doubt a professional designer could ever use it (sorry guys).
ADDSS
ADDSS is very much like PAKME, although it adds a useful Patterns section. It's so far from what I consider a useful design tool (see my paper above for more) that I can't really think of using it (sorry, again).
Knowledge Architect
Again, a tool with some good ideas (like Word integration) but far from what I'm looking for. It's fine to create a structured design document, but not to reason about difficult design problems.
In the end, it seems like most of those tools suffer from the same problems:
- The research is good; a nice metamodel is built, some of the problems faced by professional designers seem to be well understood.
- The tool does little more than organize knowledge, would get in the way of the designer thinking about thorny issues, does not help through visualization, and is at best useful at the end of the design process, possibly to fake some rationality, a-la Parnas/Clements.
That said, AREL is probably the most promising tool of the pack, but in the end I've being doing pretty much the same for years now, using (well, abusing :-) plain old use case diagrams to model goals and issues, with a few ideas taken from KAOS and the like.
Recently, I began experimenting with another standard UML diagram (the activity diagram) to model some portion of design reasoning. I'll show an example in my next post, and then show how we can change our perspective and move from design reasoning to the forcefield.
Design rationale extends beyond software engineering, and aims to capture design decisions and the reasoning behind those decisions. Now, design decisions are (ideally) taken as trade-offs between several competing forces. Those forces creates the forcefield, hence the large overlap between the two subjects.
The concept of design rationale has been around for quite a few years, but I haven't seen much progress either in tools or notations. Most often, tools fall into the “rationalize after the fact” family, while I'm more interested in reasoning tools and notations, that would help me (as a designer) get a better picture about my own thoughts while I'm thinking. That resonates with the concept of reflection in action that I've discussed in Listen to Your Tools and Materials a few years ago.
So, as I was reading a recent issue of IEEE Software (March/April 2009), I found a list of recent (and not so recent) tools dealing with design rationale in a paper by Philippe Kruchten, Rafael Capilla, Juan Carlos Dueñas (The Decision View’s Role in Software Architecture Practice), and I decided to take a quick ride. Here is a very quick summary of what I've found.
Seurat
Seurat (see also the PDF tutorial on the same website) is based on a very powerful language / model, but the tool (as implemented) is very limiting. It's based on a tree structure, which makes for a nice todo list, but makes visual reasoning almost impossible. Actually, in the past I've investigated on using the tree format myself (and while doing so, I discovered others have done the same: see for instance the Reasoning Tree pattern), but restricting visualization to (hyperlinked) nodes in a tree just does not work when you're facing difficult problems.
Sysiphus
Sysiphus seems to have recently morphed into another tool (UniCase), but from the demo of UniCase it's hard to appreciate any special support for design rationale (so far).
AREL
(see also some papers from Antony Tang on the same page; Antony also had an excellent paper on AREL in the same issue of IEEE Software)
AREL is integrated with Enterprise Architect. Integration with existing case tools (either commercial or free) seems quite a good idea to me. AREL uses a class diagram (through a UML profile) to model design rationale, so it's not limited to a tree format. Still, I've found the results rather hard to read. It seems more like a tool to give structure to design knowledge than a tool to reason about design. As I go through the examples, I have to study the diagram; it doesn't just talk back to me. I have to click around and look at other artifacts. The reasoning is not in the diagram, it's only accessible through the diagram.
PAKME
Honestly, PAKME seems more like an exercise in building a web-based collaboration tool for software development than a serious attempt at providing a useful / usable tool to record design rationale. It does little more than organize artifacts, and it requires so many clicks / page refresh to get anything done that I doubt a professional designer could ever use it (sorry guys).
ADDSS
ADDSS is very much like PAKME, although it adds a useful Patterns section. It's so far from what I consider a useful design tool (see my paper above for more) that I can't really think of using it (sorry, again).
Knowledge Architect
Again, a tool with some good ideas (like Word integration) but far from what I'm looking for. It's fine to create a structured design document, but not to reason about difficult design problems.
In the end, it seems like most of those tools suffer from the same problems:
- The research is good; a nice metamodel is built, some of the problems faced by professional designers seem to be well understood.
- The tool does little more than organize knowledge, would get in the way of the designer thinking about thorny issues, does not help through visualization, and is at best useful at the end of the design process, possibly to fake some rationality, a-la Parnas/Clements.
That said, AREL is probably the most promising tool of the pack, but in the end I've being doing pretty much the same for years now, using (well, abusing :-) plain old use case diagrams to model goals and issues, with a few ideas taken from KAOS and the like.
Recently, I began experimenting with another standard UML diagram (the activity diagram) to model some portion of design reasoning. I'll show an example in my next post, and then show how we can change our perspective and move from design reasoning to the forcefield.
Sunday, April 26, 2009
Bad Luck, or "fighting the forcefield"
In my previous post, I used the expression "fighting the forcefield". This might be a somewhat uncommon terminology, but I used it to describe a very familiar situation: actually, I see people fighting the forcefield all the time.
Look at any troubled project, and you'll see people who made some wrong decision early on, and then stood by it, digging and digging. Of course, any decision may turn out to be wrong. Software development is a knowledge acquisition process. We often take decisions without knowing all the details; if we didn't, we would never get anything done (see analysis paralysis for more). Experience should mitigate the number of wrong decisions, but there are going to be mistakes anyway; we should be able to recognize them quickly, backtrack, and take another way.
Experience should also bring us in closer contact with the forcefield. Experienced designers don't need to go through each and every excruciating detail before they can take a decision. As I said earlier, we can almost feel, or see the forcefield, and take decisions based on a relatively small number of prevailing forces (yes, I dare to consider myself an experienced designer :-).
This process is largely unconscious, and sometimes it's hard to rationalize all the internal reasoning; in many cases, people expect very argumentative explanations, while all we have to offer on the fly is aesthetics. Indeed, I'm often very informal when I design; I tend to use colorful expressions like "oh, that sucks", or "that brings bad luck" to indicate a flaw, and so on.
Recently, I've found myself saying that "bad luck" thing twice, while reviewing the design of two very different systems (a business system and a reactive system), for two different clients.
I noticed a pattern: in both cases, there was a single entity (a database table, a in-memory structure) storing data with very different timing/life requirements. In both cases, my clients were a little puzzled, as they thought those data belonged together (we can recognize gravity at play here).
Most naturally, they asked me why I would keep the data apart. Time to rationalize :-), once again.
Had they all been more familiar with my blog, I would have pointed to my recent post on multiplicity. After all, data with very different update frequency (like: the calibration data for a sensor, and the most recent sample) have a different fourth-dimensional multiplicity. Sure, at any given point in time, a sensor has one most recent sample and one set of calibration data; therefore, in a static view we'll have multiplicity 1 for both, suggesting we can keep the two of them together. But bring in the fourth dimension (time) and you'll see an entirely different picture: they have a completely different historical multiplicity.
Different update frequencies also hint at the fact that data is changing under different forces. By keeping together things that are influenced by more than one force, we expose them to both. More on this another time.
Hard-core programmers may want more than that. They may ask for more familiar reasons not to put data with different update frequencies in the same table or structure. Here are a few:
- In a multi-threaded software, in-memory structures requires locking. If your structure contains data that is seldom updated, that means it's being read more than written: if it's seldom read and seldom written, why keep it around at all?
Unfortunately, the high-frequency data is written quite often. Therefore, either we accept to slow down everything using a simple mutex, or we aim for higher performances through a more complex locking mechanism (reader/writer lock), which may or may not work, depending on the exact read/write pattern. Separate structures can adopt a simpler locking mechanism, as one is being mostly read, the other mostly written; even if you go with a R/W lock, here it's almost guaranteed to have good performance.
- Even on a database, high-frequency writes may stall low-frequency reads. You even risk a lock escalation from record to table. Then you either go with dirty reads (betting on your good luck) or you just move the data in another table, where it belongs.
- If you decide to cache database data to improve performances, you'll have to choose between a larger cache with the same structure of the database (with low frequency data too) or a smaller and more efficient cache with just the high-frequency data (therefore revealing once more that those data do not belong together).
- And so on: I encourage you to find more reasons!
In most cases, I tend to avoid this kind of problems instinctively: this is what I really call experience. Indeed, Donald Schön reminds us that good design is not for everyone, and that you have to develop your own sense of aesthetics (see "Reflective Conversation with Materials. An interview with Donald Schön by John Bennett", in Bringing Design To Software, Addison-Wesley, 1996). Aesthetics may not sound too technical, but consider it a shortcut for: you have to develop your own ability to perceive the forcefield, and instinctively know what is wrong (misaligned) and right (aligned).
Ok, next time I'll get back to the notion of multiplicity. Actually, although I've initially chosen "multiplicity" because of its familiarity, I'm beginning to think that the whole notion of fourth-dimensional multiplicity, which is indeed quite important, might be confusing for some. I'm therefore looking for a better term, which can clearly convey both the traditional ("static") and the extended (fourth-dimensional, historical, etc) meaning. Any good idea? Say it here, or drop me an email!
Look at any troubled project, and you'll see people who made some wrong decision early on, and then stood by it, digging and digging. Of course, any decision may turn out to be wrong. Software development is a knowledge acquisition process. We often take decisions without knowing all the details; if we didn't, we would never get anything done (see analysis paralysis for more). Experience should mitigate the number of wrong decisions, but there are going to be mistakes anyway; we should be able to recognize them quickly, backtrack, and take another way.
Experience should also bring us in closer contact with the forcefield. Experienced designers don't need to go through each and every excruciating detail before they can take a decision. As I said earlier, we can almost feel, or see the forcefield, and take decisions based on a relatively small number of prevailing forces (yes, I dare to consider myself an experienced designer :-).
This process is largely unconscious, and sometimes it's hard to rationalize all the internal reasoning; in many cases, people expect very argumentative explanations, while all we have to offer on the fly is aesthetics. Indeed, I'm often very informal when I design; I tend to use colorful expressions like "oh, that sucks", or "that brings bad luck" to indicate a flaw, and so on.
Recently, I've found myself saying that "bad luck" thing twice, while reviewing the design of two very different systems (a business system and a reactive system), for two different clients.
I noticed a pattern: in both cases, there was a single entity (a database table, a in-memory structure) storing data with very different timing/life requirements. In both cases, my clients were a little puzzled, as they thought those data belonged together (we can recognize gravity at play here).
Most naturally, they asked me why I would keep the data apart. Time to rationalize :-), once again.
Had they all been more familiar with my blog, I would have pointed to my recent post on multiplicity. After all, data with very different update frequency (like: the calibration data for a sensor, and the most recent sample) have a different fourth-dimensional multiplicity. Sure, at any given point in time, a sensor has one most recent sample and one set of calibration data; therefore, in a static view we'll have multiplicity 1 for both, suggesting we can keep the two of them together. But bring in the fourth dimension (time) and you'll see an entirely different picture: they have a completely different historical multiplicity.
Different update frequencies also hint at the fact that data is changing under different forces. By keeping together things that are influenced by more than one force, we expose them to both. More on this another time.
Hard-core programmers may want more than that. They may ask for more familiar reasons not to put data with different update frequencies in the same table or structure. Here are a few:
- In a multi-threaded software, in-memory structures requires locking. If your structure contains data that is seldom updated, that means it's being read more than written: if it's seldom read and seldom written, why keep it around at all?
Unfortunately, the high-frequency data is written quite often. Therefore, either we accept to slow down everything using a simple mutex, or we aim for higher performances through a more complex locking mechanism (reader/writer lock), which may or may not work, depending on the exact read/write pattern. Separate structures can adopt a simpler locking mechanism, as one is being mostly read, the other mostly written; even if you go with a R/W lock, here it's almost guaranteed to have good performance.
- Even on a database, high-frequency writes may stall low-frequency reads. You even risk a lock escalation from record to table. Then you either go with dirty reads (betting on your good luck) or you just move the data in another table, where it belongs.
- If you decide to cache database data to improve performances, you'll have to choose between a larger cache with the same structure of the database (with low frequency data too) or a smaller and more efficient cache with just the high-frequency data (therefore revealing once more that those data do not belong together).
- And so on: I encourage you to find more reasons!
In most cases, I tend to avoid this kind of problems instinctively: this is what I really call experience. Indeed, Donald Schön reminds us that good design is not for everyone, and that you have to develop your own sense of aesthetics (see "Reflective Conversation with Materials. An interview with Donald Schön by John Bennett", in Bringing Design To Software, Addison-Wesley, 1996). Aesthetics may not sound too technical, but consider it a shortcut for: you have to develop your own ability to perceive the forcefield, and instinctively know what is wrong (misaligned) and right (aligned).
Ok, next time I'll get back to the notion of multiplicity. Actually, although I've initially chosen "multiplicity" because of its familiarity, I'm beginning to think that the whole notion of fourth-dimensional multiplicity, which is indeed quite important, might be confusing for some. I'm therefore looking for a better term, which can clearly convey both the traditional ("static") and the extended (fourth-dimensional, historical, etc) meaning. Any good idea? Say it here, or drop me an email!
Monday, March 30, 2009
Notes on Software Design, Chapter 5: Multiplicity
Gravity, as we have seen, provides a least resistance path, leading to monolithic software. If gravity was the only force at play, all software would be a monolithic blob. That being not the case, there must be other forces at play. Pervasive, primitive forces just like gravity, setting up a different forcefield, so that it's more convenient to keep things apart.
Consider an amateur programmer, writing a simple program to keep track of his numerous books. He starts with a database-centric approach, and without much knowledge of conceptual modeling, he jumps into creating tables. He creates a Book table, and adds a few fields:
AuthorFirstName, AuthorLastName, Title, Publisher, ISBN, …
It doesn't take much for him to realize that an author could be present several times in his database. He may begin to realize that he could perhaps add an Author table and move AuthorFirstName and AuthorLastName to that table.
Why? He doesn't know squat about database normalization. It's just a simple matter of multiplicity. One author - many books. Different multiplicity suggests to keep things apart. It is quite a good suggestion, as different multiplicity basically requires different gravitational centers, lest we end up with an unfavorable forcefield.
Consider what happens when our amateur programmer discovers he wants to add more biographical data about authors. Without an Author table, there is not any good gravitational center that could possibly attract those data. There is only the Book table, so there they go - adding more data redundancy.
Our amateur programmer, however, might not be so eager to give in. A single table is easier to manage. No foreign keys, no referential integrity, no nothing. It's just simpler, and he doesn't live in the future. He wants to do the simplest thing that could possibly work, so he keeps the Author fields inside the Book table.
He doesn't need much more, however, to realize that many books have more than one author. One book - many authors. That's a different forcefield again, with a many-to-many relationship. Now, our amateur is rather stubborn. He wants to keep things inside a single table anyway. So he goes on and adds more fields:
AuthorFirstName1, AuthorLastName1, AuthorFirstName2, AuthorLastName2, AuthorFirstName3, AuthorLastName3, Title, Publisher, ISBN, …
Of course, at this point he can basically feel he's no longer going along the path of least resistance. Actually, he's fighting the forcefield. Sure, gravity wants him to keep things together, but multiplicity doesn't. The form he's trying to give to the Book table is not in frictionless contact with the forcefield. The forcefield wants Book and Author to stay on their own.
Multiplicity is the primordial force that keeps [software] things apart. It shouldn't come as a surprise, then, that a great emphasis is given to multiplicity in the Entity-Relationship model and also in the static view of OO models (class diagram).
Multiplicity, however, goes much deeper than that. Reusability is a special case of multiplicity. What? :-). Well, it that sounds odd, you're not thinking fourth dimensionally (as Doc said in "Back to the future").
Consider a different problem, at a different granularity. Our amateur programmer is writing another small application, to keep track of who has borrowed some of his precious books. He's doing the simplest thing again, so he's basically going GUI-centered, and he's putting all the business logic inside the form itself. When you click on "Ok", the form will validate data and store a record into some table. The form requires, among other things, a phone number, which must be validated. It's the only place where he has to validate a phone number, so he puts the validation logic right inside the OnOk method generously provided by his RAD tool.
What's wrong? Apparently, there is no multiplicity at play here. There is one function, where he's doing two distinct things (validation and insertion), and inside validation he's doing different things, but each one is intended to validate one field, so it wouldn't pay to move the field validation logic elsewhere. Gravity keeps things together.
Multiplicity is hidden in the fourth dimension: time. Reusability means being able to take something you have already written (in the past) and use it again, unchanged, in the future. It means you have multiple callers, just not at the same time. If you think fourth dimensionally, multiplicity comes out quite clearly.
Multiplicity is an interesting force, one we need to be very familiar with. It will take a few posts to give it justice. Right now, it's time for me to put my running shoes on and hit the road :-). Still, here are a few pointers to some important issues that I'm going to cover in the next weeks (or months :-)
The fractal nature of multiplicity
Maintainability
Scalability
Conway's Law
Tools and Languages - lowering costs
Good questions to ask while doing analysis and design.
Is multiplicity stronger than gravity?
Examples from patterns. On truly understanding Abstract Factory.
N-degrees of separation.
Interfaces and Multiplicity - what is separation, anyway?
Cross-cutting concerns.
Down-to-earth guidelines.
The Display problem, once again.
Consider an amateur programmer, writing a simple program to keep track of his numerous books. He starts with a database-centric approach, and without much knowledge of conceptual modeling, he jumps into creating tables. He creates a Book table, and adds a few fields:
AuthorFirstName, AuthorLastName, Title, Publisher, ISBN, …
It doesn't take much for him to realize that an author could be present several times in his database. He may begin to realize that he could perhaps add an Author table and move AuthorFirstName and AuthorLastName to that table.
Why? He doesn't know squat about database normalization. It's just a simple matter of multiplicity. One author - many books. Different multiplicity suggests to keep things apart. It is quite a good suggestion, as different multiplicity basically requires different gravitational centers, lest we end up with an unfavorable forcefield.
Consider what happens when our amateur programmer discovers he wants to add more biographical data about authors. Without an Author table, there is not any good gravitational center that could possibly attract those data. There is only the Book table, so there they go - adding more data redundancy.
Our amateur programmer, however, might not be so eager to give in. A single table is easier to manage. No foreign keys, no referential integrity, no nothing. It's just simpler, and he doesn't live in the future. He wants to do the simplest thing that could possibly work, so he keeps the Author fields inside the Book table.
He doesn't need much more, however, to realize that many books have more than one author. One book - many authors. That's a different forcefield again, with a many-to-many relationship. Now, our amateur is rather stubborn. He wants to keep things inside a single table anyway. So he goes on and adds more fields:
AuthorFirstName1, AuthorLastName1, AuthorFirstName2, AuthorLastName2, AuthorFirstName3, AuthorLastName3, Title, Publisher, ISBN, …
Of course, at this point he can basically feel he's no longer going along the path of least resistance. Actually, he's fighting the forcefield. Sure, gravity wants him to keep things together, but multiplicity doesn't. The form he's trying to give to the Book table is not in frictionless contact with the forcefield. The forcefield wants Book and Author to stay on their own.
Multiplicity is the primordial force that keeps [software] things apart. It shouldn't come as a surprise, then, that a great emphasis is given to multiplicity in the Entity-Relationship model and also in the static view of OO models (class diagram).
Multiplicity, however, goes much deeper than that. Reusability is a special case of multiplicity. What? :-). Well, it that sounds odd, you're not thinking fourth dimensionally (as Doc said in "Back to the future").
Consider a different problem, at a different granularity. Our amateur programmer is writing another small application, to keep track of who has borrowed some of his precious books. He's doing the simplest thing again, so he's basically going GUI-centered, and he's putting all the business logic inside the form itself. When you click on "Ok", the form will validate data and store a record into some table. The form requires, among other things, a phone number, which must be validated. It's the only place where he has to validate a phone number, so he puts the validation logic right inside the OnOk method generously provided by his RAD tool.
What's wrong? Apparently, there is no multiplicity at play here. There is one function, where he's doing two distinct things (validation and insertion), and inside validation he's doing different things, but each one is intended to validate one field, so it wouldn't pay to move the field validation logic elsewhere. Gravity keeps things together.
Multiplicity is hidden in the fourth dimension: time. Reusability means being able to take something you have already written (in the past) and use it again, unchanged, in the future. It means you have multiple callers, just not at the same time. If you think fourth dimensionally, multiplicity comes out quite clearly.
Multiplicity is an interesting force, one we need to be very familiar with. It will take a few posts to give it justice. Right now, it's time for me to put my running shoes on and hit the road :-). Still, here are a few pointers to some important issues that I'm going to cover in the next weeks (or months :-)
The fractal nature of multiplicity
Maintainability
Scalability
Conway's Law
Tools and Languages - lowering costs
Good questions to ask while doing analysis and design.
Is multiplicity stronger than gravity?
Examples from patterns. On truly understanding Abstract Factory.
N-degrees of separation.
Interfaces and Multiplicity - what is separation, anyway?
Cross-cutting concerns.
Down-to-earth guidelines.
The Display problem, once again.
Subscribe to:
Posts (Atom)