Sometimes, my clients ask me what to read about software design. Most often than not, they don't want a list of books – many already have the classics covered. They would like to read papers, perhaps recent works, to further advance their understanding of software design, or perhaps something inspirational, to get new ideas and concepts. I have to confess I'm often at loss for suggestions.
The sense of discomfort gets even worse when they ask me what happened to the kind of publications they used to read in the late '90s. Stuff like the C++ Report, the Journal of Object Oriented Programming, Java Report, Object Expert, etc. Most don't even know about the Journal of Object Technology, but honestly, the JOT has taken a strong academic slant lately, and I'm not sure they would find it all that interesting. I usually suggest they keep current on design patterns: for instance, the complete EuroPLoP 2009 proceedings are available online, for free. However, mentioning patterns sometimes furthers just another question: what happened to the pattern movement? Keep that conversation going for a while, and you get the final question: so, is software design literature dead?
Is it?
Note that the question is not about software design - it's about software design literature. Interestingly, Martin Fowler wrote about the other side of the story (Is Design Dead?) back in 2004. He argued that design wasn't dead, but its nature had changed (I could argue that if you change the nature of something, then it's perhaps inappropriate to keep using the same name :-), but ok). So perhaps software design literature isn't dead either, and has just changed nature: after all, looking for "software design" on Google yields 3,240,000 results.
Trying to classify what I usually find under the "software design" chapter, I came up with this list:
- Literature on software design philosophy, hopefully with some practical applications. Early works on Information Hiding were as much about a philosophy of software design as about practical ways to structure our software. There were a lot of philosophical papers on OOP and AOP as well. Usually, you get this kind of literature when some new idea is being proposed (like my Physics of Software :-). It's natural to see less and less philosophy in a maturing discipline, so perhaps a dearth of philosophical literature is not a bad sign.
- Literature on notations, like UML or SysML. This is not really software design literature, unless the rationale for the notation is discussed.
- Academic literature on metrics and the like. This would be interesting if those metrics addressed real design questions, but in practice, everything is always observed through the rather narrow perspective of correlation with defects or cost or something appealing for manager$. In most cases, this literature is not really about software design, and is definitely not coming from people with a strong software design background (of course, they would disagree on that :-)
- Literature on software design principles and heuristics, or refactoring techniques. We still see some of this, but is mostly a rehashing of the same old stuff from the late '90s. The SOLID principles, the Law of Demeter, etc. In most cases, papers say nothing new, are based on toy problems, and are here just because many programmers won't read something that has been published more than a few months ago. If this is what's keeping software design literature alive, let's pull the plug.
- Literature on methods, like Design by Contract, TDD, Domain-Driven Design, etc. Here you find the occasional must-read work (usually a book from those who actually invented the approach), but just like philosophy, you see less and less of this literature in a maturing discipline. Then you find tons of advocacy on methods (or against methods), which would be more interesting if they involved real experiments (like the ones performed by Simula labs) and not just another toy projects in the capable hands of graduate students. Besides, experiments on design methods should be evaluated [mostly] by cost of change in the next few months/years, not exclusively by design principles. Advocacy may seem to keep literature alive, but it's just noise.
- Literature on platform-specific architectures. There is no dearth of that. From early EJB treatises to JBoss tutorials, you can find tons of papers on "enterprise architecture". Even Microsoft has some architectural literature on application blocks and stuff like that. Honestly, in most cases it looks more like Markitecture (marketing architecture as defined by Hohmann), promoting canned solutions and proposing that you adapt your problem to the architecture, which is sort of the opposite of real software design. The best works usually fall under the "design patterns" chapter (see below).
- Literature for architecture astronauts. This is a nice venue for both academics and vendors. You usually see heavily layered architectures using any possible standards and still proposing a few more, with all the right (and wrong) acronyms inside, and after a while you learn to turn quickly to the next page. It's not unusual to find papers appealing to money-saving managers who don't "get" software, proposing yet another combination of blueprint architectures and MDA tools so that you can "generate all your code" with a mouse click. Yeah, sure, whatever.
- Literature on design patterns. Most likely, this is what's keeping software design literature alive. A well-written pattern is about a real problem, the forces shaping the context, an effective solution, its consequences, etc. This is what software design is about, in practice. On the other hand, a couple of conferences every year can't keep design literature in perfect health.
- Literature on design in the context of agility (mostly about TDD). There is a lot of this on the net. Unfortunately, it's mostly about trivial problems, and even more unfortunately, there is rarely any discussion about the quality of the design itself (as if having tests was enough to declare the design "good"). The largest issue here is that it's basically impossible to talk about the design of anything non-trivial strictly from a code-based perspective. Note: I'm not saying that you can't design using only code as your material. I'm saying that when the problem scales slightly beyond the toy level, the amount of code that you would have to write and show to talk about design and design alternatives grows beyond the manageable. So, while code-centric practices are not killing design, they are nailing the coffin on design literature.
Geez, I am usually an optimist :-)), but this picture is bleak. I could almost paraphrase Richard Gabriel (of "Objects have failed" fame) and say that evidently, software design has failed to speak the truth, and therefore, software design narrative (literature) is dying. But that would not be me. I'm more like the opportunity guy. Perhaps we just need a different kind of literature on software design.
Something's missing
If you walk through the aisles of a large bookstore, and go to the "design & architecture" shelves, you'll all the literary kinds above, not about software, but about real-world stuff (from chairs to buildings). except on the design of physical objects too. Interestingly, you can also find a few morelike:
- Anthologies (presenting the work of several designers) and monographs on the work of famous designers. We are at loss here, because software design is not immediately visible, is not interesting for the general public, is often kept as a trade secret, etc.
I know only one attempt to come up with something similar for software: "Beautiful Architecture" by Diomidis Spinellis. It's a nice book, but it suffers from a lack of depth. I enjoyed reading it, but didn't come back with a new perspective on something, which is what I would be looking for in this kind of literature.
It would be interesting, for instance, to read more about the architecture of successful open-source projects, not from a fanboy perspective but through the eyes of experienced software designers. Any takers?
- Idea books. These may look like anthologies, but the focus is different. While an anthology is often associated with a discussion or critics of the designer's style, an idea book presents several different objects, often out of context, as a sort of creative stimulus. I don't know of anything similar for software design, though I've seen many similar book for "web design" (basically fancy web pages). In a sense, some literature on patterns comes close to being inspirational; at least, good domain-specific patterns sometimes do. But an idea book (or idea paper) would look different.
I guess anthologies and monographs are at odd with the software culture at large, with its focus on the whizz-bang technology of the day, little interest for the past, and often bordering on religious fervor about some products. But idea books (or most likely idea papers) could work, somehow.
Indeed, I would like to see more software design literature organized as follows:
- A real-world, or at least realistic problem is presented.
- Forces are discussed. Ideally, real-world forces.
- Possibly, a subset of the whole problem is selected. Real-world problems are too big to be dissected in a paper (or two, or three). You can't discuss the detailed design of a business system in a paper (lest you appeal only to architecture astronauts). You could, however, discuss a selected, challenging portion. Ideally, the problem (or the subset) or perhaps just the approach / solution, should be of some interest even outside the specific domain. Just because your problem arose in a deeply embedded environment doesn't mean I can't learn something useful for an enterprise web application (assuming I'm open minded, of course :-). Idea books / papers should trigger some lateral thinking on the reader, therefore unusual, provocative solutions would be great (unlike literature on patterns, where you are expected to report on well-known, sort of "traditional" solutions).
- Practical solutions are presented and scrutinized. I don't really care if you use code, UML, words, gestures, whatever. Still, I want some depth of scrutiny. I want to see more than one option discussed. I don't need working code. I'm probably working on a different language or platform, a different domain, with different constraints. I want fresh design ideas and new perspectives.
- In practice, a good design aims at keeping the cost of change low. This is why a real-world problem is preferable. Talking about the most likely changes and how they could be addressed in different scenarios beats babbling about tests and SOLID and the like. However, "most likely changes" is meaningless if the problem is not real.
Funny enough, there would be no natural place to publish something like that, except your own personal page. Sure, maybe JOT, maybe not. Maybe IEEE Software, maybe not. Maybe some conference, maybe not. But we don't have a software design journal with a large readers pool. This is part of the problem, of course, but also a consequence. Wrong feedback loop :-).
Interestingly, I proposed something similar years ago, when I was part of the editorial board of a software magazine. It never made a dent. I remember that a well-known author argued against the idea, on the basis that people were not interested in all this talking about ins-and-outs, design alternatives and stuff. They would rather have a single, comprehensive design presented that they could immediately use, preferably with some source code. Well, that's entirely possible; indeed, I don't really know how many software practitioners would be interested in this kind of literature. Sure, I can bet a few of you guys would be, but overall, perhaps just a small minority is looking for inspiration, and most are just looking for canned solutions.
Or maybe something else...
On the other hand, perhaps this style is just too stuck in the '90s to be appealing in 2011. It's unidirectional (author to readers), it's not "social", it's not fun. Maybe a design challenge would be more appealing. Or perhaps the media are obsolete, and we should move away from text and graphics and toward (e.g.) videos. I actually tried to watch some code kata videos, but the guys thought they were like this, but to an experienced designer they looked more like this. (and not even that funny). Maybe the next generation of software designers will come up with a better narrative style or media.
Do something!
I'm usually the "let's do something" kind of guy, and I've entertained the idea of starting a Software Design Gallery, or a Software Design Idea Book, or something, but honestly, it's a damn lot of work, and I'm a bit skeptical about the market size (even for a free book/paper/website/whatever). As usual, any feedback is welcome, and any action on your part even more :-)
Showing posts with label link. Show all posts
Showing posts with label link. Show all posts
Thursday, February 03, 2011
Thursday, August 19, 2010
Notes on Software Design, Chapter 9. A simple property: Distance (part 2)
What is Distance in the run-time world? As I began pondering on this, it turned out I could define distance in several meaningful ways. Some were redundant with other notions I have yet to present. Some were useless for a theory of software design. In the end, I picked up a very simple definition, with some interesting ramifications.
Just like the notion of distance in the artifact world is based on the hierarchy of artifacts, the notion of distance in the run-time world is based on a hierarchy of locations. These are the locations where executable knowledge is located at any given time. So, given two pieces of executable knowledge P1 and P2, we can define an ordinal scale:
P1 and P2 are inside the CPU registers or the CPU execution pipeline - that's minimum distance.
P1 and P2 are inside the same L1 cache line
P1 and P2 are inside the L1 cache
… etc
for the full scale, see the (updated) Summary at the Physics of Software website.
Dude, I don't care about cache lines!
Yeah, well, but the processor does, and I'm looking for a good model of the real-world, not for an abstract model of computing disconnected from reality (which would be more like the math of software, not the physics).
You may be writing your code in Java or C#, or even in C++, and be blissfully unaware of what is going on under the hood. But the caches are there, and their effect is absolutely visible. For a simple, experimental, and well-written summary, see Igor Ostrovsky's Gallery of Processor Cache Effects (the source code is in C#, but results wouldn't be different in Java or C++).
Interestingly enough, most algorithms and data structures are not optimized for modern processors with N levels of cache. Still, there is an active area of research on cache-oblivious algorithms which, despite the name, are supposed to perform well with any cache line size across any number of cache levels (you can find a few links to specialized algorithms here but you'll have to work around broken links).
What about virtual memory? Again, we can ignore the magic most of the times, but when we're looking for high-performance solutions, we have to deal with it. Unconvinced? Take a look at You're Doing It Wrong, where Poul-Henning Kamp explains (perhaps with a bit too much "I know it all and you don't" attitude :-)) why textbooks are not really talking about real-world computers [anymore].
Consequences
What happens when run-time distance grows? We're bound to see a staircase-like behavior, as in the second picture in the gallery above, just with more risers/treads. When you move outside your process, you have context switching. When you move outside your computer, you have network latency. When you move outside your LAN, you also have name lookup and packet hops. We'll understand all this stuff much better as we get to the concept of friction.
There is more. When talking about artifact distance, I said that coupling (between artifacts) should decrease as distance increases. In the run-time world, coupling to the underlying platform should decrease as distance increases. This must be partially reflected in the artifacts themselves, but it is also a language / platform / transformation concern.
Knowledge at short distance can be tightly coupled to a specific hw / sw platform. For instance, all code inside one component can be tightly bound to:
This is fine at some level. I can even accept the idea that all the components inside a single application have to share some underlying assumptions. Sure, it would be better if components were relatively immune from binary issues (a plague in the C++ world). But overall (depending on the size of the application) I can "control" things and make sure everything is aligned.
But when I'm talking to another service / application over the network, my degree of control is much smaller. If everything is platform-dependent (with a broad definition of platform, mind you: Java is a platform), we're in for major deployment / maintenance issues. Even worse, it would represent a huge platform lock-in (I can't use a different technology for a new service). Things get just worse on a global network scale. This is why XML took the world by storm, why web services have been rather successful in the real world, and so on. This is also why I like technologies / languages that take integration with other technologies / languages seriously, and not religiously.
As usual, the statement above is bi-directional. That is, it makes very little sense to pursue strong decoupling from the underlying platforms at micro-level. Having a class talking to itself in XML is not a brilliant strategy. Again, design is about balance: in this case, balance between efficiency and convenience on one side, and flexibility and evolvability on the other. Balance is obtained when you can depend on your platform locally, and be increasingly independent as you move farther.
Run-time Distance is not a constant
Not necessarily, anyway; it depends on small-scale technical choices. In C++, for instance, once you get two objects in the same cache line, they will stay there for their entire life, because identity in C++ is address-based. In a garbage collected environment, this is not true anymore: objects can move freely during collection.
Moreover, once we move from the cache line to the entire cache, things come and go, they become near and distant along time. This contributes to complex performance patterns, and indeed, modern hardware makes accurate performance prediction almost impossible. I'm pretty sure there are some interesting phenomena to be studied here - a concept of oscillating distance, perhaps the equivalent of a performance beat when two concurrent threads have slightly different oscillating frequency, and so on, but I'm not currently investigating any of this - it's just too early.
At some point, distance becomes more and more "constant". Sure, a local service may migrate to LAN and then to WAN, but usually it does so because of human intervention (decisions!), and may require changes on the artifact side as well. Short-range distance is a fluid notion, changing as executable knowledge is, well, executed :-).
By the way: distance in the artifact world is not a constant, either. It is constant when the code is frozen. As soon as we change it, we change some distance relationships. In other words, when the computer is processing executable knowledge, run-time distance changes. When we process encoded knowledge (artifacts), artifact distance changes.
Distance-preserving transformations
Knowledge encoded in artifacts can be transformed several times before becoming executable knowledge. Most of those transformations are distance-preserving, that is, they map nearby knowledge to nearby knowledge (although with some jumps here and there).
For instance, pure sequences of statements (without choices, iterations, calls) are "naturally" converted into sequential machine-level instructions that not only will likely sit in the same cache line, but won't break the prefetch pipeline either. Therefore, code that is near in the artifact world will end up near in the run-time world as well.
In the C/C++ family, data that is sequentially declared in a structure (POD) is sequential in memory as well. Therefore, there is a good chance of sharing the same cache line if you keep your structures small.
Conversely, data and code in different components are normally mapped to different pages (in virtual memory systems). They won't share the same cache line (they may compete for the same cache line, but won't be present in the same line at once). So distant things will be distant.
Even "recent" advances in processor architecture are increasing the similarity between the artifact and run-time class of distance. Consider predicated execution: it's commonly used to remove branches at machine-level for short sequence of statements in if/else conditionals (see Fig. 1 in the linked paper). In terms of distance, it allows nearby code in the artifact space to stay close in the run-time space, by eliminating branches and therefore maximizing proximity in the execution pipeline.
Some transformations, however, are not distance-preserving. Inlining of code (think of C++ inline functions, for instance) will shrink distance while moving from the artifact world to the run-time world.
Aspect Oriented Programming is particularly interesting from the point of view of distance. On the artifact side, aspects allow to isolate cross-cutting concerns. Therefore, they allow to increase distance between the advice, that is factored out, and the join points. A non-distance-preserving transformation (weaving) brings the two concepts back together as we move toward execution.
Curiously enough, some non-preserving transformations work in the opposite way: they allow things to be near in the artifact space, yet be distant in the run-time world. Consider the numerous technologies (dating back to remote procedure calls) that allow you to code as if you were invoking a local function, or a method of a local object, while in fact you are executing a remote function, or a method of a remote object (through some kind of local proxy). This is creating an illusion of short distance (in the artifact world) while in fact maintaining high distance in the run-time world. As usual, whenever we create software over a thin layer of illusion, there is a potential for problems. When you look at The 8 fallacies of distributed computing , you can immediately recognize that dealing with remote objects as if they were local objects can be rather dangerous. Said otherwise, the illusion of short distance is a leaky abstraction. More on distributed computing when I'll get to the concept of friction.
A closing remark: in my previous post, I said that gravity (in the artifact world) tends to increase performance (a run-time property). We can now understand that better, and say that it is largely (not entirely) because:
- the most common transformations are distance-preserving.
- performance increases as the run-time distance decreases.
Again, friction is also at play here, but I have yet to introduce the concept.
Addenda on the artifact side
Some concepts on the artifact side are meant to give the illusion of a shorter distance, while maintaining separation. Consider extension methods in .NET or the more powerful concept of category in objective C. They both give the illusion of being very close to a class (when you use them) while in fact they are just as distant as any other class. (By the way: I've been playing with extension methods [in C#] as a way to get something like partial specialization in C++; it kinda works, but not inside generics, which is exactly were I would need it).
Distance in the Decision Space
While thinking about distance in the artifact and in the run-time world, I realized that the very first notion of distance I introduced was in the decision space. Still, I haven't defined that notion at the detail level (or lack thereof :-) at which I've defined distance in the artifact and run-time world. I have a few ideas, of course, the simplest definition being "the number of decision that must be undone + the number of [irreversible?] decisions that must be taken to move your artifacts". Those decisions would involve some mass of code. Moving that mass for that "space" would give a notion of necessary work. Anyway, it's probably too early to say more, as I have to understand the decision space better.
Coming soon, I hope, the notion of friction.
Just like the notion of distance in the artifact world is based on the hierarchy of artifacts, the notion of distance in the run-time world is based on a hierarchy of locations. These are the locations where executable knowledge is located at any given time. So, given two pieces of executable knowledge P1 and P2, we can define an ordinal scale:
P1 and P2 are inside the CPU registers or the CPU execution pipeline - that's minimum distance.
P1 and P2 are inside the same L1 cache line
P1 and P2 are inside the L1 cache
… etc
for the full scale, see the (updated) Summary at the Physics of Software website.
Dude, I don't care about cache lines!
Yeah, well, but the processor does, and I'm looking for a good model of the real-world, not for an abstract model of computing disconnected from reality (which would be more like the math of software, not the physics).
You may be writing your code in Java or C#, or even in C++, and be blissfully unaware of what is going on under the hood. But the caches are there, and their effect is absolutely visible. For a simple, experimental, and well-written summary, see Igor Ostrovsky's Gallery of Processor Cache Effects (the source code is in C#, but results wouldn't be different in Java or C++).
Interestingly enough, most algorithms and data structures are not optimized for modern processors with N levels of cache. Still, there is an active area of research on cache-oblivious algorithms which, despite the name, are supposed to perform well with any cache line size across any number of cache levels (you can find a few links to specialized algorithms here but you'll have to work around broken links).
What about virtual memory? Again, we can ignore the magic most of the times, but when we're looking for high-performance solutions, we have to deal with it. Unconvinced? Take a look at You're Doing It Wrong, where Poul-Henning Kamp explains (perhaps with a bit too much "I know it all and you don't" attitude :-)) why textbooks are not really talking about real-world computers [anymore].
Consequences
What happens when run-time distance grows? We're bound to see a staircase-like behavior, as in the second picture in the gallery above, just with more risers/treads. When you move outside your process, you have context switching. When you move outside your computer, you have network latency. When you move outside your LAN, you also have name lookup and packet hops. We'll understand all this stuff much better as we get to the concept of friction.
There is more. When talking about artifact distance, I said that coupling (between artifacts) should decrease as distance increases. In the run-time world, coupling to the underlying platform should decrease as distance increases. This must be partially reflected in the artifacts themselves, but it is also a language / platform / transformation concern.
Knowledge at short distance can be tightly coupled to a specific hw / sw platform. For instance, all code inside one component can be tightly bound to:
- An internal object model, say the C++ object model of a specific compiler version, or the C# or Java object model for a specific version of the virtual machine.
- A specific operating system, if not virtualized (native code).
- A specific hardware, if not virtualized.
This is fine at some level. I can even accept the idea that all the components inside a single application have to share some underlying assumptions. Sure, it would be better if components were relatively immune from binary issues (a plague in the C++ world). But overall (depending on the size of the application) I can "control" things and make sure everything is aligned.
But when I'm talking to another service / application over the network, my degree of control is much smaller. If everything is platform-dependent (with a broad definition of platform, mind you: Java is a platform), we're in for major deployment / maintenance issues. Even worse, it would represent a huge platform lock-in (I can't use a different technology for a new service). Things get just worse on a global network scale. This is why XML took the world by storm, why web services have been rather successful in the real world, and so on. This is also why I like technologies / languages that take integration with other technologies / languages seriously, and not religiously.
As usual, the statement above is bi-directional. That is, it makes very little sense to pursue strong decoupling from the underlying platforms at micro-level. Having a class talking to itself in XML is not a brilliant strategy. Again, design is about balance: in this case, balance between efficiency and convenience on one side, and flexibility and evolvability on the other. Balance is obtained when you can depend on your platform locally, and be increasingly independent as you move farther.
Run-time Distance is not a constant
Not necessarily, anyway; it depends on small-scale technical choices. In C++, for instance, once you get two objects in the same cache line, they will stay there for their entire life, because identity in C++ is address-based. In a garbage collected environment, this is not true anymore: objects can move freely during collection.
Moreover, once we move from the cache line to the entire cache, things come and go, they become near and distant along time. This contributes to complex performance patterns, and indeed, modern hardware makes accurate performance prediction almost impossible. I'm pretty sure there are some interesting phenomena to be studied here - a concept of oscillating distance, perhaps the equivalent of a performance beat when two concurrent threads have slightly different oscillating frequency, and so on, but I'm not currently investigating any of this - it's just too early.
At some point, distance becomes more and more "constant". Sure, a local service may migrate to LAN and then to WAN, but usually it does so because of human intervention (decisions!), and may require changes on the artifact side as well. Short-range distance is a fluid notion, changing as executable knowledge is, well, executed :-).
By the way: distance in the artifact world is not a constant, either. It is constant when the code is frozen. As soon as we change it, we change some distance relationships. In other words, when the computer is processing executable knowledge, run-time distance changes. When we process encoded knowledge (artifacts), artifact distance changes.
Distance-preserving transformations
Knowledge encoded in artifacts can be transformed several times before becoming executable knowledge. Most of those transformations are distance-preserving, that is, they map nearby knowledge to nearby knowledge (although with some jumps here and there).
For instance, pure sequences of statements (without choices, iterations, calls) are "naturally" converted into sequential machine-level instructions that not only will likely sit in the same cache line, but won't break the prefetch pipeline either. Therefore, code that is near in the artifact world will end up near in the run-time world as well.
In the C/C++ family, data that is sequentially declared in a structure (POD) is sequential in memory as well. Therefore, there is a good chance of sharing the same cache line if you keep your structures small.
Conversely, data and code in different components are normally mapped to different pages (in virtual memory systems). They won't share the same cache line (they may compete for the same cache line, but won't be present in the same line at once). So distant things will be distant.
Even "recent" advances in processor architecture are increasing the similarity between the artifact and run-time class of distance. Consider predicated execution: it's commonly used to remove branches at machine-level for short sequence of statements in if/else conditionals (see Fig. 1 in the linked paper). In terms of distance, it allows nearby code in the artifact space to stay close in the run-time space, by eliminating branches and therefore maximizing proximity in the execution pipeline.
Some transformations, however, are not distance-preserving. Inlining of code (think of C++ inline functions, for instance) will shrink distance while moving from the artifact world to the run-time world.
Aspect Oriented Programming is particularly interesting from the point of view of distance. On the artifact side, aspects allow to isolate cross-cutting concerns. Therefore, they allow to increase distance between the advice, that is factored out, and the join points. A non-distance-preserving transformation (weaving) brings the two concepts back together as we move toward execution.
Curiously enough, some non-preserving transformations work in the opposite way: they allow things to be near in the artifact space, yet be distant in the run-time world. Consider the numerous technologies (dating back to remote procedure calls) that allow you to code as if you were invoking a local function, or a method of a local object, while in fact you are executing a remote function, or a method of a remote object (through some kind of local proxy). This is creating an illusion of short distance (in the artifact world) while in fact maintaining high distance in the run-time world. As usual, whenever we create software over a thin layer of illusion, there is a potential for problems. When you look at The 8 fallacies of distributed computing , you can immediately recognize that dealing with remote objects as if they were local objects can be rather dangerous. Said otherwise, the illusion of short distance is a leaky abstraction. More on distributed computing when I'll get to the concept of friction.
A closing remark: in my previous post, I said that gravity (in the artifact world) tends to increase performance (a run-time property). We can now understand that better, and say that it is largely (not entirely) because:
- the most common transformations are distance-preserving.
- performance increases as the run-time distance decreases.
Again, friction is also at play here, but I have yet to introduce the concept.
Addenda on the artifact side
Some concepts on the artifact side are meant to give the illusion of a shorter distance, while maintaining separation. Consider extension methods in .NET or the more powerful concept of category in objective C. They both give the illusion of being very close to a class (when you use them) while in fact they are just as distant as any other class. (By the way: I've been playing with extension methods [in C#] as a way to get something like partial specialization in C++; it kinda works, but not inside generics, which is exactly were I would need it).
Distance in the Decision Space
While thinking about distance in the artifact and in the run-time world, I realized that the very first notion of distance I introduced was in the decision space. Still, I haven't defined that notion at the detail level (or lack thereof :-) at which I've defined distance in the artifact and run-time world. I have a few ideas, of course, the simplest definition being "the number of decision that must be undone + the number of [irreversible?] decisions that must be taken to move your artifacts". Those decisions would involve some mass of code. Moving that mass for that "space" would give a notion of necessary work. Anyway, it's probably too early to say more, as I have to understand the decision space better.
Coming soon, I hope, the notion of friction.
Tuesday, July 27, 2010
On Kent Beck's Responsive Design
I try to keep an eye on software design literature. I subscribe to relevant publications from IEEE and ACM; I get a copy of conference proceedings; I read blogs and, just like everybody else, I follow links and suggestions. Being aware of the potential risk of filtering out information that is not aligned with the way I think, I'm also following a few blogs that are definitely not aligned with my belief system, just to make sure I don't miss too much. I'm bound to miss something, anyway, but I can live with that :-).
A few weeks ago I realized that I wasn't following Kent Beck's blog. I don't know why: I'm not a fan of XP, but I'm rather fond of Kent. He's an experienced designer with many good ideas. So, I took a tour of his posts and discovered that I had completely missed the concept of Responsive Design. That's weird, because I'm reading/scanning quite a few agile-oriented blogs, and I've never seen any mention of it. Oh, well; time to catch up.
I did my homework and spent some time reading all posts in the Responsive Design category and watching Kent's interesting QCon presentation. Looking at blog dates, it seems like activity peaked in April 2009, but rapidly declined during 2009 to almost nothing in 2010. So, apparently, Responsive Design hasn't taken the world by storm yet. Well, design wasn't so popular in the pre-agile era, and it's not going to be popular in the post-agile era either :-).
Anyway, the presentation inspired me with a stream of reflections that I'd like to share with you guys. I would recommend that you watch the presentation first (I know, it takes about 1 hour, but I think it's worth it). What follows is not intended to be a critic. I'm mostly trying to relate Kent's perspective on "what software design is about" with mine. It's a rather long post, so I split it in three paragraphs. The first two are about concepts in the presentation itself. The third is about an interesting difference in perspective, and how it affects our thinking.
Reflections on the introductory part
The idea of taking notes as we design, to uncover our own strategies and tactics, sounded so familiar. I guess at some point some of us feel an urge to understand what we're really doing. Although I'm looking for something different from Kent, I share some of his worries: trivial or extremely complicated insights might be just around the corner :-)
The talk about the meaning of "responsive" around 0:13 is resonating very well with the concepts of forcefield, design as "shaping a material", and backtalk. From my perspective, Kent is mostly saying: am I going to look at the forcefield, and design accordingly, or am I going to force a preconceived set of design decisions upon the problem? (see also my empty cup post).
Steady flow of features. We all love that, of course, and in a sense it's the most agile, XP-ish concept in the entire talk. The part about "the right time to design" seems rather connected with the Least Responsible Moment, so I can't really spare you a link to my own little idea of Invariant Decisions.
Again, I have a feeling that there is never enough context when talking about timing. I've certainly designed systems with a large separation (in time) between design and implementation. In many cases, it worked out quite well (of course, we always considered design as something fluid, not cast in stone). I guess it has a lot to do with how much domain knowledge you can leverage. Not every project is about doing something nobody has done before. Many are "just" about doing something in a much better way. Context, context, context...
Starting with values: it's something I have to improve in my Physics of Software work. Although I've said several times that all new methods should start with a declaration of values and beliefs, so far I haven't been thorough on this side. For instance, I value honest, unbiased, free, creative communication and thinking about software design issues, where the best ideas, and not the best talkers, get to win (just because I'm a good talker doesn't mean I want to win that way :-)). That's why I'm trying to come up with a less ambiguous, wishy-washy way to think and talk about design.
"Most of the decisions I make while designing have nothing to do with the problem domain [… but ...] are shaped by the fact that I'm instructing a computer. Weird. This is not my experience. I surely reuse solutions across domains, but the problem domain is heavily shaping the forcefield. When you go down to small-scale decisions, yeah, it gets more domain-independent, but high-level design is heavily problem-dependent. For instance, while you can usually (but not always) implement a chosen data structure in a rather domain-independent way, choosing the right data structure is usually a domain-dependent choice.
Don't get me wrong: I know I can safely ignore some portions of the problem domain when I design - I do that all the time. My view is that I can ignore the (possibly very complex) functional issues that have little impact on the forcefield, and therefore on the form. This is a complex subject that would require an in-depth study.
Principles. I don't quite like the term "principle" because is so wide in meaning that you can use it for just about everything. So, while the "don't repeat yourself" is a commonly accepted design principle, Kent's examples of principles from the insurance world or from Dynamo looks much more like pre-made [meta] decisions to me.
"You should never to lose a write". Fine. It's a decision, you can basically consider that a requirement.
"We're there to aid human decision making". Fine. It's a meta-decision, meaning, it's not just a design decision: it will influence everything, from your value proposition down to functional requirements and so on. It's not really a design principle, not even a project-specific design principle: it's more akin to a metaphor or to a goal.
Aside from that distinction, I agree that sometimes constraints, whatever form they take, are actually helpful during design, because they prune out a lot of the decision space (again, see my post above on the empty cup). Of course, the wrong constraints can prevent you from finding the sweet spot, so timing is an issue here as well - some constraints should be considered fluid early on, because you may learn that they're not the right constraints for your project.
The short part about patterns and principles reminded me of my early work on SysOOD, some of which got published in IEEE Software exactly as Principles Vs. Patterns. So, while Kent wants to understand principles better, I've always been trying to eliminate principles altogether (universal principles, that is). Still, I'd like to see the "non-universal" or "project-specific" principles recast under some other name and further explored, because I agree that having a shared set of values / goals / constraints can help tremendously in any significant project. Oh, I still have the Byte smalltalk balloon issue somewhere : ).
Reflection on the 4 strategies
I see the "meat" of the talk as being about [meta] strategies to move in the decision space. Your software is at some point A in the multi-dimensional decision space; for instance, you decided early on that you would use filenames and not streams. Now you want to move your software to another point B in the decision space (where, for instance, streams are used instead of filenames). How do you go from A to B?
That's an interesting issue, where experienced designers/programmers routinely adopt different approaches compared to novices (see the empty cup post again), so it's both interesting and promising to see Kent articulate his reasoning. In a sense, it's completely orthogonal to both patterns/antipatterns (which may point you to B / away from A) and to my work (which is more concerned with understanding and articulating why B is a more desirable point than A, or to find B in the first place).
Actually, strictly speaking I'm not even sure this stuff is really about "design". In my view, design is more about finding B. The execution/implementation of design, however, is about moving to B. Therefore, Kent's strategies looks more like "design execution strategies" than "design strategies".
That said, there is a subtle, yet striking difference between Kent's perspective and mine. Kent is talking about / looking to software design in a "first-person perspective", while I'm looking through the "I'm shaping a material" metaphor. He says things like: "I'm here; how do I get there?", and he's literally mimicking small steps. I'm saying things like: "my software is here; how do I move it there?"; it may seem like a small, irrelevant distinction, but I think it's shaping our thoughts rather deeply. It surely shapes the names we use: I would have never used "stepping stone", because I'm not thinking of myself going anywhere : ). More on this on the third paragraph below.
Although this is probably the most “practical” part of the talk, concepts are a little fuzzy. Leap can be seen as a strategy, actually the simplest strategy of all: you just move there. Safe Steps is more like a meta-strategy to move inside the decision space (indeed, around 42:00 Kent talks about Safe/Small Steps as a "principle"). Once you choose Safe Steps, you have many options left, Parallel, Stepping Stones and Simplifications being 3 of them. Still, the granularity of those strategies is so coarse that it's rather surprising Kent found only 3 of them.
For instance, I probably wouldn't have used Parallel to deal with the filenames/streams issues. I can't really say without looking at the code, but assuming I wanted to take small steps, I could have created a new class, which IS-A stream and HAS-A filename, moved the existing code to use that class (a very simple and safe change, because both the stream and the filename are there), gradually moved all the filename-dependent code inside that class, removed the filename from the interface (as it's no longer used), moved the filename-dependent code in a factory (so that I could make a choice on the stream type). At that point I could have killed the temporary class, using a plain stream again. This strategy (I could call it wrap/extract/unwrap) is quite simple and effective when dealing with legacy code, and doesn't really fit in any of the strategies Kent is proposing.
Stepping Stone. While Leap and Parallel are more like strategies to implement a decision you have already taken, it seems to me that Stepping Stone is different. Although this might be my biased understanding of it, it's also the only interpretation I can give that makes Stepping Stone different from good old top-down decomposition (which Kent makes clear is different, at about 50:30).
I would say that Stepping Stone is about making progress where you can, and while doing so, shed some light on the surroundings as well. This is not very clear in the beginning, but as Kent moves forward, it's more and more obvious that he's thinking about novel situations, where you don't know exactly what you want to build.
Indeed, sometimes the forcefield is just too blurry. Still, we often can see a portion of it rather clearly, and we may have a sensible idea about the form we want to shape locally. Stepping Stone is a good strategy here: build (or design) what you understand, get feedback (or backtalk), get progress, etc. I usually keep exploring the dark areas while I'm building stepping stones. I'm also rather careful not to overcommit on stepping stones, as they may turn out not to fit perfectly with what's coming next. That's fine, inasmuch as we understand that design is a process, not a phase with a beginning and an end.
I can also build "stepping stones" when I know where I'm going to, but as I said, at that point it's hard to tell the difference between building stepping stones and executing top-down design. Oh, by the way, for the agile guys who hated me so much for saying that TDD is a limited design strategy: move around 50:20. Play. Rewind. Play again. Repeat till necessary :-). But back to Stepping Stones: I think the key phrase here is around 55:40, when Kent says that by standing on a stepping stone he can see where to go next, which I read like: a previously blurry portion of the forcefield is now visible and clear, and I can move further.
Simplification. This is more akin to a design strategy, not to an execution strategy. I could see Simplification as a way to create small steps (perhaps even stepping stones). Indeed, the difference between Stepping Stones and Simplification is not really obvious (around 1:02:00 a participant asks about the similarity and doesn't seem like Kent can explain the difference so well either). I guess simplification is about to solve one single problem by solving a simpler version of the same problem first, whereas Stepping Stones are about solving a complex problem by solving another problem first (one that, however, brings us closer to solving the initial problem). In this sense, Simplification can be used where you can see where you wanna go (but it's too big / risky / ambitious) but also where you can't see exactly where you want to go, but you can clearly see a smaller version of it. Or, from my "I'm shaping a material" perspective, you can't really see the form of the real thing, but you can see the form of a smaller version of it. As I said, this stuff is interesting but rather fuzzy, and honestly, I'm not really sold on those 4 strategies.
Overall, I'd like to see Responsive Design further developed. It would probably benefit from a better distinction between design, design execution, principles, values, goals, and so on, but it's interesting stuff in the almost flat landscape of contemporary software design literature.
Perspective Matters
As I said, Kent is looking at the decision space in first-person perspective, while I see myself moving other things around. There is an interesting implication: when you think about moving yourself, you mostly consider distance. You are who you are. You want to go elsewhere. Distance, and therefore Leaps, Small Steps, Stepping Stones, it's all that matters.
When you think about moving something else, you have to consider two more factors. That, in my opinion, allows better reasoning.
Sure, distance is an important factor. But, as I explained in my post about Inertia, another important factor is Mass. You cannot ignore mass, but it's unnatural to consider a varying mass from a first-person perspective.
A simple example: I use Visual Studio (yeah yeah I know, some of you don't like it : ). Sometimes, as I'm creating a new project, I click in the wrong place (yeah yeah I'm kinda dumb :-) and I get the wrong project type. I want a C# project and I get a VB.NET project (yikes!! :-)). That's quite a distance in the decision space. Yet there is a very simple, safe step (Leap): delete the project and start from scratch. The distance is not a problem, because mass is basically null. I can use Leap not because the distance is small, but because the mass is small. I wouldn't do that with an existing, large project (I may consider going through an IL -> C# transformation though :-).
There is more: when I discussed Inertia, I talked about software being in a state of rest or motion in the decision space. This is the third factor, and again, it's hard to think about it in first-person perspective. You want to go somewhere, but you're already going elsewhere. It's not truly natural. Deflecting a moving object, however, is quite ordinary (most real-life games involving a ball, for instance, involve deflecting a moving object).
Consider a large project. Maybe a team is busy moving a subsystem S1 to another place in the decision space. Now you want to move subsystem S2 from A to B. It's not enough to consider distance and mass. You have to consider whether or not S2 is already moving, perhaps by attraction from S1. So S2 may be small (in mass) and the distance A-B may be small too, but if S2 is already moving into a different direction your job is harder, more risky, and in need of a careful execution strategy.
A real-world example: you have a legacy C++/MFC application. Currently, the UI is talking with the application through a small, homebrew messaging component (MC), of which you are in charge. You just found a better messaging component (BMC) on the internet, say something that is more efficient, portable, and actively developed by a community. You want to move your subsystem to another place in the decision space, from MC to BMC. Part of BMC being "portable" is based on the reasonable assumption that messages are strictly C++ objects.
Meanwhile, another team is moving the UI toward .NET (that's a big mass - big distance issue, but it's their problem, not yours :-). Now, there is an attraction between your messaging component and the UI. A .NET UI may have some issues dealing with native C++ messages. So, like it or not, there is another force at play, trying to move your component to a different place (say, NMC). Now, the mass of the messaging component might be small. The distance between MC and BMC may be small too - perhaps something you can deal with by using an adapter. But probably BMC is in a very different direction than NMC, where the UI team is pushing your component to move. So it's not just a matter of distance. It's not even enough to consider mass. You need to consider distance + full-blown Inertia. Or, to cut it short, work
Of course, depending on your project, some factor may dominate over the others, and a simpler model might be enough, but usually, distance alone won't cut it.
There is more...
At the end of the presentation, Kent mentions power law distribution. This is something that has got me interested too in the past few months (I mentioned that in Facebook), but it's too early to say something about it, and I want to check a few ideas with the author of an interesting paper first, anyway.
See you guys soon with a post on Distance (not in the decision space, but in the artifact space!)
A few weeks ago I realized that I wasn't following Kent Beck's blog. I don't know why: I'm not a fan of XP, but I'm rather fond of Kent. He's an experienced designer with many good ideas. So, I took a tour of his posts and discovered that I had completely missed the concept of Responsive Design. That's weird, because I'm reading/scanning quite a few agile-oriented blogs, and I've never seen any mention of it. Oh, well; time to catch up.
I did my homework and spent some time reading all posts in the Responsive Design category and watching Kent's interesting QCon presentation. Looking at blog dates, it seems like activity peaked in April 2009, but rapidly declined during 2009 to almost nothing in 2010. So, apparently, Responsive Design hasn't taken the world by storm yet. Well, design wasn't so popular in the pre-agile era, and it's not going to be popular in the post-agile era either :-).
Anyway, the presentation inspired me with a stream of reflections that I'd like to share with you guys. I would recommend that you watch the presentation first (I know, it takes about 1 hour, but I think it's worth it). What follows is not intended to be a critic. I'm mostly trying to relate Kent's perspective on "what software design is about" with mine. It's a rather long post, so I split it in three paragraphs. The first two are about concepts in the presentation itself. The third is about an interesting difference in perspective, and how it affects our thinking.
Reflections on the introductory part
The idea of taking notes as we design, to uncover our own strategies and tactics, sounded so familiar. I guess at some point some of us feel an urge to understand what we're really doing. Although I'm looking for something different from Kent, I share some of his worries: trivial or extremely complicated insights might be just around the corner :-)
The talk about the meaning of "responsive" around 0:13 is resonating very well with the concepts of forcefield, design as "shaping a material", and backtalk. From my perspective, Kent is mostly saying: am I going to look at the forcefield, and design accordingly, or am I going to force a preconceived set of design decisions upon the problem? (see also my empty cup post).
Steady flow of features. We all love that, of course, and in a sense it's the most agile, XP-ish concept in the entire talk. The part about "the right time to design" seems rather connected with the Least Responsible Moment, so I can't really spare you a link to my own little idea of Invariant Decisions.
Again, I have a feeling that there is never enough context when talking about timing. I've certainly designed systems with a large separation (in time) between design and implementation. In many cases, it worked out quite well (of course, we always considered design as something fluid, not cast in stone). I guess it has a lot to do with how much domain knowledge you can leverage. Not every project is about doing something nobody has done before. Many are "just" about doing something in a much better way. Context, context, context...
Starting with values: it's something I have to improve in my Physics of Software work. Although I've said several times that all new methods should start with a declaration of values and beliefs, so far I haven't been thorough on this side. For instance, I value honest, unbiased, free, creative communication and thinking about software design issues, where the best ideas, and not the best talkers, get to win (just because I'm a good talker doesn't mean I want to win that way :-)). That's why I'm trying to come up with a less ambiguous, wishy-washy way to think and talk about design.
"Most of the decisions I make while designing have nothing to do with the problem domain [… but ...] are shaped by the fact that I'm instructing a computer. Weird. This is not my experience. I surely reuse solutions across domains, but the problem domain is heavily shaping the forcefield. When you go down to small-scale decisions, yeah, it gets more domain-independent, but high-level design is heavily problem-dependent. For instance, while you can usually (but not always) implement a chosen data structure in a rather domain-independent way, choosing the right data structure is usually a domain-dependent choice.
Don't get me wrong: I know I can safely ignore some portions of the problem domain when I design - I do that all the time. My view is that I can ignore the (possibly very complex) functional issues that have little impact on the forcefield, and therefore on the form. This is a complex subject that would require an in-depth study.
Principles. I don't quite like the term "principle" because is so wide in meaning that you can use it for just about everything. So, while the "don't repeat yourself" is a commonly accepted design principle, Kent's examples of principles from the insurance world or from Dynamo looks much more like pre-made [meta] decisions to me.
"You should never to lose a write". Fine. It's a decision, you can basically consider that a requirement.
"We're there to aid human decision making". Fine. It's a meta-decision, meaning, it's not just a design decision: it will influence everything, from your value proposition down to functional requirements and so on. It's not really a design principle, not even a project-specific design principle: it's more akin to a metaphor or to a goal.
Aside from that distinction, I agree that sometimes constraints, whatever form they take, are actually helpful during design, because they prune out a lot of the decision space (again, see my post above on the empty cup). Of course, the wrong constraints can prevent you from finding the sweet spot, so timing is an issue here as well - some constraints should be considered fluid early on, because you may learn that they're not the right constraints for your project.
The short part about patterns and principles reminded me of my early work on SysOOD, some of which got published in IEEE Software exactly as Principles Vs. Patterns. So, while Kent wants to understand principles better, I've always been trying to eliminate principles altogether (universal principles, that is). Still, I'd like to see the "non-universal" or "project-specific" principles recast under some other name and further explored, because I agree that having a shared set of values / goals / constraints can help tremendously in any significant project. Oh, I still have the Byte smalltalk balloon issue somewhere : ).
Reflection on the 4 strategies
I see the "meat" of the talk as being about [meta] strategies to move in the decision space. Your software is at some point A in the multi-dimensional decision space; for instance, you decided early on that you would use filenames and not streams. Now you want to move your software to another point B in the decision space (where, for instance, streams are used instead of filenames). How do you go from A to B?
That's an interesting issue, where experienced designers/programmers routinely adopt different approaches compared to novices (see the empty cup post again), so it's both interesting and promising to see Kent articulate his reasoning. In a sense, it's completely orthogonal to both patterns/antipatterns (which may point you to B / away from A) and to my work (which is more concerned with understanding and articulating why B is a more desirable point than A, or to find B in the first place).
Actually, strictly speaking I'm not even sure this stuff is really about "design". In my view, design is more about finding B. The execution/implementation of design, however, is about moving to B. Therefore, Kent's strategies looks more like "design execution strategies" than "design strategies".
That said, there is a subtle, yet striking difference between Kent's perspective and mine. Kent is talking about / looking to software design in a "first-person perspective", while I'm looking through the "I'm shaping a material" metaphor. He says things like: "I'm here; how do I get there?", and he's literally mimicking small steps. I'm saying things like: "my software is here; how do I move it there?"; it may seem like a small, irrelevant distinction, but I think it's shaping our thoughts rather deeply. It surely shapes the names we use: I would have never used "stepping stone", because I'm not thinking of myself going anywhere : ). More on this on the third paragraph below.
Although this is probably the most “practical” part of the talk, concepts are a little fuzzy. Leap can be seen as a strategy, actually the simplest strategy of all: you just move there. Safe Steps is more like a meta-strategy to move inside the decision space (indeed, around 42:00 Kent talks about Safe/Small Steps as a "principle"). Once you choose Safe Steps, you have many options left, Parallel, Stepping Stones and Simplifications being 3 of them. Still, the granularity of those strategies is so coarse that it's rather surprising Kent found only 3 of them.
For instance, I probably wouldn't have used Parallel to deal with the filenames/streams issues. I can't really say without looking at the code, but assuming I wanted to take small steps, I could have created a new class, which IS-A stream and HAS-A filename, moved the existing code to use that class (a very simple and safe change, because both the stream and the filename are there), gradually moved all the filename-dependent code inside that class, removed the filename from the interface (as it's no longer used), moved the filename-dependent code in a factory (so that I could make a choice on the stream type). At that point I could have killed the temporary class, using a plain stream again. This strategy (I could call it wrap/extract/unwrap) is quite simple and effective when dealing with legacy code, and doesn't really fit in any of the strategies Kent is proposing.
Stepping Stone. While Leap and Parallel are more like strategies to implement a decision you have already taken, it seems to me that Stepping Stone is different. Although this might be my biased understanding of it, it's also the only interpretation I can give that makes Stepping Stone different from good old top-down decomposition (which Kent makes clear is different, at about 50:30).
I would say that Stepping Stone is about making progress where you can, and while doing so, shed some light on the surroundings as well. This is not very clear in the beginning, but as Kent moves forward, it's more and more obvious that he's thinking about novel situations, where you don't know exactly what you want to build.
Indeed, sometimes the forcefield is just too blurry. Still, we often can see a portion of it rather clearly, and we may have a sensible idea about the form we want to shape locally. Stepping Stone is a good strategy here: build (or design) what you understand, get feedback (or backtalk), get progress, etc. I usually keep exploring the dark areas while I'm building stepping stones. I'm also rather careful not to overcommit on stepping stones, as they may turn out not to fit perfectly with what's coming next. That's fine, inasmuch as we understand that design is a process, not a phase with a beginning and an end.
I can also build "stepping stones" when I know where I'm going to, but as I said, at that point it's hard to tell the difference between building stepping stones and executing top-down design. Oh, by the way, for the agile guys who hated me so much for saying that TDD is a limited design strategy: move around 50:20. Play. Rewind. Play again. Repeat till necessary :-). But back to Stepping Stones: I think the key phrase here is around 55:40, when Kent says that by standing on a stepping stone he can see where to go next, which I read like: a previously blurry portion of the forcefield is now visible and clear, and I can move further.
Simplification. This is more akin to a design strategy, not to an execution strategy. I could see Simplification as a way to create small steps (perhaps even stepping stones). Indeed, the difference between Stepping Stones and Simplification is not really obvious (around 1:02:00 a participant asks about the similarity and doesn't seem like Kent can explain the difference so well either). I guess simplification is about to solve one single problem by solving a simpler version of the same problem first, whereas Stepping Stones are about solving a complex problem by solving another problem first (one that, however, brings us closer to solving the initial problem). In this sense, Simplification can be used where you can see where you wanna go (but it's too big / risky / ambitious) but also where you can't see exactly where you want to go, but you can clearly see a smaller version of it. Or, from my "I'm shaping a material" perspective, you can't really see the form of the real thing, but you can see the form of a smaller version of it. As I said, this stuff is interesting but rather fuzzy, and honestly, I'm not really sold on those 4 strategies.
Overall, I'd like to see Responsive Design further developed. It would probably benefit from a better distinction between design, design execution, principles, values, goals, and so on, but it's interesting stuff in the almost flat landscape of contemporary software design literature.
Perspective Matters
As I said, Kent is looking at the decision space in first-person perspective, while I see myself moving other things around. There is an interesting implication: when you think about moving yourself, you mostly consider distance. You are who you are. You want to go elsewhere. Distance, and therefore Leaps, Small Steps, Stepping Stones, it's all that matters.
When you think about moving something else, you have to consider two more factors. That, in my opinion, allows better reasoning.
Sure, distance is an important factor. But, as I explained in my post about Inertia, another important factor is Mass. You cannot ignore mass, but it's unnatural to consider a varying mass from a first-person perspective.
A simple example: I use Visual Studio (yeah yeah I know, some of you don't like it : ). Sometimes, as I'm creating a new project, I click in the wrong place (yeah yeah I'm kinda dumb :-) and I get the wrong project type. I want a C# project and I get a VB.NET project (yikes!! :-)). That's quite a distance in the decision space. Yet there is a very simple, safe step (Leap): delete the project and start from scratch. The distance is not a problem, because mass is basically null. I can use Leap not because the distance is small, but because the mass is small. I wouldn't do that with an existing, large project (I may consider going through an IL -> C# transformation though :-).
There is more: when I discussed Inertia, I talked about software being in a state of rest or motion in the decision space. This is the third factor, and again, it's hard to think about it in first-person perspective. You want to go somewhere, but you're already going elsewhere. It's not truly natural. Deflecting a moving object, however, is quite ordinary (most real-life games involving a ball, for instance, involve deflecting a moving object).
Consider a large project. Maybe a team is busy moving a subsystem S1 to another place in the decision space. Now you want to move subsystem S2 from A to B. It's not enough to consider distance and mass. You have to consider whether or not S2 is already moving, perhaps by attraction from S1. So S2 may be small (in mass) and the distance A-B may be small too, but if S2 is already moving into a different direction your job is harder, more risky, and in need of a careful execution strategy.
A real-world example: you have a legacy C++/MFC application. Currently, the UI is talking with the application through a small, homebrew messaging component (MC), of which you are in charge. You just found a better messaging component (BMC) on the internet, say something that is more efficient, portable, and actively developed by a community. You want to move your subsystem to another place in the decision space, from MC to BMC. Part of BMC being "portable" is based on the reasonable assumption that messages are strictly C++ objects.
Meanwhile, another team is moving the UI toward .NET (that's a big mass - big distance issue, but it's their problem, not yours :-). Now, there is an attraction between your messaging component and the UI. A .NET UI may have some issues dealing with native C++ messages. So, like it or not, there is another force at play, trying to move your component to a different place (say, NMC). Now, the mass of the messaging component might be small. The distance between MC and BMC may be small too - perhaps something you can deal with by using an adapter. But probably BMC is in a very different direction than NMC, where the UI team is pushing your component to move. So it's not just a matter of distance. It's not even enough to consider mass. You need to consider distance + full-blown Inertia. Or, to cut it short, work
Of course, depending on your project, some factor may dominate over the others, and a simpler model might be enough, but usually, distance alone won't cut it.
There is more...
At the end of the presentation, Kent mentions power law distribution. This is something that has got me interested too in the past few months (I mentioned that in Facebook), but it's too early to say something about it, and I want to check a few ideas with the author of an interesting paper first, anyway.
See you guys soon with a post on Distance (not in the decision space, but in the artifact space!)
Friday, February 12, 2010
Form vs. Function: a Space Odyssey
I was teaching Object Oriented Design past week, and I mentioned the interplay between form and function (form follows function; function follows form). I'm rather cautious not to spend too much time on philosophy, although a little philosophy shouldn't hurt, and people who know me tend to expect a short philosophical digression every now and then.
Function follows form: that is to say, the shape of an object will suggest possible uses. Form follows function: the intended usage of an object will constrain, and therefore guide, its physical shape. This is true for software objects as well. It's just a different material, something we can't immediately see and shape with our hands.
Realizing that function follows form is a pivotal moment in the development of intelligence. You probably remember the opening of 2001: A Space Odyssey. The apes are touching the monolith and, shortly after, one of them is playing with a bone and bang! - it's not just a bone anymore: it's a tool. Function follows form. This chapter is known as "The Dawn of Man", and rightly so.
Watch a little more, and you'll see a doughnut-shaped space station. That's a very good example of form following function (exercise for the reader :-)
By the way, if you have never seen that apes stuff till now :-), here it is, at least until it gets removed from YouTube...
Function follows form: that is to say, the shape of an object will suggest possible uses. Form follows function: the intended usage of an object will constrain, and therefore guide, its physical shape. This is true for software objects as well. It's just a different material, something we can't immediately see and shape with our hands.
Realizing that function follows form is a pivotal moment in the development of intelligence. You probably remember the opening of 2001: A Space Odyssey. The apes are touching the monolith and, shortly after, one of them is playing with a bone and bang! - it's not just a bone anymore: it's a tool. Function follows form. This chapter is known as "The Dawn of Man", and rightly so.
Watch a little more, and you'll see a doughnut-shaped space station. That's a very good example of form following function (exercise for the reader :-)
By the way, if you have never seen that apes stuff till now :-), here it is, at least until it gets removed from YouTube...
Wednesday, March 19, 2008
(Simple) Metrics
I've been using metrics for a long time (certainly more than 10 years now). I've been using metrics to control project quality (including my own stuff, of course), to define acceptance criteria for outsourced code, to understand the way people work, to "smell" large projects before attempting a refactoring activity, to help making an informed refactor / rewrite decision, to pinpoint functions or classes in need of a careful review, to estimate residual bugs, an so on.
Of course, I use different metrics for different purposes. I also combine metrics to get the right picture. In fact, you can now find several tools to calculate (e.g.) code metrics. You can also find many papers discussing (often with contradictory results) the correlation between any given metric and (e.g.) bug density. In most cases, those papers are misguided, as they look for correlation between a single metric and the target (like bug density). Reality is not that simple; it can be simplified, but not to that point.
Consider good old cyclomatic complexity. You can use it as-is, and it can be useful to calculate the minimum reasonable number of test cases you need for a single function. It's also known that functions with higher cyclomatic complexity tend to have more bugs. But it's also well known that (on average) there is a strong, positive correlation between cyclomatic complexity (CC) and lines of code (LOC). That's really natural: long functions tend to have a complex control flow. Many people have therefore discounted CC, as you can just look at the highly correlated (and easier to calculate) LOC. Simple reasoning, except it's wrong :-).
The problem with that, again, is trying to use just one number to understand something that's too complex to be represented by a single number. A better way is to get both CC and LOC for any function (or method) and then use quadrants.
Here is a real-world example, albeit from a very small program: a smart client invoking a few web services and dealing with some large XML files on the client side. It has been written in C# using Visual Studio, therefore some methods are generated by the IDE. Also, the XML parser is generated from the corresponding XSD. Since I'm concerned with code which is under the programmer's control, I've excluded all the generated files, resulting in about 20 classes. For each method, I gathered the LOC and CC count (more on "how" later). I used Excel to get the following picture:
As you can see, every method is just a dot in the chart, and the chart has been split in 4 quadrants. I'll discuss the thresholds later, as it's more important to understand the meaning of each quadrant first.
The lower-left quadrant is home for low-LOC, low-CC methods. These are the best methods around: short and simple. Most code ought to be there (as it is in this case).
Moving clockwise, the next you get (top-left) is for high LOC, low CC methods. Although most coding standards tend to somehow restrict the maximum length of any given method, it's pretty obvious that a long method with a small CC is not that bad. It's "linear" code, likely doing some initialization / configuration. No big deal.
The next quadrant (top-right) is for high LOC, high CC methods. Although this might seem the worst quadrant, it is not. High LOC means an opportunity for simple refactoring (extract method, create class, stuff like that). The code would benefit from changes, but those changes may require relatively little effort (especially if you can use refactoring tools).
The lower-right quadrant is the worst: short functions with high CC (there are none in this case). These are the puzzling functions which can pack a lot of alternative paths into just a few lines. In most cases, it's better to leave them alone (if working) or rewrite them from scratch (if broken). When outsourcing, I usually ask that no code falls in this quadrant.
For the project at hand, 3 classes were in quadrant 3, so candidate for refactoring. I took a look, and guess what, it was pretty obvious that those methods where dealing with business concerns inside the GUI. There were clearly 3 domain classes crying to be born (1 shared by the three methods, 1 shared by 2, one used by the remaining). Doing so brought to better code, with little effort. This is a rather ordinary experience: quadrants pinpoint problematic code, then it's up to the programmer/designer to find the best way to fix it (or decide to leave it as it is).
A few words on the thresholds: 10 is a rather generous, but somewhat commonly accepted threshold for CC. The threshold for LOC depends heavily on the overall project quality. I've been accepting a threshold of 100 in quality-challenged projects. As the quality improves (through refactoring / rewriting) we usually lower the threshold. Being a new development, I adopted 20 LOC as a rather reasonable threshold.
As I said, I use several different metrics. Some can be used in isolation (like code clones), but in most cases I combine them (for instance, code clones vs. code stability gives a better picture of the problem). Coupling and cohesion should also be considered as pairs, never as single numbers, and so on.
Quadrants are not necessarily the only tool: sometimes I also look at the distribution function of a single metric. This is way superior to what too many people tend to do (like looking at the "average CC", which is meaningless). As usual, a tool is useless if we can't use it effectively.
Speaking of tools, the project above was in C#, so I used Source Monitor, a good free tool working directly on C# sources. Many .NET tools work on the MSIL instead, and while that may seem like a good idea, in practice it doesn't help much when you want a meaningful LOC count :-).
Source Monitor can export in CSV and XML. Unfortunately, the CSV didn't contain the detailed data I wanted, so I had to use the XML. I wrote a short XSLT file to extract the data I needed in CSV format (I suggest you use the "save as" feature, as unwanted spacing / carriage returns added by browsers may cripple the result). Use it freely: I didn't put a license statement inside, but all [my] source code in this blog can be considered under the BSD license unless otherwise stated.
Of course, I use different metrics for different purposes. I also combine metrics to get the right picture. In fact, you can now find several tools to calculate (e.g.) code metrics. You can also find many papers discussing (often with contradictory results) the correlation between any given metric and (e.g.) bug density. In most cases, those papers are misguided, as they look for correlation between a single metric and the target (like bug density). Reality is not that simple; it can be simplified, but not to that point.
Consider good old cyclomatic complexity. You can use it as-is, and it can be useful to calculate the minimum reasonable number of test cases you need for a single function. It's also known that functions with higher cyclomatic complexity tend to have more bugs. But it's also well known that (on average) there is a strong, positive correlation between cyclomatic complexity (CC) and lines of code (LOC). That's really natural: long functions tend to have a complex control flow. Many people have therefore discounted CC, as you can just look at the highly correlated (and easier to calculate) LOC. Simple reasoning, except it's wrong :-).
The problem with that, again, is trying to use just one number to understand something that's too complex to be represented by a single number. A better way is to get both CC and LOC for any function (or method) and then use quadrants.
Here is a real-world example, albeit from a very small program: a smart client invoking a few web services and dealing with some large XML files on the client side. It has been written in C# using Visual Studio, therefore some methods are generated by the IDE. Also, the XML parser is generated from the corresponding XSD. Since I'm concerned with code which is under the programmer's control, I've excluded all the generated files, resulting in about 20 classes. For each method, I gathered the LOC and CC count (more on "how" later). I used Excel to get the following picture:
As you can see, every method is just a dot in the chart, and the chart has been split in 4 quadrants. I'll discuss the thresholds later, as it's more important to understand the meaning of each quadrant first.
The lower-left quadrant is home for low-LOC, low-CC methods. These are the best methods around: short and simple. Most code ought to be there (as it is in this case).
Moving clockwise, the next you get (top-left) is for high LOC, low CC methods. Although most coding standards tend to somehow restrict the maximum length of any given method, it's pretty obvious that a long method with a small CC is not that bad. It's "linear" code, likely doing some initialization / configuration. No big deal.
The next quadrant (top-right) is for high LOC, high CC methods. Although this might seem the worst quadrant, it is not. High LOC means an opportunity for simple refactoring (extract method, create class, stuff like that). The code would benefit from changes, but those changes may require relatively little effort (especially if you can use refactoring tools).
The lower-right quadrant is the worst: short functions with high CC (there are none in this case). These are the puzzling functions which can pack a lot of alternative paths into just a few lines. In most cases, it's better to leave them alone (if working) or rewrite them from scratch (if broken). When outsourcing, I usually ask that no code falls in this quadrant.
For the project at hand, 3 classes were in quadrant 3, so candidate for refactoring. I took a look, and guess what, it was pretty obvious that those methods where dealing with business concerns inside the GUI. There were clearly 3 domain classes crying to be born (1 shared by the three methods, 1 shared by 2, one used by the remaining). Doing so brought to better code, with little effort. This is a rather ordinary experience: quadrants pinpoint problematic code, then it's up to the programmer/designer to find the best way to fix it (or decide to leave it as it is).
A few words on the thresholds: 10 is a rather generous, but somewhat commonly accepted threshold for CC. The threshold for LOC depends heavily on the overall project quality. I've been accepting a threshold of 100 in quality-challenged projects. As the quality improves (through refactoring / rewriting) we usually lower the threshold. Being a new development, I adopted 20 LOC as a rather reasonable threshold.
As I said, I use several different metrics. Some can be used in isolation (like code clones), but in most cases I combine them (for instance, code clones vs. code stability gives a better picture of the problem). Coupling and cohesion should also be considered as pairs, never as single numbers, and so on.
Quadrants are not necessarily the only tool: sometimes I also look at the distribution function of a single metric. This is way superior to what too many people tend to do (like looking at the "average CC", which is meaningless). As usual, a tool is useless if we can't use it effectively.
Speaking of tools, the project above was in C#, so I used Source Monitor, a good free tool working directly on C# sources. Many .NET tools work on the MSIL instead, and while that may seem like a good idea, in practice it doesn't help much when you want a meaningful LOC count :-).
Source Monitor can export in CSV and XML. Unfortunately, the CSV didn't contain the detailed data I wanted, so I had to use the XML. I wrote a short XSLT file to extract the data I needed in CSV format (I suggest you use the "save as" feature, as unwanted spacing / carriage returns added by browsers may cripple the result). Use it freely: I didn't put a license statement inside, but all [my] source code in this blog can be considered under the BSD license unless otherwise stated.
Subscribe to:
Posts (Atom)