Sunday, July 31, 2005

The perils of abstraction

When we build large systems, abstraction is our best friend. We can't constantly worry about all the tiny details of each and every component, class, library, API, etc. Still, abstraction is not a substitute for understanding. We can safely abstract away what we know, not what we don't know.
Real-world example: I know how to do asynchronous I/O using completion ports in Windows. I know how it works (right down into the kernel, although this depth is not really necessary) and when to use it. I've also built a small framework of classes to abstract away some details and crystallize some sensible design decisions. It is easier to build systems over the abstract view of my miniframework; however, I can always look under the hood if needed, because I'm abstracting away something that I know.
We can also try to abstract away things we don't really know. We routinely do that with communication protocols: we don't need to know the gory details of TCP/IP to open a socket and send/receive a few data. Well, maybe we do need to know :-) if we want to squeeze good performances out of it. Abstracting the unknown is the primary purpose of many libraries, components and technologies.
Whenever we use abstraction to shield ourselves from missing knowledge, however, we are taking a chance. Two things can go wrong with abstracting the unknown:
- the abstraction is not working the way it should. In this case, people often resort to exploratory programming, trying out things (parameters, flags, execution order) and looking for a combination that works. This is just a way to avoid looking under the hood, but we end up with a system that seems to work, but you don't know why.
- much more dangerous: we build our system on fragile foundations, because we are misinterpreting the abstraction.
Real world example again (this is actually what inspired me to write this post). I was reading an article on the Caching Framework Application Block in .NET. The author says that you can host the CacheService in-process, out of process, or in another (shared) computer, which is right. He then goes on to say that you can choose your storage to be Singleton, Memory Mapped File, and Sql Server. He says that Singleton is a hash table, therefore not thread-safe, therefore to be used only in process. That Memory Mapped File is thread-safe. That Sql Server is persistent and can handle large date volumes. Everything in italics is basically wrong. Seems like he didn't know the underlying abstractions, ending up with the wrong reasons to choose the storage. Singleton is a hash table, and a hash table can be easily turned into a thread-safe container simply by using a synchronization primitive. Indeed, even in an in-process scenario, you may have multiple threads, so it would be ridiculous if the Singleton storage wasn't thread safe. However, since the hash table will be stored into the address space of one single process, it won't be easily shared among processes. A Memory Mapped File is not thread safe by itself. We still need to build synchronization around it to avoid conflicts. Still, a MMF can be easily shared between processes, and is the ideal candidate for an out of process scenario. Again, MMF are easily shared between processes but not between computers. Opening your MMF on a shared server does not reliably work. If you want to share your cache between multiple computers, you need a standalone process communicating over the network and keeping your stuff cached. A database is a ready-made implementation of that, which accidentally is also persistent and can handle large storages (hardly useful for a cache anyway!).
In this specific case, the misunderstanding wasn't really that dangerous, but I see this happening all the time: people committing to (e.g.) COM+ or J2EE not because they need those services, they understand what is going on under the abstraction, and want the details abstracted away, but because they don't want to understand what is going on, and they hope the abstractions will take care of everything.
Conclusion: there is no free lunch - we need to understand the potential gain and potential risk of our decisions, including abstracting away the unknown, weight the two, and choose responsibly.

Friday, July 29, 2005

Observability

Software design is about -ilities: extendibility, reusability, scalability, etc. As designers, we recognize that software is not simply about function but is also about form. We shape our software to obtain specific - ilities that we consider important for the project. This is, by the way, one of the reasons why I’m highly skeptical about some XP predicaments like building the easiest thing and then refactoring - software architecture is much more than piecemeal improvements over simplistic code.
A property that is seldom considered in OOP/OOD is observability: by this, I mean the possibility of (easily) understanding the system state by observing, for instance, a memory dump. Object oriented programs are often structurally more complex than procedural programs. More exactly, they have a more complex static and dynamic structure. Their run-time state is frequently shaped as a relatively complex network of interconnected, fine-grained objects. A direct observation of a memory dump is not trivial; changing data on the fly, on a running system, is even more complex.
Observability is not a fundamental requirement for most systems. However, high-availability, critical systems may benefit from observability. A very common strategy when building those systems is to move all the data to a well-known, static global area. Not only dynamic allocation is banned (as it often is on critical systems) but in extreme cases also local, stack-based data is avoided, so that all the program state is easily observable. Diversion: it has also been suggested (although I definitely do not agree :-) that this strong separation of code and data leads to easier maintainability: see "The Separation Principle: A Programming Paradigm" by Yasushi Kambayashi and Henry F. Ledgard, IEEE Software, March/April 2004.
Recently, I've been dealing with a system where observability was considered important. The team was willing to use object oriented techniques (C++), and in theory they could even reuse several portions of code from a larger system. However, in the large system observability wasn't considered important, and the original developers used dynamic allocation in many places. The team was then confronted with a choice: reuse existing, debugged, known code, but give up observability (and accept dynamic allocation as well) or copy/paste the code and change it to use only global variables, possibly introducing bugs.
Here is where the power of C++ shows up immediately (assuming we don't give up too easily and we know the language): we can have observability and reusability, sharing code with systems doing dynamic allocations. All we have to do in the small, observable system is:
- create a structure for the global state: this allows easy observability.
- in this structure, provide enough space for all the entities you need at run-time.
- overload operator new for the different classes that are dynamically instantiated. Operator new should just return the address of one existing slot in the global state. You know in advance how many objects you need (otherwise, you won't have an observable system anyway), so you’re sure that there will be a free slot when new is called.
That's pretty much it, since most embedded systems never release memory: they build their structures at startup, and keep them alive forever.
Note that the large system won’t redefine operator new, and it will probably allocate and free a number of objects during its lifetime. Still, by redefining operator new in the small observable system, we don’t have to link the standard allocator code, and we obtain an important design property without compromising reuse.
For a real-world example of the importance of observability, see this report on the mars rover . Last diversion :-) : over the years, I've found that too many programmers don’t know about priority inversion (mentioned in the article). If you build multithreaded programs, especially real-time stuff, you gotta know it!

Wednesday, July 27, 2005

How's C++ ?

C++ is not doing so well. It's falling behind for Windows development, and that's enough to say it's not doing so well. It is suffering from excessive stability (!) as requested by ANSI/ISO standards. It is suffering from not being a single product (like Delphi or VB) just like Unix was suffering years ago, until Linux came by. It is also suffering from lack of tools, partially because of its complex grammar, partially because nobody is willing to invest too much time / money on a shrinking market.
C++ is suffering because a standard library came too late in a balkanized world full of custom implementation of (e.g.) String buried down into large frameworks (e.g. MFC) leading to all kind of integration troubles.
C++ is also suffering because too many programmers didn't invest the necessary time to become good C++ programmers. They are not using smart pointers, RAII, exceptions, the standard algorithms and containers, they are not really using OOP / OOD, they never really grew outside C. Therefore, productivity is low and more than a few managers promptly considered alternative (easier) languages.
C++ is also slowly growing in other communities, first and foremost in embedded programming. However, here is meeting a strong resistance from the old generation of die-hard "assembly and just a little C" programmers.
Change is always met with resistance, and resistance is not necessarily bad. If your target platform is a 8 bit processor with 2 KB or RAM and a whoopee 8 KB of ROM, you may not want C++. Better yet: you may not want abstraction at all. You may enjoy the power of what I call perfect knowledge of global state. You know that at a certain point you can reuse a location (maybe a few bits only) because it will be overwritten a few instructions later. You know you don't have to calculate a specific value because a previous portion has "accidentally" left it on a specific register. And so on. Been there, done that :-).
There are, of course, serious scalability problems with the notion of perfect knowledge of global state and in general with the lack of abstraction. In many cases, the few bytes you save and the few clock cycles you save have little or no economic sense. Sure, you have to factor in development time, lost opportunities due to longer time-to-market, and all those money stuff that the true nerd doesn't want to hear about.
Very recently, a customer told me that an outsourced (actually offshored) product based on old 8-bits technology and entirely programmed in assembly could be manufactured at a lower cost here, using ARM processors and C/C++. That's very common indeed. Most often, conservatism is not based on a careful economic and risk analysis, but on pride and prejudice (sorry for the trivial pun :-).
Anyway, considering that several "embedded systems" are now (necessarily) equipped with powerful processors and considerable memory, there are very little (or no) sound technical arguments against C++, and many technical arguments to use C++, because when your application grows, perfect knowledge of global state is a risky business. As usual, the hardest step will be a real transition to OOP, and that will take some time. UML is still lagging in embedded programming, even more than in other fields (this is pretty obvious given that abstraction is not considered valuable).
Sideways notes: saving memory does not necessarily means tweaking bits (although tweaking bits is easy). A well-known approach (well, maybe not so well-known :-) is to create a small custom language for the problem at hand, write a compact interpreter for the language, and then store the "program" in this custom language. This is a very old technique, used for instance to move Zork to the "personal computer" in the eighties: see How to Fit a Large Program Into a Small Machine by Marc S. Blank / S. W. Galley, Creative Computing, July 1980.
A few readers may also be interested in a simple pattern language to save memory:
Proceedings of the Memory Preservation Society - Patterns for managing limited memory by James Noble and Charles Weir.

Monday, July 25, 2005

"Pure Something" considered harmful

Talking about glue languages reminds me how much I dislike languages that cannot easily call components written in anything else. Guess that gets only worse when you get proud of it and call the idea "Pure Something" :-))).
For a real-world example, I really would like to integrate one of those Java-based UML tools with a COM-based project management tool: both have a complex object model based on different technologies, and getting the two to talk is a lot more work than I'm wanting to do. Looking around for some library to make it easier is not the answer: it's still too much work. I want this kind of gluing to be a snap.
I like the idea of a compiler switch enforcing strict compatibility and portability constraints and stuff like that. But I don't like the idea of building a platform that is inherently poor at any form of interaction with anything else.

Herb Sutter on C++/CLI

I have found a two-parts video interview with Herb Sutter:
part 1 part 2
For those who don't know him, Herb is one of the foremost C++ experts, and since 2002 has been working (guess what) as a program manager for Visual C++.
In the video, he talks about deterministic destruction, garbage collection, safety, and the underestimated difference between memory release and object disposal (my Italian readers may want to read my paper C++, Java, C#: qualche considerazione which also covers deterministic destruction). Maybe we'll see something better in the CLR in the future, but I won't hold my breath.
He also talks a little about the future of C++. He says that C++ has been successful, so it's now hard to change (which is partially true). He also says that C++ is a language with a tradition of trusting the programmer (true) and therefore is exposing more of the underlying platform (true). So, in his view, C++/CLI will be exposing more of the CLR than "any" other .NET language. Maybe, but I still see little value in that except as a (possibly fantastic :-) glue technology. Still, even fantastic glue is just glue. You don't build a system with a glue language.
He's also mentioning that colleges are getting back to teaching C++ from Java/C#; I've no data to back up this claim, and from what I see, I'm not so keen to believe it.
Herb is not the only guy in the video from the VC++ team, but as you listen to the others, you probably feel like you'd better uninstall Visual Studio right away :-). The final talk on Phoenix is interesting, I would definitely like to play with that in the future :-).

Sunday, July 24, 2005

Multiple Inheritance

"Modern" languages like Java and C# don't have multiple inheritance. The funny thing is that most programmers don't like multiple inheritance not because of the technical difficulties that it brings into the internal object model, or because some of those difficulties tend to emerge at the language level (e.g. the difference between regular multiple inheritance and virtual multiple inheritance in C++). They don't like it because they have been told so in the huge Java propaganda, just like some have been convinced that all implementation inheritance is/was bad during the huge COM propaganda.
Anyway, in these days I was playing a little with Context Attributes under .NET. Context Attributes are very powerful and possibly the only creative contribution of the .NET platform. However, to apply a Context Attribute to a class, that class must derive from ContextBoundObject. Some of you may be more familiar with ServicedComponent, that is derived from ContextBoundObject and provides the base class for all classes using COM+ services. ContextBoundObject is an abstract class, not an interface.
Unfortunately, I wanted to apply some Context Attributes to a class derived from Form. Now, this is one of those catch-22 situations. Given the .NET library as it is, I just can't. Using wrappers and adapters is not the answer: the whole point of Context Attributes is to write less code using AOP-like features. If you have to write wrappers, you can just as well avoid the attributes altogether.
Sure, the library is not perfect: Form and ContextBoundObject have a common ancestor (MarshalByRefObject). Moreover, it seems like ContextBoundObject is not adding any intelligence to MarshalByRefObject, except a [Serializable()] attribute and possibly playing a marker class role. Most likely, they could have done that using a [ContextBound] pseudo-attribute implying a [Serializable], so that any class derived from MarshalByRefObject could have been activated in a context if desired. Too late, I guess.
And here we have it: when we put strong limits into the language, we need perfect libraries. We need libraries where all classes fit nicely together. This is wishful thinking when a single large library is concerned, and ridiculous when you find yourself integrating code from different libraries or frameworks.
Give me Multiple Inheritance back :-).
Meanwhile, the weather is nice out here, and it's time to get into the sun :-).

Microsoft Visual C++ 7.x still lousy at optimizations

I just answered a question from one of my customers about Visual C++ 7.x, Return Value Optimization and Named Value Optimization, and I think the answer could be interesting for a few of my readers as well (my Italian readers may want to look at my paper Return Value Optimization, Named Value Optimization e Costruttori Operazionali for a review of those concepts).
The sad truth is that the compiler is still lousy at optimizing object oriented code: that is, it's not any better than VC 6.0 on those specific optimizations.
More exactly:
- RVO is partially implemented. Copy construction is optimized away in trivial cases (returning anonymous objects built in the return statement without any further manipulation) but not when you build an anonymous object and change it via (e.g.) some operator.
- NVO is not implemented even in trivial cases.
Not long ago, I did some experiments comparing C++ and C# on some real-world numeric code I was writing. C# came really close to C++: actually, unless C++ code was nearly perfect :-), and without careful tweaking of compiling options for C++, C# was even faster. Of course there are little reasons for that (especially on numeric code), except the fact that the JIT knows the target processor. Of course, the easiest answer is that VC++ is just not so good at optimizing code. Seems like the easiest answer is probably true :-)).
I won't go into any (extremely deserved :-) anti-Microsoft rambling here, but this is exactly why so many programmers tend to hate them :-).

Saturday, July 23, 2005

Trust and Beliefs

Prelude:I have received an email with a PDF attachment and a single line of text: "numbers are numbers". Someone is trying to convince me of his theories using numbers. I'll write the interlude below before reading the document :-).
Interlude
People trust information aligned with their belief system without further checking. After all, they're already convinced, therefore they don't need any further proof. They will gladly accept any information that confirms what they already think, and pass it around, spreading the idea.
People also tend to trust information from influential sources; sometimes, the mere fact of seeing the information in print, not necessarily on an influential journal, is enough to take it at face value. Even more so if it's also confirming what we think.
Academic papers, for instance, frequently quote the Standish Group Chaos Report about frequent failure of software projects. Authors tend to believe that practitioners aren't that good at software engineering, so little attention is paid to the scientific foundations of the report. It's confirming their opinions, therefore, it must be true. Only a few papers have criticized the report: it's not a surprise that they're coming from people with an academic background but also a deep experience in real-world software development. Readers interested in this diversion :-) may want to read "How Large Are Software Cost Overruns? Critical Comments on the Standish Group's CHAOS Reports" by Magne Jorgensen and Kjetil Johan Molokken-Ostvold, or "IT Failure Rates--70% or 10-15%?" by Robert L. Glass, IEEE Software, May/June 2005. As for me, it's time to read the document I received :-).
Postlude
The document is about open source in the enterprise. I'm not going to debate the document itself - it would be largely pointless: I agree with a few statements, and disagree with others. However, it's probably interesting to see how easy it is to reinforce a belief (in this case, the belief that open source is the One Right Way). "Numbers" begin with a couple of charts surrounded by this text: "A 2003 survey by Forrester Inc. shows over 70% of $1B+ North American companies are now running the Linux open source operating system". Let's play with that :-).
- The Forrester report is not freely available (isn't that funny? :-)). Actually, the report ("Your Open Source Strategy") is 24 pages long and is sold for 675$. How big is the chance that the guy sending me the email has actually bought and read the report? Pretty narrow :-). He's trusting a source reported in a document, because it's aligned with his beliefs system.
- How big is a $1B+ company? I mean, how many employees, computers, operating systems can we expect to find there? Sorry, I don't have any numbers here :-(. I would say several thousands employees anyway. Now, what is the chance that someone is using any given operating system in such a company? MS-DOS? SunOs? Solaris? QNX? Linux? Windows Me :-))? Far from zero, I would say. So, is that 70% still so impressive?
- The paper goes on quoting from the Forrester report: "When Forrester asked, 'What products do you use or plan to use?' the answers were:". Here comes a chart, showing a staggering Linux: 100%. Well, I guess, when you ask someone using Linux if he's indeed using Linux, you're bound to get a 100% "yes" :-). The confusion between "use" and "plan to use" just adds to the general mess.
- A last comment (I'm a nice guy, really :-). Forrester Research question on money saving is: "In What Ways Do You Think Open Source Software Might Save You Money?". Again, if you read the statement just as a support for your belief system, you may find the numbers interesting. But if you read the question again neutrally, you may notice that they didn't ask whether or not they will be saving money, and if so, how. They just asked how. Given the price of the report, I would expect a more scientific approach to gathering data :-).
Scherzo
I guess I've lost a few readers :-). Open source is one of those religious issues you can't even think to question. Hey, don't hack my web site: I'm running on Linux too! :-)). Anyway, if you read so far, consider this: in "Democracy and Filtering" (Communications of ACM, December 2004) Cass R. Sunstein argues that the internet is giving readers a huge power to "filter" what they see, that is, to read only what they find congenial, easily avoiding any exposure to alternative points of view. He says "the implication is that groups of people, especially if they are like-minded, will end up thinking the same thing they thought before - but in a more extreme form, sometimes in a much more extreme form". That's where reason ends and a product / language / tool / approach / whatever / is turned into religion. Consider pieces like this a free antidote :-)))).
Post Scriptum
Some meta-reflections on my repeated quoting above (appearing in a post about trusting stuff in print and filtering) are definitely needed :-)))).

Thursday, July 21, 2005

Teaching at University

I think I know how to teach to software professionals: I've been doing that for almost 15 years now, and in most cases I can make a few days of training both productive and fun.
In spring 2005, however, I've taught a short UML course at University of Trieste. The dynamics were quite different. I've taught several courses where more than a few people were fresh out of university, but the dynamics inside the university were quite different. And by that I mean: I could have done it better!
I'll be repeating the experience this fall, and after thinking more about what could be improved, I'm radically changing the order in which I'll present the material. I'll start, nonetheless, with the State Diagram, move up to the Class Diagram, and later provide a few key points on the Component Diagram, the Activity Diagram, the Use Case diagram.
Of course, it's not just a matter of moving things around: my aim is to present the material in a way that will truly make sense to students, most likely with a strong passion for coding but little real-world experience, a strong drive to code but less experience with modeling. In a few months, I'll tell you if this different approach is, as I'm expecting, so much better!

Wednesday, July 20, 2005

Highly Configurable Applications are Languages Too

In the past few months I've been consulting for a company trying to redesign a payroll system, mostly targeted at the public administration (my Italian readers will probably understand the almost unbearable complexity of that :-)). They already have a working system, highly configurable to handle the peculiarities of the different institutions.
Highly configurable applications are both a boon and a hindrance. They can save enormous development time in the long run. But they can be easily misused; they can easily get out of control; it's often hard to understand why the application is misbehaving, or even why is it working properly :-).
Back to the payroll: when they first tried to improve over the existing version by using object technology, it seemed like objects didn't work so well. They could see no advantages in using objects to mimic tables or queries, basically writing the same code in Java instead of using PL/SQL. In fact, there were no advantages.
It was necessary to rethink the system from a different perspective, from where objects could be actually quite useful. The system we are designing now is basically built on the idea that we need a very powerful language to define rules and formulas, and that most of the business knowledge must go into the language, not into the formulas, so that all our configurations will be simple to write, simple to read, simple to maintain.
In fact, all highly configurable applications are languages. They're simply not designed that way. That's why they become so brittle. I can think of several industrial applications where adding (e.g.) a sensor requires the editing of a few (obscure) XML or INI files, perhaps some changes into a database through custom programs, and so on. Make a single mistake, and you'll break the integrity of the system, with unpredictable consequences.
You may have never thought of those configurations as a language - but that's what they are. Unfortunately, it's often a composite language, growth out of needs, not from careful design, and it usually shows. One of my customers is suffering configuration problems almost every day, in plants all around the planet. I guess our next highly configurable application will be significantly different from the past :-)

Tuesday, July 19, 2005

Needs Vs. Demands

Leaving for Udine today, I was waiting for my train. A barely understandable voice announced that the train for Venice would shortly arrive on platform 3. A woman asked me if they had just announced the train from Venice. I replied "No, it's the train from Albenga to Venice". She asked again. I added more details: "Yes, it's the train from Albenga to Milan and then Venice". She mumbled something to herself. I had other things to think about, so I didn't ask the obvious question, "where are you headed to". They repeated the announcement, so she got the name of the first stop (Genoa). "See, it's the train to Genoa". "Yes, it's from Albenga to Genoa, Milan, Venice". "I'll stop in Genoa".
Of course, she could have asked that from the very beginning: "is this train for Genoa?". After all, that was her real problem. Instead, she somehow assumed that the train for Genoa was also from Venice (not to Venice) and she started asking from there. She had a (wrong) solution in her mind, and that was her starting point for her questions. She won't start by telling her real problem. Sure, given a little more motivation :-)), I could have asked the right question and given the right answer. After all, the problem domain was trivial and well known - people want to go somewhere, they don't usually want to take a train from somewhere.
This is not always the case for software applications: the domain can be arbitrarily complicated and largely unknown. Still, the users will keep asking to solve what they believe is their real problem - only to find out that what they asked for isn't really that useful. It is our duty to understand the real problem and design a reasonable solution within the possibilities of current technology. That's why I'm highly skeptical about approaches strongly based on user's stories - they tend to replicate [possibly outdated] solutions, not to make real progress.
For a slightly related paper, see my When past solutions cause future problems, published a few years ago in IEEE Software. Food for thought: if you've ever heard the expression "IT productivity paradox", how many relationships can you find with the above? :-))

Tools

A nice article on buying tools from Embedded.com. I don't necessarily agree on everything, but it's quite interesting anyway.

Monday, July 18, 2005

Brushing up slides on Testing for Testers

I just finished brushing up my slides "Testing for Testers", a course aimed at the guys with a mission to break our software :-). I improved the section or test plan minimization, adding material on "all-pairs" testing. I've also tried out a free tool for all-pairs generation, which seems good enough, although all the copy/paste from excel is not that handy.
For those of you who may have an interest in the theory behind the tool, I recommend reading "The AETG System: An Approach to Testing Based on Combinatorial Design", IEEE Transactions on Software Engineering, July 1997.
I'll be teaching the course this week for a well-known Process Control company in Udine. This is one of the several steps we have taken to improve the software development process (and the skills of everyone involved).
From my long-standing presence in the industrial automation field (I've several customers working on machineries of every size and scale :-) I would say that most companies are still entirely unaware of how much software-centered they have become. Many players in this field still think to themselves as "heavy industry" companies. Meanwhile, they've got millions of lines of code hanging around. It's not rare to find companies with a tradition of "anybody can be a programmer" depending more and more on software for innovation. Some are getting the message, and (slowly) learning how to really grow their software factories. The others... well, I guess there won't be any "others" in the long run :-).

Sunday, July 17, 2005

Thinking Like Users

How do we move away from corporate thinking and toward users? The question is quite broad and would (will?:-) require several postings. Here are a few suggestions, straight from my everyday experience:
- Stop thinking "features" and "information", think "goals". If you design your application asking yourself "what set of functionality do we need" or "what is the information architecture here", you're doomed. You'll begin thinking like a developer, or like your company, or both. Instead, ask yourself "a user just started my application. He wants something, right now. What's that?".
- Use cases may help [a little]. Use cases help as long as you focus on why would a user go through that use case, that is, what is the real, ultimate goal. Thinking in use case format brings up some issues too early for my taste, but it's generally better than thinking about "information" and slightly better than "feature". For the record: a toolbar like "banking, credit, trading, ..." is not about goals and is not about services. Is about information architecture.
- Involve the user - but be careful! Although some approaches recommend a strong user involvement, and although we must certainly talk to users a lot, having a user dictating the interaction is just plain wrong. Unless, of course, he's the only user of your application. This goes in pair with a longstanding debate on requirements (see, for instance, "Eureka! Why Analysts Should Invent Requirements" by James Robinson, IEEE Software, July/August 2002, also available online from the author's website). I'm strongly on the side that we should take the time to learn the real problem from the users and then invent the requirements.
- Read at least some classics on HCI. You should at least be familiar with Schneiderman's Eight Golden Rules, like "Design dialogs to yield closure". Of course, you'll have mastered the concept when you'll learn when closure is desirable (most times) and when is not!
- Understand the difference between short-lived corporate applications and long-lived, large-market applications. The dynamics are totally different. Short-lived applications can be created from "user's requirements", that is, by having the users asking for a [quick] solution to his perceived problems. Users stories, or use cases, can be fine here. Long-lived, large-market, possibly embedded applications can't be built on the basis of a few users asking to solve what they believe it's their problem. It's our job to understand the real problem and to see how technology can actually go and solve that problem. You can't expect your users to innovate your products.
- Don't optimize the interaction for the worst case. The most frequent reason to make 99.9% of your users unhappy is that the interface is optimized for the 0.1% extreme users. An easy criticism to my simplified interface for internet banking is: what if I have more than one account on the same bank? So the programmer starts with one more page where you have a list of accounts, or a list of credit cards, or a list of whatever. Don't. Optimize for the 99.9%. The 0.1% must be able to work with your product (that is, your application must handle the worst case), but don't make the extremes drive the interaction. Of course, in some cases the extremes are a valuable niche, in which case they probably deserve a different application!
- Stop thinking "User Interface"! I've got the idea from an interesting (although somewhat conventional) paper, "The obstacles and myths of usability and software engineering", from Ahmed Seffah and Eduard Metzker, Communications of the ACM, December 2004. They say that talking about "user interface" gives the impression that user-centered design and usability engineering are "only for decorating a thin component sitting on top of the software", and I largely agree. While I understand the importance of building business objects detached from the (I'm gonna say it! :-) user interface, those business objects must be discovered by designing the user interaction, not the other way around!
- Learn the personas approach. It's still the best game in town. If you find you can't play that game, please leave interaction design to someone else in your team. Quick self-check: if you can easily program your VCR to record your favorite show next week, and you either believe anybody can do that, or that they should simply read the manual and learn how to do it, you have a long road in front of you before you can be an effective interaction designer (sorry :-).

What I've seen and what I've got :-)

I mentioned "seeing" a better approach to solve a probability distribution problem. Well, here is what I've seen (ok, I didn't see those colors, but that was the shape, anyway :-).
As usual, that was only the first step. It took some effort to move from vision to reality. Here is what I've got: the red and blue curves are two sections of the surface, instantiated on a specific problem. The purple curve helps finding the best x coordinate, and therefore a pair of points on the red and blue curve. I'm very, very close to a solution :-).
Now all I have to do is write down a few pages to explain myself :-) what I've done, to make sure I'm [still] on the right track. Then I'll try out the idea on a few experimental data I have collected, and see how good it is at predicting the future :-).

Saturday, July 16, 2005

"An organization consists of conversations"

The relationship between software engineers and business (management, marketing, etc) has never been idyllic - for many reasons I won't touch yet. Still, at some point, many talented software engineers are asked to outgrow their role and move into management, or in other business roles.
This is never an easy step. It's not just about new concepts and skills to master - software engineers are usually quick to grasp new ideas; it's about learning new values, being able to shift into a different mindset, and then back when is needed. It takes time (and some effort), and most people don't take that time. They assume they will succeed solely on the basis of technical excellence. Good luck :-).
Meanwhile, I see a minority of excellent software engineers growing a sincere interest (and appreciation) for management and organizational issues. They may appreciate this pamphlet: Notes on the Role of Leadership and Language in Regenerating Organizations. I've stumbled into it about one year ago, and found it extremely interesting. Here is my favorite paragraph:
Ultimately,
an organization consists of conversations:
who talks to whom, about what.
Each conversation
is recognized, selected, and amplified
(or ignored) by the system.
Decisions, actions, and a sense of valid purpose
grow out of these conversations.

Now, what is the conversational nature of your company? :-)

Tuesday, July 12, 2005

Company Information Model Vs. Users Information Model

In another posting, I mentioned the difference between building an application around user's conceptual models and around the information architecture adopted inside the company. I believe a real-world example can be useful here. This time it's not from one of the applications I've been working on, but from something I use quite often: internet banking.
The [web] application provided by my bank is entirely built around the company information model. After a login screen, you get a top bar with entries like trading, banking, credit, planning, etc. On the left, there is another bar, with "shortcuts" to the most common services, like wire transfer. The rest of the page is wasted advertising products (does anybody ever read that stuff?).
My routine goes like this: I click on "Statement" on the left. I get into another screen where I can see my Balance (good :-) and I can choose the contents of the statement: all the transactions in the current month, in the latest 3 months, or within 2 dates (= 6 combo boxes) . I normally choose current month. And I get (guess what?) a list :-), although usually not a very long list.
Occasionally I check my credit card (more or less the same workflow as above), or I pay for something (a slightly different road).
Again, this is so commonplace that you may think it's fine. Unfortunately, it's not. This is not the way I (among many others :-) look at my banking account. Consider a different approach. I log in and I immediately get a useful screen: it tells me when I've been there the last time; my previous balance; my current balance; the difference (don't make me think :-); how much money went in and how much went out (this is not the same as the difference between balances!). Everything is an hyperlink, so if I'm interested in the (short!) details of the money that went in, it's just one click away.
Of course, this is just the tip of the iceberg. We can redesign the whole user interaction according to what is useful for the majority of users, while leaving "advanced" features available (with a few more clicks).
The problem (just like with grids) is that it's much easier to build the interface the way they built it. Because in that way, you don't have to know the user. You only have to provide information, using the company information model, which is well understood (internally). Of course, when we don't really know the user, we have no choice but to create a generic application that "leaves the user in control". To add insult to injury, this sense of control it's something most developers like, so they're quite happy to build applications this way. That's why they like grids too :-). They can see all the information there, and feel somewhat "in control". And they may honestly believe that users want that too. Unfortunately (for us all) this is not true. More on this another time :-).

I'll be away for a few days, so unless I get some friendly WiFi hotspot on the road, I'll not be posting for a short while. See you soon!

Monday, July 11, 2005

Grids, lists, and the like

I don't like grids. The only grid-based program I like is the spreadsheet, for reasons I'll explain later. The heavy presence of grids is often the symptom of a terminal disease: the computer as a filing cabinet or, as I call it, the "useless computer".
The usual grid-based program works like this:
- the user can insert new information into the program, usually through a form.
- the user can later search for information, usually by scrolling through a long grid and/or by applying filters and sorting on some fields.
- the user can update the information, either in-place or in the same form as above, or see more details, again in a form.
- alternatively, once he has found the information, he's sent to another grid, e.g.: now that you've found your customer, pick the products.
This is so commonplace that people tend to believe it's good. However, the computer is totally useless here. Sure, it's faster than the old filing cabinet. There are also a few constraints checked here and there, so inconsistencies are somewhat trapped. Still, it's a file cabinet on steroids. Duh. Very exciting... : (.
This kind of application is common because it's easy to write. Extremely easy. Almost in the realm of end-user programming. It takes little understanding of the (real) business. Little or no understanding of the user. Little or no interaction design. With a modern tool, it doesn't take much understanding of technology either. Instead, moving away from the "useless computer" approach takes a large effort in understanding the business and the user. It takes creativity, and may require a larger investment: for instance, the barcode has been conceived to move some applications away from the grid paradigm, but without a large dissemination effort, it would have been useless. Often the solution to the grid disease must be found in a stronger integration between applications, sometimes in a stronger integration with the physical world (see the barcode, and its modern RFID incarnation).
Another real-world example can be useful here. Some time ago, one of my customers (a bank) showed me a prototype for a backoffice application. The application was intended to enable users to keep track of a complex workflow. The user would select a customer and guess what? He would get a lengthy list of documents, provided by the customer or to the customer, in a [nice] grid with a date field, a status field, and so on. Very common sense. Very commonplace. And totally wrong.
The user would have to understand, from the list of documents, the next steps in the workflow. Are there any missing documents? What I've got to do next? Why is not this !@#$@# computer helping me?
Consider now a different interface: since the workflow is so important, show the workflow. Show the user the steps he has been through, and display documents associated to each step with a click. Highlight the next steps. Help him visualize what he's got to do next.
More complicated to design and implement? Sure. More useful? You bet!
Back to the spreadsheet: here we have a grid too, but the interaction model is completely different. You write numbers and formulas and you play what-if scenarios. The computer is extremely useful. It's not acting as a fast filing cabinet. It's a thinking tool. Exactly what I want.

Sunday, July 10, 2005

Running, Oxygen, Probability Density

Maybe I've been around customers like Technogym and Mywellness a little too much :-), but I enjoy running over the hills around here. Today I've also met a couple of deers; they were really curious and we looked at each other for a while before they disappeared into the woods.
I like that hyper-oxygenated state I enter after a while: I've got quite a few good ideas while running, and now I always bring a tiny notebook and a pencil with me. The mind tends to wander a lot, and if I don't jot down a few key points, there is a good chance I'll forget everything on the way home.
Today I've literally seen :-) a new approach that could help me solve a difficult problem with probability density. I've been tinkering with it for almost a year now, and I'm getting closer to a solution, but I'm still too far from something I can use in the real world. My aim here is to bring together a de-facto approach in estimation ("expert estimation", that is, the expert is "guessing" the effort), some research from Simula Labs in Norway (see the excellent papers from Magne Jorgensen on improving the realism of expert estimation), and statistical techniques, like good old 3-Points PERT. There is still a lot of work ahead, but as Thomas Edison said, genius is one percent inspiration and ninety-nine percent perspiration. And that brings back the idea of running :-)).

UML 2 Manuale di Stile

Il capitolo sulle Protocol State Machine e' completato ed online. Giusto il tempo di mandare a tutti gli abbonati la mail di notifica e poi, visto che qui in liguria il tempo e' fresco ma soleggiato a dispetto delle previsioni, la scelta sara' tra un po' di mare, una corsa sulle colline, ecc ecc :-)).

Saturday, July 09, 2005

UML 2 Manuale di Stile

Il capitolo sulle Protocol State Machine e' quasi finito. E' breve ma dice (spero) tutto quanto serve per capire questo nuovo concetto di UML 2.
Con un po' di fortuna domani sara' disponibile online, gratis come sempre.

Friday, July 08, 2005

C Omega

An interesting paradigm shift for concurrency and data access, based on C#. Nice to see Luca Cardelli is working on this stuff! A preview compiler is available for free. See also a paper from Cardelli on the theory behind the new language.

Thursday, July 07, 2005

HCI, Usability, and beyond...

As a consultant, I'm often involved in improving the usability of software products. Sometimes, I'm called in after the fact, and I can only suggest a few changes here and there. In other cases, I've the chance to influence the human-computer interaction from the very beginning. In both cases, the way I approach interaction design depends heavily on my customer.
Some of the companies I work with are somewhat conservative. They are honestly interested in improving the user's experience. They value usability and ergonomics standard like ISO 9241 and ISO/IEC 11581 (which can be very useful at times, especially to settle down controversy by appealing to an international standard :-). They love the idea of making everything easier: "Don't Make me Think", by Steve Krug, is usually on their bookshelves. But they won't (for instance) change an interface based on their terminology and information architecture (which they consider "professional") to another based on the terminology and the (possibly fuzzy) mental model of a casual user, although the latter would be so much easier to understand for anybody outside the company.
Some of the companies I work with, however, enjoy a true freedom in the design of the user interaction. Here, I try to apply the best method I've ever found to create a pleasant user experience: the concept of personas, as introduced by Alan Cooper in "The Inmates Are Running The Asylum" (my Italian readers can also get a good translation from Apogeo, "Il Disagio Tecnologico").
If you are involved in interaction design, do yourself a favor: get the book. It's a short and nice read. You don't have to agree on each and every detail, but if you find yourself in strong disagreement, please leave GUI design to someone else :-)).
Beware: many software developers find the idea of personas too child-like. Or, they think it's not their job to think like that, or to get to know the user like that, and so on. Cooper thinks a programmer can't possibly double as a good interaction designer. I don't agree on that (well, Cooper himself has been a programmer, so... :-). But I'm pretty sure that interaction design it's not for everyone.
Next time: why I don't like grids. Gee, this may put an old friend of mine out of the publishing business :-).

Commenti e lingua... [post periodico]

Scrivo il blog in Inglese perche' ho molti visitatori stranieri e non avrei il tempo e le energie :-) di scriverne uno anche in Italiano. Tuttavia non createvi problemi nel commentare in Italiano, Inglese, Francese, Spagnolo, Sanscrito :-). Per le altre lingue, non credo riuscirei a capire bene, ma fate pure :-).

Wednesday, July 06, 2005

"Inside Microsoft .NET IL Assembler"

I finished reading this book from Serge Lidin (Microsoft Press). It doesn't go in depth as much as I wanted, but it's still one of the best references on the subject.
If you want to learn more about MSIL, because you want to read some dumps in ILDASM, tweak some code at the IL level, or dynamically generate code through Reflection.Emit, this book is a nice and easy read.
Unlike traditional assembly books, you won't find much information about performance. This is partially unavoidable, since IL is just... intermediate :-), and the actual performance depends on the final machine code. Still, some examples of translation into Intel assembly would have been useful, e.g. when the switch/case is introduced.
Overall, the book provides you valuable information on IL, like the ability to do a tail function call, but leaves you wanting more. On the other hand, some details I need are not strictly part of MSIL, but more on the side of the CLR. Those details are somewhat covered in two classics on CLR (from Richter and Box), which unfortunately are a little boring to read :-).

Tuesday, July 05, 2005

From Gastroenterology to Fitness Equipments (and somehow back :-)

In 1991, I designed an open platform for (soft) real-time data acquisition, online processing and visualization of physiological data, initially targeted for gastroenterology, and later extended to cardiology and so on.
The system, written in C++ with small portions in assembly, had to run under Windows 3.0 (and was later upgraded to follow the OS evolutions), and the minimum suggested hardware was a 386sx at 16 MHz. Besides doing most of the design, I also coded a large portion of the application.
I had to find a good balance between good object-oriented design and performance. Extendibility was a must - the whole idea of an open platform was that we could plug in new drivers, new real-time algorithms, new kind of visualization. Performance was also critical, and given the non-preemptive nature of Windows 3.x, a good deal of effort went into making everything fast and non-blocking. In the beginning, I even used self-modifying code in some critical points to squeeze a few clock cycles, and abused :-) of a few undocumented APIs. In the end, I brought into the company an higher awareness of what made a good hardware for Windows, they redesigned their flagship hardware to radically improve performances, and we could remove a few tricky portions here and there.
The company (Synectics Medical AB, Stockholm) at a point merged with Dantec Technologies, and was later bought by Medtronics. After all those years, that system is still sold! An unofficial source told me that after I left the company, they tried to design a newer, more abstract version of the system, but didn't succeed: performance wasn't good enough. Software design is about finding balance between many forces. Ignoring performance in a performance-critical systems is simply not good, as it is ignoring reusability, extendibility, maintainability, and so on. As usual, finding balance is harder than optimizing for just one property.
Several years later, I started working for an Italian company, a well-known leader in fitness equipments. My experience with medical systems certainly helped my learning their domain quickly (you can't design a good system without an appropriate understanding of the problem domain). Now we are designing a system which is very close to the medical field, and to the kind of applications I developed in that field.
Many things are different, however. Sampling rate is much smaller. Computers (CPU, RAM speed and amount, graphic cards, etc) are enormously more powerful. We still need some kind of extendibility, but on a different scale. We can afford to buy components that I had to hand-optimize. We can afford to use XML descriptors, interpreted at run-time, where I had to create custom drivers in C++ or even in assembly. The overall application is also less ambitious, so we can certainly benefit from a scaled-down architecture. Scheduling is also more compressed in these days than it was 14 years ago. In fact, planning is now becoming more important than it used to be, exactly because we have shorter schedules. Where many people find activities like planning and design as time-wasters, I see them as time-savers. Assuming, of course, that we take the time to learn how to plan and how to design. Unfortunately, too many software professionals don't take that time, and then pretend coding it's all that matters.

Monday, July 04, 2005

Dropping features

Software rarely gets smaller: we tend to add more and more features. This is probably intrinsic in the software business model or, as Donald Norman argues in The Invisible Computer, in the PC software business model.
As we add more features, some become obsolete. And as we design new versions of a software product, intended to replace the current version, we often have the chance to remove some feature as well. This rarely happens.
Common arguments against dropping feature are:
"users may like it"
"users may feel lost without it"
"there is no direct replacement for that in the new product" (or, the replacement requires some roundabout)
"I would use it"
If you listen carefully, this is more exactly phrased like: "We don't know the way our software is used. People may or may not use that feature frequently or at all".
Of course, the darkest side of this is that if you don't know what users are not using, you may also have a confused picture on how to improve the product, but let's ignore that.
What I find interesting is that we know the solution to the problem (tracing users behavior) and we routinely don't build a decent tracing systems inside the software. Note that the kind of tracing I'm proposing is not related to bug identification: we need an high-level, compact log of feature usage, ideally per-user (although we don't need to identify the user, it would be useful to know how many users are indeed using a feature).
The reason we tend not to build a tracing system is that it takes precious time out of the mainstream development and maintenance. This is where I'd like to see more support from new tools and libraries, but looking at what we're getting, it seems like most people are substantially more interested in having just another way to display a sexy grid on a form.
Anyway, this sounds like a perfect problem for an AOP (Aspect Oriented Programming) solution. All we need is a "FeatureUsage" aspect that can be applied to any method that acts as the entry point for an high-level feature. Well, the aspect may have a granularity parameter to control the tracing. Again, this sounds very similar to the debug-style tracing, but differs enough in nuances (including being enabled all the time on user's machine) that it would probably deserve a slightly different design.
The blurb above about tools and libraries not getting any wiser reminds me of a remotely related paper, "Traipsing through the QA Tools Desert" by Terry Coatta (Quality Assurance Vol. 3, No. 1 - February 2005). Well, I do hope someone will act on his words. My "FeatureUsage" aspect is much easier to get, and I guess sooner or later I'll spend a few hours thinking about it and writing down a .NET implementation using attributes.

Sunday, July 03, 2005

Villa Gregoriana

Another day nearby Rome. I visited Villa Gregoriana in Tivoli, which was closed till a few months ago (as it was for more than 30 years).
The waterfall is probably a little unimpressive by today standards, but it must have been a daunting engineering project back in 1832.
Nature all around was fine but not stunning... well, guess I've definitely got high standards :-).
I managed to read about 60% of one of 3 the books I bought yesterday, "when yes means no (or yes or maybe)". Overall nice, though I'm afraid it's a little stereotypical and possibly already outdated.

Saturday, July 02, 2005

Domus Aurea

A sunny day in Rome with my girlfriend. I visited the Domus Aurea, which I had never seen before. For one reason or another, it didn't inspire me any thoughts on software architecture :-). That's somehow peculiar, considering that a few years ago I collected a few pictures in Rome (like this), and then in other towns, for a never completed article on the evolution of buildings and software.
Later, on the way to Piazza Navona, I've bought 3 new books. Again, nothing about software, although somehow connected with the business of consulting. Last time I checked, I had more than 2000 books. But that was quite a few years ago :-).
Multiple Dispatch in .NET is still too slow. I need some fresh ideas to get better performance. But that's not for tonight ;-).

Friday, July 01, 2005

Build in-house, or buy a customization?

Some questions come up over and over, but they don't get any easier to answer.
Make or buy is a classic, and in practice is often answered in unclear, emotional terms.
One of my customers is facing a though choice. They could create a product in-house, reusing a few components from other projects and using a few commercial components as well. Alternatively, they could ask a third-party to customize an existing, similar product, which would sensibly shorten development time.
We have full data for the in-house effort:
  • requirements are reasonably clear
  • we have an overall architectural model
  • we have two alternative schedules for implementation, both based on the concepts of Incremental Funding and Minimum Marketable Features (see "Software by Numbers", from Mark Denne and Jane Cleland-Huang), so the product should start paying for itself when is roughly 50% "complete".
  • we have a reasonable estimate for development cost and time, and to some extent for maintenance costs over the next few years.
  • we know the in-house process, the kind of documentation we will have, the defect density we can expect from the development team, etc.

The third-party is reasonably eager to provide an extremely favorable estimate (cost and even more so time) so to get a contract signed.
The real question is: how do we make an informed choice? Management shouldn't (and with any luck, won't :-) choose based only on time and money. Will product quality be comparable with that of in-house products (or better)? What is the third-party strategy for customization, e.g. do they have a modular, plug-in based architecture, or will they simply create a branch of their main product? How do they feel about a joint design and code review?
Over time, I've developed a comprehensive approach to deal with this kind of issues. In fact, while I appreciate the need to trust your guts when you make this kind of decision, I find that answering a detailed set of questions provides you with two benefits:
1) you have a clear set of values to consider, where "value" means both something that is valuable to know, and a precise answer to ponder.
2) since in most cases the answers must come through extensive talking with the third-party, you get to know them much better, well under the sugarcoating of glossy brochures and nice presales. This will help your guts too :-).