CONFERENCE SPONSORS

 

 

 

 

An Overview from Sofcon 2007

• Context: Why There’s Nothing Like the Software Industry (and Never Has Been)
• Buzzwords from Sofcon
• How Software Firms Are Driving Down Costs (and Why They Have To)
• The Future of Innovation
◦ From Trickle-Down to Bubble-Up
◦ Service Innovation: from SaaS to the Systems View
◦ From the Killer App to the Killer Data
◦ What Open Source Opens Up
• The Art of the Long View and the Big Picture (‘Is Planning Dead?’)

Context: Why There's Nothing Like the Software Industry (and Never Has Been)

Industries that change the world tend to deliver broad-based enabling technologies: goods that can be used in many ways, for many purposes. This was true in the past of steam engines and railroads. It was true of telecommunications, electric power and motor vehicles. All produced immense fortunes and employed millions; all transformed society while becoming integral to its fabric.

The industry playing that role today, and playing it as never before, is software. It had long been a dream of inventors to make something that would embody intelligent operation — that could think and work along with us. For years, people imagined this might be done by creating androids (as in Karel Capek’s 1921 stageplay R.U.R., which coined the word “robot”). What we’ve gotten instead, with software, is even more miraculous: an “embodiment” of intelligent operation that takes practically no physical space.
• Software can be, and is, put to work anywhere. It can be embedded in any device or called up from afar, on demand.
• Software is used to design almost everything — including the hardware that it runs on.
• It is a chief driver of productivity gain in other industries; it stores knowledge and creates new knowledge.
• For an item so complex, software can be made fairly quickly and cheaply — no wet labs or earth-moving equipment needed — and it can be changed and added to endlessly.
• In just a few decades, the software industry has produced the world’s richest man … and has also become the main avenue of upward mobility for people in some of the world’s poorest nations.
• For the mystically inclined, software is the soul of every machine, including our mortal brains. (What lives on after you die? The software!)

At Sofcon 2007, the vast scope and protean nature of this industry was a recurring theme:
• There are now about 12 million programmers worldwide (Evans Data Corp. projects 17 million by 2009). This is an amazing global beehive — and Tim Chou, the former head of Oracle On Demand, noted that it is highly dispersed across economic sectors: “Citibank actually has more programmers than Oracle.”
• Or as the venture capitalist Ann Winblad put it: “Everybody is a software company, not just the companies in the software industry.” For those within the industry who serve enterprise markets, she added, this means that “we have to provide platforms not just to do things, or to improve revenue or to lower total cost of ownership, but we’re providing platforms for innovation to every customer in the world … [we’re expected to give them] an opportunity to differentiate their core advantages through software.”
• New uses of software keep being found. Various speakers cited them across a dizzying range of fields and functions— from poultry farming to publishing, from finance to charitable fundraising, and from sales optimization to search optimization.

Growing Old or Growing Up?

The software industry is also “maturing” in a curious way. On the one hand, many of the classic and usually ominous signs of maturity are present. Margins are being squeezed (Ray Lane of Kleiner Perkins cited data showing that 75% of industry profits go to only three companies: Microsoft, Oracle and SAP); consolidation among big firms is well under way; and the distinguished scholar David Messerschmitt warned that a lot of the “low-hanging fruit” has been picked.

Yet at the same time there’s been a great surge of new entrants and new ideas. Time and again, speakers at Sofcon referred to areas in which “we’re just getting started” or the business models “haven’t settled yet.” Michael Cusumano of MIT asked: is the industry really entering a mature commodity phase, or is it just “in between platform transitions”?

And what the entrepreneur Kim Polese said about open source seems true of software in general. She said “the challenge is abundance” — not a dearth of opportunities but how to find the right ones and how to execute.

The following Sofcon summaries may help shed light on that.

And for those who like their wisdom bite-sized, we’ll lead with:

Buzzwords back to top

Okay, some are phrases. Here, with definitions, are the most popular from Sofcon.

“The triple-A business model” — one that won’t get you far. VCs are tired of hearing simple-minded pitches for web-based, ad-revenue startups built on “Adsense, Ajax and Arrogance.”

“Data exhaust” — the data created when anyone transacts, searches for, or interacts with anything. Like motor vehicle exhaust, it’s not to be spewed about heedlessly. But opportunities abound in capturing it and using it wisely and well.

“Sweet spots” —zones in which productivity and profit margins are high, and the rates of return can increase with scale. Conversely, sour spots are zones of stagnant or diminishing returns. The entire service sector of the economy was once thought to be a sour spot, but now we are learning that it ain’t necessarily so.

“Not a lot of love” — alas, what enterprise customers feel for their software vendors.

“Service systems” — what you ought to be thinking about instead of the delivery model. A service system is the whole agglomeration of what’s involved in performing a service; it is that which causes the venture to prosper when gotten right. IBM’s Paul Maglio defined a service system as “a value co-creation configuration of people and technologies, connected to other systems by value propositions and shared information.” The phrase then became a source of fun, as other speakers kept asking Paul to repeat this mouthful, but they did so with good reason: it’s a pretty good definition.

“Platform” — a magical entity on which other things run or are built. The word can be used in many senses (try searching for “platform” through the Sofcon transcripts). Sometimes it’s nice to own a platform, sometimes it’s nice to have a free one you can use, and sometimes there are too many of them or they’re changing, and the trick is to deal with that.

“Mature industry” — a term we never thought we’d hear applied to software. Many of us aren’t very happy to hear it, as the usual meaning is “all downhill from here.” But the message from Sofcon was: take heart. Though margins are tightening and lots of software is being commoditized (an equally dreaded word), a number of the “mature” aspects of the industry — the ubiquity of software and programmers, the shrinking costs — are setting the stage for new forms of innovation and growth.

“Free” and “Open” — the apparent future of everything in software, but not quite, since people who create it must somehow earn a living. Thus much business thinking revolves around matters such as how to make it “free but not free” (Google, Visible Path, etc.), and the many ways that open source can fit into the new “ecologies of innovation.”

How Software Firms Are Driving Down Costs (and Why They Have To) back to top

While it’s not strictly true that the price of everything tends to go to zero, the software industry has entered the type of spiral seen in many industries before it: firms keep finding new ways to cut costs and give customers more for less, but the same factors that enable them to do that also exert downward pressure on revenues and margins.

There are some interesting twists in how it’s happening in software, however. Factors like economies of scale, which were keys in driving down costs in manufacturing, aren’t playing as much of a role in software. (Although as Ray Lane noted, the real reason for a lot of M&A in our industry is not to “acquire functionality” but to spread operating costs across a larger customer base.) Meanwhile, software seems to get an unusual amount of cost reduction from innovations in business models and organizational models (including the open source movement, which has no real parallel in other industries).

Also, whereas cost- and price-cutting measures in other industries have usually been more available to larger firms — thus tending to produce a few big survivors with many casualties — many of those in software are quite accessible to small shops and startups or are even targeted to them: Amazon Web Services, virtual servers in the cloud, find-a-coder global outsourcing networks and more. Thus entry barriers stay low, and as Michael Cusumano pointed out, while it’s long been possible to start a software company with “two guys in a garage,” now it’s arguably getting easier to start a pretty sophisticated one and even bootstrap it.

But our Sofcon experts agreed that when cost advantages and price pressures are working in tandem, the trick is to stay ahead of the game. More than ever, execution counts. Here are a few of the pro-and-con cards that have to be played skillfully, plus a couple of wild cards:

Globalization of the industry
• Why it’s an advantage: plenty of talent out there, able to do things cheaply.
• Why it’s a pressure: plenty of talent out there, able to do things cheaply.

Open Source
• Advantage: free, open software you can use in many ways.
• Pressure: it’s hard to compete with free.

New Revenue Models

The most popular of these come under the catch-all term “free but not free” — that is, you offer your software (or what your software does) free to users, but make your revenue in other ways: through ad sales, through related activities, through charging for special versions of the free thing.
• Advantage: gets lots of software into the hands of people who need or want it — free! — and you can build from there.
• Pressure: you’d better know what you’re doing. VC Ann Winblad of Hummer Winblad said:

“You don’t know the number of startups that come to us and say, ‘Our revenue model is advertising.’ We ask them, ‘Have you ever sold advertising? Do you know about things like make-goods? Do you know what kind of CPMs are prevalent across your industry? Do you know where the advertising dollars are going today? Do you know whether you sell to an agency or the advertiser directly?’ If you can’t answer those questions … you certainly should not be hoping that you have a ton of page views and get purchased before you actually have to execute against your revenue model.”

Virtualization
• Advantage: virtual servers, virtual testing environments and the like can give you much more bang for your IT buck.
• Pressure: no apparent downsides here … unless your firm happens to make some kind of software that’s going to be disintermediated or obsoleted by virtualization.

Software as a Service (SaaS)
• Advantage: customers like it for all the well-known reasons — they don’t have to maintain the stuff; they can “pay by the drink,” etc. And as a software provider, SaaS + web marketing can let you bring something to market without fielding a sales army.
• Pressure: Saas squeezes the cash flow into startups — instead of up-front license fees for your new product, you’ll get strings of monthly payments for your new (ahem) service. (Investor Bill Burnham: “working-capital planning is critically important.”) And in existing product firms that wish to convert to SaaS, it can impose steep transition costs: everything from development to sales and support has to be done differently.

Finally, one factor that is strictly a negative pressure on prices … and a recently discovered dynamic that requires a bit of navigating.

Perceived Lack of Value

Many customers, especially in the enterprise market, aren’t sure their software is worth what they’re paying. Nick Carr’s book Does IT Matter? struck a chord that still resonates, and as Ray Lane said at Sofcon:

“If you test what customers think of Oracle or CA or Microsoft or SAP, there’s no one saying ‘I’m buying from these companies because they‘re innovative. They are fresh. They re offering me high value.’ These customers have allocated 80% of their budgets to maintenance [and] they don’t see the value, other than that they have to do it.”

Lane’s ominous bottom line: “There’s not a stable environment between the customer and the provider.”

Sweet and Sour Spots for Companies Shifting into Services

Software product firms tend to turn into “hybrids” as they age. They typically start out getting most of their revenues from product sales, then draw an increasing share of revenue from services— including both maintenance and other services, such as consulting or contracting. Michael Cusumano’s latest research asked: how does this affect profit margins? The findings he shared at Sofcon were striking:
• Up to a point, usually around 22% of total revenues, services have a positive impact on margins: doing more service correlates with higher rates of return.
• But then the effect turns negative for a long period — margins are actually depressed — until service revenues reach about 63% of the total. From there, margins pick up again.

Thus we find “sweet spots” at the low and high ends of the service spectrum with a big sour spot in between. Why is this so? Cusumano hypothesized that at the low end, the services are mostly routine things like maintenance that are fairly easy to do. Then the going gets tougher as firms branch into more demanding or unfamiliar services — until they learn how to systematize the services for better returns.

For more of Cusumano’s new research on the economics of the industry see his presentation. But the story above has a moral emphasized by other speakers as well. There’s a great deal of upside to be gained in how we approach service work. (See “Service Innovation: from SaaS to the Systems View” later in this overview.)

The Future of Innovation back to top

New sources and modes of innovation are emerging in software; advantage will go to firms that can master them. Several were discussed in depth at Sofcon.

From Trickle-Down to Bubble-Up back to top

Whereas major innovations used to appear first in the corporate market and then “trickle down” to the consumer market — computers, mobile phones — they’re now likely to originate in the consumer space and bubble up into enterprises. Tim Chou:

“Historically, if you had wanted to know about the future, chances are you would have gone to Bell Labs (or Xerox PARC) and wandered around … Now, in the world of software, this is completely reversed. The research lab that is being run right now is the research lab of the consumer Internet. And if all of us in the world of business are going to figure out what the future looks like, we should become students of that laboratory — whether that is Amazon, or MySpace, or Wikipedia, or YouTube, or World of Warcraft.”

With tools like wikis and social-networking software bubbling into the enterprise market, Ray Lane, former president of Oracle, has coined the term “the personal enterprise” for the process of incorporating them. Benefits to the end-user firm include:
• The tools “deliver individual productivity and value.”
• Since they are usually web-hosted, they don’t consume a lot of IT resources.
• Acquisition & implementation doesn’t have to be a massive ordeal. Just grab tools from the web, try them, and if people use them and they work well, go forward.

Some requirements for the provider:
• The software indeed must be easy to implement — little to no learning curve for the user; little to no data entry needed; small IT footprint.
• The software must be capable of meeting enterprise standards for reliability, privacy and security.

And since piecemeal adoption of “personal” tools can be chaotic, there’s a business opening here. Lane’s VC firm (Kleiner Perkins) is backing a company called SpikeSource “that’s basically aggregating open source software.” The value proposition to the client, he says, is: “Here is a suite that has social networking in it. It has blogs in it; it has news feeds … The CIO gets only one thing in the enterprise and individuals can use it. Maybe in any one category, it’s not the best of breed. That’s okay. You’ve saved tons of money, implemented it today, and it’s available today.”

Online Role-Playing Games

Games can suggest ideas for new business methods, since they’re task-oriented and take place in virtual societies. For example, Tim Chou described World of Warcraft as a place where people from many backgrounds “form groups to achieve objectives” — which is just what you’re trying to do in a globally networked, outsourced business, so why not study how it’s done in the game? One nice thing about WoW, he noted, is that the game has a credentialing system — players attain skill levels, earn talent points, etc. — so you can tell instantly how good a mage or a warrior someone is. That’s more of a problem when you’re putting together an ad hoc software team.

But then another Sofcon speaker, Martin Griss, pointed out that at least one real-world company has built its business model around the gaming trick. The company is TopCoder. It stages online coding tournaments. Programmers from around the world compete to earn rating points; their standings are posted on the TopCoder website. Client firms can thus hire the talent they need: grandmasters for tough jobs, pretty-good coders for lesser work, and so on.

(Meanwhile, real firms and orgs are literally moving into Second Life, a gaming world that simulates our modern-day world. The American Cancer Society raises real money by holding virtual walkathons there; IBM, Sun and Microsoft have presences in Second Life; Sweden has opened an embassy. One big use in the future may be virtual market-testing. A private toymaking firm already uses Second Life for rapid prototyping and testing of new product ideas, and the MacArthur Foundation could soon be testing new concepts for grant programs.)

Service Innovation: from SaaS to the Systems View back to top

What’s beyond software as a service? This was a major theme at Sofcon, with two entire sessions devoted to it. Some key thoughts:
• When software people think about “service,” they tend to start from the software end. They think in terms of SaaS (a delivery model that essentially turns a product into a service) or they think of ancillary services that can be sold around the product — support, consulting, whatever. And that, said Paul Maglio of IBM Almaden Research, is “a narrowly construed view.”
• The real opportunities lie in starting from the other end. Think first about services that you could provide, or that are now being provided, by you or others. Think of them in terms of a service system, of which software would be (or is) one element. Then ask: how could the system be designed (or optimized) to deliver maximum value?

Firms like Amazon and eBay have done this, of course. They’ve conceived great service systems of which software is a vital part but only a part. (Another key thought, by the way: these are “value co-creation systems” where the customers and users participate.) And you don’t have to launch a big global startup to take the service-system approach. You can look at a niche market, or look for missing links in existing service systems; you can look at how to improve your firm’s internal systems.

A true science of service system design is just emerging. Sofcon speakers Paul Maglio, John Zysman, Bob Glushko and Shelley Evenson presented concepts derived from their work and that of others. For instance:
• Glushko urged thinking in terms of the system’s “front stage” (the part the customer/user sees) and “backstage” (the enabling stuff). Useful questions include: where is it best to draw the line in each case? (McDonald’s hands you a ready-made meal while Benihana prepares the food at your table. Both work, but for different reasons.)
• Zysman noted that service systems are embedded in social and political systems, and must be congruent with the “rules and roles” inherent in those larger systems. (A health care service that works in the U.S. may not fly in France or Denmark, where health care is run differently.) The upside, he said, is that by thinking innovatively about rules and roles, we can come up with unique value propositions.
• Evenson suggested there’s still plenty of room for service-system integrators. In traveling to Sofcon, she said, she used a host of services — she listed many more than the typical air/car/hotel bundle offered by the travel sites — and since they’re all part of a single “experience” for the user, why can’t someone make them work together more seamlessly?

Evenson also stressed the ongoing demand for more speed in services of all kinds: “Service is now.” And that’s more than a sad sign of the impatience of yuppies, for if quality can be held constant, then speed = productivity.

Which, as many speakers noted, brings up the genuine need for more rigorous thinking about service systems. With the world’s economies shifting more and more to service work, we can’t afford to fall prey to Baumol’s disease, the oft-cited failure of service work to deliver rising rates of return along with scale.

Thus far the software industry has played a big part in fighting Baumol’s disease, largely by automating routine tasks. But automation will no longer be enough. As John Zysman said, we’re now at the next frontier: the “algorithmic transformation” of services.

From the Killer App to the Killer Data back to top

In the 1967 film The Graduate, the famous word of advice for up-and-comers was “plastics.” For software innovators today the word may well be “data.”

Time and again Sofcon speakers referred to the opportunities that abound in searching data, in capturing or aggregating data, in creating metadata, and in manipulating, bundling and presenting data.

Venture capitalist Scott Russell, talking about the kinds of startup ideas he favors:

“What I really like is building application software where you control some of the data and the content … where you price your system not based on the software coding itself, so you don’t get into this maintenance/revision cycle, but where you own something in the content that you can resell to vendors. For example, there are lots in financial services. At a Reuters or a Bloomberg they write tons of software, tons of trading systems, and yet they really are selling you data and communications … And the stuff that they know about traders and markets now is unbelievable. The entire financial services industry doesn’t work without those guys.”

Similarly, VC and public-market investor Bill Burnham, speaking of good stock picks in software:

“I might go for something like a Tele Atlas or a NAVTEQ on the mapping side and GPS — very large databases which are incredibly valuable, and then a very heavy software stack on top, and they get integrated into a lot of downstream applications.”

He summed up this type of play as “the analytics platform with all the data.”

One reason data is hot is that there’s so much of it. Tim Chou spoke of the vast amounts residing in the “deep web” — the part that’s not publicly searchable, which exceeds the Google-able “surface web” as the Marianas Trench exceeds a swimming pool. And more is being generated every day: VC Ann Winblad used the term “data exhaust” for the huge volumes of info created when firms like Amazon, eBay or Google mediate transactions for their users.

Much of the data on the deep web is proprietary or confidential. It’s sitting behind corporate and institutional firewalls; it ranges from personal records to trade secrets and military secrets. Typically this stuff makes news when it is hacked into or somehow used illegitimately, but many possible legitimate uses have barely been tapped. Chou said, for example, that just as Amazon recommends books for customers, his bank’s site ought to be able to tell him that “everyone with my profile bought WebEx stock yesterday.”

Ann Winblad cited three big areas of opportunity:
• Smarter tools for internal search. Big organizations are proliferating internal sites and portals, yet employees often can’t find what they need. (“When they look for articles about chip heat dissipation, their search results might bring up who’s bringing chips to the company picnic.”)
• Data integration. One example is Salesforce’s AppExchange, by which data-rich applications like Mapquest and business databases such as Hoover’s can be plugged into sales management software. Windblad sees a need for many more “integration platforms” that let users merge “silos of information.”
• Finally she sees growing use of “very granular, almost real-time data,” as in trading platforms — which “opens up a lot of opportunity in predictive analysis and business optimization.”

The creation of metadata (“data about data,” based on new systems for organizing data) is central to many efforts. Scott Russell:

“If you look at the search infrastructure we have today that’s been built to deliver answers, I think you’re going to see that search infrastructure used to create metadata. In essence, to crawl lots of different data in a distributed fashion, either within an enterprise or across enterprises, or wherever it may be. And with a particular view, extracting metadata and creating a proprietary database out of it.

“There’s a lot of companies doing what’s called log analysis, and then correlating log analyses or extracting data from logs across enterprises in the security space. They’re also doing it in the sales space and the performance space, et cetera. So the business intelligence players are all thinking very hard about getting into this business, because they see that as the next logical space they’re going to have to be in — not just in the reporting space, but actually in helping coalesce all those different data streams, extracting metadata from it, and using it to improve performance.”

As the journey progresses, “there are hundreds of Googles to be had from here to there,” Tim Chou said.

Also, though it’s a bit of a departure from the concept of “data” as discussed here, the drive to cut costs and stretch resources in data centers is spawning trends that reach far beyond the centers. A big one is virtualization — of servers, of testing environments, of resources on PCs, and more. This is slashing costs and cycle times for big enterprises and startups alike, while starting to trigger some basic re-thinking about development and business models. Virtualization firms like VMware, Akimbi and Scalant were mentioned as big recent winners; Ann Winblad noted that penetration is still low and “there is a lot of the virtualization story still to unfold.”

What Open Source Opens Up back to top

Open source was the subject of a specific session at Sofcon, and also came up often in a session featuring VCs and investors (whose basic angle was: yes, open source is hot, but how exactly does one make money from something that’s free?). Experts from both sessions agreed that open source fits into the emerging “ecology of innovation” in a variety of ways:
• Open source software provides building blocks that a firm can use to enter a market or catch up.
• Companies are using open source for the “commoditized” developmental infrastructure, then building their value-added proprietary products on top of that. (Tony Wasserman, an industry veteran and professor at Carnegie Mellon West, said that about half of the exhibitors he talked to at the recent Web 2.0 Conference were doing this.)
• For-profit firms can use open source as a competitive weapon. “IBM’s embrace of Linux as a way to essentially head off Windows NT in the data center was masterful, and I think you’re going to see more of that,” Bill Burnham said; while others mentioned IBM’s use of the Eclipse development platform to eat into Microsoft’s early lead in developer tools for the Windows Server environment. To which Burnham added:

“There’s something like 120 people at IBM working full time on Eclipse — who probably have a big picture of Visual Studio with a bulls-eye on it as well. I think there’s an idealized conception of open source as a bunch of nymph-like programmers all hugging and saying this is great, and the reality is that there are board rooms with people saying, ‘How do I inflict the maximum hurt on my competitors? What part of the value chain am I going to commoditize and convince people that it’s open source as a way to lock them into my other piece of infrastructure?’”
• Firms can use the open source community as a development lab-slash-testbed for new products and features: Netscape and Mozilla, Red Hat and Fedora.

As for business opportunities and how to seize them:
• Many lie in bundling, certifying, and above all supporting open source products for enterprise clients. Kim Polese, CEO of SpikeSource, explained how her firm does this with a unique business model that’s part-closed and part-open. The firm uses an automated testing system, which is proprietary, along with a “federated support” system whereby “we create a network of support partners — committers who are experts in the [open source] codes — and flow revenue back to them.”
• Polese also mentioned new firms like Black Duck and Palamida, which provide open-source asset management to enterprises: checking how much of the firm’s software is open source, seeing that licenses and IP are in order, etc.
• For startups trying to decide if (or how much of) their own code ought to be open source, Ann Winblad had this advice: if it’s a platform type of product, make it open, but if it’s a specialized app, probably not. Citing her own experience as cofounder of an early enterprise software firm, she said:

“We were in what would today be called the ERP space for mid-market. And that becomes highly verticalized very fast. So we started out saying, ‘Let’s write the verticals ourselves,’ and then we thought, well, that was a stupid idea. We didn’t really know anything about freight and hospitals and whatever. And we didn’t have any way to sort of start the market on writing verticals on top of our core … So we just gave away our source code … And before we knew it, we had verticals all over the place. And no one ever just swiped our whole general ledger, payables, receivables, order processing. They left that intact because they knew that was a young piece of code and we were maintaining it.

“For companies that have the opportunity to be a platform, it really is a very, very good model to open the platform and let innovation flourish, so to speak. For an application company that’s very narrow in focus, I’m not sure how being an open source company really gets leverage for them.”

Some caveats and qualifiers:

Jim Herbsleb and his colleagues at Carnegie Mellon have been studying cases in which firms use the open-source community as part of their “value chain” — very often, to build the commodity or developmental infrastructure, which they then can use as a springboard for their own value-added software. Herbsleb warned that these value chains work differently than the usual kind, and “companies have to change their philosophies in some fundamental ways.” These may include
• Getting used to the transparency of the open source process
• Ceding control to the open source community (which will decide, for instance, who can be a committer)
• Gearing up your firm to innovate faster than ever. Said Herbsleb:

“If there’s this commodity base on top of which everyone builds their value-added software, well, as soon as you show something to the community — you know, you put some cool thing out there and charge money for it — then a bunch of geeks [will] decide ‘Gee, that’s cool. We want to build one of those’ … and they’ll make it free. So next year you, as the value-added provider of something, have to innovate and find something beyond that that you can charge for. This is sort of the price of having this openness. Everyone can get in the game and there’s tremendous pressure to stay ahead of it. So you have to put your resources that you’ve saved from not having to invest so much in the commodity piece into the more innovated pieces — which is what most companies would like to do anyway.”

The Art of the Long View and the Big Picture back to top

With the software industry focusing more and more on rapid-fire innovation and the perpetual beta — release it now; keep tinkering — one must ask: is planning dead? Bob Glushko was one Sofcon speaker who raised that topic rather bluntly:

“I think the culture of software as a service and agile development and all this kind of quick, continuous release is almost anti-intellectual in a way, because it forces you to look at such a short term perspective. You don’t really sit back and think about the big issues. You just get stuff happening — bang, bang, feature of the week … [Whereas if] you have a chance to think long term thoughts, frankly, that’s not a bad thing. Being careful and systematic are not crimes in this country.”

Other speakers dwelled in depth on the need to preserve (or expand) long-term, big-picture thinking. A dialogue on agile vs. traditional development brought out the limits of the agile approach and how to blend the two. And David Messerschmitt, a former department head at Cal-Berkeley, issued a “call for collective action” on industry-wide issues.

Where Agile Meets Planning

Adam Blum, VP of Engineering at Mobio Networks, spoke of being a dedicated Rails user:

“I’m spending about 30% of my time writing Rails code, and I love it … As a Rails developer, you can be very agile … You can get things done incredibly rapidly. You can show stuff to customers and say, ‘Look at this. Do you like it or not?’ and change it in front of their eyes … But it’s very clear that the actual development of the Rails platform is not that agile … The gap between 1.1.2 of Rails and 1.2 was quite large. It was about double what was expected, something like eight months… So it’s the users of that framework that are agile. But as for the creators, if you look at not just the timelines, but some of the design discussions that are going on, it’s clear that there are some ‘traditional artifacts’ involved …

“There’s big time involved in truly reusable stuff that has to work with a huge community of people and has to maintain backward compatibility. There’s real work and real analysis that has to take place.”

Nor does it end the matter just to say that “platforms” must be systematically planned so that app development can then be agile ever after. As new kinds of applications are written, we begin to see new commonalities among them, along with the need for new kinds of development tools — and the result, as Blum’s dialogue partner Martin Griss put it, is “kind of a rising tide of ‘What’s the platform?’”

Thus it’s an ongoing struggle to find the right balance, in each case, between step-by-step planned development and agile approaches that dispense with many of the steps.
• Blum pointed out that the agile methods prescribe rules and procedures of their own — the one-month sprint, pair programming, etc. — and one of his main pieces of advice was: if you’ve got a team that’s already performing pretty fast and well, don’t muck them up by trying to impose an entire new system. Pick and choose the pieces you want.
• To which Griss’s yes-but was: “Sure, you can pick and choose. ‘We are going to do agile, but we don’t like pair programming because these guys don’t like to talk to each other,’ right? And a test-driven development model — ‘You know, I’ll write the test after I’ve got some of the code’ … I think you have to pick and choose in some consistent way.” (He recommended Boehm and Turner’s book Balancing Discipline and Agility: A Guide to the Perplexed.)

One can certainly find many situational mixes. Griss noted that open source projects have widely differing degrees of structure and hierarchy. An audience member pointed out that open source, in general, tends to short-cut a classic planning step — “there is no real requirements process” — but in many cases that’s OK, because “the developers are users. They’re power users. They know what the product should do.”

The bottom line is that it’s hard to draw a clear line between “let’s plan it” and “let’s wing it,” at least as far as development is concerned. However in terms of where the software industry as a whole is headed, perhaps the time has come for more planning and less winging.

On Collective Action

In the closing speech at Sofcon, David Messerschmitt raised a point that’s been seldom discussed anywhere: compared to other advanced industries, the software industry takes relatively little “collective action” to better itself for the long term.

By this he meant more than lobbying or standard-setting. He meant firms working together — often in concert with government, academe, and/or firms in client industries — to identify, and tackle, key issues that could either hold the industry back or advance it greatly in the years ahead.

Messerschmitt, former chair of EE and Computer Science at Berkeley and a member of the National Academy of Engineering, argued that the time for such action has come. Software has become mission-critical to the world and does great things, but it often beleaguers the users or just doesn’t perform as they would like. Consumer software gets only mediocre scores in satisfaction surveys, while enterprise customers have a host of chronic complaints — about the burdens of legacy software and “gratuitous changes,” about problems with security and interoperability, about software that dictates how they must run their businesses.

Moreover improvement will only get tougher, he said, as most of the “low-hanging fruit” in innovation has now been picked and complexity keeps rising. So he called for an industry-wide focus on “providing greater value to the user and consumer.” This would entail, among other things, thinking about how to evolve software “in a more graceful and measured fashion, so that the end users are spending less time fighting legacies, and so that we software suppliers are spending less time porting our stuff from one platform to another.”

Roadmapping and Key Issues

One big collective step that would help, Messerschmitt said, is a “technology roadmapping” process similar to what the U.S. semiconductor industry did in the 1980s (which helped that industry rebound from a genuine state of crisis). The idea is to sketch out a desired future state of the industry, targeting areas where concerted effort is needed in order to get there.
• This would include “identifying knowledge gaps.” For instance, the semiconductor industry saw that making more complex chips would call for advances in computer-aided design. By creating centers of excellence at universities, the industry got toolsets along with a new wave of talent skilled at developing them — a strategy that paid double dividends in other knowledge-gap areas as well.

Some issues the software industry might want to address, according to Messerschmitt:
• Managing “complementarities.” Since software always runs on and for something else, the industry has a large number of “complementers” — hardware makers, cell-phone service providers, etc. — and it would help to have more coordination on matters such as R&D investment and infrastructure development.
• Componentization. Messerschmitt noted that in the past, industries like munitions and motor vehicles benefited by working collectively to develop interchangeable building blocks in terms of parts and design. He called for a new approach in software that would involve componentizing “software and hardware together.”

Also, a couple of meta-areas for collective action:

Harnessing end-user innovation — Software is a field in which sophisticated users are already highly prone to develop or improve their own tools. Messerschmitt suggested a number of ways to systematically boost the process:
• Through “end-user organizations in particular vertical industries” working together to brainstorm new kinds and uses of software.
• By “a group of software firms affiliating with a group of end-user firms … to identify innovations that would be of benefit”
• Through new modes of community or collaborative software development that are similar to, but distinct from, the open source model.

Rethinking software education — At many universities, enrollments in computer science are down and few women study the field. There’s also a sense that we’re educating CS majors to be good at the technology, but not at understanding the needs and social contexts of users.

Academe and industry together might explore solutions such as: redesigning degree programs; making CS mainly a graduate-degree field open to a broad mix of undergrad majors; and creating a user-oriented companion field to CS, much as architecture is a companion to civil engineering.

Send-off Note: A More Connected Future

Keynes famously said, “In the long run we are all dead.” At Sofcon, various speakers in various ways said: In the long run we are all more connected.

Software to begin with is inherently a technology that connects people (and lets people “talk to machines,” and lets the machines talk to each other, etc.) [PLEASE ADD A LINE-SPACE AFTER THIS PARAGRAPH]

As for the estimated 12 million programmers in the world today, never in human history has such a vast beehive of intellectual activity in one field burgeoned so quickly.

As for the industry, Dave Messerschmitt’s closing talk implied that moving to higher orders of collective action would be a key step in the industry’s maturity — in the sense of growing up, not growing old.

And Bob Glushko of Cal-Berkeley noted that the emerging movement to think in terms of “service system design” would drive the industry toward “a much more multidisciplinary perspective” — really thinking about how the code and the business models and the delivery models fit in with social systems and how people wish to live their lives.

Somewhere in all of that, there will surely be plenty of ideas you can take to the bank. Altogether, it’s the great endeavor of our time.

back to top

© 2007 Carnegie Mellon West