The Software Supply Chain Problem

May 19 08

The Software Supply Chain Problem

Paul Weinstein

Last week a dust up occurred in part of the software industry relating to a security issue in a key software toolkit. Apparently two years ago, someone ran an analysis tool on the source code to the security toolkit OpenSSL in the Debian Linux distribution. The tool reported an issue within the OpenSSL package included by Debian, so the Debian team decided that they needed to fix this “security bug”. Alas the solution broke a critical element of OpenSSL, its random number generator, (Long story short, a truly random number generator is critical to software encryption tools such as OpenSSL.) The end result is that for the past two years security applications on Debian and Debian related distributions have been “hackable” and need to be rebuilt.

Each side in the matter is blaming the other. A member of the OpenSSL team suggested that “had Debian [submitted its code changes], we (the OpenSSL Team) would have fallen about laughing, and once we had got our breath back, told them what a terrible idea this was.” Debian developers on the other hand have noted that the email address provide by the OpenSSL team is incorrect and that overall documentation on the part of the OpenSSL team is lacking.

As with our own service issue from a few months back pointing fingures isn’t as helpful as discovering where the chain broke and why. In both cases the issues are eerily similar, a break down in customer/vendor communication.

In Boston Ben Hyde deftly makes a connection between his local butcher’s meat packing industry and his own and in the process wonders what might be the fallout of interdependent web applications circa 2008. Here in Chicago, the former hog butcher for the world, I think we are just starting to see questions and concerns of “quality control” starting to percolate into the public consciousness as the software supply chain between “suppliers”, “vendors” and “customers” grows in sophistication.

Last Labor Day the Chicago Park District recently revealed a statue at the corner of Pulaski and Foster, just a short walk from my home here in the Albany Park neighborhood, in honor of the local park’s namesake, Samuel Gompers. Samuel Gompers was an American labor organizer, union leader and founder of the American Federation of Labor (AFL). Unlike some of his contemporaries, Gompers doesn’t seem to have considered himself a Socialist, Anarchist, or even a Communist, which in today’s political world would probably place him and his beliefs somewhere near the center of America’s political spectrum. Although at the time he’s ideals clearly fell progressively left of center.

Upton Sinclair, a junior contemporary of Gompers, was, no doubt about it, a Socialist actively advocating socialist views. In fact, while he gained particular fame for his 1906 novel The Jungle, which dealt with conditions in the U.S. meat packing industry, in turn causing a public uproar that partly contributed to the passage of the Pure Food and Drug Act and the Meat Inspection Act in 1906, Sinclair himself felt the meaning of his work had been lost on the general public. His outcry wasn’t about the conditions of the meat so much as it was about the human tragedy lived by the workers in the plants handling the meat.

And yet, The Jungle did ultimately bring about change. Perhaps not the change originally intended by its author, but change did come to the growing complexity of the American food supply chain of the early 20th century, a supply chain in which the quality control problems of the time started to get dealt with as regulations and greater customer awareness started to take hold.

A Zoomshare service outage, while problematic, is correctable. A security breach from improperly patched software from two years ago is a little harder to correct….

Recently TJX Cos., a discount retailer that operates T.J. Maxx, Marshalls, HomeGoods and A.J. Wright stores started mailing notifications to customers about a recently arrived at settlement to a class action suit in relation to a January 2007 report that computers that handle customer transactions at a number of its chains were broken into.

What if – and this is just a hypothetical here – what if the TJX issue was related to the Debian/OpenSSL fiasco? Who would legally be on the hook? TJX? Debian? OpenSSL? All three?

What are the implications? We are already seeing customers and regulators react. Services such as Zoomshare post Privacy Policies and Terms of Service. States such as California have passed laws requiring immediate notification if customer data is compromised.

It seems easy to wonder if the computer industry is one Upton Sincalr expose away from greater public and governmental outcry. Even without a “man-of-the-people” individual looking to correct some of the inequities in the IT industry one can see changes are brewing as the overall complexity of our systems grow – along with our greater dependence.