Matthew Tippett
© 1999; Tippett Information Consulting Pty Ltd
http://www.ticons.com.au/
Appropriate support frameworks are considered by some to be one of the fundamental building blocks for widespread acceptance of Open Source Software in the wider corporate community. Provision of support for Open Source Software can be achieved in a sustainable manner by maintaining a strong focus on the Open Source Community and working with the Community.
There are a number of possible frameworks that implement support within an Open Source Environment, each with distinct benefits and drawbacks. In most cases all can provide suitable levels of support within their defined limitations. Examination is made of the current state of the Open Source support market, presenting a number of currently existing support providers.
Open Source Software (OSS)
There are some terms that are worthwhile defining within the context of this paper. These terms are sometimes confused or misconstrued, and so a solid term of reference is useful.
Open Source - Products developed where the source code is freely available. In addition to the code being freely available, the licensing of these products is such that the freedom to modify and distribute is always maintained. The definitive definition of Open Source is available at [OSD].
Proprietary - Products or protocols that are licensed, or held in such a manner that they are not freely available. Typical proprietary products are binary only products where there is no disclosure of source code at all. Even some products where the source is freely available are considered proprietary since the owners of the copyright have a license which restricts the core freedoms that truly Open Source software include.
Most of the real world support mechanisms outlined in this paper are primarily targeted towards Linux distributions. The use of the word Linux is very common in their literature. In reality most Linux distributions tend to be built from a number of OSS products that are not Linux. Of all the OSS products, Linux is currently receiving the largest amount of media attention, which may explain some of the reasoning behind Linux being the major target.
When an organisation takes on supporting Linux as it is known now, they are actually taking on the support of products such as Apache and Samba in addition to Linux (which is just the kernel). These products are just as suitable to be supported in their own right.
When an organisation begins to use a product they will most likely need support at some time. Support as defined by The Jargon File[TJF] is as follows.
support n.
After sale hand holding; something many software vendors promise but few deliver. To hackers, most support people are useless -- because by the time a hacker calls support he or she will usually know the software and the relevant manuals better than the support people (sadly, this is not a joke or exaggeration). A hacker's idea of `support' is a tête-à-tête with the software's designer.
Light heartedness aside, in essence support means helping an organisation understand, utilise, fault find and maintain a product. Whether a product is Proprietary or Open Source, Commercial or Community, it still needs support.
In reality there is no single definition of support. Each organisation has its own unique requirements which make up its own definition of support. Variables including financial constraints, in house knowledge and reliance on the products, all begin to paint support in many different colours.
As a consequence of this multi-coloured definition of support, there is no single support framework that will assist all organisations.
Although it has been highlighted above that there is no precise definition of what support is, we can look closely at what organisations typically need in terms of support.
Some of these needs are valid. Others just give comfort in a corporate environment that if something goes wrong, all bases are covered. This may even be if they do not really assist an organisation to actually support the product.
Following is a list and discussion of some the major needs that are required from an organisation that provides support.
Access to Information - For an organisation to make appropriate decisions about a particular technology it is important for them to have easy access to pertinent information. This could be in the form of case studies, white papers, tuning guides or just the raw source code.
Product Knowledge - An organisation providing support must have in depth knowledge of the products that it is supporting. A customer should be able to contact the supporting organisation and request particular information and within a reasonable time frame get a qualified answer.
Appropriate Escalation - When something does go wrong, a support mechanism should have a well defined procedure for escalating the problem to someone who can give a definitive answer.
Quantified Process - When a support agreement is created, the supporting organisation should give transparency as to how it will supply support. This gives the customer an understanding of what occurs when they do log a support request.
Someone Accountable - If a problem does prove intractable, it is sometimes felt that having an organisation that can be held accountable for what has happened is useful.
On occasions this need is sometimes confused by using the phrase 'someone to blame' or 'someone to sue'. Very rarely in the corporate world is a problem so intractable that a support organisation has been taken to court. The onus is really on the customer to ensure that the product (and the purchased support) suites their needs.
Suitable Responsiveness - Timeliness in response, escalation and resolution. Although covered under the support agreement, a supporting organisation must ensure that it provides responsiveness to the satisfaction of the customer.
Guidance on Implementation - Not all organisations have appropriate knowledge to successfully implement a solution. The offerings that a support organisation provides should include guidance on implementation. This is quite important for the supporting organisation since they will effectively take on responsibility for the implementation that a customer has. A poor implementation will make a system difficult to support in the future.
Information Transfer - Most organisations prefer to be able to train an internal person to do first level support. Consequently a large number of support organisations provide training services either directly or through other organisational branches.
With respect to the accountability, it has been commonly held that for software to be used in the corporate environment, it must have an entity behind the software that is capable of being sued. This in reality is hollow rhetoric. Almost all proprietary and OSS have disclaimers protecting the programs authors (corporate or otherwise) from almost all liability.
Hence accountability isn't so much that the software will work, and if it does not that the vendor can be sued, but that an organisation can be assured that it will be able to keep a support organisation accountable under the terms of a support agreement.
In the proprietary world, sites like Microsoft's Support Site provide most of the information sharing needs online, with references to programs that will allow organisations to co-ordinate the non online components of support such as training.
In a lot of ways, support of products in a proprietary world is a lot easier. An organisation such as Microsoft have all the information itself. It can provide a one-stop shop for all support requirements for its products. The vendor pays for the support engineers and the developers and so can easily provide solutions to all the needs outlined above. These costs are forwarded on to the user in maintenance, licensing and support costs.
When an organisation takes on support for a product that it has not developed itself, it will usually partner (either formally or through support channels) with the company that developed the software. This allows the organisation to provide solutions to these needs using resources within itself and some resources from the company who developed the software that they are supporting.
In an OSE the rules are very different. As indicated previously the community is the vendor, and consequently some of the needs are handled by the community already. The list of needs outlined above are addressed below in the context of the OSE.
Access to Information - Ultimately since the source code is available, the ultimate source of information is immediately present. Coupled with this there are typically multiple knowledge-bases around in the form of mailing list archives, bug tracking systems or documentation provided with the product.
In some areas the OSC does need to work hard to produce more suitable information in areas such as application tuning and troubleshooting guidelines. The Linux Documentation Project (LDP) has already done much work this area, but there are still voids that are not currently filled.
Product Knowledge - Almost all OSS projects have a core set of developers and users. These people tend to offer themselves to the user community through mechanisms such as mailing lists or Usenet News groups.
Product knowledge is gained through working with, supporting and further developing the product. Like proprietary software, this knowledge is brought through long term experience with a product.
Alternatively using Linux Certification allows an organisation to provide engineers with at least a reference body of knowledge. Open Source Certification (as it covers more that just Linux) is something that is relatively new to the Open Source community and over the coming months will begin to grow.
Assisting in the further development of a product is an area that an organisation should seriously look at. It begins to place the organisation in a more traditional position. Organisations such as Linuxcare and SGI use an alternative approach and have hired existing OSS developers and use these people to assist them in both the development and the support of that product. In both cases core Samba developers are now under the employ of these companies, allowing continued development of Samba, which both companies are now supporting or selling.
Appropriate Escalation - This particular need is difficult to maintain in the OSE. In a proprietary environment the core developers are typically managed by the same organisation that supports the product. If needed the developers can be tasked internally to assist the support personnel in solving a problem.
Within the OSE an organisation may have highly trained professionals to deal with support requests, but typically they do not reach the all encompassing knowledge that the core developers have. As the definition of support in [TJF] indicates, the highest understanding of a software product comes from the system designer. In the case of OSS projects, the core developers carry this knowledge.
For an organisation supporting OSS to achieve this level of escalation, the organisation must either become intimately involved in the OSS project or, as indicated above, bring some parts of the OSS development in house, leaving the core developers available internally to discuss support related problems.
Quantified Process - An organisation will define the limits of how it will offer support. This process will have to be defined in a similar way to proprietary support mechanisms.
This process may vary depending on what level of support is being offered. A quantified process might entail some of the following features.
Someone Accountable - The agreement between the support provider and their customers will always define the level of accountability, irrespective of the software model (OSS or proprietary) that the software has been developed in.
In the proprietary world, the software license typically disclaims the software vendor from most liability for the product. OSS products also disclaim legal liability.
In the OSE, developers tend to concentrate on developing the products to completion, rather than producing a product to be released by a certain date. As a consequence of this differing focus, most OSS projects typically have higher reliability and if problems do occur, they get fixed quickly and the patches made publicly available.
In reality it is not so much someone to be held accountable when something goes wrong, it is more someone to lean on to get things fixed when something goes wrong.
Suitable Responsiveness - For any support organisation, proprietary or Open Source, the initial level of response will depend on the agreement between the organisation and the customer.
There are occasions where the support organisation will need to escalate a problem to the point where it receives the attention of the OSS developer.
If an organisation does not have the required developer in house, how the particular support framework implemented by the organisation integrates with the OSC will reflect the responsiveness of the OSC to the organisation.
Guidance on Implementation - Since in the OSE the community takes on the development and management of the software, the community also tends to provide case studies on implementations of the software. This may be in contributed examples or white papers. A lot of this work has been done through the LDP, although few of these references are targeted at the corporate environment.
Apart from the vendor equivalent written information, organisations offering support can provide guidance and direction on implementation strategies. This is not necessarily a support organisations prime role, but if they provide solutions to the some of the above mentioned needs then they are suitably armed to provide this information.
Information Transfer - OSS support organisations get the benefit of real world corporate experience with the software. They can provide feedback into the community in the forms of reference material. Providing this information back to the community allows the organisation to both fulfil this need and assist in the general level of knowledge about implementations of the OSS products which they are supporting.
Information transfer can also be accomplished through techniques such as training and certification. In the OSE other organisations can provide solutions to this need, reducing the burden on the support provider. Strategic partnerships can be formed with specialist organisations to allow customers easy access to training and certification.
Once the support needs that an organisation is intending to fulfil are identified it is necessary to package the solutions to these needs in a support framework. This framework is what prospective customers will look at when evaluating if a support organisation fulfills its requirements for support.
Since support means many things to many people, there are a number of frameworks that address particular sets of needs.
In this framework an agreement is made to manage and maintain part or all of an organisation's OSS based systems. Typically these organisations support not only their own products, but a number of other vendor's product. For most of these organisations OSS products become just another product supported alongside their other proprietary products.
This sort of framework is commonly targeted at corporate enterprises. A single support organisation that fulfils most of the enterprises support needs fits comfortably with management and allows streamlining of problem resolution.
Examples of this are in the support offerings of HP.
In this framework the support organisation provides complete support for all the OSS products an organisation uses. Very similar to the corporate support services, except that OSS products are the only type of products supported.
Linuxcare offer this form of support.
In this framework a vendor of a suite of products provide support for the products they produce. You contact their help desk and receive support through on-site support, telephone or email mechanisms.
You always know that for a particular product you can contact a particular support organisation to get your support. You know that from a shrinked wrapped point of view you are getting support from the closest thing to a vendor.
This sort of support is the style that most proprietary software companies offer. In the OSE, Caldera and Red Hat offer an example of this framework in its standard support offerings. Cygnus have been providing this type of support for gcc and other value added products for a number of years.
This framework is where a vendor authorises a number of external support providers to provide support on behalf of the vendor. This allows the support organisation to provide support in areas that may have no organisational presence.
Again, both Red Hat Software and Caldera have this sort of support framework in place.
This framework is when a support organisation is formed under which other organisations operate. A customer contacts only one organisation, with the support task forwarded on to an appropriate member organisation to handle.
Most importantly, this mechanism is vendor neutral. It is intended to provide support for a wide range of products through by utilising a larger number of consultants.
The Linux Group and Linux Support Services follow this support framework.
This framework is where organisations assist each other to communicate about problems that they are having through a mechanism that allows them to have these problems tracked and resolved.
From a customer point of view, they can choose a number of support providers who they know all have appropriate levels of expertise and pull from the same common pool of knowledge if they cannot resolve the problem themselves.
The Open Source Support Initiative is an example of this sort of framework.
Some OSS projects have co-ordinated efforts for providing support. Be this in issue tracking or bug-tracking systems, or highly managed and moderated mailing lists. You post a problem and you know that the problem will not be forgotten. There is no real onus on anyone to actually solve your problem, but it will be tracked. Most online systems strive to bring to completion all problems being tracked by their system.
There are also product neutral online systems, the free portion of Linux Support Services.
A large number of Open Source projects offer this sort of mechanism, this includes the Samba and Apache projects.
Historically independent consultants, or one man bands, have been the primary means of obtaining support for OSS products.
There are numerous small consultancies that offer OSS support. The major disadvantage of this sort of support mechanism is that typically support for the OSS products goes through a small number of people, and are very sensitive to staff movements and external resource pressures.
Traditionally support for Linux and most OSS has been done through ad-hoc methods. You have a problem with a product, and you email a mailing list or post a message to a newsgroup. It is ad-hoc since you have no guarantee that there will be a solution to your problem.
This method has supported the OSC for many years, the 80/20 rule applies very well to Ad-Hoc methods - 80% of problems can be solved through these methods, the remaining 20% have been usually resolved through consultancies - typically internally or through the hiring of specialised consultants to solve the problems.
In reality, the support frameworks above are intended to fill in the 20%
Of the frameworks described above, the most suited frameworks for the OSE are those which are centred around the community.
Linux and almost all of the true Open Source applications that reside on top of it are a product of the Open Source Community (OSC). It is not difficult to perceive of the Community as a somewhat loosely coupled vendor. It does serve most of the major services that a traditional software vendor provides, albeit on a smaller, more product oriented way. These services include continued development, issue management, support and problem resolution.
Most software products are one of many that a company may sell or distribute. Due to organisational constraints a company typically has a support group that deals with support issues independently of the group that actually develops the software. In the Open Source Environment (OSE) the developers and other project participants are the ones most suitable to support the product. This is since most OSS developers work with, develop for, provide assistance for and just have plain pride in the product. Most major OSS projects have mechanisms in place, similar to larger organisations, that allow for discussion and problem resolution on a per product level. The OSC does indeed support the products.
Dealing with a vendor that is not a tangible entity is a relatively new concept to the business community. The game is now played by different rules. You cannot strike an agreement with one person on behalf of an OSS project. To gain benefits from the community an organisation must become part of the community, dealing with individual members as appropriate.
If, for example, an organisation has a critical reliance on software that is OSS, the risk can be reduced by the either forming alliances with the community (through development or active participation), or by using support mechanisms that are closely aligned to the OSC.
Of the frameworks described above, the most suited frameworks for the OSE are those which are centred around the community. This can be easily understood in the context of The Community is the Vendor, since in the proprietary world any organisation that provides support for products other than its own typically receives input and assistance from the product's vendor.
The support frameworks from Commercial Consortiums and below have the lowest entry cost to the OSC. These frameworks can easily integrate with the community and work with community assets to assist in support. Due to the alliance nature of these frameworks, the community effectively becomes another ally.
The vendor based frameworks and above are more prone to being controlled by vendor or product oriented forces. This may result in pressures being applied that result in the community coming second best to commercial interests, effectively cutting the organisation's own wrist with regard to the vendor it is really supporting, the OSC.
Imagine a scenario where an organisation has developed a support mechanism where they maintain tight controls on the inflow and outflow of information. This information can be invaluable to the OSS developers who develop the products that are supported by the organisation. If the information is withheld, the OSS developers may begin to feel alienated by the support organisation and become unresponsive to the organisation.
Any organisation that is going to be supporting OSS will need access to information to provide appropriate levels of support. For any organisation of any size, this information will primarily come from the OSC.
If the organisation begins to alienate the developers of a particular OSS project they can leave themselves in a situation where they will be forced to rely on their own understanding of the issues and solutions to do with that particular product. Therefore information being commonly available is critical to both the OSC, and to organisations supporting OSS products.
An organisation can only afford to finance a finite number of support engineers. No single organisation can be everything to everyone. By adopting a community based model, an organisation can leverage the collective knowledgebase that exists within community and fulfil, as Eric Raymond put it, Linus Torvald's Law of 'Given enough eyeballs, all bugs are shallow'[ERW]. No single organisation could afford to employ all the best support engineers, working with them through the community is the next best thing.
An example of this scalability within the OSC, Mindcraft was sponsored by Microsoft in early 1999 to do a comparative benchmark[MND] of Linux/Samba/Apache against Windows NT. Although there where political issues regarding the comparison, there where a number of scalability issues that have since been found and rectified within the Linux kernel. These fixes have already been tested and promoted to the stable series of kernels. Very few organisations would be able to afford to employ the number and calibre of the people that solve these sorts of problems with their current speed.
Similarly when other industry wide security flaws have been found, the OSC has become legendary for resolving problems quickly. This is only possible through the communication of many developers throughout the OSC.
With community based efforts, information tends to be pooled. Documents on particular problems that have been found will be collated and managed. This already occurs in the HOWTO section of the Linux Documentation Project (LDP)[LDP]. This knowledgebase becomes a community asset.
Non-community based efforts will tend to try to keep their own knowledgebase and references. The more independent support organisations, the greater the duplication of effort.
Of course there are some disadvantages in a community based model. In most cases these problems can be minimised through being a responsible member of the OSC, or analysing as to whether the net benefit of being within the OSC satisfies an organisations requirements.
By adopting a community based model for support, an organisation providing support may find that they are placed in a situation where they must begin to release some of the income derived from support to a number of third parties.
By concentrating effort on and building competency in a number of core products, an organisation can reduce the possibility of this happening. This is since the organisation will need to contact third parties less since they already have the knowledge in house.
Since in a community based model an organisation is dealing directly with members of the OSC, there may be situations where the political or ideological stances may be different. The risk of this sort of thing happening can be greatly minimised by an organisation being a contributing member to the OSC.
In recent months (especially as this paper was being completed) there have been many announcements of support mechanisms becoming active for Linux. It is beneficial to summarise some of the activities.
HP are offering 24x7 support services with 0-2 hr response times on Linux based Operating systems and HP's Linux applications.
Linuxcare offer either frontline support services, where support is handled by Linuxcare on anything up to a 24x7 basis, or backline support, where Linuxcare assist an organisations internal IT support group in solving problems within an organisation.
Caldera offer up to 24x7 support for all Caldera products. They also have independent support providers who offer support on behalf of Caldera.
Although not entirely apparent from their online support information, Red Hat appear to offer both Vendor-based and Vendor-program support frameworks.
800Linux provides support for Linux based products up to 24x7 with one hour call back and on site support.
A consortium of consultants that can provide support for Linux based systems. Primarily based in Australia, this consortium can provide support in most states.
Originally Linux Support Services started out as a free online Linux support mechanism, out of this grew a commercial version. The commercial section of Linux Support Services addresses the needs of the Corporate sector, while the free version addresses the needs of the general user community..
An affiliation framework, the OSSI provides consultants with access to other consultants who can assist in the resolution of support problems.
Despite a perception in the media about the lack of support for OSS, the market is truly alive and kicking. Unfortunately most of the larger support frameworks currently in place target vendor specific implementations (such as Red Hat or Caldera), and ignore the fact that most of the OSS products that are bundled with these products are suitable to be supported in their own right.
As OSS in the commercial setting begins to mature, more support mechanisms will become available that will target the products individually. Ultimately allowing organisations to use OSS on a range of platforms, Open Source or otherwise, and still receive appropriate levels of support.
The author of this paper feels that in the long term support mechanisms that are based around the products of the community, rather than vendors particular implementation can leverage the knowledge and swift reactiveness that the OSC provides.
I would like to thank the following people (in alphabetical order) who have assisted me with the development of this paper.
Since this paper is intended to be a non-online reference as well some of the links embedded above, are included below in their expanded form.
| 800Linux | http://www.800Linux.com/ |
| Caldera | http://www.calderasytems.com/ |
| HP Linux Support | http://www.hp.com/ssg/ds/ds56.html |
| Linuxcare | http://www.linuxcare.com/ |
| Linux Support Services | http://www.linux-support.net/ |
| Microsoft's Support Pages | http://support.microsoft.com/ |
| Open Source Support Initiative | http://ossi.ticons.com.au/ |
| SGI | http://www.sgi.com/ |
| Redhat Support | http://www.redhat.com/cgi-bin/support |
| Samba | http://www.samba.org/ |
| The Linux Group | http://www.linux.com.au/ |