The above stated characteristics of technical documents tend to drive the following business goals:

For businesses where there are serious legal liability and safety issues, such as aircraft manuals, additional business goals include:

While these two additional requirements could be of valuable in any business, the cost of satisfying them is quite high because of the necessary sophistication of information management systems that can manage the sheer number of versions and dependency links. Thus they are usually only set as system requirements when the potential cost of not having them is high, as is the case for safety critical equipment. The short service life cycle of most consumer products means that there is little need to maintain more than a few year's worth of old versions and it may be sufficient in many cases to simply keep offline archives of source files and delivered renditions as they are produced.

From these requirements and goals it should be clear that the ability to do effective re-use of information objects is a major component of any sophisticated technical documentation authoring and management strategy. Other strategy components include the production of many delivery forms from a single XML source (“single sourcing”), the use of standards such as XSLT, XSL-FO, and SVG to reduce the cost of producing documents in many national languages, and so on.

While re-use has obvious and compelling value, it is important to caution that true and wide-ranging re-use is hard, often harder than most people realize it will be.

Re-use is hard for at least two reasons. First, the technology infrastructure required to make re-use practical for authors is non-trivial and must be implemented carefully and thoroughly. Second, it requires document authors to work collaboratively and often under greater time pressure than they are used to or would prefer. It often requires them to be much more aware of the details of how the source is constructed and how to work with versioning systems in sophisticated ways. That is, writers have to be aware of both how to do re-use appropriately and how to do it correctly using their tools, and do it in close cooperation with other writers under strict time and budget pressures.

As a technical writer by training and practice I know that these are pressures that many technical writers do not like and do not necessarily adapt to well. Many, if not most, technical writers are essentially creative, introverted people who like to work more or less in isolation, focusing on the words they are writing and how they are presented without much concern with how those words actually get stored and processed in their computer. Any XML-based documentation system changes that simply by breaking the direct connection between what you type and what you see. Most first-generation XML system do not change the typical one-document, one writer development process. Most writers can adapt to this type of environment, especially once they see how structured document authoring can in fact make their job easier.

However, any re-use-based documentation system completely breaks this way of working, forcing writers to be information engineers much more than creative writers. This can be a severe and jarring change for many writers. Implementors of such systems must be sensitive to the potential cultural resistance and work practice disruption, as well as the fact that you are, in absolute terms, making writers' jobs harder at the same time that you are making them more productive and efficient. That is, it is always the case that labor saving devices mean that fewer people can do more work, which means that the people left to do the work always have, as individuals, more work to do. This is certainly true for re-use-enabling authoring and production systems. While fewer authors can produce more published products more quickly, the intellectual skills and effort required to do so are greater than before because the actual work activity has become more sophisticated and complex.

There is no well-established terminology for re-use in the technical documentation domain. In this paper and in my professional practice I use the following terminology.^[1] Also, see Figure 1.

The diagram in Figure 1 illustrates the three basic layers in any information management system. At the bottom is the storage management layer, which manages access to storage objects. In the middle is the semantic layer, which manages access to semantic data objects constructed from the syntactic content of storage objects. In practice the semantic layer might be something like a DOM or an abstract API over a relational database. At the top is the application layer, which represents processing applied to semantic objects and storage objects in order to perform some task (which may be to create more semantic objects or storage objects). These layers can be found in any non-trivial system, from complex enterprise information management systems to XSLT scripts applied against individual XML files. In many systems the three layers may not be clearly distinguished or exposed.

Figure 1. Information Management System Layers

If the only concern with semantic re-use was how to implement the data processing required to produce specific deliverable renditions from XML documents then content management would not be a concern. The actual processing of XInclude links, for example, is fairly straightforward and there are a number of systems that do it or implementations that can be quickly integrated into purpose-built systems[Xincl Impl]. That is, given that the documents involved, processing them is easy.

The challenge is in enabling the authoring of documents that do semantic re-use. It is here that challenging issues of content management are unavoidable, especially as the length of the documents' life cycles and the scope of re-use increases.

The key content management challenges imposed by re-use include:^[2]

Thus it is not possible to discuss XInclude and authoring without also discussing content management. One potential danger with XInclude is that it is easy enough to implement that you may implement a system that provides more functionality than authors can practically take advantage of due to lack of comprehensive management facilities. By the same token, it is possible to create relatively low-cost content management and authoring systems that enable the productive use of XInclude by not immediately providing all the link management and tracking facilities that may eventually be needed and by imposing editorial constraints that avoid some of the more challenging linking use cases, such as disallowing links from re-used information objects that are used in more than one use context, which largely avoids the issue of per-context link management.

It should also be noted up front that when using XInclude or similar mechanisms based on embedded XML markup for representing relationships it is always possible to answer questions like “where used” using brute force approaches, which may be slow but that are easy to implement. In addition, simple conventions that make it possible to quickly find the root documents of bounded object sets also make things easier. For example, if you use the convention that every compound document is stored using the same directory structure and naming convention then it is easy to write an XSLT or Java process that finds all the compound documents, calculates their bounded object sets and then reports on how many of those bounded object sets contain a specific document.

Obviously a more complete solution would maintain the knowledge of bounded object set participation dynamically as documents are added to the system and new versions are created, but that is significantly more involved than the brute force method. If you have a relatively low volume of documents and don't always need instant answers to these questions, then a brute force solution may be adequate and, due to its lower cost, actually provide attractive value.

The key point here is that because all the relationships are defined using XML markup there is no absolute requirement to have a sophisticated content management system in place before you can start taking advantage of XInclude. If you do not have an high performance or scalability requirement immediately then it is possible to implement a lot of functionality with suboptimal, but acceptable, performance for a reasonable cost. For example, by using CVS as the base versioning storage management system, you can implement commit-time processing to maintain indexes of bounded object set members or information object where-used information using simple, workaday relational databases and a little bit of Java or Python code.

It is only when you need to support hundreds of authors or thousands of units of publication or many versions over time that it becomes absolutely necessary to step up to more involved systems that are expressly engineered to address the performance and scale challenges.

I find it best to define specialized inclusion elements that make it possible to fully constrain the occurrence of specific references to specific element types. Unfortunately, the current XInclude specification does not provide the same sort of “type” mechanism that XLink[XLink] does, so document types that uses these specializations are not, strictly speaking, conforming XInclude documents. However, they can easily be transformed into conforming XInclude documents if it ever became an issue.^[5]

Using this approach, the first example above can be refined into this:

<!ELEMENT sect
  (title,
   (sect_body |
    (intro?,
     (subsect |
      subsect_include)+)))
>

<!ELEMENT subsect_include
  EMPTY
>
<!ATTLIST subsect_include
  href
    CDATA
    #IMPLIED
  xpointer
    CDATA
    #IMPLIED
  xinclude
    NMTOKEN
    #FIXED "xi:include"
>

The subsect_include is a specialized xi:include element, as indicated by the xinclude= attribute. The href= and xpointer= attributes are taken directly from the XInclude specification.

The xinclude= attribute is my private convention for indicating that a given element type is to be interpreted as one of the element types defined in the XInclude specification. It has no magic. However, XML processors can easily be written to look for this attribute and act on it. For example, an XSLT transform that converts these specialized elements into the proper XInclude elements might include this template:

<xsl:template match="*[@xinclude = 'xi:include']">
  <xi:include>
    <xsl:copy-of select="@href | @xpointer"/>
    <xsl:apply-templates/>
  </xi:include>
</xsl:template>

When defining specialized inclusion elements I like to use a consistent naming convention so that authors can predict what the XInclude element for any normal element would be.

Creating specialized XInclude elements solves the problem of constraining where references can occur but it does not do anything about constraining what the reference can point to. That is, there is still a requirement to constrain the type of thing referenced.

An obvious way to do this is to add another fixed attribute that lists the element types that can be selected. I typically call this attribute reftype, short for “reference type constraint”. The value the attribute allows is dependent on your application's needs. The simplest approach is to specify a single target element type, e.g.:

<!ATTLIST subsect_include
  href
    CDATA
    #IMPLIED
  xpointer
    CDATA
    #IMPLIED
  xinclude
    NMTOKEN
    #FIXED "xi:include"
  reftype
    NMTOKEN
    #FIXED “subsect”
>

This is sufficient when there is always a one-to-one mapping between referencing elements and their allowed target. However, some applications may be more flexible.

For example, many document types use a DITA-like extension mechanism[DITA] to allow controlled specialization of base element types. This means both that it might not be possible to predict what the full set of possible targets are and that it would be meaningful, to the processing application, to refer to any specialization of some base form.

One way to enable this would be to allow XPath expressions that reference targets must match, rather than simple element type names.

To continue the previous example, say that we have a new attribute, base-type= that indicates the base element type of which a given element type is a specialization. For example, a specialized form of subsection might be declared like so:

<!ELEMENT concepts_and_terminology
  (%subsection_content;)
>
<!ATTLIST concepts_and_terminology
  %subsect_atts;
  base-type
    NMTOKEN
    #FIXED "subsect"
>

Here the base-type= attribute of the concepts_and_terminology element indicates formally that concepts_and_terminology is a specialization of subsect. This means that the subsect_include element can be redeclared to allow reference to any specialization of subsect:

<!ATTLIST subsect_include
  href
    CDATA
    #IMPLIED
  xpointer
    CDATA
    #IMPLIED
  xinclude
    NMTOKEN
    #FIXED "xi:include"
  reftype
    CDATA
    #FIXED "subsect | *[@base-type = 'subsect']"
>

Here the reftype= attribute has been changed from “NMTOKEN” to “CDATA” and the fixed value is now an XPath expression that allows either the original subsect element or any direct specializations of it.^[6]

From an implementation standpoint there should be no great difficulty implementing this sort of XPath-based check in either XML authoring tools or in processing tools (although XSLT-based processors will require something like the EXSLT eval() extension function to do dynamic evaluation of XPaths). For example, this type of check requires only a couple of lines of code in Arbortext's Epic Editor[Epic] scripting language.

The XInclude specification makes it clear that XInclude links always point to the targets in their original source context (what I call “as authored”). For example, if you include an element that has as a descendant another inclusion, that second inclusion is resolved relative to the source document that contains second inclusion, not the (abstract) document that results from resolving the first inclusion.

This same principle applies to all other uses of direct addresses: the addresses point to the targets at their source locations, not their locations following resolution of the includes.

By “direct addresses” I mean things like ID references, but also any other form of direct address, such as ID-based XPointers, document-type-specific pointers, and so on. “Indirect addresses” in this context would be addresses that address things based on potentially non-unique properties, such as containing a particular string in their content or having particular metadata values. For these types of address, it's just as good to resolve them in the context of the resolved result as it is in terms of the source documents—the result should be the same.

This rule that all direct addresses must be in terms of source location means that most, if not all, references in a given document type may be cross-document references, meaning that the reference crosses XML document boundaries. This means that normal ID references cannot be used because ID references are only meaningful within a single XML document—the XML specification provides no direct facility for doing ID references to elements in other documents.

To address any given element then you must first address the XML document that contains it. Having addressed the document you can then address a target element or elements however you care to, i.e., via unique ID, tree-based XPath, etc. This two-part address is reflected explicitly in the design of the xi:include element: the href= attribute addresses the document that contains the inclusion target (via URI), the xpointer= attribute addresses individual elements. If only a URI is specified, then the referenced element is implicitly the document's root element. If only an XPointer is specified, the document is implicitly the document that contains the xi:include element itself.

This same approach can be used with any element that needs to do referencing and must be done with elements that do ID references (unless you also provide an indirect addressing mechanism, such as that defined by the XML Indirection facility[XInd]—I'm ignoring that option here for the sake of simplicity, but you may likely find that something like XIndirect is the best way to solve certain link management problems).

For example, consider this typical cross reference element:

<!ELEMENT xref
  EMPTY
>
<!ATTLIST xref
  refid
    IDREF
    #REQUIRED
>

This element must be refactored to enable cross-document addressing. There are several ways this can be done:

Option 1 is probably the least attractive: it requires a fairly complete XPointer-aware processor and may present interoperability problems, as discussed in the XInclude specification itself.

Option 2 is fairly simple to implement and does not require any modification to existing documents as the name of the refid= attribute is not changed and all valid IDs are also valid name tokens. It also allows you to limit the syntax used for pointing to documents, which might be simple filenames, full URLs, or some content-management-specific document pointer. The downside is that whatever you define will necessarily be document type specific, meaning that you must implement the resolution processing in all contexts that need it (e.g., within the editor, within XSLT transforms, etc.). However, because the processing is based on normal ID resolution, it's usually not very hard to implement.

Option 3 is probably the best overall solution as it is the most standards-based and is consistent with the XInclude specification itself. Once you have implemented (or acquired an implementation) of XInclude references you should be able to use that same processing for other element types. However, this type of markup may be unfamiliar to authors or may require migration of existing documents to use the new attributes.

An important implication of the fact that elements are always addressed in the context of the document that contains them is that there is absolutely no need to ensure that element IDs are unique across documents. This is because all ID references are always qualified by the document that contains the ID. This means that the full address of any element is really a document ID and an element ID. This implies that any XML-aware content management system that requires that all element IDs be unique across the entire repository is either not thinking clearly about how XML really works or is providing a convenience for finding elements by providing a set of repository-wide IDs for elements in addition to whatever local, document-specific IDs they might have.

If your content management system also enforces and makes explicit the fact that documents, once created, are invariant, then it is also the case that you do not need to depend on element IDs in order to address elements reliably. When documents are truly invariant then position-based addresses are just as good as ID-based references because the element will never move.

Note that this discussion does not address the inherent problem of managing references to elements as new versions of the referenced documents are created. Most XML management systems address this problem by insisting on the use of element IDs and requiring that IDs, once assigned, not be changed. However, this is a limited solution and is not a general solution. This fundamental issue of versioned hyperdocument management is discussed in detail in [Heintz] and [RefTrack].

Regardless of how you implement your addressing, authors will require user interface support for creating references. This user interface support will be very similar to the user interface required for creating XInclude references, as discussed below.

It is unrealistic and impractical to expect authors to directly specify the href= and xpointer= attributes for creating XInclude references. Therefore you must provide some sort of user interface by which authors can select reference targets. This type of interface poses several challenges that must be carefully considered before implementation.

The first challenge is how to determine what the set of potential reference targets is and how to get the list of target documents from the content management system. The answer will depend entirely on the business rules that govern your application and the type of content management system being used. For example, if you are using a system of naming conventions for directories on a normal file system then the problem may be simply a matter of examining the directory structure. But if you are using any sort of client/server system (including simple file-based systems such as CVS) then you must provide a way for the server to quickly communicate to the client the information needed to enable navigation to and selection of reference target documents. Depending on the scale of the repository, the network performance, and similar variables this may pose a serious performance challenge and require sophisticated implementation techniques in order to provide a satisfactory user experience. For example, a system that could have tens or hundreds of thousands of potential target documents will have to make sure that the client is not presented with the entire list in one go.

The task may be further complicated by the granularity of reference allowed.

From the standpoint of XInclude processors it doesn't matter whether references are to entire documents or to single elements within documents—the data processing is the same and it's no more difficult to do one than the other.

However, from an authoring standpoint, and in particular, from a user interface standpoint, it is usually much more difficult to present a tree of elements from which to select than it is to present a list of documents (files).

For example, in Epic editor, there is a built-in control for a file chooser dialog, but producing a list of target elements would require signficant customization work, especially if that list should reflect only those elements that are actually allowed targets for a given reference (for example, only showing the allowed element types and their ancestors and hiding all other element types). From an implementation standpoint, this is one or two orders of magnitude difference in cost—the first requires a few lines of code, the second requires the implementation of specialized user interface components that must reflect potentially complex business rules.

Another challenge is reflecting in this type of interface the important properties of the reference targets, such as their titles, unique IDs, national languages (if working on multi-language documents), and so on. This suggests that it would be difficult to create a generic user interface component that would work with any document type without at least some degree of configuration.

Finally, if selection of individual elements is allowed it is probably also a requirement to assign unique IDs to reference targets that do not already have them. This can lead to another content management problem, the problem of referent management, which is discussed in [XInd]. In short the problem is this: you are creating a reference from document A to an element in document B. The element you referring to does not have an ID so your authoring system adds one. Unfortunately, you do not have write access to document B because another author has it locked while they are working on it. This leaves you no choice but to either contact the author working on document B and ask them to add an ID to the element you're pointing to and commit their changes so you can complete your reference or simply wait until document B's author is done and releases the lock. Neither option is particularly attractive and neither may in fact be possible at any given time. For example, document B's author might be in a time zone 12 hours distant or not at a point where they can commit their changes or whatever, or they might be on vacation, having forgotten to release their lock.

One solution is to put IDs on all elements that might ever be reference targets. However this adds a lot of data to the files, most of which will never be used, and is in any case unnecessary and does not address another issue, which is the problem of creating version-aware references.

Another solution is an indirect approach, described [XInd] and [RefTrack], that allows you to create ID-based pointers to elements without the need to either have IDs on all elements or to modify the reference target when you create the reference.

Many authors would probably most prefer to see the entire content of re-used elements within the editor so that they always have full context for whatever they are working on. However this can present some serious challenges, both in terms of enabling it in the editor and in communicating with the content management system. For example, a system I worked on for a major commercial aircraft company supported the creation of entire aircraft maintenance manuals using an XInclude-type use-by-reference mechanism (the system predates XML and XInclude by several years). An aircraft maintenance manual may extend to 100 000 pages or more and be composed of thousands of individual component documents. We quickly discovered that trying to resolve all the references in the editor could not perform under any circumstances. This is usually the case, even where documents are much smaller, simply because network bandwidth is usually the limiting factor in client/server systems, so even pulling across a score of files needed for a 100-page document could still require a minute or more on typical corporate internal networks. The problem can be even worse when the repository is accessed through a VPN and, worst case, a dial-up connection.

In addition the currently available XML authoring tools do not make it easy to reflect the referenced content at the point of reference in a complete and robust way. It can be done (we did it for the aforementioned aircraft manual system) but it's not pretty and not trivial.

A simpler solution is to reflect just the titles of the referenced objects at the point of reference. This is usually easy to do and usually provides enough context for authors. Even when authors have the option of reflecting the entire content they will want the option of showing just the titles for performance purposes.

A simple approach to reflecting titles is to add an attribute to inclusion elements to take the title text. This attribute can easily be set at the time the reference is created. Most editors make it easy to reflect attribute values through in-editor style sheets. The only potential problem with this approach is that titles might change, requiring the titles to be periodically checked and updated. But this processing can be done either at author request or as a background process so it shouldn't be a serious issue in practice.

I prefer to put this type of process-specific or utility attribute into a distinct name space that helps make it clearer that the attribute is not a critical property of the base content. For example, you might define a namespace like “http://innodata-isogen.com/xinclude/utilities” and declare the title-holding element like this:

<!ATTLIST subsect_include
  href
    CDATA
    #IMPLIED
  xpointer
    CDATA
    #IMPLIED
  inod-util:reftitle
    CDATA
    #IMPLIED
  xmlns:inod-util
    CDATA
    #FIXED "http://innodata-isogen.com/xinclude/utilities"
  xinclude
    NMTOKEN
    #FIXED "xi:include"
  reftype
    CDATA
    #FIXED "subsect | *[@base-type = 'subsect']"
>

An instance of subsect_include would look like this:

<section>
  <title>Getting Started</title>
  <subsect_include
    href=”subsect_01.xml”
    inod-util:reftitle=”Before You Begin”/>
  ...
</section>