While SAX[SAX], the simple API for XML, is a broadly, almost universally implemented standard among Java parsers, many SAX parsers have serious bugs. The lack of a complete SAX conformance test suite has been a severe hindrance to interoperability. For example, about half of SAX parsers call endDocument() even after reporting a fatal error, while the other half don’t. Existing XML test suites mostly focus on whether the parser correctly answers boolean questions of well-formedness or validity, while ignoring the much more complex questions of whether the parser correctly reports document content in the correct order. Indeed the XML specification is mostly silent on exactly which parts of the document a parser is required to report. Not surprisingly this has led to a number of inconsistencies between parsers as well as outright bugs in more than a few implementations.

This paper demonstrates a conformance suite written in Java that tests parsers which claim to implement the SAX API. The framework asks the parser to read a collection of input documents and then logs the methods the parser invokes and their arguments. This log takes the form of an XML document that can be compared against the expected results.

The documents in the test set are derived from the W3C XML conformance test suite. The software includes a framework for testing parsers against this document collection and measuring their conformance to both the core and optional parts of both XML and SAX. Conformance results for major parsers including Xerces, Crimson, and Piccolo are reported. A number of areas in which deficiencies in the SAX specification have led to varying parser behavior are identified.