I was startled to see the level of passion in Alex Brown’s 2010-03-31 post, Microsoft Fails the Standards Test. Alex has two concerns: (1) dwindling OOXML standards-maintenance attention and resources; and (2) Microsoft silence with regard to implemented support for the strict level of IS 29500 and any retirement of the transitional level as the only level supported in Microsoft implementations of OOXML.
Perhaps the most level-headed analysis is the “Wow” from Andy Updegrove in his 2010-04-01 post, Alex Brown: “Without action, the entire OOXML Project is now surely headed for failure.”
For me, the most peculiar aspect of the reactions I see is not that Alex has the concerns he announced, but that others treat his expression of concern for the future as a declaration of the actual present. Furthermore, these observers who proudly pontificate that there is no action, there will be no action, and there was never going to be any action, excitedly congratulate Alex on having awakened from being hood-winked. It is as if nothing has happened since 1998 and the book is closed on Microsoft forever.
Expectations Against Observable Reality
I want to look at just one part of this situation: expectations around IS 29500 implementation in Microsoft products. The desire to have Microsoft abandon to-be-deprecated transitional provisions of IS 29500 in favor of producing only documents in the strict IS 29500 format is tied into that expectation.
I am eminently qualified to address this topic. I have no information on what Microsoft is actually doing to incorporate support for IS 29500 in its products. I have no idea what strategy Microsoft has, if any, with regard to the retirement of support for IS 29500 compliant transitional documents. Microsoft doesn’t tell me anything about product efforts and I am happy to keep it that way. Microsoft doesn’t seek my advice on the matter either. So I am perfectly positioned to speculate, with my standing as a standards, interoperability, and architectural armchair astronaut unblemished.
What I am going to report is my observation of the simple state of affairs and how difficult it is to erase the past and jump to implementations that only produce what are called strict IS 29500 documents. Expecting that to have been achieved in two years is about as unobservant as belief that all Microsoft needed to have done was adopt ODF as its native format in the first place.
Can I Has Have Me Strict Now Please?
The ISO/IEC International Standard for OOXML, IS 29500:2008, has two major levels that can be implemented. There is a strict IS 29500 format that is the subject of the main part of the specification. There is also a transitional IS 29500 format that mainly includes everything allowed in the strict format along with some other provisions that it was agreed would be retired but retained now for compatibility purposes. The idea was that there would become a time when consumers might accept transitional documents but the routinely-expected output would be a strict IS 29500 document. The strict-transitional differentiation and the notion that production of non-strict documents would be discouraged for new documents was a creation of the DIS 29500 Ballot Resolution meeting in February 2008. It was ratified in the approval of DIS 29500 as an International Standard on April 2, 2008.
In IS 29500:2008 as it was first published in November, 2008, the set of strict documents is essentially a subset of the transitional documents. In addition, it is the transitional documents that include the most provisions of ECMA-376:2006, the specification supported by the already-distributed Microsoft Office System 2007 (and any Office 2003 configuration, like mine, to which the OOXML Compatibility Pack has been added). Various translators also accept or emit ECMA-376 documents. The .docx, .xlsx, and .pptx documents that are growing in numbers every day in production use satisfy the provisions of ECMA-376:2006 and tend to use transitional provisions of IS 29500 rather than preferable strict counterparts.
There is already a legacy situation with OOXML concerning the need for products to support ECMA-376:2006 in the documents that are accepted and produced.
Well, Not So Fast, Sparky
In addition, SC34 WG4, the standards-development working group that maintains IS 29500, has now created a situation in which there is more than one strict IS 29500.
As part of the early maintenance work, some found it disturbing that there was no way to differentiate provisions of ECMA-376:2006 from IS 29500 transitional and of IS 29500 transitional from IS 29500 strict. The same namespaces were used for all of them. This issue was being addressed before the ink was dry on IS 29500:2008. In 2009, the SC34 WG4 arrived at a set of amendments, most of which were designed to separate the namespace used for the strict provisions of IS 29500 from the namespace used for transitional documents with their additional/alternative too-be-deprecated provisions.
If Microsoft had already implemented a way to produce strict IS 29500:2008 documents as defined before these amendments, those documents would now be considered transitional documents. Their XML parts would employ the transitional namespaces, not the recently-adopted ones for strict IS 29500.
It has been a few months since the amendment solidified, and one would hope that Microsoft is looking at how to enact the production of strict IS 29500 documents in some customer-respectful manner. Whether that is something that could possibly appear in the soon-to-be-released Microsoft Office 2010 products is not something I will even guess about. Microsoft’s commitment to support IS 29500 is not specific on this topic, and there may be residual difficulties in how the separation of strict out from under transitional has been executed in the amendments and any anticipatory implementation work.
I can see ways to work through a gradual migration where the expected output is strict IS 29500 documents. But I would expect transitional/ECMA-676 documents to be accepted for a long time and to be produced at least as long as “Save As … 97-2003 Document” also exists.
Appendix: Arriving at Separated Strict and Transitional Namespaces: The Progression
Although it has been two years since there was agreement on what IS 29500:2008 would be, it was not until November 25 that the specifications were publicly available. In less than one year after that, amendments creating a substantial difference in the separation of strict OOXML documents from transitional OOXML documents was formulated and put out for ballot. The official Corrigenda and Addenda carrying those and other changes are not yet available to the public. Here is the progression.
- ISO/IEC International Standard 29500:2008 Office Open XML File Formats. First edition 2008-11-15. This is the first official publication of IS 29500 for OOXML after the 2007 balloting and a subsequent Ballot Resolution Meeting in early 2008. The specification is in four parts:
- Part 1: Fundamentals and Markup Language Reference. This is where the strict provisions are specified.
- Part 2: Open Packaging Conventions. There are some transitional considerations in packaging. These are being treated by separate defect reports. The OPC are adaptable for non-OOXML usage.
- Part 3: Markup Compatibility and Extensibility. A set of independent features by which extensions can be added to an XML Document using a set of specific conventions that allow for graceful degradation when the extensions are not understood.
- Part 4: Transitional Migration Features. Additional features that are not included in the strict provisions. As originally formulated, the strict provisions were to be a subset of the transitional OOXML documents and all of ECMA-376 was embraced by transitional OOXML with a few deviations.
- SC34 N 1246 ISO/IEC 29500-1:2008/FPDAM1, Part 1: Fundamentals and Markup Language Reference – AMENDMENT 1, 2009-08-04 available as a public 1.56MB downloadable Zip file consisting of the proposed draft amendment and the corrected (RNG and W3C) schemas.
This public document was taken to a four-month SC34 FPDAM Ballot that closed 2009-12-04.
- SC34 N 1251 ISO/IEC 29500-4:2008/FPDAM1, Part 4: Transitional Features – AMENDMENT 1, 2009-08-04 available as a public 1.59MB downloadable Zip file consisting of the proposed draft amendment and the corrected (RNG and W3C) schemas.
This public document was taken to a four-month SC34 FPDAM Ballot that closed 2009-12-04.
- SC34 N 1253 IS 29500
99:2008 Defect Report Log [At Closure of the DCOR1 and FDAM1 Sets], 2009-08-04 edition available as a public 3.91MB downloadable PDF file.
Defect Item 08-0012 was submitted by Ms. Ruth Schneider of the National Body (SNV) for Switzerland. It was circulated to SC34 on 2008-11-03. The defect involves the inability to distinguish between ECMA-376:2006 schemas and those of IS 29500 because the same namespaces are used. From the log on this item we can see that there were many ways to deal with explicit versioning and that no reasonable solution stood out. Finally, at a meeting in Prague on 2009-03-24, a small break-out group returned with proposed requirements for any versioning solution:
- To push producers gently towards strict conformance [emphasis mine, orcmid:]
- To improve interoperability
- To do no evil to the 29500 ecosystem (i.e., files, end users, and implementations)
The option to change the namespace for strict schemas and only strict schemas emerged in subsequent discussions. It took until the 2009-06-22/24 meeting in Copenhagen to finish wading through the considerations and agree that the only change would be new namespaces for strict schemas and the associated narratives. Along the way there were was stumbling around versioning, conformance attributes, and features of transitional that had no existence in ECMA-376:2006 and were there now merely to preserve transitional as the superset. Although the final changes needed to accomplish
thisthe namespace separations were contributed by Microsoft expert Shawn Villaron of the ECMA delegation(and Microsoft), the issues and inter-dependencies were heavily discussed among all of the WG4 participants.
In the end, resolution of defect item 08-0012 involved over 300amendment entries in N1246 for IS 29500 Part 1 and over 200 in N1251 for Part 4. This is about 75% of all of the amendment entries in the FPDAM1 set, all on behalf of this single defect and its simply-stated disposition.
- Brad Smith’s 2009-12-16 Microsoft Statement on European Commission Decision. This declaration from Microsoft establishes a “public undertaking” with regard to interoperability. Part of this undertaking includes a Warranty Agreement covered in a 2009-12-16 Annex (downloadable Microsoft Office Word .doc file). The timing of this announcement has nothing to do with events at ISO/IEC JTC1 SC34 (and some of the “undertaking” documents were first uploaded with 2009-10-06, dates). The “undertaking” is in the spirit of the Interoperability Principles and other agreements with regulatory authorities (c.f., my 2008-02-26 Interoperability by Design post). It also puts some serious teeth into Microsoft commitments by asserting that “Microsoft will make available legally-binding warranties that will be offered to third parties.”
In my reading of the available on-line documents, legally-binding warranties are provided for an unstated fee to parties that intend to provide interoperability with Microsoft implementations of various protocols and industry standards. Under such warranties, there is assurance that successors to Microsoft Office Word 2007, Excel 2007, and PowerPoint 2007 will support IS 29500 rather than ECMA-376. Nothing in the “undertaking” distinguishes between strict and transitional provisions of IS 29500.
- Subsequent Creation of Documents. There are no more-recent public documents
,as of 2010-04-03. On 2010-03-05, documents SC34 N 1382 and SC34 N 1384 appeared as final FDAM1 amendment texts on the SC34 document repository. I presume that these reflect the disposition of FPDAM1 ballot comments and perhaps other feedback on N 1246 and N 1251, respectively.
- New Defects from the Namespace Change. I can’t imagine that the massive changes involved in separation of strict into its own namespaces would not be accompanied by new defects,
especially inconsidering howthedependent the transitional provisions were on the formerly-strict-but-same-namespace provisions. This seems to be borne out in the Outstanding Action Items 7-9 of the 2010-02-18 SC34 WG6 Teleconference minutes (available as a public 574kB downloadable PDF file). Although there will need to be those, perhaps other, repairs, there is probably little that would further delay an implementation being able to recognize its supported strict features under either namespace so long as the provisions are identical. Preventing the clashing of unique-to-strict and unique-to-transitional provisions will require care in all cases. I am blissfully ignorant of how the mixing of features might complicate the internal, “in-memory” document model of Microsoft and other existing products. The added complexity of testing and building a relevant document corpus for all of the use cases strikes me as seriously daunting.
[update 2010-04-05T23:41Z And I am finally scrapping all of that useless white space that was on the end of my blog text.
update 2010-04-05T23:11Z I misquoted Andy Updegrove’s blog title quoting Alex Brown and have fixed it, thanks to the good eye of Rob Weir.
update 2010-04-05T16:49Z Too be more careful when there are twoo many words to get right the first time. Sometimes, I just can’t bear it.
update 2010-04-05T15:11Z I took inspiration from Alex Brown’s comment to tidy up some wording and add an after-thought about the unseen but material impacts of a simple namespace change in a world of comingled strict and transitional features.]
