Markup languages are an increasingly important method for data-representation and exchange. This article documents the package sgml2pl, a foreign library for SWI-Prolog to parse SGML and XML documents, returning information on both the document and the document's DTD. The parser is designed to be small, fast and flexible.
Markup languages have recently regained popularity for two reasons. One is document exchange, which is largely based on HTML, an instance of SGML, and the other is for data exchange between programs, which is often based on XML, which can be considered a simplified and rationalised version of SGML.
James Clark's SP parser is a flexible SGML and XML parser. Unfortunately it has some drawbacks. It is very big, not very fast, cannot work under event-driven input and is generally hard to program beyond the scope of the well designed generic interface. The generic interface however does not provide access to the DTD, does not allow for flexible handling of input or parsing the DTD independently of a document instance.
The parser described in this document is small (less than 100 kBytes executable on a Pentium), fast (between 2 and 5 times faster than SP), provides access to the DTD, and provides flexible input handling.
The document output is equal to the output produced by xml2pl, an SP interface to SWI-Prolog written by Anjo Anjewierden.
This package allows you to parse SGML, XML and HTML data into a Prolog data structure. The high-level interface defined in sgml provides access at the file-level, while the low-level interface defined in the foreign module works with Prolog streams. Please use the source of sgml.pl as a starting point for dealing with data from other sources than files, such as SWI-Prolog resources, network-sockets, character strings, etc. The first example below loads an HTML file.
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> <html> <head> <title>Demo</title> </head> <body> <h1 align=center>This is a demo</title> <p>Paragraphs in HTML need not be closed. <p>This is called `omitted-tag' handling. </body> </html> |
?- load_html_file('test.html', Term),
pretty_print(Term).
[ element(html,
[],
[ element(head,
[],
[ element(title,
[],
[ 'Demo'
])
]),
element(body,
[],
[ '\n',
element(h1,
[ align = center
],
[ 'This is a demo'
]),
'\n\n',
element(p,
[],
[ 'Paragraphs in HTML need not be closed.\n'
]),
element(p,
[],
[ 'This is called `omitted-tag\' handling.'
])
])
])
].
|
The document is represented as a list, each element being an atom to
represent CDATA or a term element(Name, Attributes, Content).
Entities (e.g. <) are expanded and included in the
atom representing the element content or attribute value.
1
These predicates are for basic use of the library, converting entire and self-contained files in SGML, HTML, or XML into a structured term. They are based on load_structure/3.
load_structure(File, ListOfContent, [dialect(sgml)]).
load_structure(File, ListOfContent, [dialect(xml)]).
load_html_file(File, Term) :-
dtd(html, DTD),
load_structure(File, Term,
[ dtd(DTD),
dialect(sgml),
shorttag(false)
]).
|
SGML or XML files are loaded through the common predicate load_structure/3. This is a predicate with many options. For simplicity a number of commonly used shorthands are provided: load_sgml_file/2, load_xml_file/2, and load_html_file/2.
stream(StreamHandle) or a file-name. Options is a
list of options controlling the conversion process.
A proper XML document contains only a single toplevel element whose name matches the document type. Nevertheless, a list is returned for consistency with the representation of element content. The ListOfContent consists of the following types:
ListOfAttributes is a list of Name=Value pairs for
attributes. Attributes of type CDATA are returned literal. Multi-valued
attributes (NAMES, etc.) are returned as a list of atoms.
Handling attributes of the types NUMBER and NUMBERS depends on
the setting of the number(+NumberMode) attribute through
set_sgml_parser/2 or load_structure/3. By
default they are returned as atoms, but automatic conversion to Prolog
integers is supported. ListOfContent defines the content for the
element.
<?...?>), Text holds the text of the processing instruction. Please note that the
<?xml ...?> instruction is handled internally.
The Options list controls the conversion process. Currently defined options are:
<!DOCTYPE ...>
declaration is ignored and the document is parsed and validated against
the provided DTD. If provided as a variable, the created DTD is
returned. See implicitdtd.
1 is different from
01. For this reason the default is to handle numeric attributes as
tokens. If conversion to integer is enabled, negative values are silently
accepted.
error(limit_exceeded(max_errors, Max), _)
SGML2PL has four modes for handling white-space. The initial mode can be
switched using the space(SpaceMode) option to
load_structure/3 and set_sgml_parser/2. In XML
mode, the mode is further controlled by the xml:space attribute,
which may be specified both in the DTD and in the document. The defined
modes are:
Consider adjacent <b>bold</b> <ul>and</ul> <it>italic</it> words. |
The parser can operate in two modes: sgml mode and xml mode, as
defined by the dialect(Dialect) option. Regardless of this
option, if the first line of the document reads as below, the parser is
switched automatically into XML mode.
<?xml ... ?> |
Currently switching to XML mode implies:
<element [attribute...] /> is recognised as
an empty element.
<), gt
(>), amp (&), apos (')
and quot (").
ELEMENT, etc.).
_) and colon (:) are
allowed in names.
<!ATTLIST pre xml:space nmtoken #fixed preserve> |
Using the dialect xmlns, the parser will interpret XML namespaces. In this case, the names of elements are returned as a term of the format
URL:LocalName
If an identifier has no namespace and there is no default namespace it is returned as a simple atom. If an identifier has a namespace but this namespace is undeclared, the namespace name rather than the related URL is returned.
Attributes declaring namespaces (xmlns:ns=url)
are reported as if xmlns were not a defined resource.
In many cases, getting attribute-names as url:name
is not desirable. Such terms are hard to unify and sometimes multiple
URLs may be mapped to the same identifier. This may happen due to poor
version management, poor standardisation or because the the application
doesn't care too much about versions. This package defines two
call-backs that can be set using set_sgml_parser/2 to deal
with this problem.
The call-back xmlns is called as XML namespaces are noticed.
It can be used to extend a canonical mapping for later use
by the urlns call-back. The following illustrates this behaviour.
Any namespace containing rdf-syntax in its URL or that is used as
rdf namespace is canonised to rdf. This implies that any
attribute and element name from the RDF namespace appears as
rdf:name.
:- dynamic
xmlns/3.
on_xmlns(rdf, URL, _Parser) :- !,
asserta(xmlns(URL, rdf, _)).
on_xmlns(_, URL, _Parser) :-
sub_atom(URL, _, _, _, 'rdf-syntax'), !,
asserta(xmlns(URL, rdf, _)).
load_rdf_xml(File, Term) :-
load_structure(File, Term,
[ dialect(xmlns),
call(xmlns, on_xmlns),
call(urlns, xmlns)
]).
|
The DTD (Document Type Definition) is a separate entity in sgml2pl, that can be created, freed, defined and inspected. Like the parser itself, it is filled by opening it as a Prolog output stream and sending data to it. This section summarises the predicates for handling the DTD.
dtd using the call:
...,
absolute_file_name(dtd(Type),
[ extensions([dtd]),
access(read)
], DtdFile),
...
|
<?xml ...?> switches the DTD to XML mode and encountering
unknown elements adds these elements to the DTD object. Re-using a DTD
object to parse multiple documents should be restricted to situations
where the documents processed are known to be error-free.
omit(OmitOpen, OmitClose), where both
arguments are booleans (true or false representing whether the
open- or close-tag may be omitted. Content is the content-model
of the element represented as a Prolog term. This term takes the
following form:
list(Type) is used. Finally, the DTD construct
(a|b|...) is mapped to the term
nameof(ListOfValues).
Default describes the sgml default. It is one required,
current, conref or implied. If a real default is present, it
is one of default(Value) or fixed(Value).
system(+File) andor
public(+PublicId).
As this parser allows for processing partial documents and process the DTD separately, the DOCTYPE declaration plays a special role.
If a document has no DOCTYPE declaraction, the parser returns a list holding all elements and CDATA found. If the document has a DOCTYPE declaraction, the parser will open the element defined in the DOCTYPE as soon as the first real data is encountered.
Some documents have no DTD. One of the neat facilities of this library
is that it builds a DTD while parsing a document with an implicit DTD. The resulting DTD contains all elements encountered in
the document. For each element the content model is a disjunction of
elements and possibly #PCDATA that can be repeated. Thus,
if we found element y and CDATA in element x, the model
is:
<!ELEMENT x - - (y|#PCDATA)*> |
Any encountered attribute is added to the attribute list with the type CDATA and default #IMPLIED.
The example below extracts the elements used in an unknown XML document.
elements_in_xml_document(File, Elements) :-
load_structure(File, _,
[ dialect(xml),
dtd(DTD)
]),
dtd_property(DTD, elements(Elements)),
free_dtd(DTD).
|
dtd(DTD) option.
file(File)
option.
<?xml ...> is encountered. See xml for details.
qualify_attributes option below.
encoding= attribute in the header. Explicit
use of this option is only required to parse non-conforming documents.
Currently accepted values are iso-8859-1 and utf-8.
<!DOCTYPE
declaration has been parsed, the default is the defined doctype. The
parser can be instructed to accept the first element encountered as the
toplevel using doctype(_). This feature is especially
useful when parsing part of a document (see the parse option to
sgml_parse/2.
<tag/value/>.
This option is intended to support syntax-sensitive editors. Such an editor should load the DTD, find an appropriate starting point and then feed all data between the starting point and the caret into the parser. Next it can use this option to determine the elements allowed at this point. Below is a code fragment illustrating this use given a parser with loaded DTD, an input stream and a start-location.
...,
seek(In, Start, bof, _),
set_sgml_parser(Parser, charpos(Start)),
set_sgml_parser(Parser, doctype(_)),
Len is Caret - Start,
sgml_parse(Parser,
[ source(In),
content_length(Len),
parse(input) % do not complete document
]),
get_sgml_parser(Parser, allowed(Allowed)),
...
|
Input is a stream. A full description of the option-list is below.
source(Stream), this implies reading is stopped as soon as the
element is complete, and another call may be issued on the same stream
to read the next element.
call(on_begin, Pred) to parse individual elements after
validating their headers.
doctype
declaration.
allowed(Elements) option of get_sgml_parser/2.
It disables the parser's default to complete the parse-tree by closing
all open elements.
Handler(+Tag, +Attributes, +Parser).
Handler(+Tag, +Parser).
Handler(+CDATA, +Parser), where CDATA is an atom representing
the data.
Handler(+Text, +Parser), where
Text is the text of the processing instruction.
<!...>) has been read. The named handler is
called with two arguments: Handler(+Text, +Parser),
where Text is the text of the declaration with comments removed.
This option is expecially useful for highlighting declarations and comments in editor support, where the location of the declaration is extracted using get_sgml_parser/2.
Handler(+Severity, +Message, +Parser), where
Severity is one of warning or error and Message is
an atom representing the diagnostic message. The location of the error
can be determined using get_sgml_parser/2
If this option is present, errors and warnings are not reported using print_message/3
Handler(+NameSpace, +URL, +Parser).
See xmlns for details.
In some cases, part of a document needs to be parsed. One option is to
use load_structure/2 or one of its variations and extract
the desired elements from the returned structure. This is a clean
solution, especially on small and medium-sized documents. It however is
unsuitable for parsing really big documents. Such documents can only be
handled with the call-back output interface realised by the
call(Event, Action) option of sgml_parse/2.
Event-driven processing is not very natural in Prolog.
The SGML2PL library allows for a mixed approach. Consider the case where
we want to process all descriptions from RDF elements in a document. The
code below calls process_rdf_description(Element) on each element
that is directly inside an RDF element.
:- dynamic
in_rdf/0.
load_rdf(File) :-
retractall(in_rdf),
open(File, read, In),
new_sgml_parser(Parser, []),
set_sgml_parser(Parser, file(File)),
set_sgml_parser(Parser, dialect(xml)),
sgml_parse(Parser,
[ source(In),
call(begin, on_begin),
call(end, on_end)
]),
close(In).
on_end('RDF', _) :-
retractall(in_rdf).
on_begin('RDF', _, _) :-
assert(in_rdf).
on_begin(Tag, Attr, Parser) :-
in_rdf, !,
sgml_parse(Parser,
[ document(Content),
parse(content)
]),
process_rdf_description(element(Tag, Attr, Content)).
|
The parser can deal with ISO Latin-1 and UTF-8 encoded files, doing decoding based on the encoding argument provided to set_sgml_parser/2 or, for XML, based on the encoding attribute of the XML header. The parser reads from SWI-Prolog streams, which also provide encoding handling. Therefore, there are two modes for parsing. If the SWI-Prolog stream has encoding octet (which is the default for binary streams), the decoder of the SGML parser will be used and positions reported by the parser are octet offsets in the stream. In other cases, the Prolog stream decoder is used and offsets are character code counts.
In some cases applications wish to process small portions of large SGML, XML or RDF files. For example, the OpenDirectory project by Netscape has produced a 90MB RDF file representing the main index. The parser described here can process this document as a unit, but loading takes 85 seconds on a Pentium-II 450 and the resulting term requires about 70MB global stack. One option is to process the entire document and output it as a Prolog fact-base of RDF triplets, but in many cases this is undesirable. Another example is a large SGML file containing online documentation. The application normally wishes to provide only small portions at a time to the user. Loading the entire document into memory is then undesirable.
Using the parse(element) option, we open a file, seek
(using seek/4) to the position of the element and
read the desired element.
The index can be built using the call-back interface of sgml_parse/2. For example, the following code makes an index of the structure.rdf file of the OpenDirectory project:
:- dynamic
location/3. % Id, File, Offset
rdf_index(File) :-
retractall(location(_,_)),
open(File, read, In, [type(binary)]),
new_sgml_parser(Parser, []),
set_sgml_parser(Parser, file(File)),
set_sgml_parser(Parser, dialect(xml)),
sgml_parse(Parser,
[ source(In),
call(begin, index_on_begin)
]),
close(In).
index_on_begin(_Element, Attributes, Parser) :-
memberchk('r:id'=Id, Attributes),
get_sgml_parser(Parser, charpos(Offset)),
get_sgml_parser(Parser, file(File)),
assert(location(Id, File, Offset)).
|
The following code extracts the RDF element with required id:
rdf_element(Id, Term) :-
location(Id, File, Offset),
load_structure(File, Term,
[ dialect(xml),
offset(Offset),
parse(element)
]).
|
While processing an SGML document the document may refer to external data. This occurs in three places: external parameter entities, normal external entities and the DOCTYPE declaration. The current version of this tool deals rather primitively with external data. External entities can only be loaded from a file and the mapping between the entity names and the file is done using a catalog file in a format compatible with that used by James Clark's SP Parser, based on the SGML Open (now OASIS) specification.
Catalog files can be specified using two primitives: the predicate sgml_register_catalog_file/2 or the environment variable SGML_CATALOG_FILES (compatible with the SP package).
If no files are registered using this predicate, the first query on the catalog examines SGML_CATALOG_FILES and fills the catalog with all files in this path.
Two types of lines are used by this package.
DOCTYPE doctype file PUBLIC"Id"file
The specified file path is taken relative to the location of the catolog file. For the DOCTYPE declaraction, sgml2pl first makes an attempt to resolve the SYSTEM or PUBLIC identifier. If this fails it tries to resolve the doctype using the provided catalog files.
Strictly speaking, sgml2pl breaks the rules for XML, where system identifiers must be Universal Resource Indicators, not local file names. Simple uses of relative URIs will work correctly under UNIX and Windows.
In the future we will design a call-back mechanism for locating and processing external entities, so Prolog-based file-location and Prolog resources can be used to store external entities.
The library sgml_write provides the inverse of the parser, converting the parser's output back into a file. This process is fairly simple for XML, but due to the power of the SGML DTD it is much harder to achieve a reasonable generic result for SGML.
These predicates can write the output in two encoding schemas depending on the encoding of the Stream. In UTF-8 mode, all characters are encoded using UTF-8 sequences. In ISO Latin-1 mode, characters outside the ISO Latin-1 range are represented using a named character entity if provided by the DTD or a numeric character entity.
The sgml2pl package is a parser. Output is generally much easier achieved directly from Prolog. Nevertheless, it contains a few building blocks for emitting markup data. The quote funtions return a version of the input text into one that contains entities for characters that need to be escaped. These are the XML meta characters and the characters that cannot be expressed by the document encoding. Therefore these predicates accept an encoding argument. Accepted values are ascii, iso_latin_1, utf8 and unicode. Versions with two arguments are provided for backward compatibility, making the safe ascii encoding assumption.
The current parser is rather limited. While it is able to deal with many serious documents, it omits several less-used features of SGML and XML. Known missing SGML features include
<!ATTLIST #NOTATION name attributes>. Those data attributes
are provided when you declare an external CDATA, NDATA, or SDATA entity.
XML does not include external CDATA, NDATA, or SDATA entities, nor any of the other uses to which data attributes are put in SGML, so it doesn't include data attributes for notations either.
Sgml2pl does not support this feature and is unlikely to; you should be aware that SGML documents using this feature cannot be converted faithfully to XML.
<tag/content/ is a valid abbreviation for
<tag>content</tag>, which can also be written as
<tag>content</>.
Empty start tags (<>), unclosed start tags
(<a<b) and unclosed end tags (</a<b) are not
supported.
In XML mode the parser recognises SGML constructs that are not allowed in XML. Also various extensions of XML over SGML are not yet realised. In particular, XInclude is not implemented because the designers of XInclude can't make up their minds whether to base it on elements or attributes yet, let alone details.
Installation on Unix system uses the commonly found configure,
make and make install sequence. SWI-Prolog should be
installed before building this package. If SWI-Prolog is not installed
as pl, the environment variable PL must be set to
the name of the SWI-Prolog executable. Installation is now accomplished
using:
% ./configure % make % make install |
This installs the foreign libraries in $PLBASE/lib/$PLARCH and the Prolog library files in $PLBASE/library, where $PLBASE refers to the SWI-Prolog `home-directory'.
The Prolog representation for parsed documents is based on the SWI-Prolog interface to SP by Anjo Anjewierden.
Richard O'Keefe has put a lot of effort testing and providing bug reports consisting of an illustrative example and explanation of the standard. He also made many suggestions for improving this document.
| dtd/2 | Find or build a DTD for a document type |
| dtd_property/2 | Query elements, entities and attributes in a DTD |
| free_dtd/1 | Free a DTD object |
| free_sgml_parser/1 | Destroy a parser |
| get_sgml_parser/2 | Get parser options |
| html_write/3 | Write an HTML document from a parser-compatible term |
| load_dtd/2 | Read DTD information from a file |
| load_dtd/3 | Read DTD information from a file |
| load_html_file/2 | Parse HTML file into Prolog term |
| load_sgml_file/2 | Parse SGML file into Prolog term |
| load_structure/3 | Parse XML/SGML/HTML data into Prolog term |
| load_xml_file/2 | Parse XML file into Prolog term |
| new_dtd/2 | Create a DTD object |
| new_sgml_parser/2 | Create a new parser |
| open_dtd/3 | Open a DTD object as an output stream |
| set_sgml_parser/2 | Set parser options (dialect, source, etc.) |
| sgml_parse/2 | Parse the input |
| sgml_register_catalog_file/2 | Register a catalog file |
| sgml_write/3 | Write an SGML document from a parser-compatible term |
| xml_is_dom/1 | Validate an SGML/XML DOM term. |
| xml_name/1 | Test atom for valid XML name |
| xml_name/2 | Test atom for valid XML name |
| xml_quote_attribute/2 | Quote text for use as an attribute |
| xml_quote_attribute/3 | Quote text for use as an attribute |
| xml_quote_cdata/2 | Quote text for use as PCDATA |
| xml_quote_cdata/3 | Quote text for use as PCDATA |
| xml_write/3 | Write an XML document from a parser-compatible term |
number(+Code). With the introduction of wide characters
in the 5.5 branch this is no longer needed