From The Linux Foundation
Contents
|
How to Add a Library to LSB
- Note: This procedure is in the process of being replaced by: LSB Library Import Tools
Overview
To add a library to the LSB, a few things are required:
- The library must have a stable ABI and be considered "best practice" or in common use across the major Linux distributions.
- The library interfaces must be added to the LSB database
- The interfaces, description, arguments, results must be added the LSB specification
- Tests need to be defined to test the behavior of the interfaces and added to the LSB test suites.
To illustrate the process, we'll walk through adding a common library, libbz2, to the LSB.
Build a version of the library with debugging enabled
wget http://www.bzip.org/1.0.4/bzip2-1.0.4.tar.gz tar -xzf bzip2-1.0.4.tar.gz cd bzip2-1.0.4 CC="gcc -g" make -f Makefile-libbz2_so
(-g enables debugging information in the compiled output. libbz2 doesn't use a "configure" script but provides the above mentioned Makefile to compile a shared library.)
file libbz2.so.1.0.4 libbz2.so.1.0.4: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), not stripped
Display symbols from the dynamic section:
readelf -DWs libbz2.so.1.0.4
Looking at this you'll see entries like:
50 39: 0000ed00 7 FUNC GLOBAL DEFAULT 10 BZ2_bzflush
Those entries with an Ndx entry of 10, are the interfaces that we're interested in in defining, specifying in the LSB, and creating tests for. So we could filter this down to just a list of the interesting interfaces:
readelf -DWs libbz2.so.1.0.4 | grep "GLOBAL DEFAULT 10" | awk '{print $9}' > interfaces.list
You should carefully compare this list with the header files and remove any interfaces that are exported, but not really part of the supportable and public API of the library (Rather than edit the file, we'll create another list of excluded interfaces that we'll use for libtodb). [1]
So, for the case of libbz2, our original list of 33 interfaces would get trimmed some more:
for i in `cat interfaces.list`;do grep -q $i bzlib.h;if [ "$?" -eq 1 ]; then echo $i;fi;done BZ2_blockSort BZ2_compressBlock BZ2_indexIntoF BZ2_bsInitWrite BZ2_hbMakeCodeLengths BZ2_bz__AssertH__fail BZ2_hbAssignCodes BZ2_decompress BZ2_hbCreateDecodeTables
These interfaces will become part of our exclude list in the next step.
Add the library to the LSB database
It's best to get a copy of the LSB database you can setup locally. The transactions to examine and add a library are too numerous to effectively be able to do it over the internet, and you'll want to work out the issues on a local copy before official submission anyway: Spec Database
Install MySQL - instructions for this will vary according to your Linux distribution.
All LSB tools and tests can be checked out from bzr. The repositories can be browsed at: http://bzr.freestandards.org/lsb/
You will need to install bzr to check out source: Bazaar-NG
Check out the specdb:
bzr branch http://bzr.freestandards.org/lsb/devel/specdb/ specdb
Read 00README:
SPECDB
======
The files in this directory reflect the current state of the LSB
database. There are some basic rules that must be followed when
using these files:
make restore
This will create a new database copy using the environment
variables LSBUSER, LSBDBPASSWD, LSBDBHOST and LSBDB.
*** NEVER use "make restore" on the central database
(dbservices.freestandards.org) unless that database has been hoplessly
lost! "make restore" is ONLY intended for users requiring to
initialize local copies of their own database.
You'll note the mention of LSB* environment variables. These should be set appropriately to access your local database copy, in ~/.bashrc or similar:
export LSBUSER=stew export LSBDBPASSWD=foobar export LSBDBHOST=norris export LSBDB=lsb
So, once these are set, "make restore" should populate your local LSB database.
Now that we have our library and the database, we can use the tool "libtodb" to examine the library and it's headers, and eventually add it to the database: LibtoDB. This is an interative process, and depending on the complexity of the library in question, and whether you want to exclude certain interfaces from the LSB (due to the fact they may not have a stable ABI or may be deprecated), it can take several iterations of using libtodb to get a satisfactory import into the database that will allow one to build the stub headers and specification.
Check out libtodb and friends:
bzr branch http://bzr.freestandards.org/lsb/devel/scripts/ scripts
At this point you have a decision to make if your library is 'complex' and has multiple header files associated with it. If it is a simple library with a single header covering all of the exported symbols you are pretty much ready to go. If it is a complex library like OpenSSL with many headers exposing different portions of the ABI you have a choice to make. Either you need to fix things such that the entire libraries interface can be imported from a single header or you need to go through a process of importing exports relative to each individual header file. Choosing to put everything into a single header should only be done with the advise and consent of the upstream package maintainer since doing so may break some applications that depend on the existing upstream header layout and side effects of header inclusion (symbols defined, symbols not defined). For complex libraries I have worked out a sort of 'pipeline' to assist in importing these libraries. It requires a bit more time to setup, but at that point is automated and the results will be more true the upstream package than the LSB header modification approach. addopenssl.sh in the scripts directory is an example of this pipeline in action. [1]
If you look at one of the libraries that are already in the LSB, you'll see the captured data used during the import:
[scripts]$ find libtodb_data/jpeg/ libtodb_data/jpeg/ libtodb_data/jpeg/jpeglib.h.def libtodb_data/jpeg/jpeglib_ex.list libtodb_data/jpeg/jpeglib_opaque.list libtodb_data/jpeg/libs libtodb_data/jpeg/libs/libjpeg.so.62.0.0
First we want to make a constants file from the library's header file, in this case bzlib.h:
./mkconstfile -h bzlib.h > bzlib.h.def
The above tool tends to pull in macros imported in the header file that aren't actually defined in the header file. What you really want is just those macros that are part of the library in question. Using Tracy's "pipeline" method you can narrow this down to a more succinct list:
cpp -dD bzlib.h | ./trim_macros.pl -m bzlib.h | grep -v "#define HEADER" \
| ./mkconstfile -h - | grep -v "^<macro>" | sort -u > bzlib.h.def
Even with this, the file will still need to be edited, until we end up with something like this:
<>#define BZ_FINISH 2 <>#define BZ_FINISH_OK 3 <>#define BZ_FLUSH 1 <>#define BZ_FLUSH_OK 2 <>#define BZ_MAX_UNUSED 5000 <>#define BZ_OK 0 <>#define BZ_RUN 0 <>#define BZ_RUN_OK 1 <>#define BZ_STREAM_END 4 <unknown>#define BZ_EXPORT <unknown>#define _BZLIB_H
Then create the excluded interfaces list:
for i in `cat interfaces.list`;do grep -q $i bzlib.h;if [ "$?" -eq 1 ]; \
then echo $i;fi;done > bzlib_ex.list
Likewise, opaque interfaces would be listed in bzlib_opaque.list. The opaque list is important for cropping the exposed API into a portable ABI. If the API exposes internal structures that are not guaranteed to be stable across many releases it is best to cause the ABI to treat these as opaque types. The opaque file lists types that should be treated as opaque. These will have to be manually adjusted, even in the complex pipeline. [1]
For libbzip2, an opaque listing will not be necessary.
Before proceeding with the import with libtodb, two more choices need to be made. Those are the module the new library will be part of, and the standard that will be referred do.
For the purposes of our example, we'll create a new LSB module: LSB_Toolkit_Compression. To be able to use this, we'll need to add an entry to the "Module" table in the LSB database, using mysql, picking the next unused module number. Alternatively, the library could have gone into an existing module. In this case, the libtodb -M 9 argument below would be adjusted to refer to the appropriate module number. The DB Navigator can be used to browse the modules.
INSERT INTO Module VALUES (9, 'LSB_Toolkit_Compression', 'LSB Toolkit Compression Module', '3.2', NULL, '3.2');
The standards question comes down to whether there is good API-level specifications available for the library. If so, the LSB should reference that specification, and only a small addition needs to be made to the LSB specification. If not, the specification text has to be written and added to the LSB itself, in which case LSB itself (specification ID 10) becomes the reference specification - that's the case for the example we're using, the argument below will be -s 10. If a new standard is to be referenced, it must be inserted into the database Standard table before the import, and the -s argument adjusted. Again, the DB Navigator can be used to browse the existing standards.
As a first pass, run libtodb without the "-i" argument which inserts the interfaces into the database.
./libtodb -q -H bzlib.h -L "Bzip Compression Library" -M 9 -s 10 -S libbz2.so.1 \
-A IA32 -c bzlib.h.def -x bzlib_ex.list libbz2.so.1.0.4 > out.log
So we're doing a test run of adding libbz2 to the compression module (-M 9), standard is LSB (-s 10), SONAME is libbz2.so.1 (-S) host architecture is IA32 (-A), and we've specified our constants file (-c) and excludes file (-x). So if we look at out.log, we find 26 items in the proposed list of interfaces to add to LSB:
$ grep "DWARF" out.log DWARF: BZ2_bzflush: Function DWARF: BZ2_bzDecompressInit: Function DWARF: BZ2_bzlibVersion: Function DWARF: BZ2_bzBuffToBuffDecompress: Function DWARF: BZ2_rNums: Data DWARF: BZ2_bzread: Function DWARF: BZ2_bzReadOpen: Function DWARF: BZ2_bzReadGetUnused: Function DWARF: BZ2_bzCompress: Function DWARF: BZ2_bzCompressInit: Function DWARF: BZ2_bzDecompress: Function DWARF: BZ2_bzerror: Function DWARF: BZ2_bzopen: Function DWARF: BZ2_bzReadClose: Function DWARF: BZ2_bzCompressEnd: Function DWARF: BZ2_bzDecompressEnd: Function DWARF: BZ2_bzWrite: Function DWARF: BZ2_bzBuffToBuffCompress: Function DWARF: BZ2_bzWriteClose64: Function DWARF: BZ2_bzWriteClose: Function DWARF: BZ2_bzdopen: Function DWARF: BZ2_bzWriteOpen: Function DWARF: BZ2_bzclose: Function DWARF: BZ2_bzRead: Function DWARF: BZ2_crc32Table: Data DWARF: BZ2_bzwrite: Function
BZ2_rNums and BZ2_crc32Table are data structures, and can be added to our exclude list also.
If the list looks reasonable, then you can proceed with actually adding the library to the database, by including the "-i" argument:
./libtodb -i -q -H bzlib.h -L "Bzip Compression Library" -M 9 -s 10 -S libbz2.so.1 \
-A IA32 -c bzlib.h.def -x bzlib_ex.list libbz2.so.1.0.4 > out.log
Module used is LSB_Toolkit_Compression
Standard used for interface(s) is LSB
Architecture has id 1(All)
Host Architecture has id 2(IA32)
Library has id 263
LibGroup has id 493
Header has id 488
Header Group has id 929
24 interfaces added to database.
Generate the LSB headers and Stub libraries
Headers
Now that you've imported the header into the database, you should be able to regenerate the LSB headers.
$ bzr branch http://bzr.freestandards.org/lsb/devel/build_env $ cd build_env/headers $ make headers-All
You should get a generated bzlib.h header as part of this build, generated from the database.
Since we added our own module, we'll need to edit "mkfilelists" to be aware of it, so that the new header appears in the correct header/library package. There are two packages that include generated header files and libraries, lsb-build-base (core modules headers/libraries) and lsb-build-desktop (desktop modules headers/libraries). Starting from the headers directory in the previous step:
$ cd ..
$ bzr diff mkfilelists
=== modified file 'mkfilelists'
--- mkfilelists 2007-04-12 12:37:31 +0000
+++ mkfilelists 2007-04-30 16:03:47 +0000
@@ -16,7 +16,8 @@
lsbversion = sys.argv[1]
-core_modules = ["LSB_Core", "LSB_Graphics", "LSB_Cpp"]
+core_modules = ["LSB_Core", "LSB_Graphics", "LSB_Cpp",
+ "LSB_Toolkit_Compression"]
desktop_modules = ["LSB_Toolkit_Gtk", "LSB_Toolkit_Qt",
"LSB_Graphics_Ext", "LSB_XML", "LSB_Toolkit_Qt3"]
So we've just added the LSB_Toolkit_Compression to the core_modules group. We can now generate the lists:
$ ./mkfilelists $ ls -1 */*filelist headers/core_filelist headers/desktop_filelist stub_libs/core_filelist stub_libs/desktop_filelist
Stub libraries
Now we can build the stub libraries.
$ cd stub_libs $ make distclean $ make dbfiles $ make
You should find libbz2 (appropriate for the build arch):
$ find . -name 'libbz2*' ./IA32/libbz2.c ./IA32/libbz2.Version ./IA32/libbz2.o ./IA32/libbz2.so ./IA64/libbz2.c ./IA64/libbz2.Version ./PPC32/libbz2.c ./PPC32/libbz2.Version ./PPC64/libbz2.c ./PPC64/libbz2.Version ./S390/libbz2.c ./S390/libbz2.Version ./S390X/libbz2.c ./S390X/libbz2.Version ./x86-64/libbz2.c ./x86-64/libbz2.Version
Test the new headers / stub libraries
So, we now have a stub library and an LSB header. It should be possible at this point to copy these to /opt/lsb and use them with lsbcc.
headers$ sudo cp bzlib.h /opt/lsb/include/ cd ../stub_libs/IA32 sudo cp libbz2.so /opt/lsb/lib
We need to alter lsbcc to accept -lbz2 as an LSB library, rather than an external library to link statically. lsbcc is part of the same build_env tree we checked out earlier:
cd lsbdev-cc
gendiff . .bz2
--- ./lsbcc.c.bz2 2007-03-13 16:56:28.000000000 -0400
+++ ./lsbcc.c 2007-03-13 16:56:52.000000000 -0400
@@ -270,6 +270,7 @@
"rt",
"util",
"z",
+ "bz2",
"GL", /* graphics module */
"ICE",
"SM",
lsbcc should compile ok with this change:
make
Now we can compile a test app using our LSB header and the patched lsbcc (bztest.c):
#include <stdlib.h>
#include "bzlib.h"
#define OFILE "outfile.bz2"
#define BUFSTR "01234567890123456789012345678901234567890123456789"
#define WRITE_ERR "BZ2_bzWrite set bzerror to"
int main(void)
{
BZFILE* b;
FILE* f;
char buf[] = BUFSTR;
int bzerror;
f = fopen(OFILE, "w");
if (!f) {
printf("Cannot open %s for writing\n", OFILE);
exit(1);
}
b = BZ2_bzWriteOpen(&bzerror, f, 9, 0, 0);
if (bzerror != BZ_OK) {
BZ2_bzWriteClose(&bzerror, b, 0, NULL, NULL);
printf("Cannot open %s for bzWrite\n", OFILE);
exit(1);
}
BZ2_bzWrite(&bzerror, b, buf, strlen(buf));
if (bzerror == BZ_IO_ERROR) {
BZ2_bzWriteClose(&bzerror, b, 0, NULL, NULL);
printf("%s %d\n", WRITE_ERR, bzerror);
exit(1);
}
BZ2_bzWriteClose(&bzerror, b, 0, NULL, NULL);
if (bzerror == BZ_IO_ERROR) {
printf("Cannot close %s\n", OFILE);
exit(1);
}
}
Compile the test app with our patched lsbcc and our new header/stub lib:
./lsbcc bztest.c -lbz2 -o bztest ldd bztest linux-gate.so.1 => (0xffffe000) libbz2.so.1 => /lib/libbz2.so.1 (0xf7ef1000) libpthread.so.0 => /lib/libpthread.so.0 (0xf7edd000) libm.so.6 => /lib/libm.so.6 (0xf7eb8000) libc.so.6 => /lib/libc.so.6 (0xf7d98000) /lib/ld-lsb.so.3 (0xf7f0f000)
An unpatched lsbcc would have statically linked libbz2.a:
lsbcc -lbz2 bztest.c -o bztest ldd bztest linux-gate.so.1 => (0xffffe000) libpthread.so.0 => /lib/libpthread.so.0 (0xf7f20000) libm.so.6 => /lib/libm.so.6 (0xf7efb000) libc.so.6 => /lib/libc.so.6 (0xf7ddb000) /lib/ld-lsb.so.3 (0xf7f41000)
Run the app (simply creates a bzip file outfile.bz2):
./bztest ls -l outfile.bz2 -rw-r--r-- 1 stew stew 51 Mar 13 19:16 outfile.bz2 bzcat outfile.bz2 01234567890123456789012345678901234567890123456789
It's possible that you might encounter issues here and have to go back to the libtodb step and re-import your library into the database, after making corrections. You can restore your local LSB database using (in the specdb directory):
make restore
You would then have a pristine database ready to re-add your module and library.
Regenerate checkers
add content here on regenerating in misc-test
Add the library to the specification
The spec content is in the lsbspec branch of bzr:
bzr branch http://bzr.freestandards.org/lsb/devel/lsbspec
In this tree there is a document named docbook_tutorial.txt. This describes an overview of how the specification is built and edited.
Create Toolkit_Compression module tree
Since we've opted to create a new module, we'll need the directory structure to support it. Looking at the existing structure, it looks like we can perhaps model our new lib from Graphics-Ext/generic/libjpeg, with the thought that Toolkit_Compression may someday have other compression libs (like gzip, zip, lzma, etc.). If we were integrating into an existing module, the steps below would be modified appropriately.
cd lsbspec install -d Toolkit_Compression/intro install -d Toolkit_Compression/generic/libbz2
We need a toplevel makefile (in Toolkit_Compression):
SUBDIRS= generic intro
all:
for dir in $(SUBDIRS);do (cd $$dir && make all);done
autobuild:
for dir in $(SUBDIRS);do (cd $$dir && make gensrc all);done
gensrc:
for dir in $(SUBDIRS);do (cd $$dir && make gensrc);done
source:
for dir in $(SUBDIRS);do (cd $$dir && make all);done
clean:
for dir in $(SUBDIRS);do (cd $$dir && make clean);done
rm -f $(HTML)
spotless:
for dir in $(SUBDIRS);do (cd $$dir && make spotless);done
rm -f $(HTML)
And one in intro:
.SUFFIXES: .sgml .cpp .m4
FILES=intro.sgml
TABLES=standards.sgml
.cpp.sgml:
cpp -I../../../code/headers $*.cpp | grep -v '^# ' >$@
.m4.sgml:
m4 -Uindex -Uformat $*.m4 >$@
all: $(FILES) $(TABLES)
gensrc:
../../mkstandardsgmltable -a All -s "'ISOC99','libbz2'" >standards.sgml
../../mklibsgmltable -a All -m LSB_Toolkit_Compression >libraries.sgml
clean:
rm -f $(FILES)
spotless: clean
rm -f $(TABLES)
intro.sgml: standards.sgml libraries.sgml
And one in generic:
SUBDIRS= libbz2
all:
for dir in $(SUBDIRS);do (cd $$dir && make all);done
gensrc:
for dir in $(SUBDIRS);do (cd $$dir && make gensrc);done
source:
for dir in $(SUBDIRS);do (cd $$dir && make source);done
clean:
for dir in $(SUBDIRS);do (cd $$dir && make clean);done
rm -f $(HTML)
spotless:
for dir in $(SUBDIRS);do (cd $$dir && make spotless);done
rm -f $(HTML)
And one in generic/libbz2 (only listing our 2 sample functions for the moment):
.SUFFIXES: .sgml .cpp .m4
MANPAGES= BZ2_bzRead.sgml\
BZ2_bzWrite.sgml
FILES=libbz2.sgml
TABLES= bz2.sgml
.cpp.sgml:
cpp -I../../../code/headers $*.cpp | grep -v '^# ' >$@
.m4.sgml:
m4 -Uindex -Uformat $*.m4 >$@
all: $(FILES) $(TABLES)
gensrc:
../../../mklibspec -a All -c 3 -l libbz2 >bz2.sgml
clean:
rm -f $(FILES)
spotless: clean
rm -f $(TABLES)
libbz2.sgml: bz2.sgml
libbz2.sgml: $(MANPAGES)
And generic/libbz2/libbz2.m4:
<PART ID="toclibbz2"> <TITLE>BZ2 library</TITLE> <CHAPTER id=libbz2> <TITLE>Libraries</TITLE> include(bz2.sgml) </CHAPTER> </PART>
We also need an into/intro.m4 document and intro/terms.sgml. I won't paste those here. I basically copied them from a neighboring directory and edited intro.m4 appropriately.
Write our libbz2 interface documents
Then we need to write the "foo.sgml" document for each of our imported interfaces (in generic/libbz2), let's pick a couple for examples, BZ2_bzWrite and and BZ2_bzRead:
BZ2_bzWrite.sgml:
<refentry id="libbz2.BZ2.bzWrite.1"> <refmeta> <refentrytitle>BZ2_bzWrite</refentrytitle> <refmiscinfo>libbz2</refmiscinfo> </refmeta> <refnamediv> <refname>BZ2_bzWrite</refname> <refpurpose>Absorbs len bytes from the buffer buf, eventually to be compressed a nd written to the file</refpurpose> <indexterm id="ix.libbz2.BZ2.bzWrite.1"> <primary>BZ2_bzWrite</primary> </indexterm> </refnamediv> <refsynopsisdiv> <funcsynopsis> <funcsynopsisinfo> #include <bzlib.h> </funcsynopsisinfo><funcprototype> <funcdef>void <function>BZ2_bzWrite</function> </funcdef> <paramdef>int <parameter>*bzerror</parameter> </paramdef> <paramdef>BZFILE <parameter>*b</parameter> </paramdef> <paramdef>void <parameter>*buf</parameter> </paramdef> <paramdef>int <parameter>len</parameter> </paramdef> </funcprototype> </funcsynopsis> </refsynopsisdiv> <refsect1> <title>Description</title> <para> The function <function>BZ2_bzWrite</function> shall absorb len bytes from the buffer buf, eventually to be compressed and written to the file. </para> </refsect1> <refsect1> <title>Errors</title> <para> Possible assignments to bzerror: <VARNAME>BZ_PARAM_ERROR</VARNAME> if b is NULL or buf is NULL or len < 0 <VARNAME>BZ_SEQUENCE_ERROR</VARNAME> if b was opened with BZ2_bzReadOpen <VARNAME>BZ_IO_ERROR</VARNAME> if there is an error writing the compressed file. <VARNAME>BZ_OK</VARNAME> otherwise </para> </refsect1> </refentry>
BZ2_bzRead.sgml:
<refentry id="libbz2.BZ2.bzRead.1"> <refmeta> <refentrytitle>BZ2_bzRead</refentrytitle> <refmiscinfo>libbz2</refmiscinfo> </refmeta> <refnamediv> <refname>BZ2_bzRead</refname> <refpurpose>Reads up to len (uncompressed) bytes from the compressed file b into the buffer buf. If the read was successful, bzerror is set to BZ_OK and the num ber of bytes read is returned. If the logical end-of-stream was detected, bzerro r will be set to BZ_STREAM_END, and the number of bytes read is returned. All ot her bzerror values denote an error.</refpurpose> <indexterm id="ix.libbz2.BZ2.bzRead.1"> <primary>BZ2_bzRead</primary> </indexterm> </refnamediv> <refsynopsisdiv> <funcsynopsis> <funcsynopsisinfo> #include <bzlib.h> </funcsynopsisinfo><funcprototype> <funcdef>void <function>BZ2_bzRead</function> </funcdef> <paramdef>int <parameter>*bzerror</parameter> </paramdef> <paramdef>BZFILE <parameter>*b</parameter> </paramdef> <paramdef>void <parameter>*buf</parameter> </paramdef> <paramdef>int <parameter>len</parameter> </paramdef> </funcprototype> </funcsynopsis> </refsynopsisdiv> <refsect1> <title>Description</title> <para> The function <function>BZ2_bzRead</function> shall read up to len (uncompressed) bytes from the compressed file b into the buffer buf. If the read was successfu l, bzerror is set to BZ_OK and the number of bytes read is returned. If the logi cal end-of-stream was detected, bzerror will be set to BZ_STREAM_END, and the nu mber of bytes read is returned. All other bzerror values denote an error. </para> </refsect1> <refsect1> <title>Errors</title> <para> Possible assignments to bzerror: <VARNAME>BZ_PARAM_ERROR</VARNAME> if b is NULL or buf is NULL or len < 0 <VARNAME>BZ_SEQUENCE_ERROR</VARNAME> if b was opened with BZ2_bzWriteOpen <VARNAME>BZ_IO_ERROR</VARNAME> if there is an error reading from the compressed file. <VARNAME>BZ_UNEXPECTED_EOF</VARNAME> if the compressed file ended before the logical end-of-stream was detected <VARNAME>BZ_DATA_ERROR</VARNAME> if a data integrity error was detected in the compressed stream <VARNAME>BZ_DATA_ERROR_MAGIC</VARNAME> if the stream does not begin with the requisite header bytes (ie, is not a bzip2 data file). This is really a special case of BZ_DATA_ERROR. <VARNAME>BZ_MEM_ERROR</VARNAME> if insufficient memory was available <VARNAME>BZ_STREAM_END</VARNAME> if the logical end of stream was detected. <VARNAME>BZ_OK</VARNAME> otherwise </para> </refsect1> </refentry>
Generate content from the database
The remaining documents get generated from the database:
[Toolkit_Compression]$ make gensrc for dir in generic intro;do (cd $dir && make gensrc);done make[1]: Entering directory `/mnt/LSB/bzr/devel/lsbspec/Toolkit_Compression/generic' for dir in libbz2 ;do (cd $dir && make gensrc);done make[2]: Entering directory `/mnt/LSB/bzr/devel/lsbspec/Toolkit_Compression/generic/libbz2' ../../../mklibspec -a All -c 3 -l libbz2 >bz2.sgml Skipped 22 missing interfaces make[2]: Leaving directory `/mnt/LSB/bzr/devel/lsbspec/Toolkit_Compression/generic/libbz2' make[1]: Leaving directory `/mnt/LSB/bzr/devel/lsbspec/Toolkit_Compression/generic' make[1]: Entering directory `/mnt/LSB/bzr/devel/lsbspec/Toolkit_Compression/intro' ../../mkstandardsgmltable -a All -s "'ISOC99','libbz2'" >standards.sgml ../../mklibsgmltable -a All -m LSB_Toolkit_Compression >libraries.sgml make[1]: Leaving directory `/mnt/LSB/bzr/devel/lsbspec/Toolkit_Compression/intro'
You'll note the mention of 22 missing interfaces. These are the BZ2* interfaces we haven't yet written documents for. Looking at our directory now, we see:
Toolkit_Compression]$ find . . ./intro ./intro/makefile ./intro/standards.sgml ./intro/intro.m4 ./intro/libraries.sgml ./intro/terms.sgml ./generic ./generic/libbz2 ./generic/libbz2/makefile ./generic/libbz2/BZ2_bzWrite.sgml ./generic/libbz2/bz2.sgml ./generic/libbz2/BZ2_bzRead.sgml ./generic/libbz2/libbz2.m4 ./generic/makefile ./makefile
Create the "book" structure
I directory up, you'll find a directory named "book". Since this a new module, we'll need a similar structure here to generate the remaining documents:
[book]$ find Toolkit_Compression Toolkit_Compression Toolkit_Compression/makefile Toolkit_Compression/buildversion Toolkit_Compression/contents Toolkit_Compression/Toolkit_Compression.sgml.sed
buildversion:
3.1
contents:
&toolkit-compression-generic-bz2;
makefile: ARCH=generic DOC=Toolkit_Compression
include ../Makefile.common </pre>
Toolkit_Compression.sgml.sed:
<!DOCTYPE BOOK PUBLIC "-//OASIS//DTD DocBook V4.1//EN" [ <!ENTITY % entities SYSTEM "../../entities"> <!ENTITY contents SYSTEM "contents"> <!ENTITY specversion "VERSION"> %entities; <!ENTITY copyrightyear "2005"> <!ENTITY copyrightholder "Free Standards Group"> <!ENTITY license "&fdl;"> <!ENTITY doccopyright SYSTEM "../../matters/fsgcopyright.sgml"> ]> <BOOK> <BOOKINFO> <TITLE>LSB Toolkit Compression Specification</TITLE> <COPYRIGHT> <YEAR>2005</YEAR> <YEAR>2006</YEAR> <HOLDER>Free Standards Group</HOLDER> </COPYRIGHT> &legal; </BOOKINFO> &toolkit-compression-intro; &contents; </BOOK>
To include this in the spec document build:
[lsbspec]$ gendiff . .libbz2
--- ./book/makefile.libbz2 2007-03-16 12:53:31.000000000 -0400
+++ ./book/makefile 2007-03-16 12:54:03.000000000 -0400
@@ -17,7 +17,7 @@
LSB-CXX-PPC32 LSB-CXX-PPC64 LSB-CXX-S390 LSB-CXX-S390X \
Packaging-generic Packaging-AMD64 Packaging-IA32 Packaging-IA64 \
Packaging-PPC32 Packaging-PPC64 Packaging-S390 Packaging-S390X \
- Graphics-Ext XML $(QT4_SUBDIRS)
+ Graphics-Ext XML Toolkit_Compression $(QT4_SUBDIRS)
# $(GTK_SUBDIRS) $(QT3_SUBDIRS)
all:
--- ./makefile.libbz2 2007-03-15 17:48:52.000000000 -0400
+++ ./makefile 2007-03-15 17:49:18.000000000 -0400
@@ -1,5 +1,5 @@
-DOCDIRS=ELF Graphics LSB Packaging Graphics-Ext Toolkit_Gtk Toolkit_Qt3 Toolkit_Qt XML Desktop
+DOCDIRS=ELF Graphics LSB Packaging Graphics-Ext Toolkit_Gtk Toolkit_Qt3 Toolkit_Qt XML Desktop Toolkit_Compression
BOOKDIRS=book booksets
SUBDIRS= $(DOCDIRS) $(BOOKDIRS)
--- ./entities.libbz2 2007-03-16 13:04:21.000000000 -0400
+++ ./entities 2007-03-16 13:08:03.000000000 -0400
@@ -184,6 +184,11 @@
<!ENTITY toolkit-gtk-s390-gtk SYSTEM "Toolkit_Gtk/S390/GTK/GTK.sgml">
<!ENTITY toolkit-gtk-s390x-gtk SYSTEM "Toolkit_Gtk/S390X/GTK/GTK.sgml">
+<!-- Toolkit-Compression Chapters -->
+<!-- Generic -->
+<!ENTITY toolkit-compression-intro SYSTEM "Toolkit_Compression/intro/intro.sgml">
+<!ENTITY toolkit-compression-generic-bz2 SYSTEM "Toolkit_Compression/generic/libbz2/bz2.sgml">
+
<!-- Graphics-Ext Chapters -->
<!-- Generic -->
<!ENTITY graphics-ext-intro SYSTEM "Graphics-Ext/intro/intro.sgml">
Writing tests
Ideally, tests should be designed to test each one of the newly added interfaces. A complete test should check not only "normal" function behavior, but also verify the interface correctly handles corner and error cases, returning expected error codes.
The LSB uses a test framework known as "TET" - Test Environment Toolkit. TET includes APIs for C as well as modules for ksh, posix_sh, perl, and python.
Creating tests to verify ABI compliance can be approached in a number of ways, depending on what the library in question might provide. If the vendor already provides a test suite, it may be possible to adapt or wrap these tests in a fashion that will provide "TET" output in the form of a journal file, than can then be processed by the TET tools and submitted for certification.
LSB provides the packages lsb-tet3-lite as well as lsb-tet3-lite-devel which can be used to develop tests.
To run TET code, the environment variable TET_ROOT must be set. For these examples, setting it to `pwd` should be sufficient:
export TET_ROOT=`pwd`
Perl 4 example (from the tet tarball)
#!/usr/bin/perl
@iclist=(ic1);
@ic1=("tp1","tp2");
$tet'startup="startup";
$tet'cleanup="cleanup";
sub startup {
&tet'infoline("This is the startup");
&tet'delete("tp2", "deleted in startup");
}
sub cleanup {
&tet'infoline("This is the cleanup");
}
sub tp1{
&tet'infoline("This is the tp1 test case");
&tet'result("PASS");
}
sub tp2{
&tet'infoline("This is the tp2 test case");
&tet'result("PASS");
}
require "$ENV{\"TET_ROOT\"}/lib/perl/tcm.pl";
Run the test:
./tc1
Results are in tet_xres:
15|0 3.7 1|TCM Start 520|0 0 14116 1 1|This is the startup 400|0 1 2 15:37:39|IC Start 200|0 1 15:37:39|TP Start 520|0 1 14116 1 1|This is the tp1 test case 220|0 1 0 15:37:39|PASS 200|0 2 15:37:39|TP Start 520|0 2 14116 1 1|deleted in startup 220|0 2 6 15:37:39|UNINITIATED 410|0 1 2 15:37:39|IC End 520|0 0 14116 1 1|This is the cleanup
Python example (from the tet tarball)
#!/opt/lsb/appbat/bin/python
from pytet import *
import sys
def startup():
print "tc1: Calling startup"
tet_delete(3, "Marking test 3 as uninitiated")
def cleanup():
print "tc1: Calling cleanup"
def test1():
try:
var = tet_getvar("PYTET_TC1_VAR");
except:
tet_infoline("Failed to get a value for PYTET_TC1_VAR")
tet_result(TET_UNRESOLVED)
return
tet_infoline("PYTET_TC1_VAR is set to " + var);
tet_result(TET_PASS)
def test2():
tet_infoline("platform = " + sys.platform)
tet_result(TET_PASS)
def test3():
tet_result(TET_NOTINUSE)
testlist = { 1:test1, 2:test2, 3:test3 }
pytet_init(testlist, startup, cleanup)
Run the test:
./tc1 tc1: Calling startup tc1: Calling cleanup
Results are in tet_xres:
15|0 3.7-lite 3|TCM Start 400|0 1 1 15:30:21|IC Start 200|0 1 15:30:21|TP Start 520|0 1 00013957 1 1|Failed to get a value for PYTET_TC1_VAR 220|0 1 2 15:30:21|UNRESOLVED 410|0 1 1 15:30:21|IC End 400|0 2 1 15:30:21|IC Start 200|0 2 15:30:21|TP Start 520|0 2 00013957 1 1|platform = linux2 220|0 2 0 15:30:21|PASS 410|0 2 1 15:30:21|IC End 400|0 3 1 15:30:21|IC Start 200|0 3 15:30:21|TP Start 520|0 3 00013957 1 1|Marking test 3 as uninitiated 220|0 3 6 15:30:21|UNINITIATED 410|0 3 1 15:30:21|IC End
C example (from the tet tarball)
#include <stdlib.h>
#include <tet_api.h>
/*
* The tet_startup / tet_cleanup functions will be explained
* in detail later in the course, as will the tet_testlist
* structure
*/
void (*tet_startup)() = NULL, (*tet_cleanup)() = NULL;
void tp1();
struct tet_testlist tet_testlist[] = { {tp1,1}, {NULL,0} };
void tp1()
{
(void) tet_printf("test case: %s, TP number: %d ",
tet_pname, tet_thistest);
tet_result(TET_PASS);
/* tet_result(TET_FAIL);*/
}
Compile the test:
cc -I/opt/lsb-tet3-lite/inc/tet3 -o tc1 tc1.c /opt/lsb-tet3-lite/lib/tet3/tcm.o \
/opt/lsb-tet3-lite/lib/tet3/libapi.a
Run the test:
./tc1
Results are in tet_xres:
15|0 3.7-lite 1|TCM Start 400|0 1 1 15:48:52|IC Start 200|0 1 15:48:52|TP Start 520|0 1 00014320 1 1|test case: ./tc1, TP number: 1 220|0 1 0 15:48:52|PASS 410|0 1 1 15:48:52|IC End
Many more examples are include with the tet tarball source.
Whatever API you choose, the final test setup should be put together such that it can readily be integrated with the existing test suites. You can review the current test suites in bzr under the directories "something-test"
libbz2 test example
Now if we go back to our libbz2 example, we'll make tests for the BZ2_bzRead, BZ2_bzWrite interfaces, using the TET C API:
You can build and run the tests like this:
tar xzf bzip2-test-0.1lsb.tar.gz cd bzip2-test-0.1lsb make test
Prerequisites to build are lsb-tet3-lite-devel and whatever your distribution package for libbzip2-devel may be called.
Looking at the bzRead.c and bzWrite.c test code, you'll see we've setup a test for each of the possible return code scenarios for each function, testing not only normal operation but that the error codes returned when the function fails agree with the specification. (bzRead test2,10 are incomplete (UNTESTED)).
Each test follows the sample API code above, with a START/RESULT/END, as well as some explanation text of the assertion being tested. Although the text isn't required, it can be helpful for the end user in debugging test failures:
200|0 1 16:11:31|TP Start 520|0 1 00016060 1 1|Test 1 520|0 1 00016060 1 2|When BZ2_bzRead is called with appropriate arguments, the function shall return the number of bytes read and set bzerror to BZ_OK 220|0 1 0 16:11:31|PASS 410|0 1 1 16:11:31|IC End
It's important to try and catch every possible exit path from your test so that meaningful results are captured in the journal file. If the test exits with no result code, it makes it very difficult for the user to debug why the test counts and/or journal analysis may be inconsistent.
That's about it. To complete libbzip2, we'd continue to edit spec files for the remaining functions and write test cases for each. The new library could then either be integrated into an existing LSB module or become a module of it's own and added to the next LSB release.
libbz2 test example
Now if we go back to our libbz2 example, we'll make tests for the BZ2_bzRead, BZ2_bzWrite interfaces, using the TET C API:
You can build and run the tests like this:
tar xzf bzip2-test-0.1lsb.tar.gz cd bzip2-test-0.1lsb make test
Prerequisites to build are lsb-tet3-lite-devel and whatever your distribution package for libbzip2-devel may be called.
Looking at the bzRead.c and bzWrite.c test code, you'll see we've setup a test for each of the possible return code scenarios for each function, testing not only normal operation but that the error codes returned when the function fails agree with the specification. (bzRead test2,10 are incomplete (UNTESTED)).
Each test follows the sample API code above, with a START/RESULT/END, as well as some explanation text of the assertion being tested. Although the text isn't required, it can be helpful for the end user in debugging test failures:
200|0 1 16:11:31|TP Start 520|0 1 00016060 1 1|Test 1 520|0 1 00016060 1 2|When BZ2_bzRead is called with appropriate arguments, the function shall return the number of bytes read and set bzerror to BZ_OK 220|0 1 0 16:11:31|PASS 410|0 1 1 16:11:31|IC End
It's important to try and catch every possible exit path from your test so that meaningful results are captured in the journal file. If the test exits with no result code, it makes it very difficult for the user to debug why the test counts and/or journal analysis may be inconsistent.
That's about it. To complete libbzip2, we'd continue to edit spec files for the remaining functions and write test cases for each. The new library could then either be integrated into an existing LSB module or become a module of it's own and added to the next LSB release.
Credits
[1] Thanks to Tracy Camp for assistance with this document and for the detailed explanations of the issues involved with more complex libraries like OpenSSL (some of which are pasted verbatim into this document). Also thanks to Mats Wichmann for pointing me in the right direction(s) and helping me get up to speed working on the development side of LSB, vs. being on the customer side, certifying a distribution.