New way of getting device info from dyndrivers

[Rafael Laboissiere]

Preamble
========

The big recent cvs commit regarding the dyndrivers database was on the top
of my TODO list. It is a necessary step towards clean
configuration/building and also packaging for Debian.

I am not yet completely happy with my implementation, but my changes
(apparently) did not break PLplot. My simple test script succeeded, at
least. I am committing without previous discussion here because, as Alan
uses to say, getting novelties in cvs HEAD is the best way to foster
discussions.


The Problem
===========

The way PLplot used to get information about the available devices provided
by the dyndrivers was through the DATA_DIR/drivers.db file. This file was
generated at configuration time and parsed when the library was initialized.

This approach has two drawbacks:

1) Information about the devices are scattered in different places (namely
in the driver source file and in configure.ac). This is ugly and may
result in unnecessary maintenance burden.

2) Since the list of available devices is hardcoded in the drivers.db it is
almost impossible to do clean packaging of Plplot. In Debian, for
instance, packaging is granular in order to reduce the dependencies:
plplot-xwin, plplot-tk, plplot-gd, etc. Users can install a subset of
the available packages at will. However, they will always get the full
list of available devices when plinit is called. That is not a critical
problem, but annoying.


The Solution
============

I have elaborated a full fix for this problem, but I just committed an
intermediary solution for it. Here is how it works:

1) drivers.db does not exist anymore.

2) In each driver file, there is a global declaration like this:

char* DEVICE_INFO_gd = "jpeg:JPEG file:0:gd:40:jpeg\n"
"png:PNG file:0:gd:39:png";

containing the entries that used to go into drivers.db. If a driver
provides more than one device, their entries must be separated by a
newline character ('\n').

3) When the library is initialized, if dyndrivers are enabled, the drivers
directory is scanned for the *.la files. Each found driver is dlopened
and the DEVICE_INFO_<driver> symbol is read and put in a temporary file.

4) This temporary file plays the role of the old drivers.db file.

Noticed that I did minimal changes to Geoff's code in plcore.c. In the
plInitDispatchTable function, I replaced the initial code (where the
drivers.db were scanned) by the scanning of the drivers directory described
in point 3 above.

Also, all references to drivers.db and DRIVERS_DB have disappeared from the
sources.


Drawbacks and improvements
==========================

I see two potential problems with my approach:

1) Portability. I am using new libc functions (POSIX tmpfile, opendir,
readdir and closedir). Although am I following the recommended procedure
found in the Autoconf docs (i.e. using AC_HEADER_DIRENT and a couple of
tests with HAVE_DIRENT_H), I am sure that there will be some weird system
out there (MacIntosh, say) for which my code won't compile. If that
happens, we have to port that part of the code.

2) With my approach, I have to open all and every module before using them.
This may appear as a regression as regards the "cache file" approach
provided by the use of drivers.db. In terms of performance, with our
current computer power, the overload is negligible. However, as I wrote
above, my original design was much better, but harder to implement (of
course). It looks like this:

a) Forget about that entries a la drivers.db.

b) In each driver file <driver>.c define symbols DEV_DESC_*, DEV_SEQ_*,
DEV_TAG_*, etc. These symbols can be used when filling fields in
function plD_dispatch_init_<device>. This would further reduce the
maintenance problem due to duplication of information.

c) At build time (not configuration time), a small C program dlopen the
<driver>.c files, get the symbols described above and write the
associated device entries in <driver>.rc (or whichever name).

d) Those <driver>.rc are installed in the drivers directory, along with
the <driver>.la and <driver>.so files.

e) When PLplot is initialized, the <driver>.rc files are scanned.

If I have some time before the 5.2.1 release, I will try to implement
this idea.


Postscript
==========

In doing my changes, I noticed that the following drivers have never had
entries in drivers.db: plbuf.c and next.c. Does anyone know why?

I introduced the DEVICE_INFO_<driver> symbol in all drivers listed in
variable EXTRA_LTLIBRARIES of drivers/Makefile.am. I also noticed that the
pstex entry was wrongly written in drivers.db. Apparently, this bug have
never bothered our users...

[Alan W. Irwin]

Post by Rafael Laboissiere
3) When the library is initialized, if dyndrivers are enabled, the drivers
directory is scanned for the *.la files. Each found driver is dlopened
and the DEVICE_INFO_<driver> symbol is read and put in a temporary file.

Are you using "dlopen" in the generic sense here? The actual call should
be to lt_dlopenext, and you have to be careful of the argument (follow
exactly what I did with drvnam and drvspec so the file name comes out exactly
the same to get it to work cross-platform).

Post by Rafael Laboissiere
c) At build time (not configuration time), a small C program dlopen the
<driver>.c files,

Same comment. My initial use of lt_dlopenext worked fine on Linux but died
horribly on OSF1 until I made my final changes to the lt_dlopenext argument.

Post by Rafael Laboissiere
Postscript
==========
In doing my changes, I noticed that the following drivers have never had
entries in drivers.db: plbuf.c and next.c. Does anyone know why?

IIRC next.c is completely unmaintained and kept in CVS only as an
encouragment for somebody, someday, to get it going again.

plbuf.c is not a driver in the traditional sense. I believe it is in the
wrong directory and should be put in src instead. It's code is put directly
into the libplplot library (see src/Makefile.am).

Post by Rafael Laboissiere
I also noticed that the
pstex entry was wrongly written in drivers.db. Apparently, this bug have
never bothered our users...

I have forgotten the details, but IIRC, I checked that drivers.db anomaly
before, and I am pretty sure it is necessary in order for pstex to work.
Something to do with the way pstex depends on ps or vice versa.

Joao, do you remember about this?

Alan
__________________________
Alan W. Irwin
email: ***@beluga.phys.uvic.ca
phone: [phone]

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the Canadian Centre for Climate Modelling and
Analysis (www.cccma.bc.ec.gc.ca) and the PLplot scientific plotting software
package (plplot.org).

__________________________

Linux-powered Science
__________________________

[Joao Cardoso]

...

Post by Alan W. Irwin

Post by Rafael Laboissiere
I also noticed that the
pstex entry was wrongly written in drivers.db. Apparently, this bug have
never bothered our users...

I have forgotten the details, but IIRC, I checked that drivers.db anomaly
before, and I am pretty sure it is necessary in order for pstex to work.
Something to do with the way pstex depends on ps or vice versa.
Joao, do you remember about this?

No. I only remember that the pstex configuration changed a couple of times.
The pstex needs the ps driver, directly calling the ps driver entry points.

If it works with Rafael changes, that's fine for me. For a quick test use any
C example and the scripts/pstex2eps script to generate an eps.

Post by Alan W. Irwin
Alan

[Rafael Laboissiere]

Post by Joao Cardoso
No. I only remember that the pstex configuration changed a couple of times.
The pstex needs the ps driver, directly calling the ps driver entry points.
If it works with Rafael changes, that's fine for me. For a quick test use
any C example and the scripts/pstex2eps script to generate an eps.

I cleaned up this mess, see my last cvs commit. For dyndrivers, it
seems that when pstex.so is dynopened, ps.so must be also be present in the
drivers directory. I added the following lines to configure.ac:

if test "$enable_dyndrivers" = yes -a "$enable_pstex" = yes ; then
enable_ps=yes
fi

I also fixed the dependencies in drivers/Makefile.am and changed "ps" to
"pstex" in DEVICE_INFO_pstex (pstex.c file).

[Rafael Laboissiere]

Post by Alan W. Irwin
Are you using "dlopen" in the generic sense here? The actual call should
be to lt_dlopenext, and you have to be careful of the argument (follow
exactly what I did with drvnam and drvspec so the file name comes out
exactly the same to get it to work cross-platform).

Of course, I used the libltdl interface (lt_dl* functions). In particular,
the modules are open with lt_dlopenext.

Post by Alan W. Irwin
IIRC next.c is completely unmaintained and kept in CVS only as an
encouragment for somebody, someday, to get it going again.

Okay. This is something that will be absent of the 5.2.1 tarball, thnaks to
"make dist".

Post by Alan W. Irwin
plbuf.c is not a driver in the traditional sense. I believe it is in the
wrong directory and should be put in src instead. It's code is put
directly into the libplplot library (see src/Makefile.am).

Why are we waiting to do the move? The only drawback that I see is that we
will loose the commit history if we move the file, since braindead CVS does
not know how to move files in the repository tree.

[Maurice LeBrun]

Post by Rafael Laboissiere
Why are we waiting to do the move? The only drawback that I see is that we
will loose the commit history if we move the file, since braindead CVS does
not know how to move files in the repository tree.

I say move it. It's been a while since any significant changes have been made
to it IIRC, but soon I have plans for it. And, one can always see the rev
history by going to the drivers dir and doing a 'cvs log' even if the file
has been cvs rm'ed. Be sure to mention where it came from & what version
in the first commit of the file in the new location.

[Rafael Laboissiere]

Post by Maurice LeBrun
I say move it. It's been a while since any significant changes have been made
to it IIRC, but soon I have plans for it. And, one can always see the rev
history by going to the drivers dir and doing a 'cvs log' even if the file
has been cvs rm'ed. Be sure to mention where it came from & what version
in the first commit of the file in the new location.

Alan, could you take care of this?

[Joao Cardoso]

Post by Rafael Laboissiere
Preamble
========
The big recent cvs commit regarding the dyndrivers database was on the top
of my TODO list. It is a necessary step towards clean
configuration/building and also packaging for Debian.
I am not yet completely happy with my implementation, but my changes
(apparently) did not break PLplot. My simple test script succeeded, at
least. I am committing without previous discussion here because, as Alan
uses to say, getting novelties in cvs HEAD is the best way to foster
discussions.

That's true, but the target should be to get a full-working 5.2.1, that should
be, in my opinion, a bug-correcting release.
Introducing such stuff can compromise it.
But we have not defined what 5.2.1 should be, and I'm too conservative.

I have myself lots of new stuff, and I'm refraining myself to commit them. I
hope that 5.2.2 or even 5.3.0 follows shortly.

Post by Rafael Laboissiere
The Problem
===========
The way PLplot used to get information about the available devices provided
by the dyndrivers was through the DATA_DIR/drivers.db file. This file was
generated at configuration time and parsed when the library was initialized.
1) Information about the devices are scattered in different places (namely
in the driver source file and in configure.ac). This is ugly and may
result in unnecessary maintenance burden.
2) Since the list of available devices is hardcoded in the drivers.db it is
almost impossible to do clean packaging of Plplot. In Debian, for
plplot-xwin, plplot-tk, plplot-gd, etc. Users can install a subset of
the available packages at will. However, they will always get the full
list of available devices when plinit is called. That is not a critical
problem, but annoying.
The Solution
============
I have elaborated a full fix for this problem, but I just committed an
1) drivers.db does not exist anymore.
char* DEVICE_INFO_gd = "jpeg:JPEG file:0:gd:40:jpeg\n"
"png:PNG file:0:gd:39:png";

OK, but we need a registration text file, manually maintained, to guide new
drivers writers on how to obtain (and register) new ids for the new drivers.

Post by Rafael Laboissiere
containing the entries that used to go into drivers.db. If a driver
provides more than one device, their entries must be separated by a
newline character ('\n').
3) When the library is initialized, if dyndrivers are enabled, the drivers
directory is scanned for the *.la files. Each found driver is dlopened
and the DEVICE_INFO_<driver> symbol is read and put in a temporary file.
4) This temporary file plays the role of the old drivers.db file.

Why is the tmp file needed? Why not keep the collected info in an internal
structure?

Post by Rafael Laboissiere
Noticed that I did minimal changes to Geoff's code in plcore.c. In the
plInitDispatchTable function, I replaced the initial code (where the
drivers.db were scanned) by the scanning of the drivers directory described
in point 3 above.

Ah, you use a tmp file just to reuse Geoff's code. But there is no reason to
not change that also latter, right?

Post by Rafael Laboissiere
Also, all references to drivers.db and DRIVERS_DB have disappeared from the
sources.
Drawbacks and improvements
==========================
1) Portability. I am using new libc functions (POSIX tmpfile, opendir,
readdir and closedir). Although am I following the recommended
procedure found in the Autoconf docs (i.e. using AC_HEADER_DIRENT and a
couple of tests with HAVE_DIRENT_H), I am sure that there will be some
weird system out there (MacIntosh, say) for which my code won't compile.
If that happens, we have to port that part of the code.

Don't create the file :)

Post by Rafael Laboissiere
2) With my approach, I have to open all and every module before using them.
This may appear as a regression as regards the "cache file" approach
provided by the use of drivers.db. In terms of performance, with our
above, my original design was much better, but harder to implement (of
a) Forget about that entries a la drivers.db.
b) In each driver file <driver>.c define symbols DEV_DESC_*, DEV_SEQ_*,
DEV_TAG_*, etc. These symbols can be used when filling fields in
function plD_dispatch_init_<device>. This would further reduce the
maintenance problem due to duplication of information.
c) At build time (not configuration time), a small C program dlopen the
<driver>.c files,

You mean "open" and not dlopen(), right?

Post by Rafael Laboissiere
get the symbols

No, you mean dlopen() the <driver>.so (I don't know the current suffix)

What do you really mean? Anyway, see bellow.

Post by Rafael Laboissiere
described above and write the
associated device entries in <driver>.rc (or whichever name).
d) Those <driver>.rc are installed in the drivers directory, along with
the <driver>.la and <driver>.so files.
e) When PLplot is initialized, the <driver>.rc files are scanned.
If I have some time before the 5.2.1 release, I will try to implement
this idea.

I prefer the full dynamic one, i.e., reading the already build drivers. I
don't think that performance will suffer. Do you have some figures or only
feelings?

Your second idea implies that the small program that scans the source files
needs information from the configure step to know what drivers are desired by
the user and supported by the system. This complicates matters.
With the first idea, by contrary, only already build drivers, i.e, user
desired and system supported, are scanned.

Also, with the second idea I think that the driver-ids (the magic numbers) can
go away -- ah, no, for historical reasons the xwin driver must be number one,
the tk driver number 2, etc. hmm, there is another solution for this but let
discuss it latter.

If performance is really an issue, then one could implements a mixing of the
two ideas: don't generate drivers.db at configure nor build time. At run time
if drivers.db does not exists, scan the drivers and build it. This way the
performance loss will only occurs once.
If a new driver is latter added to the directory (not probable, but possible,
after all this is the only advantage of dyndrivers), one could first compare
the time-stamp of all drivers versus the drivers.db file, and rebuild
drivers.db if a driver is more recent. Reading time-stamp is fast, I believe.

Joao

[Rafael Laboissiere]

Post by Joao Cardoso
That's true, but the target should be to get a full-working 5.2.1, that
should be, in my opinion, a bug-correcting release. Introducing such stuff
can compromise it. But we have not defined what 5.2.1 should be, and I'm
too conservative.

Well, my changes does not introduce new features to PLplot. BTW, they are
transparent for the users and only slightly important for the developers
(actually, I think my changes improve maintainability).

Post by Joao Cardoso
I have myself lots of new stuff, and I'm refraining myself to commit them.
I hope that 5.2.2 or even 5.3.0 follows shortly.

If you are planning to do changes that will either destabilize the code or
introduce new features like changes in the API, I think you should refrain
yourself until 5.2.1 is out.

Post by Joao Cardoso
Ah, you use a tmp file just to reuse Geoff's code. But there is no reason to
not change that also latter, right?

You got it. I have not against improving the code and putting all the
information in a structure, that can also be cached in the <driver>.rc
files. The important point here is that we should not have a unique file
(like drivers.db) that contains the information about all available drivers.

Post by Joao Cardoso

Post by Rafael Laboissiere
1) Portability. I am using new libc functions (POSIX tmpfile, opendir,
readdir and closedir). Although am I following the recommended
procedure found in the Autoconf docs (i.e. using AC_HEADER_DIRENT and a
couple of tests with HAVE_DIRENT_H), I am sure that there will be some
weird system out there (MacIntosh, say) for which my code won't compile.
If that happens, we have to port that part of the code.

Don't create the file :)

[Rafael Laboissiere]

[ Sorry if this message is duplicated. I sent it originally to Joao only and
tried to bounce it from my MUA. Since I did not receive it from the
listserver, I am trying again, as a forward.

Post by Joao Cardoso
That's true, but the target should be to get a full-working 5.2.1, that
should be, in my opinion, a bug-correcting release. Introducing such stuff
can compromise it. But we have not defined what 5.2.1 should be, and I'm
too conservative.

Well, my changes does not introduce new features to PLplot. BTW, they are
transparent for the users and only slightly important for the developers
(actually, I think my changes improve maintainability).

Post by Joao Cardoso
I have myself lots of new stuff, and I'm refraining myself to commit them.
I hope that 5.2.2 or even 5.3.0 follows shortly.

If you are planning to do changes that will either destabilize the code or
introduce new features like changes in the API, I think you should refrain
yourself until 5.2.1 is out.

Post by Joao Cardoso
Ah, you use a tmp file just to reuse Geoff's code. But there is no reason to
not change that also latter, right?

You got it. I have not against improving the code and putting all the
information in a structure, that can also be cached in the <driver>.rc
files. The important point here is that we should not have a unique file
(like drivers.db) that contains the information about all available drivers.

Post by Joao Cardoso

Post by Rafael Laboissiere
1) Portability. I am using new libc functions (POSIX tmpfile, opendir,
readdir and closedir). Although am I following the recommended
procedure found in the Autoconf docs (i.e. using AC_HEADER_DIRENT and a
couple of tests with HAVE_DIRENT_H), I am sure that there will be some
weird system out there (MacIntosh, say) for which my code won't compile.
If that happens, we have to port that part of the code.

Don't create the file :)

[Rafael Laboissiere]

drivers_DATA = $(drivers_LTLIBRARIES:.la=.rc)
%.rc: %.la get_drv_info
@echo get_drv_info $< $@

Please, take that "@echo" out. I used it for testing the idea here.
It works, anyway.

I guess I am typing too fast and posting without reading what I wrote...

[Rafael Laboissiere]

Post by Rafael Laboissiere
c) At build time (not configuration time), a small C program dlopen the
<driver>.c files, get the symbols described above and write the
associated device entries in <driver>.rc (or whichever name).

Just to make things a bit more concrete, here is the design that I have in
mind. I will take the ps.c driver as an example. The following global
variables will be defined in the driver source:

char* DEVICES_ps = "ps:psc";

char* DESCRIPTION_ps = "PostScript File (monochrome)";
int TYPE_ps = plDevType_FileOriented;
int SEQNUM_ps = 29;
char* SYMTAG_ps = "psm";

char* DESCRIPTION_psc = "PostScript File (color)";
int TYPE_psc = plDevType_FileOriented;
int SEQNUM_psc = 30;
char* SYMTAG_psc = "psc";

Of course, these variables should be used in the plD_dispatch_init_*
functions. In the case of ps.c:

ps_dispatch_init_helper( pdt,
DESCRIPTION_ps, "ps",
TYPE_ps, SEQNUM_ps,
(plD_init_fp) plD_init_psm );

ps_dispatch_init_helper( pdt,
DESCRIPTION_psc, "psc",
TYPE_psc, SEQNUM_psc,
(plD_init_fp) plD_init_psc );

This will insure that things are not defined twice in different places (like
with the current DEVICE_INFO_* variable).

The small C program that will generate the <driver>.rc file from the
<driver>.la file, would do: (1) dlopen the module; (2) get the DEVICES_*
symbol and parse its device components (in the ps.c exemple, that would be
"ps" and "psc"); (3) for each one of the devices found, get the symbols
DESCRIPTION_<dev>, TYPE_<dev>, SEQNUM_<dev>, and SYMTAG_<dev>; (4) create
the temp file that is parsed by Geoff's code. (This could be improved along
Joao's suggestion of creating a structure instead of writing the information
in a temp file.)

We have to decide first whether it is not worth abandoning the current
approach and adopting this "cached info" approach above. There is also
Joao's suggestion for dynamically building the drivers.db, but I am too lazy
to implement that (and I am not sure it is a superior approach).

What do you think? I accept suggestions for better variable names.

[João Cardoso]

On Wednesday 05 February 2003 13:55, Rafael Laboissiere wrote:
| * Rafael Laboissiere <***@psy.mpg.de> [2003-02-04 22:37]:
| > c) At build time (not configuration time), a small C program
| > dlopen the <driver>.c files, get the symbols described above and
| > write the associated device entries in <driver>.rc (or whichever
| > name).
|
| Just to make things a bit more concrete, here is the design that I
| have in mind. I will take the ps.c driver as an example. The
| following global variables will be defined in the driver source:
|
| char* DEVICES_ps = "ps:psc";
|
| char* DESCRIPTION_ps = "PostScript File (monochrome)";
| int TYPE_ps = plDevType_FileOriented;
| int SEQNUM_ps = 29;
| char* SYMTAG_ps = "psm";
|
| char* DESCRIPTION_psc = "PostScript File (color)";
| int TYPE_psc = plDevType_FileOriented;
| int SEQNUM_psc = 30;
| char* SYMTAG_psc = "psc";
|
| Of course, these variables should be used in the plD_dispatch_init_*
| functions. In the case of ps.c:
|
| ps_dispatch_init_helper( pdt,
| DESCRIPTION_ps, "ps",
| TYPE_ps, SEQNUM_ps,
| (plD_init_fp) plD_init_psm );
|
| ps_dispatch_init_helper( pdt,
| DESCRIPTION_psc, "psc",
| TYPE_psc, SEQNUM_psc,
| (plD_init_fp) plD_init_psc );
|
| This will insure that things are not defined twice in different
| places (like with the current DEVICE_INFO_* variable).
|
| The small C program that will generate the <driver>.rc file from the
| <driver>.la file, would do: (1) dlopen the module; (2) get the
| DEVICES_* symbol and parse its device components (in the ps.c
| exemple, that would be "ps" and "psc"); (3) for each one of the
| devices found, get the symbols DESCRIPTION_<dev>, TYPE_<dev>,
| SEQNUM_<dev>, and SYMTAG_<dev>; (4) create the temp file that is
| parsed by Geoff's code. (This could be improved along Joao's
| suggestion of creating a structure instead of writing the information
| in a temp file.)
|
| We have to decide first whether it is not worth abandoning the
| current approach and adopting this "cached info" approach above.

I decided to give it a try but got a seg fault:

[***@feup] gdb x01c
(gdb) run -dev xwin
Starting program: /usr/local/test/lib/plplot5.2.0/examples/c/x01c -dev
xwin

Program received signal SIGSEGV, Segmentation fault.
0x0806a46e in lt_dlsym (handle=0x807b0e0,
symbol=0xbfffef50 "DEVICE_INFO_pstex") at ltdl.c:3330
3330 lensym = LT_STRLEN (symbol) + LT_STRLEN
(handle->loader->sym_prefix)
(gdb) where
#0 0x0806a46e in lt_dlsym (handle=0x807b0e0,
symbol=0xbfffef50 "DEVICE_INFO_pstex") at ltdl.c:3330
#1 0x08053d38 in plInitDispatchTable () at plcore.c:1638
#2 0x08049b9d in plMergeOpts (options=0x8073500,
name=0x11 <Address 0x11 out of bounds>, notes=0x11) at plargs.c:699
#3 0x08049362 in main ()
#4 0x400674a2 in __libc_start_main () from /lib/libc.so.6


| There is also Joao's suggestion for dynamically building the
| drivers.db, but I am too lazy to implement that (and I am not sure it
| is a superior approach).
|
| What do you think? I accept suggestions for better variable names.

[Rafael Laboissiere]

Well, all the 20 demos run fine for me.

Post by João Cardoso
(gdb) run -dev xwin
Starting program: /usr/local/test/lib/plplot5.2.0/examples/c/x01c -dev
xwin
Program received signal SIGSEGV, Segmentation fault.
0x0806a46e in lt_dlsym (handle=0x807b0e0,
symbol=0xbfffef50 "DEVICE_INFO_pstex") at ltdl.c:3330
[...]

That is very strange. Are you sure that you have the newest cvs sources?

[João Cardoso]

On Wednesday 05 February 2003 17:15, Rafael Laboissiere wrote:
| * João Cardoso <***@fe.up.pt> [2003-02-05 16:55]:
| > I decided to give it a try but got a seg fault:
| >
| > [***@feup] gdb x01c
|
| Well, all the 20 demos run fine for me.
|
| > (gdb) run -dev xwin
| > Starting program: /usr/local/test/lib/plplot5.2.0/examples/c/x01c
| > -dev xwin
| >
| > Program received signal SIGSEGV, Segmentation fault.
| > 0x0806a46e in lt_dlsym (handle=0x807b0e0,
| > symbol=0xbfffef50 "DEVICE_INFO_pstex") at ltdl.c:3330
| > [...]
|
| That is very strange. Are you sure that you have the newest cvs
| sources?

Pretty sure, I had to cvs update to see your changes. The configure line

make maintainer-clean
./bootstrap.sh

Running aclocal (GNU automake) 1.7.2... done
Running libtoolize (GNU libtool) 1.4.3... done
Running autoheader (GNU Autoconf) 2.57... done
Running automake (GNU automake) 1.7.2... done
Running autoconf (GNU Autoconf) 2.57... done

./configure --enable-octave --enable-dyndrivers --disable-static

--with-double --prefix=/usr/local/test

Joao

[Rafael Laboissiere]

Post by Alan W. Irwin
If you had different configure options than Joao and me but are too tired
to re-run the test, could you at least tell us what those configuration
options are? I might find they work on my system which would imply the bug
is configure option dependent and not autotools version dependent.

Just a comment on this: what makes me think that the problem is related to

Post by Alan W. Irwin
| > (gdb) run -dev xwin
| > Starting program: /usr/local/test/lib/plplot5.2.0/examples/c/x01c
| > -dev xwin
| >
| > Program received signal SIGSEGV, Segmentation fault.
| > 0x0806a46e in lt_dlsym (handle=0x807b0e0,
| > symbol=0xbfffef50 "DEVICE_INFO_pstex") at ltdl.c:3330
| > [...]

The bug has been caused by a call to the function lt_dlsym in ltdl.c, more
precisely when it is called with argument "DEVICE_INFO_pstex". Of course,
this is related to my last changes in the code. However, I can hardly see
what I am doing wrong, since I cannot replicate the bug. I tend to believe
that it is related to version problems in the Autotools.

That said, I really hope that this is a configuration option problem, which
would be easier to fix...

[João Cardoso]

On Wednesday 05 February 2003 16:55, João Cardoso wrote:
| On Wednesday 05 February 2003 13:55, Rafael Laboissiere wrote:
| | * Rafael Laboissiere <***@psy.mpg.de> [2003-02-04 22:37]:
| | > c) At build time (not configuration time), a small C program
| | > dlopen the <driver>.c files, get the symbols described above and
| | > write the associated device entries in <driver>.rc (or whichever
| | > name).
| |
| | Just to make things a bit more concrete, here is the design that I
| | have in mind. I will take the ps.c driver as an example. The
| | following global variables will be defined in the driver source:
| |
| | char* DEVICES_ps = "ps:psc";
| |
| | char* DESCRIPTION_ps = "PostScript File (monochrome)";
| | int TYPE_ps = plDevType_FileOriented;
| | int SEQNUM_ps = 29;
| | char* SYMTAG_ps = "psm";
| |
| | char* DESCRIPTION_psc = "PostScript File (color)";
| | int TYPE_psc = plDevType_FileOriented;
| | int SEQNUM_psc = 30;
| | char* SYMTAG_psc = "psc";
| |
| | Of course, these variables should be used in the
| | plD_dispatch_init_* functions. In the case of ps.c:
| |
| | ps_dispatch_init_helper( pdt,
| | DESCRIPTION_ps, "ps",
| | TYPE_ps, SEQNUM_ps,
| | (plD_init_fp) plD_init_psm );
| |
| | ps_dispatch_init_helper( pdt,
| | DESCRIPTION_psc, "psc",
| | TYPE_psc, SEQNUM_psc,
| | (plD_init_fp) plD_init_psc );
| |
| | This will insure that things are not defined twice in different
| | places (like with the current DEVICE_INFO_* variable).
| |
| | The small C program that will generate the <driver>.rc file from
| | the <driver>.la file, would do: (1) dlopen the module; (2) get the
| | DEVICES_* symbol and parse its device components (in the ps.c
| | exemple, that would be "ps" and "psc"); (3) for each one of the
| | devices found, get the symbols DESCRIPTION_<dev>, TYPE_<dev>,
| | SEQNUM_<dev>, and SYMTAG_<dev>; (4) create the temp file that is
| | parsed by Geoff's code. (This could be improved along Joao's
| | suggestion of creating a structure instead of writing the
| | information in a temp file.)
| |
| | We have to decide first whether it is not worth abandoning the
| | current approach and adopting this "cached info" approach above.

Now that HEAD is working again. lets continue.

I measure the time that your code in plInitDispatchTable() takes to load
all drivers, and it is only 20ms (wall clock) for all 33 drivers, on a
***@700MHz (with a small but continuous load of 1). It looks like it is
pretty efficient.

Your concerns that dlopen() would load all libraries that a driver needs
does not apply, as this would only happens if some drivers code is
executed, which is not the case. As libtool's info says:

"Unresolved symbols in the module are resolved using
its dependency libraries"

As the symbols you are looking for are in the modules, no further
loading will occur.

Thus, given that the current implementation is efficient enough, avoids
the need of another program to build the drivers.rc file, is fully
dynamic because if more drivers are added to the directory they will be
recognized without further intervention,... lets keep it as is.

Joao

|
| I decided to give it a try but got a seg fault:
|
| [***@feup] gdb x01c
| (gdb) run -dev xwin
| Starting program: /usr/local/test/lib/plplot5.2.0/examples/c/x01c
| -dev xwin
|
| Program received signal SIGSEGV, Segmentation fault.
| 0x0806a46e in lt_dlsym (handle=0x807b0e0,
| symbol=0xbfffef50 "DEVICE_INFO_pstex") at ltdl.c:3330
| 3330 lensym = LT_STRLEN (symbol) + LT_STRLEN
| (handle->loader->sym_prefix)
| (gdb) where
| #0 0x0806a46e in lt_dlsym (handle=0x807b0e0,
| symbol=0xbfffef50 "DEVICE_INFO_pstex") at ltdl.c:3330
| #1 0x08053d38 in plInitDispatchTable () at plcore.c:1638
| #2 0x08049b9d in plMergeOpts (options=0x8073500,
| name=0x11 <Address 0x11 out of bounds>, notes=0x11) at
| plargs.c:699 #3 0x08049362 in main ()
| #4 0x400674a2 in __libc_start_main () from /lib/libc.so.6
|
| | There is also Joao's suggestion for dynamically building the
| | drivers.db, but I am too lazy to implement that (and I am not sure
| | it is a superior approach).
| |
| | What do you think? I accept suggestions for better variable names.
|
| -------------------------------------------------------
| This SF.NET email is sponsored by:
| SourceForge Enterprise Edition + IBM + LinuxWorld
| http://www.vasoftware.com
| _______________________________________________
| Plplot-devel mailing list
| Plplot-***@lists.sourceforge.net
| https://lists.sourceforge.net/lists/listinfo/plplot-devel

[Rafael Laboissiere]

Post by João Cardoso
I measure the time that your code in plInitDispatchTable() takes to load
all drivers, and it is only 20ms (wall clock) for all 33 drivers, on a
pretty efficient.
Your concerns that dlopen() would load all libraries that a driver needs
does not apply, as this would only happens if some drivers code is
"Unresolved symbols in the module are resolved using
its dependency libraries"
As the symbols you are looking for are in the modules, no further
loading will occur.
Thus, given that the current implementation is efficient enough, avoids
the need of another program to build the drivers.rc file, is fully
dynamic because if more drivers are added to the directory they will be
recognized without further intervention,... lets keep it as is.

Thanks for addressing those two points which I was too lazy to evaluate
(efficiency and loading of libraries). Although I am not totally happy with
the current implementation, I will keep it as is for now.

I am pretty convinced that the memory management problems that you had when
using lt_dlclose come only from the libltdl code (use of a private realloc
along with a system's malloc). This has been discussed in the libtool
mailing list and fixed in the libtool's cvs tree. There are good news here:
version 1.5 is coming soon:

http://mail.gnu.org/archive/html/libtool/2003-02/msg00014.html

Next step: Debian packages.

[Rafael Laboissiere]

Post by Maurice LeBrun
I've not been following the discussion very closely, but does this mean that
the code currently loads all dynamic drivers at startup time? If true, this
seems against the spirit of dynamic drivers, and adds unnecessarily to the
memory used by the application.

This is what I thought at the beginning and this is why I proposed the
alternative design with the <driver>.rc files. However here is what Joao

Post by Maurice LeBrun
Your concerns that dlopen() would load all libraries that a driver needs
does not apply, as this would only happens if some drivers code is
"Unresolved symbols in the module are resolved using
its dependency libraries"
As the symbols you are looking for are in the modules, no further
loading will occur.

I hope that this is true for all architectures.

At any rate, as Alan pointed out, in the current design the driver moudles
should be dlclosed after the plD_DEVICE_INFO_<dirver> variable is obtained.

[João Cardoso]

On Monday 10 February 2003 07:13, Rafael Laboissiere wrote:
| * Maurice LeBrun <***@gazoo.ph.utexas.edu> [2003-02-09 21:13]:
| > I've not been following the discussion very closely, but does this
| > mean that the code currently loads all dynamic drivers at startup
| > time? If true, this seems against the spirit of dynamic drivers,
| > and adds unnecessarily to the memory used by the application.
|
| This is what I thought at the beginning and this is why I proposed
| the alternative design with the <driver>.rc files. However here is
| what Joao wrote some days ago:
|
| * João Cardoso <***@fe.up.pt> [2003-02-07 14:51]:
| > Your concerns that dlopen() would load all libraries that a driver
| > needs does not apply, as this would only happens if some drivers
| > code is executed, which is not the case. As libtool's info says:
| >
| > "Unresolved symbols in the module are resolved using
| > its dependency libraries"
| >
| > As the symbols you are looking for are in the modules, no further
| > loading will occur.

Further, the libtool docs says that RTLD_LAZY is used when dlopen() is
effectively called.

But one thing is what the docs says, the other what is done.
What valgrind reports (in the attached program) is that libs are loaded:

...
==20614== discard syms in /usr/lib/libjpeg.so.62.0.0 due to munmap()
...

This seems to indicate that libjpeg was loaded, and latter unloaded,
even if not used, i.e., no code from it was executed and no symbol
belonging to it requested.

But the dlopen() man page also says:

If the library exports a routine named _init, then that
code is executed before dlopen returns.

and:

[***@feup] nm -p /usr/lib/libjpeg.so | fgrep _init
0001aaa0 T jpeg_mem_init
00001fd4 T _init

a _init function is defined in the library, meaning that it must be
executed. This must also happens for other libraries, as valgrind gives
the same kind of warning for other libraries.

Of course on most systems the libraries are not loaded, as they are
already loaded by the several programs the user is working with, but
there are of course exceptions.

Back to the drivers.rc solution, Rafael.

The attached program also shows that the bug we found does not seems to
be in the libtool library, but in the system's libdl, as the program
calls dl*() directly.

Joao

|
| I hope that this is true for all architectures.
|
| At any rate, as Alan pointed out, in the current design the driver
| moudles should be dlclosed after the plD_DEVICE_INFO_<dirver>
| variable is obtained.

[Rafael Laboissiere]

Thanks for your useful report on library loading, Joao.

Post by João Cardoso
Back to the drivers.rc solution, Rafael.

I will give it a try one of these days.

[João Cardoso]

On Monday 10 February 2003 15:44, Rafael Laboissiere wrote:
| Thanks for your useful report on library loading, Joao.
|
| * João Cardoso <***@fe.up.pt> [2003-02-10 15:19]:
| > Back to the drivers.rc solution, Rafael.
|
| I will give it a try one of these days.

If dlopen() is called with RTLD_NOW instead of RTLD_LAZY in the program
that I attached in my last e-mail, it work's OK (and its output is
similar to drivers.db).

But libtool docs says that RTLD_LAZY is used by default, so we might
have a problem. However, a comment in libtool-1.4.3/libltdl/ltdl.c
seems to indicate that this is an issue in some platforms:

/* We may have to define LT_LAZY_OR_NOW in the command line if we
find out it does not work in some platform. */

With the small program is also easier to check for memory leaks with
valgrind, and most of the leaks come from libdl.
tmpfile() also contributes a bit.

Joao

man ld.so
LD_DEBUG=help ./po

[Maurice LeBrun]

Post by Rafael Laboissiere

Post by Rafael Laboissiere
c) At build time (not configuration time), a small C program dlopen the
<driver>.c files, get the symbols described above and write the
associated device entries in <driver>.rc (or whichever name).

Just to make things a bit more concrete, here is the design that I have in
mind. I will take the ps.c driver as an example. The following global
char* DEVICES_ps = "ps:psc";
char* DESCRIPTION_ps = "PostScript File (monochrome)";
int TYPE_ps = plDevType_FileOriented;
int SEQNUM_ps = 29;
char* SYMTAG_ps = "psm";
char* DESCRIPTION_psc = "PostScript File (color)";
int TYPE_psc = plDevType_FileOriented;
int SEQNUM_psc = 30;
char* SYMTAG_psc = "psc";
...
What do you think? I accept suggestions for better variable names.

If you are introducing variables into the global name space (i.e. accessible
via "extern"), at least give them a plplot-specific introducer like "pl_" or
even better "pldev_".

[Rafael Laboissiere]

Post by Maurice LeBrun
If you are introducing variables into the global name space (i.e.
accessible via "extern"), at least give them a plplot-specific introducer
like "pl_" or even better "pldev_".

Point taken.

The functions exported by the the drivers have the "plD_" prefix. Would
that be appropriate? Should I use something like: plD_DESCRIPTION_<dev>
with the variable name in upper case to emphasize the fact that they are
exported variables?

[Maurice LeBrun]

Post by Rafael Laboissiere

Post by Maurice LeBrun
If you are introducing variables into the global name space (i.e.
accessible via "extern"), at least give them a plplot-specific introducer
like "pl_" or even better "pldev_".

Point taken.
The functions exported by the the drivers have the "plD_" prefix. Would
that be appropriate? Should I use something like: plD_DESCRIPTION_<dev>
with the variable name in upper case to emphasize the fact that they are
exported variables?

Sure.

[Rafael Laboissiere]

Post by Rafael Laboissiere

Post by Maurice LeBrun
If you are introducing variables into the global name space (i.e.
accessible via "extern"), at least give them a plplot-specific introducer
like "pl_" or even better "pldev_".

Point taken.
The functions exported by the the drivers have the "plD_" prefix. Would
that be appropriate? Should I use something like: plD_DESCRIPTION_<dev>
with the variable name in upper case to emphasize the fact that they are
exported variables?

Sure.

I just committed the change for the extern DEVICE_INFO_<driver> variables.
They are called now plD_DEVICE_INFO_<driver>.

[Alan W. Irwin]

I say move it (plbuf.c). [...] Be sure to mention where it came from & what version
in the first commit of the file in the new location.

Done.

Alan
__________________________
Alan W. Irwin
email: ***@beluga.phys.uvic.ca
phone: [phone]

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the Canadian Centre for Climate Modelling and
Analysis (www.cccma.bc.ec.gc.ca) and the PLplot scientific plotting software
package (plplot.org).

__________________________

Linux-powered Science
__________________________

[Alan W. Irwin]

Post by Joao Cardoso
Also, with the second idea I think that the driver-ids (the magic numbers)
can go away -- ah, no, for historical reasons the xwin driver must be
number one, the tk driver number 2, etc. hmm, there is another solution for
this but let discuss it latter.

Please, notice that these magic number (seqnum) are not so "magic". They
only give a hint about how to order them. Think on them like "priority
level". BTW, the gnome driver also has seqnum = 1.

I assume you are talking about your new code here. For the old code the
numbers had to be unique (and seqnum was 6 for the gnome driver). Of course,
that was a maintenance burden so if they just become non-unique priority
numbers now, that is a significant improvement.

Alan
__________________________
Alan W. Irwin
email: ***@beluga.phys.uvic.ca
phone: [phone]

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the Canadian Centre for Climate Modelling and
Analysis (www.cccma.bc.ec.gc.ca) and the PLplot scientific plotting software
package (plplot.org).

__________________________

Linux-powered Science
__________________________

[Rafael Laboissiere]

Post by Alan W. Irwin
I assume you are talking about your new code here. For the old code the
numbers had to be unique

Not necessarily. In the old code, the entries were sorted using the seq
filed of the PLDispatchtable entries using qsort. This means that, when two
drivers claim the same number, of of them will (randomly) get first in the
list. However, it is guaranteed that they will appear both before (above)
drivers with greater (lesser) sequence numbers.

This is way I see this sequence number as a kind of "priority level".

Post by Alan W. Irwin
(and seqnum was 6 for the gnome driver).

Right, I thought I had put "1" in the past.

Post by Alan W. Irwin
Of course, that was a maintenance burden so if they just become non-unique
priority numbers now, that is a significant improvement.

This is exactly the idea behind my last proposal.

[Rafael Laboissiere]

[Errata. I am typing too fast...]

Post by Rafael Laboissiere
drivers claim the same number, of of them will (randomly) get first in the

^^
*one* of them

Post by Rafael Laboissiere
This is way I see this sequence number as a kind of "priority level".

^^^

*why*

[Alan W. Irwin]

[Rafael Laboissiere]

Rafael, you should be able to reproduce this bug if you do exactly what I
did since we have very similar systems. Joao's choice of configure options
may be important or there may be something left over in your build or
install location that makes it work on your system but not on ours. But the
fresh checkout and the above rm -rf should take care of that potential
difference between us.

I cannot replicate the bug here. I did exactly what you suggested: fresh
cvs checkout in a empty dir, ./bootstrap.sh, make, and make install (haveing
rm -rf the install destination before). All x??c.c examples compile and
work perfectly.

I am puzzled.

[Alan W. Irwin]

Post by Rafael Laboissiere

Rafael, you should be able to reproduce this bug if you do exactly what I
did since we have very similar systems. Joao's choice of configure options
may be important or there may be something left over in your build or
install location that makes it work on your system but not on ours. But the
fresh checkout and the above rm -rf should take care of that potential
difference between us.

I cannot replicate the bug here. I did exactly what you suggested: fresh
cvs checkout in a empty dir, ./bootstrap.sh, make, and make install (haveing
rm -rf the install destination before). All x??c.c examples compile and
work perfectly.
I am puzzled.

So am I!

You didn't say so, but I presume you configured with exactly same options
and have latest stable autotools like Joao and me (not Debian ones).

Or maybe it is gcc version?

I have Debian woody

Version: 2:2.95.4-14

If everything is the same between us, then try valgrind on x01c in case
there are memory problems but they just happen not to have any nasty
consequences for your particular machine (often memory management bugs give
different results on different machines because they are history dependent).

If valgrind shows no problems at all, then I am really REALLY puzzled.
Bitrot has set in, its the end of the world as I know it, I am going
crazy....;-)

Or should that be crazier....;-)

Alan
__________________________
Alan W. Irwin
email: ***@beluga.phys.uvic.ca
phone: [phone]

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the Canadian Centre for Climate Modelling and
Analysis (www.cccma.bc.ec.gc.ca) and the PLplot scientific plotting software
package (plplot.org).

__________________________

Linux-powered Science
__________________________

[Rafael Laboissiere]

Post by Alan W. Irwin
You didn't say so, but I presume you configured with exactly same options
and have latest stable autotools like Joao and me (not Debian ones).

I have :

$ ./bootstrap.sh
Running aclocal (GNU automake) 1.7.2... done
Running libtoolize (GNU libtool) 1.4.2a... done
Running autoheader (GNU Autoconf) 2.57... done
Running automake (GNU automake) 1.7.2... done
Running autoconf (GNU Autoconf) 2.57... done

I am not using the latest version of libtool (1.4.3). That may explain the
difference. I will try to use 1.4.3, but that is not going to be before
this weekend (I have two busy days before me).

Post by Alan W. Irwin
Or maybe it is gcc version?
I have Debian woody
Version: 2:2.95.4-14

I have here:

Version: 2:2.95.4-17

[Joao Cardoso]

Post by Rafael Laboissiere

Post by Alan W. Irwin
You didn't say so, but I presume you configured with exactly same options
and have latest stable autotools like Joao and me (not Debian ones).

$ ./bootstrap.sh
Running aclocal (GNU automake) 1.7.2... done
Running libtoolize (GNU libtool) 1.4.2a... done
Running autoheader (GNU Autoconf) 2.57... done
Running automake (GNU automake) 1.7.2... done
Running autoconf (GNU Autoconf) 2.57... done
I am not using the latest version of libtool (1.4.3).

I noticed that "file x01c" in the build tree reports an executable ELF, not a
script file as is usual with libtool. That's why I make x01c in the install
directory, but I still get an executable!

Post by Rafael Laboissiere
That may explain the
difference. I will try to use 1.4.3, but that is not going to be before
this weekend (I have two busy days before me).

Post by Alan W. Irwin
Or maybe it is gcc version?
I have Debian woody
Version: 2:2.95.4-14

mine is gcc-3.2

Post by Rafael Laboissiere
Version: 2:2.95.4-17

[Alan W. Irwin]

Post by Rafael Laboissiere

Post by Alan W. Irwin
You didn't say so, but I presume you configured with exactly same options
and have latest stable autotools like Joao and me (not Debian ones).

$ ./bootstrap.sh
Running aclocal (GNU automake) 1.7.2... done
Running libtoolize (GNU libtool) 1.4.2a... done
Running autoheader (GNU Autoconf) 2.57... done
Running automake (GNU automake) 1.7.2... done
Running autoconf (GNU Autoconf) 2.57... done

[then got diverted into version stuff which might be relevant and forgot to
mention the configure options].

If you had different configure options than Joao and me but are too tired
to re-run the test, could you at least tell us what those configuration
options are? I might find they work on my system which would imply the bug
is configure option dependent and not autotools version dependent.

Joao's further comment about his quite different gcc version seems to take
that version dependence out of contention.

Alan

__________________________
Alan W. Irwin
email: ***@beluga.phys.uvic.ca
phone: [phone]

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the Canadian Centre for Climate Modelling and
Analysis (www.cccma.bc.ec.gc.ca) and the PLplot scientific plotting software
package (plplot.org).

__________________________

Linux-powered Science
__________________________

[Rafael Laboissiere]

Post by Alan W. Irwin
If you had different configure options than Joao and me but are too tired
to re-run the test, could you at least tell us what those configuration
options are? I might find they work on my system which would imply the bug
is configure option dependent and not autotools version dependent.

Here is exactly what I am doing.

$ rm -rf plplot
$ cvs -d ***@cvs.plplot.sourceforge.net:/cvsroot/plplot co plplot
$ cd plplot
$ ./bootstrap.sh
$ destdir=/var/tmp/plplot
$ ./configure --prefix=$destdir --disable-tcl --disable-itcl \
--disable-python --enable-dyndrivers --with-double
$ rm -rf $destdir
$ make install
$ cd examples/c
$ make x01c
$ ./x01c -dev xwin

All the C demos compile and work fine here.

I am now using Libtool 1.4.3 from Debian unstable. Please noticed that I had
to make changes in bootstrap.sh in order to get this version of Libtool
working properly with Autoconf 2.57 and Automake 1.7.2 (see my last cvs
commit).

If you guys cannot replicate my successful build with the latest version of
bootstrap.sh, I will be completely puzzled.

[Alan W. Irwin]

Post by Rafael Laboissiere

Post by Alan W. Irwin
If you had different configure options than Joao and me but are too tired
to re-run the test, could you at least tell us what those configuration
options are? I might find they work on my system which would imply the bug
is configure option dependent and not autotools version dependent.

Here is exactly what I am doing.
$ rm -rf plplot
$ cd plplot
$ ./bootstrap.sh
$ destdir=/var/tmp/plplot
$ ./configure --prefix=$destdir --disable-tcl --disable-itcl \
--disable-python --enable-dyndrivers --with-double
$ rm -rf $destdir
$ make install
$ cd examples/c
$ make x01c
$ ./x01c -dev xwin

I confirm that set of options works, and we are back in a rational universe
again after 24 hours where I wasn't sure what kind of universe we were in
....;-)

I will need more investigation to see whether Joao's set of options still
don't work. There is some ambiguity here because I realize there was a
possibility that my old "system" versions of autotools were interfering.
(My path pointed to the new versions, but some of the old configuration
files might have been interfering.) So the first thing I did was remove
those system versions completely.

apt-get --purge remove autoconf libtool
Reading Package Lists... Done
Building Dependency Tree... Done
The following packages will be REMOVED:
autoconf* autoconf2.13* automake1.5* libtool*
0 packages upgraded, 0 newly installed, 4 to remove and 0 not upgraded.
Need to get 0B of archives. After unpacking 5022kB will be freed.
Do you want to continue? [Y/n]
(Reading database ... 92196 files and directories currently installed.)
Removing automake1.5 ...
Removing libtool ...
Removing autoconf ...
Removing autoconf2.13 ...
dpkg - warning: while removing autoconf2.13, directory /etc/autoconf2.13'
not empty so not removed.
Removing diversion of /usr/bin/autoconf to /usr/bin/autoconf2.50 by
autoconf2.13'
Removing diversion of /usr/bin/autoheader to /usr/bin/autoheader2.50 by
autoconf2.13'
Removing diversion of /usr/bin/autoreconf to /usr/bin/autoreconf2.50 by
autoconf2.13'
Purging configuration files for autoconf2.13 ...
***@starling> ls /etc/autoconf2.13/
***@starling> rmdir /etc/autoconf2.13/

(note the autoconf2.13 above which always adds some version uncertainty if
you are using the Debian versions of the autotools.) That is why I keep
urging Rafael to get rid of his Debian versions of autotools to be
consistent with the rest of us, but in this case it apparently does not make
a difference because I confirm his result when I am using the latest stable
version of autotools from FSF.

Fresh plplot checkout as of 16:46 UT.

./bootstrap.sh '-I /home/software/autotools/install/share/libtool/'
Running aclocal (GNU automake) 1.7.2... done
Running autoheader (GNU Autoconf) 2.57... done
Running automake (GNU automake) 1.7.2...configure.ac: installing ./install-sh'
configure.ac: installing ./mkinstalldirs'
configure.ac: installing ./missing'
configure.ac:456: installing ./config.guess'
configure.ac:456: installing ./config.sub'
configure.ac:456: required file ./ltmain.sh' not found
Makefile.am:25: required directory ./libltdl does not exist
bindings/c++/Makefile.am: installing ./depcomp'
drivers/Makefile.am: installing ./compile'
Makefile.am:25: required directory ./libltdl does not exist
done
Running libtoolize (GNU libtool) 1.4.3... done
Running autoconf (GNU Autoconf) 2.57... done

I think you need quotes around the -I option to bootstrap.sh as above
(contrary to your commit message). Otherwise, the $1 in bootstrap.sh will
just catch the -I and miss the rest.

rm -rf /usr/local/plplot_at

./configure --prefix=/usr/local/plplot_at --disable-tcl --disable-itcl \
--disable-python --enable-dyndrivers --with-double > & configure.out

make >& make.out
make install >& make_install.out

All the *.out files looked fine.

cd /usr/local/plplot_at/lib/plplot5.2.0/examples
cd c ; make ; cd c++ ; make ; cd f77 ; make ; cd ..
./plplot-test.sh
Plplot library version: 5.2.0
Opened x01c.ps
Opened x02c.ps
Opened x03c.ps
Opened x04c.ps
Opened x05c.ps
Opened x06c.ps
Opened x07c.ps
Opened x08c.ps
Opened x09c.ps
Opened x10c.ps
Opened x11c.ps
Opened x12c.ps
Opened x13c.ps
Opened x15c.ps
Opened x16c.ps
Opened x18c.ps
Opened x19c.ps
Opened x01cc.ps
Opened x01f.ps
Opened x02f.ps
Opened x03f.ps
Opened x04f.ps
Opened x05f.ps
Opened x06f.ps
Opened x07f.ps
Opened x08f.ps
Opened x09f.ps
Opened x10f.ps
Opened x11f.ps
Opened x12f.ps
Opened x13f.ps
Opened x16f.ps

All these results are identical with the plplot-5.2.0 results.

Joao, do you confirm this set of options is working for you as well
(after removing all traces of old autotools)?

Next, I will try Joao's set of options starting from fresh checkout.

Alan

Alan W. Irwin
email: ***@beluga.phys.uvic.ca
phone: [phone]

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the Canadian Centre for Climate Modelling and
Analysis (www.cccma.bc.ec.gc.ca) and the PLplot scientific plotting software
package (plplot.org).

__________________________

Linux-powered Science
__________________________

[Rafael Laboissiere]

Post by Alan W. Irwin
I think you need quotes around the -I option to bootstrap.sh as above
(contrary to your commit message). Otherwise, the $1 in bootstrap.sh will
just catch the -I and miss the rest.

That's fixed, see cvs commit log.

[Alan W. Irwin]

Post by Alan W. Irwin
Next, I will try Joao's set of options starting from fresh checkout.

Everything done identically from fresh checkout except for

./configure --prefix=/usr/local/plplot_at --enable-octave \
--enable-dyndrivers --disable-static --with-double > & configure.out

AND THE ANSWER IS....

immediate segfault from x01c and x08c (I didn't bother to try anything else).

So we really do live in a rational universe. The problem Joao found, and
I confirmed yesterday still exists, and it depends on the exact configure
options above and disappears if you use Rafael's options.

Rafael, do you confirm this?

Joao, I still think it is important for you to confirm that Rafael's exact
set of options works on your system so we know we are all on the same page.

Alan

__________________________
Alan W. Irwin
email: ***@beluga.phys.uvic.ca
phone: [phone]

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the Canadian Centre for Climate Modelling and
Analysis (www.cccma.bc.ec.gc.ca) and the PLplot scientific plotting software
package (plplot.org).

__________________________

Linux-powered Science
__________________________

[Alan W. Irwin]

P.S. the only options that should affect the C examples are
--enable-dyndrivers --with-double (which are common to Rafael and Joao's
options) and --disable-static (which Joao has and Rafael does not).

Thus, I bet there is some problem with the present dynamic drivers
configuration or implementation that is incompatible with --disable-static.
I hope that strong clue allows Rafael to find and fix the problem without
too much further effort.

Alan
__________________________
Alan W. Irwin
email: ***@beluga.phys.uvic.ca
phone: [phone]

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the Canadian Centre for Climate Modelling and
Analysis (www.cccma.bc.ec.gc.ca) and the PLplot scientific plotting software
package (plplot.org).

__________________________

Linux-powered Science
__________________________

[Rafael Laboissiere]

Post by Alan W. Irwin
P.S. the only options that should affect the C examples are
--enable-dyndrivers --with-double (which are common to Rafael and Joao's
options) and --disable-static (which Joao has and Rafael does not).
Thus, I bet there is some problem with the present dynamic drivers
configuration or implementation that is incompatible with --disable-static.
I hope that strong clue allows Rafael to find and fix the problem without
too much further effort.

Well, when I use:

./configure --prefix=/var/tmp/plplot --enable-octave --enable-dyndrivers \
--disable-static --with-double \
--disable-python --disable-tcl --disable-itcl

then everything works fine. Notice that the --disable-static option has
been included above.

Weird.

[Rafael Laboissiere]

Post by Rafael Laboissiere

Post by Alan W. Irwin
P.S. the only options that should affect the C examples are
--enable-dyndrivers --with-double (which are common to Rafael and Joao's
options) and --disable-static (which Joao has and Rafael does not).
Thus, I bet there is some problem with the present dynamic drivers
configuration or implementation that is incompatible with --disable-static.
I hope that strong clue allows Rafael to find and fix the problem without
too much further effort.

./configure --prefix=/var/tmp/plplot --enable-octave --enable-dyndrivers \
--disable-static --with-double \
--disable-python --disable-tcl --disable-itcl
then everything works fine. Notice that the --disable-static option has
been included above.

I investigated this further and I think that you are following the wrong
trail with --disable-static. Also, your assumption that the only options
that affect the C examples are --enable-dyndrivers, --with-double, and
--disable-static is wrong. The option --disable-tcl *does* affect the C
example.

This is the minimal pair that I found:

C examples segfault:
./configure --enable-dyndrivers

C examples work fine:
./configure --enable-dyndrivers --disable-tcl

I think I am getting close to the source of the problem and that has nothing
to do with --disable-static.

Alan, could you please confirm the minimal pair above?

[Alan W. Irwin]

Post by Rafael Laboissiere
./configure --enable-dyndrivers
./configure --enable-dyndrivers --disable-tcl
Alan, could you please confirm the minimal pair above?

Not precisely. ./configure --enable-dyndrivers --prefix=blah gives me *no*
segfault for ./x01c -dev xwin. However, when I ran x01c with valgrind, I
did get lots of "invalid reads from memory" messages and an
eventual segfault.

The most important point, however, is you do see segfaults on your machine
so you have confirmed there is a severe problem, and you therefore have
something that you can debug for your situation.

Good luck in figuring this out!

Alan
__________________________
Alan W. Irwin
email: ***@beluga.phys.uvic.ca
phone: [phone]

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the Canadian Centre for Climate Modelling and
Analysis (www.cccma.bc.ec.gc.ca) and the PLplot scientific plotting software
package (plplot.org).

__________________________

Linux-powered Science
__________________________

[Rafael Laboissiere]

Post by Alan W. Irwin
The most important point, however, is you do see segfaults on your machine
so you have confirmed there is a severe problem, and you therefore have
something that you can debug for your situation.
Good luck in figuring this out!

I think I found the source of the problem, see my last cvs commit. Could
you guys confirm that HEAD works for you now?

[João Cardoso]

On Friday 07 February 2003 08:30, Rafael Laboissiere wrote:
| * Alan W. Irwin <***@beluga.phys.uvic.ca> [2003-02-06 18:52]:
| > The most important point, however, is you do see segfaults on your
| > machine so you have confirmed there is a severe problem, and you
| > therefore have something that you can debug for your situation.
| >
| > Good luck in figuring this out!
|
| I think I found the source of the problem, see my last cvs commit.
| Could you guys confirm that HEAD works for you now?

Yes,

[***@feup] ./bootstrap.sh
Running aclocal (GNU automake) 1.7.2... done
Running autoheader (GNU Autoconf) 2.57... done
Running automake (GNU automake) 1.7.2... done
Running libtoolize (GNU libtool) 1.4.3... done
Running autoconf (GNU Autoconf) 2.57... done

./configure --enable-octave --enable-dyndrivers --disable-static
--with-double --prefix=/usr/local/test

And after install x01c works.

Joao

[Rafael Laboissiere]

Post by João Cardoso
Yes,
Running aclocal (GNU automake) 1.7.2... done
Running autoheader (GNU Autoconf) 2.57... done
Running automake (GNU automake) 1.7.2... done
Running libtoolize (GNU libtool) 1.4.3... done
Running autoconf (GNU Autoconf) 2.57... done
./configure --enable-octave --enable-dyndrivers --disable-static
--with-double --prefix=/usr/local/test
And after install x01c works.

Uff... I am relieved. I can go out for the weekend knowing that HEAD is not
in a completely bad shape.

The memory management problems are still there, though...

[Alan W. Irwin]

Post by Rafael Laboissiere

Post by Alan W. Irwin
The most important point, however, is you do see segfaults on your machine
so you have confirmed there is a severe problem, and you therefore have
something that you can debug for your situation.
Good luck in figuring this out!

I think I found the source of the problem, see my last cvs commit. Could
you guys confirm that HEAD works for you now?

Yes with a qualification.

No segfaults for ./x01c with the "Joao" configuration.

However,

valgrind --num-callers=100 ./x01c -dev psc -o temp.ps

reports 3 memory management errors (all described as "Conditional jump or
move depends on uninitialised value(s)") having to do with the call to
lt_dlopenext at plcore.c line 1634. Memory management errors with this
description are often non-consequential because they typically come from
code which jumps depending on the truth of condition1 OR condition2 where
condition1 depends on the uninitialized values and condition2 is true (and
thus the jump occurs correctly despite the memory management error). To get
rid of this valgrind error message when it occurred directly within PLplot
code, I simply used condition2 OR condition1 so that condition1 was never
executed when condition2 was true.

The actual jump with the memory management problem occurs some 20 layers
deeper into libltl and the libraries it calls so ordinarily you would
dismiss it as a problem with library code, but I pursued this further and
think it is solely a problem with the xwin device. First, note that the
exact same valgrind test produces no such messages for the 5.2.0 version
with the psc device. However, *with 5.2.0* the same 3 valgrind error
messages are produced using -dev xwin. So I believe -dev xwin is just not
quite set up correctly to be dynamic, but the rest of the devices are okay.
The reason I say this is the new code loops over every device (as a
replacement for drivers.db) and I attribute the 3 valgrind messages I found
above to the xwin device.

Summary: the new code loops over every device so will trigger the
complete collection of valgrind errors for all devices (just xwin in this
case) while the old code just looked at the user-specified device so gets no
valgrind errors (except when the user specified xwin).

So to become valgrind-clean we should do the following:

(1) Figure out what is wrong with the dynamic setup of -dev xwin compared to
*all* the other devices.

(2) Deal with the many memory leaks (valgrind defines these to be unfreed
memory at end of programme). valgrind found all the ones specific to PLplot
are generated by the dynamic driver code and some even had clobbered
pointers by end of programme. For more details about these problems, please
see the PROBLEMS file, and
http://sourceforge.net/mailarchive/message.php?msg_id=1785861.

(3) Deal with the recent problem Rafael had with using lt_dlclose. That
problem may automatically get solved once the memory leaks are dealt with.

Alan
__________________________
Alan W. Irwin
email: ***@beluga.phys.uvic.ca
phone: [phone]

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the Canadian Centre for Climate Modelling and
Analysis (www.cccma.bc.ec.gc.ca) and the PLplot scientific plotting software
package (plplot.org).

__________________________

Linux-powered Science
__________________________

[Maurice LeBrun]

Post by Alan W. Irwin
Summary: the new code loops over every device so will trigger the
complete collection of valgrind errors for all devices (just xwin in this
case) while the old code just looked at the user-specified device so gets no
valgrind errors (except when the user specified xwin).

I've not been following the discussion very closely, but does this mean that
the code currently loads all dynamic drivers at startup time? If true, this
seems against the spirit of dynamic drivers, and adds unnecessarily to the
memory used by the application.

[j***@fe.up.pt]

Post by Alan W. Irwin

Post by Rafael Laboissiere

Post by Alan W. Irwin
The most important point, however, is you do see segfaults on your

machine

Post by Rafael Laboissiere

Post by Alan W. Irwin
so you have confirmed there is a severe problem, and you therefore have
something that you can debug for your situation.
Good luck in figuring this out!

I think I found the source of the problem, see my last cvs commit. Could
you guys confirm that HEAD works for you now?

Yes with a qualification.
No segfaults for ./x01c with the "Joao" configuration.

Try removing the tkwin driver from the install directory and uncommenting the
lt_dlclose (dlhand) in plcore.c that Rafael last cvs commit has commented.
x01c now works for me. tkwin is a bad-boy!

Joao

PS-there is no need to remove the tkwin.* files, it's enough to

Post by Alan W. Irwin
mv tkwin.la tkwin.la-

To get the tkwin (not) working again is enough to

Post by Alan W. Irwin
mv tkwin.la- tkwin.la

as Rafael code only tries to load files that have a .la extension.
That's the beauty of the scheme! (that should only be a bitmore robust.)

[Rafael Laboissiere]

Post by j***@fe.up.pt
Try removing the tkwin driver from the install directory and uncommenting the
lt_dlclose (dlhand) in plcore.c that Rafael last cvs commit has commented.
x01c now works for me. tkwin is a bad-boy!

I can replicate this bug here. The tkwin driver has probably some memory
management problems that make surface when lt_dlopenext/lt_dlclose are used.
That's weird, eh?

[Alan W. Irwin]

Post by Maurice LeBrun

Post by Alan W. Irwin
Summary: the new code loops over every device so will trigger the
complete collection of valgrind errors for all devices (just xwin in this
case) while the old code just looked at the user-specified device so gets no
valgrind errors (except when the user specified xwin).

I've not been following the discussion very closely, but does this mean that
the code currently loads all dynamic drivers at startup time? If true, this
seems against the spirit of dynamic drivers, and adds unnecessarily to the
memory used by the application.

Good question.....

Here is my understanding of our current situation. Rafael's code is designed
on startup to momentarily load each driver to collect and store required
information about it (as opposed to storing the required information in
drivers/drivers.db). And Joao did some timing to show this loading of all
drivers was very fast. However, at the moment "momentarily" is until exit
from PLplot and consequently PLplot uses more memory then it should because
of a bug in libltdl that doesn't allow unloading. But once that bug is fixed
it is only a matter of uncommenting one line in plcore.c to get a leaner,
meaner PLplot.

Alan

__________________________
Alan W. Irwin
email: ***@beluga.phys.uvic.ca
phone: [phone]

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the Canadian Centre for Climate Modelling and
Analysis (www.cccma.bc.ec.gc.ca) and the PLplot scientific plotting software
package (plplot.org).

__________________________

Linux-powered Science
__________________________

[Alan W. Irwin]

Post by Rafael Laboissiere

Post by j***@fe.up.pt
Try removing the tkwin driver from the install directory and uncommenting the
lt_dlclose (dlhand) in plcore.c that Rafael last cvs commit has commented.
x01c now works for me. tkwin is a bad-boy!

I can replicate this bug here. The tkwin driver has probably some memory
management problems that make surface when lt_dlopenext/lt_dlclose are used.
That's weird, eh?

I don't completely trust such interpretation because I have seen segfaults
come and go depending on the placement (!) of print statements in the code.
It is very hard to track down the real reason for segfaults until you use a
tool like valgrind.

valgrind does indicate a problem with xwin even when lt_dlclose is commented
out. So I suggest the best test is to uncomment lt_dlclose and exclude both
xwin and tkwin. Under those circumstances, then I would believe the above
interpretation (problem somewhere in either xwin or tkwin or both) if the
valgrind result is clean. Otherwise not.

I don't have time to run this specific test now, but I will do so in the
next few days if nobody else is curious enough to try it.

Alan
__________________________
Alan W. Irwin
email: ***@beluga.phys.uvic.ca
phone: [phone]

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the Canadian Centre for Climate Modelling and
Analysis (www.cccma.bc.ec.gc.ca) and the PLplot scientific plotting software
package (plplot.org).

__________________________

Linux-powered Science
__________________________

[Rafael Laboissiere]

Post by Alan W. Irwin
I don't completely trust such interpretation because I have seen segfaults
come and go depending on the placement (!) of print statements in the code.
It is very hard to track down the real reason for segfaults until you use a
tool like valgrind.
valgrind does indicate a problem with xwin even when lt_dlclose is
commented out. So I suggest the best test is to uncomment lt_dlclose and
exclude both xwin and tkwin. Under those circumstances, then I would
believe the above interpretation (problem somewhere in either xwin or tkwin
or both) if the valgrind result is clean. Otherwise not.

I understand your point and I agree that such bugs are hard to be tracked
down. However, the simple fact that dlopening/dlclosing tkwin.la _with
everything else kept equal_ makes it a quite good candidate for culprit.
Indeed, there are tones of malloc, calloc, ckcalloc, etc calls in the source
files (drivers/tk*.c and bindings/tk/*.c). I cannot tell if they are
suspicious or not, though.

That said, it is possible that the code in xwin.c is equally bad...

Post by Alan W. Irwin
I don't have time to run this specific test now, but I will do so in the
next few days if nobody else is curious enough to try it.

we are looking forward for the results of your valgrind tests.

[Alan W. Irwin]

Post by Rafael Laboissiere
I can replicate this bug here. The tkwin driver has probably some memory
management problems that make surface when lt_dlopenext/lt_dlclose are used.
That's weird, eh?

I don't completely trust such interpretation....

I ran the valgrind test I suggested with lt_dclose uncommented and excluding
certain drivers. This test was based on the 10 February software. The point
of this test is to look for memory management problems in the current set of
drivers with this old code rather than to look at the code Rafael just
checked in which (I believe) avoids making this test of
lt_dlopenext/lt_dlclose for every driver.

I found I got an absolutely clean valgrind result (except for well known
unfreed memory warnings at programme end) if I excluded xwin, tkwin, tk, and
gd. Here was the list of drivers for the clean result:

ls ../../driversd/*.so
../../driversd/cgm.so* ../../driversd/ljiip.so*
../../driversd/pstex.so*
../../driversd/dg300.so* ../../driversd/null.so* ../../driversd/tek.so*
../../driversd/hpgl.so* ../../driversd/pbm.so* ../../driversd/xfig.so*
../../driversd/impress.so* ../../driversd/plmeta.so*
../../driversd/ljii.so* ../../driversd/ps.so*

ldd shows that each of these is dynamically linked just to libplotd, libm,
libc, and libdl. (cgm is also linked to the cd library, but that is a
static library so its a different case than gd, xwin, tkwin, and tk which
all are linked to extra dynamic libraries.) If I added in gd, then I got
valgrind errors, but no segfault. In a previous test with gd, xwin, and tkwin
(and tk?) we got segfaults.

There does seem to be a pattern developing that all drivers linked to extra
dynamic libraries have memory management problems of some kind that are
reported by valgrind. I cannot draw a definitive conclusion from this, but
there is certainly the possibility that a libtool (libltdl) bug could be the
cause of this pattern, and I will certainly check gd, xwin, tkwin, and tk
again once a new version of libtool is released. For now, though, it is
best to avoid lt_dlclose (which I think the latest code from Rafael does).
In this case (commented out lt_dlclose) we do get valgrind errors, but so
far (knock on wood for luck!) no segfaults have showed up.

I am completely sold on autotools as a convenient way to set up the
configuration of our software. However, I am sure looking forward to the
day that autotools stabilizes!

Alan
__________________________
Alan W. Irwin
email: ***@beluga.phys.uvic.ca
phone: [phone]

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the Canadian Centre for Climate Modelling and
Analysis (www.cccma.bc.ec.gc.ca) and the PLplot scientific plotting software
package (plplot.org).

__________________________

Linux-powered Science
__________________________

[Rafael Laboissiere]

[...] For now, though, it is best to avoid lt_dlclose (which I think the
latest code from Rafael does).

Yes, it does.