Surviving smoke testing

Do you have a question? Post it now! No Registration Necessary.  Now with pictures!

Let me start with obvious: smoke testing is a very useful tool.

But HOW ON THE EARTH people manage to live with the results it provides?!

Let's leave aside the topic that the web interface is majorly buggy
(the counts go up and down in a complicated dance, counts in different
"views" do not match etc) - even the puzzling buggy interface provides
heaps of useful information.  But I'd like to know tricks people
employ to fight the major limitation of smoke testings: that "original
reports from testers" have the most crucial information cut out.

  [The problem is that the "try" is broken into 3 steps
   (configuration, build, and test), and only the log of the last of
   executed steps is reported.  It would not be important for
   distributions which do essentially no configuration, and
   essentially no "building".

   But the distributions with non-trivial configuration and/or
   non-trivial build are exactly ones which could benefit most from
   widest possible testing - and to debug problems found during the
   testing stage, one would need to inspect logs of configuration step
   and the build step...]

As an appetizer, here is my trick: it tries to load the module, and if
loading fails, it reports the parameters passed to MakeMaker during
the configuration stage.

The code needs 3 parameters: which module to load, the name of one of
the test files (to find in which directory one is executed), and which
directories contain active Makefile.PL's.  You modify these
parameters, and put the code in one of the test files.

One example of the output in

 (Unfortunately, this particular report does not help - I still do not
  know why PIC is not enabled...  Looks like assembler flags should
  include -KPIC...)


#!/usr/bin/perl -w
use strict;

$| = 1;
print "1..1\nok 1\n";    # report success

exit if eval 'use Math::Pari; 1';

# If failed, report build parameters

sub report_build_parameters ($) {
  my ($makefile, $in) = (shift, '');
  warn "# reporting $makefile header:\n# ==========================\n";
  open M, "< $makefile" or die "Can't open $makefile";
  $in = <M> while defined $in and $in !~ /MakeMaker \s+ Parameters/xi;
  $in = <M>;        # Reload till first non-empty line after "Param" one:
  $in = <M> while defined $in and $in !~ /\S/;
  warn $in and $in = <M> while defined $in and $in =~ /^#/;
  close M;
  warn "# ==========================\n";

my ($base_d, $in) = (-f "t/000_load-problem.t" ? '.' : '..', '');

Site Timeline