List of files
-------------

- titan_builder.sh: Wrapper script for titan_builder.py.  It sets some basic
                    environment variables for platforms with tools installed
                    to non-standard locations.
- titan_builder.py: Build process control functions.  It shouldn't be run
                    directly, use the wrapper script instead.
- titan_builder_cfg.py: The configuration file.  It contains the build
                        configurations, build slaves etc.
- product_handler.py: Helper functions to build the VOB products.
- titan_publisher.py: Different publishing related functions.
- utils.py: General purpose functions.

Brief overview
--------------

When titan_builder.sh is started, it checks out a build configuration specific
version of TITAN from CVS.  TITAN gets configured according to the build
configuration (e.g. variables in Makefile.cfg).  When it's ready, TITAN is
distributed to each slave attached to the given build configuration.  The
builds are performed sequentially.  All script files are distributed as well
and the script starts executing in slave mode on the slave machine.  The master
and the slave can be the same machines.  In this case, using different build
directories for them is strongly recommended.  When all passes are finished on
the slave machine the master starts the build on the next slave machine etc.,
then the remaining build configurations are built.

After all that, the master collects all logs from the slave machines.  The
slaves can produce three kinds of outputs: CSV, TXT, HTML.  The CSV file is
essential, since the master will consider each slave lost without such a file.
The CSV file is used to generate the e-mail notification message as well.
Finally, the master publishes the logs and cleans things up.  The compiled
sources will remain available on each slave until the next build.

To run VOB product tests the VOB or a similar directory structure of products
should be available on master machine.  SSHFS can be used for this purpose.
The performance test package is not part of the official TITAN test suite,
hence it needs to be distributed for the slaves as well.  Currently it's just
an archive reachable on the master machine.  It's recommended to use
configspec 129 and the TCC_Common VOB.

All shell commands are executed using the `run_cmd()' function.  It has a
timeout parameter to kill the command.

To make sure, that always the latest version of the scripts runs, one can use
a simple script, which checks out the sources from CVS first:

#!/bin/bash

rm -rf *.{py,sh} TTCNv3/etc/autotest
cvs co TTCNv3/etc/autotest && ln -sf TTCNv3/etc/autotest/*.{py,sh} .
./titan_builder.sh -c tests_on_x86_64_linux,tests_on_solaris,tests_on_i386_solaris,limited_tests_on_ubuntu910 1>/dev/null

Prerequisites
-------------

- Python >= 2.3
- Configured password-less SSH connections between the master and the slave
- For scheduled execution use `cron'

Command line options
--------------------

-c [c1,c2,...]: Select the build configurations to run.  If no build
                configuration is selected, then all configurations will be
                run.  It's a comma separated list.
-d: Print all the available build configurations present in the configuration
    file.  Slaves attached to a given build configuration are listed as well.
-h: Display available options.
-s: Start the script in slave mode.  This option should only be used by
    titan_builder.py, unless everything is in place.

Configuration
-------------

It's a Python source file, take care of the syntax.

- Common options

    buiddir:STR The build directory of the master.
    logdir:STR  The logs of the master go here.
    htmldir:STR All HTML files will be published here.
    vob:STR     The VOB products will be copied from here.
    archive:INT Archive the logs after a specified number of days.
    cleanup:INT Move the archived logs somewhere else after a specified number of days.
    measureperiod:INT Reset scores after a given number of days.
    cleanupslave:{slave:STR,dir:STR} The destination of abandoned logs.

- Recipients

    {name:STR,address:STR}

    The names and e-mail addresses of the recipients.

- Slaves

    slaves[name:STR]:{ip:STR,user:STR,configs:LIST}

    The IP address, the user's name, the build configurations attached to the
    slave.  Currently one build configuration is supported for each slave.

- Product descriptions

    products[kind:STR]:{name:STR,action0:BOOL,action1:BOOL,...}

    `kind' is always a directory under the VOB's root directory (e.g.
    TestPorts, ProtocolModules).  `name' is the product name.  Actions can be
    enabled individually.  The predefined actions are the following:

    - semantic: `compiler -s'
    - translate: `compiler'
    - compile: `make'
    - run: `make run'

- Build configurations

    version:STR        Version of TITAN to use.  It can be a CVS tag or date.
                       If it's not set the HEAD will be taken.
    license:STR        Location of the license file.
    gui:BOOL           The `GUI' part in Makefile.cfg.
    jni:BOOL           The `JNI' part in Makefile.cfg.
    debug:BOOL         The `DEBUG' part in Makefile.cfg.
    compilerflags:STR  The `COMPILERFLAGS' in Makefile.cfg.
    ldflags:STR        The `LDFLAGS' in Makefile.cfg.
    gccdir:STR         This will affect `CC' and `CXX'.
    *cc:STR            Value of `CC' in synch with the previous option.
    *cxx:STR           Value of `CXX' in synch with the previous option.
    flex:STR           Replace `FLEX'.
    perl:STR           Location of the `perl' interpreter.
    bison:STR          Replace `BISON'.
    jdkdir:STR         Replace `JDKDIR'.
    qtdir:STR          Replace `QTDIR'.
    xmldir:STR         Replace `XMLDIR'.
    openssldir:STR     Replace `OPENSSL_DIR'.
    regtest:BOOL       Run regression tests.
    perftest:BOOL      Run performance tests.  The location of the testsuite
                       must be known, since it's not part of CVS.  It should
                       be available for the master locally.
    perftestdir:STR    Location of the performance tests.
    *cpsmin:INT        Minimum CPS value for performance tests.
    *cpsmax:INT        Maximum CPS value for performance tests.
    functest:BOOL      Run function tests.
    vobtest:BOOL       Run product tests.
    *vobtest_logs:BOOL Save logs for product tests.
    rt2:BOOL           Run tests with both run-times.
    builddir:STR       Everything will be done here.  It should be different
                       from the master's.
    logdir:STR         Place of the logs.
    *pdfdir:STR        Local directory to copy .pdf files from.  If not present
                       no *.pdf files will be there.  If it's an empty string
                       the *.pdf files will be faked with empty files.
    *xsdtests:BOOL     Disable regression tests for `xsd2ttcn'.  It's very time
                       consuming.
    measure:BOOL       Enable `quality' measurements.

(`*' is for optional fields.)

Platform specific settings
--------------------------

If some essential tools (e.g. GCC) are installed to non-standard locations on a
given platform, all of these environmental settings can be placed into a
configuration file `${HOME}/.titan_builder'.  This file is always sourced
first, before starting the actual build.  An example file:

#!/bin/bash

# Basic configuration for `bangjohansen' (172.31.21.76).

PATH=${HOME}/apps/bin:/mnt/TTCN/Tools/gcc-3.4.6-sol10/bin:/mnt/TTCN/Tools/binutils-2.17-sol10/bin:/usr/local/bin:/usr/bin:${PATH}
LD_LIBRARY_PATH=${HOME}/apps/lib:/mnt/TTCN/Tools/gcc-3.4.6-sol10/lib:/mnt/TTCN/Tools/binutils-2.17-sol10/lib:/usr/local/lib:/usr/lib:${LD_LIBRARY_PATH}

TTCN3_LICENSE_FILE=${HOME}/.TTCN3/license_2536.dat
CVSROOT=esekits1064.rnd.ki.sw.ericsson.se:/proj/TTCN/cvs_root
CVS_SERVER=/usr/local/bin/cvs
CVS_RSH=ssh
EDITOR=mcedit

export PATH LD_LIBRARY_PATH CVSROOT CVS_SERVER EDITOR TTCN3_LICENSE_FILE CVS_RSH

When something goes wrong
-------------------------

E.g. no e-mail message is received, garbage or missing sections in the e-mail
or on the generated HTML page...

- Possible reasons and possible solutions

  - Problems with e-mail system: In case of Postfix check the `mailq'
    command.  All messages still in the queue should be printed.  An example
    output of the `mailq' command:

    -Queue ID- --Size-- ----Arrival Time---- -Sender/Recipient-------
    5A57D160B02     1696 Fri Oct 16 09:32:05  ferenc.kovacs@ericsson.com
    (host mwux020.eth.ericsson.se[159.107.148.18] said: 452 4.4.5
    Insufficient disk space; try again later (in reply to MAIL FROM command))
    ferenc.kovacs@ericsson.com

    To clear all the messages: `postsuper -d ALL'.

  - Exception in the Python code: Check the titan_builder.err-log file of the
    master and the slaves.  All exceptions should go into this file.  The
    other log file titan_builder.log can be useful as well.

  - Disk quota exceeded.

  - The SSHFS mounted partitions (e.g. used for HTML publishing) get umounted.
    The common cause is a segfault:

    Dec 12 07:00:50 tcclab1 kernel: sshfs[20289]: segfault at 6507a ip
    7fbf99315064 sp 44288ea8 error 4 in libc-2.5.so[7fbf992a2000+139000]

    Check /var/log/messages for details.

    If SSHFS locks up and the mounted partitions cannot be umounted like:

    titanrt@tcclab1:~> fusermount -u ~/public_html
    fusermount: failed to unmount /home/titanrt/public_html: Device or resource busy

    Simply `killall -9 sshfs' and try to remount the partitions.  If SSHFS
    locks up you may not be able to execute any command in the parent
    directory of these mount points.  The commands will simply hang.  If it
    doesn't work, try the following:

    titanrt@tcclab1:~> lsof | grep public_html
    lsof: WARNING: can't stat() fuse file system /home/titanrt/public_html
    Output information may be incomplete.
    bash 2378  titanrt cwd unknown /home/titanrt/public_html/titan_builds (stat: Transport endpoint is not connected)
    ssh  13480 titanrt cwd unknown /home/titanrt/public_html/titan_builds (stat: Transport endpoint is not connected)
    titanrt@tcclab1:~> kill -9 2378

    Unfortunately SSHFS is very unstable and hangs regularly, at least the 1.7
    version used on the master machine, which was released in 2006.  Another
    thing that may help:

    titanrt@tcclab1:~> killall -9 sshfs 
    tcclab1:/home/titanrt/titan_nightly_builds # umount -l vobs/ttcn
    tcclab1:/home/titanrt/titan_nightly_builds # umount -l vobs

  - It's possible that some kind of network problem prevents the nightly tests to
    check out the latest script files. If there're broken links in the test
    directory and the test results are missing this might be the case. To make
    the script run next morning the freshbuild.sh needs to be run first, since
    freshbuild.sh is run by cron.

Licenses
--------

The script can be run by anyone.  Currently, `titanrt' is used for this
purpose, however, the actual user can really be anyone, since it is set from
the environment.  It is a general user available on most of our supported
platforms, it can be created manually in need.  To eliminate user dependence,
all non-absolute paths in the configuration file are prefixed automatically
with the current user's home directory.

To run the tests on all of the supported platforms we're using `hostid' based
license files.  The license file numbers at the moment:

  tcclab1 (172.31.21.7, 0x67666473): 4812
  tcclab2 (172.31.21.49, 0x67666473): 4812
  rhea (159.107.193.33, 0x83dbd963): 5628
  bangjohansen (172.31.21.76, 0x380f076e): 2551
  
Statistics generation
---------------------

Please note, that the statistics automatically generated by the test system
are useless, unless they're monitored continuously.  Invalid failures will not
be taken into account by the test system.  The statistics are extracted from
the `report.txt' files.  Only the `Reg. tests', `Func. tests', `Perf. tests'
are important.  Weekends are never counted, but holidays need to be handled by
hand.

Rules:

All of them pass  -> 2 points
Any of them fails -> 0 points
Any of them lost  -> 1 points

Sample output:

First period: 2011-03-01 03:01:00
Overall score: 90/100 (90%) Commitment 
This period: 2011-06-01 03:01:00
Period score: 9/10 (90%) Commitment
