Roundup (0.3.0)

An Issue-Tracking System for Knowledge Workers

Contents


Installation

Prerequisites

Either:
Python 2.0 with pydoc installed. See http://www.lfw.org/ for pydoc.
or
Python 2.1 or later
Download the latest version from http://www.python.org/.

Getting Roundup

Download the latest version from http://roundup.sf.net/.

Installing Roundup

  1. Run:
    python setup.py install
  2. If you would prefer the scripts installed in somewhere other than /usr/local/bin, add "--install-scripts=<dir>" to the command:
    python setup.py install --install-scripts=<dir>
  3. The command:
    python setup.py install --help
    gives all the options available for installation.


Getting Started

The following instructions assume that you have installed roundup. If you haven't, you may still proceed - just preface all commands with "./" ie. "./roundup-admin init".

The Instance

We'll be referring to the term instance a lot from now on. An instance is a directory in your filesystem that is where all the information about a live issue tracker database is stored. The data that is entered as issues, the users who access the database and the definition of the database itself all reside there:
  1. Hyperdatabase
    This is the lowest component of Roundup and is where all the issues, users, file attachments and messages are stored.
  2. Database schema
    This describes the content of the hyperdatabase - what fields are stored for issues, what user information, etc. Being stored in the instance, this allows it to be customised for a particular application. It also means that changes in the Roundup core code do not affect a running instance.
  3. Web Interface
    The web interface templates are defined in the instance too - and the actual CGI interface class is defined (mostly using base classes in the Roundup core code) so it, like the database, may be customised for each instance in use.
Instances are created using the roundup-admin tool.

Command Line Tool

To initiliase a new instance, run roundup-admin init. You will be asked a series of questions:
  1. Instance home directory
  2. Schema to use
  3. Database back-end to use
  4. Administration user "admin" password.
Roundup is configurable using an instance_config.py file in the instance home. It should be edited before roundup is used, and may have the following variable declarations:
  1. MAILHOST
    The SMTP mail host that roundup will use to send mail
  2. MAIL_DOMAIN
    The domain name used for email addresses
  3. ISSUE_TRACKER_WEB
    The web address of the issue tracker's web interface

The email addresses used by the system by default are:

  1. ISSUE_TRACKER_EMAIL - issue_tracker@MAIL_DOMAIN
    submissions of issues
  2. ADMIN_EMAIL - roundup-admin@MAIL_DOMAIN
    roundup's internal use (problems, etc)
Note: We run the instance as group "issue_tracker" and add the mail and web user ("mail" and "apache" on our RedHat 7.1 system) to that group, as well as any admin people.

E-Mail Interface

Set up a mail alias called "issue_tracker" as:
|/usr/bin/python /usr/local/bin/roundup-mailgw <instance_home>
In some installations (e.g. RedHat 6.2 I think) you'll need to set up smrsh so sendmail will accept the pipe command. In that case, symlink /etc/smrsh/roundup-mailgw to /usr/local/bin/roundup-mailgw and change the command to:
|roundup-mailgw <instance_home>
To test the mail gateway on unix systems, try:
echo test |mail -s '[issue] test' issue_tracker@your.domain

Web Interface

This software will work through apache or stand-alone.

Stand-alone:

  1. Edit roundup-server at the top - ROUNDUP_INSTANCE_HOMES needs to know about your instance. *** command-line option
  2. roundup-server [hostname port] (hostname may be "")
  3. Load up the page /<instance name>/index where instance name is the name you nominated in ROUNDUP_INSTANCE_HOMES. *** command-line option
Apache:
  1. Make sure roundup.cgi is executable. Edit it at the top - ROUNDUP_INSTANCE_HOMES needs to know about your instance.
  2. Edit your /etc/httpd/conf/httpd.conf and make sure that the /home/httpd/html/roundup/roundup.cgi script will be treated as a CGI script.
  3. Add the following to your /etc/httpd/conf/httpd.conf:
    ------8<------- snip here ------8<-------
    RewriteEngine on
    RewriteCond %{HTTP:Authorization} ^(.*)
    RewriteRule ^/roundup/roundup.cgi(.*) /home/httpd/html/roundup/roundup.cgi$1 [e=HTTP_CGI_AUTHORIZATION:%1,t=application/x-httpd-cgi,l]
    ------8<------- snip here ------8<-------
    
    note: the RewriteRule must be on one line - no breaks
  4. Re-start your apache to re-load the config
  5. Load up the page /roundup/roundup.cgi/<instance name>/index where instance name is the name you nominated in ROUNDUP_INSTANCE_HOMES.

Users

Users and permissions

By default, roundup automatically creates one user when the instance database is initialised (using roundup-admin init). The user is "admin" and the password is the one you supply at that time.

If users attempt to use roundup in any manner and are not identified to roundup, they will be using the database in a read-only mode. That is, if roundup doesn't know who they are, they can't change anything. This has the following repurcussions:

Command-line interface
The data modification commands (create, init, retire, set) are not available without a login, and if one is not supplied on the command line (-u user:pass) then it will be prompted for.
E-Mail interface
Users are identified by e-mail address - a new user entry will be created for any e-mail address that is not recognised, so users are always identified by roundup.
Web interface
Unidentified users have read-only access. If the users database has an entry with the username "anonymous", then unidentified users are automatically logged in as that user. This gives them write access.

There has been only a half-hearted attempt to restrict certain activities to the "admin" user. For example, the "extended" schema web interface enables some fnuctionality for the "admin" user. On the fil-side, it is possible to obtain the admin user's password using the read-only access on the command line (it would also be possible to access the database files directly to obtain this information).

Adding users

To add users, use one of the following interfaces:
  1. On the web, access the URL .../<instance name>/newuser to bring up a form which may be used to add a new user.
  2. On the command-line, use:
    roundup-admin -i <instance home> create user username=bozo password=bozo address=richard@clown.org
    Supply the admin username and password. roundup-admin will print the id of the new user.
  3. Any e-mail sent to roundup from an address that doesn't match an existing user in the database will result in a new user entry being created for that user.

Issues

To add issues, use one of the following interfaces:
  1. On the web, access the URL .../<instance name>/newissue to bring up a form which may be used to add a new issue.
  2. On the command-line, use:
    roundup-admin -i <instance home> create issue title="test issue"
    Supply the admin username and password. roundup-admin will print the id of the new issue.
  3. Any e-mail sent to roundup with the subject line containing [issue] will automatically created a new issue in the database using the contents of the e-mail.


User Guide

Command Line Tool

Usage: roundup-admin [-i instance home] [-u login] [-c] <command> <arguments>

Options:
-i instance home specify the issue tracker "home directory" to administer
-u the user[:password] to use for commands
-c when outputting lists of data, just comma-separate them

Command Help
history history designator

Lists the journal entries for the node identified by the designator.

find find classname propname=value ...

Find the nodes of the given class with a given property value. The value may be either the nodeid of the linked node, or its key value.

list list classname [property]

Lists all instances of the given class along. If the property is not specified, the "label" property is used. The label property is tried in order: the key, "name", "title" and then the first property, alphabetically.

retire retire designator[,designator]*

This action indicates that a particular node is not to be retrieved by the list or find commands, and its key value may be re-used.

create create classname property=value ...

This creates a new entry of the given class using the property name=value arguments provided on the command line after the "create" command.

get get property designator[,designator]*

Retrieves the property value of the nodes specified by the designators.

spec spec classname

This lists the properties for a given class.

set set designator[,designator]* propname=value ...

Sets the property to the value for all designators given.

init init [template [backend [admin password]]]

The command will prompt for the instance home directory (if not supplied through INSTANCE_HOME or the -i option. The template, backend and admin password may be specified on the command-line as arguments, in that order.

freshen freshen

**DO NOT USE**

This currently kills databases!!!!

This action should generally not be used. It reads in an instance database and writes it again. In the future, is may also update instance code to account for changes in templates. It's probably wise not to use it anyway. Until we're sure it won't break things...

help help [command]

Short help about roundup-admin or the specific command.

morehelp morehelp

All available help from the roundup-admin tool.

All commands (except help) require an instance specifier. This is just the path to the roundup instance you're working with. A roundup instance is where roundup keeps the database and configuration file that defines an issue tracker. It may be thought of as the issue tracker's "home directory". It may be specified in the environment variable ROUNDUP_INSTANCE or on the command line as "-i instance".

A designator is a classname and a nodeid concatenated, eg. bug1, user10, ...

Property values are represented as strings in command arguments and in the printed results:

When multiple nodes are specified to the roundup get or roundup set commands, the specified properties are retrieved or set on all the listed nodes.

When multiple results are returned by the roundup get or roundup find commands, they are printed one per line (default) or joined by commas (with the -c) option.

Where the command changes data, a login name/password is required. The login may be specified as either "name" or "name:password".

If either the name or password is not supplied, they are obtained from the command-line.

Web Interface

Index views may be modified by the following arguments:
    :sort    - sort by prop name, optionally preceeded with '-'
             to give descending or nothing for ascending sorting.
    :group   - group by prop name, optionally preceeded with '-' or
             to sort in descending or nothing for ascending order.
    :filter  - selects which props should be displayed in the filter
             section. Default is all.
    :columns - selects the columns that should be displayed.
             Default is all.
    propname - selects the values the node properties given by propname
             must have (very basic search/filter).
Not sure what to put in here...

E-Mail Gateway

Incoming messages are examined for multiple parts:

Message content summary

The "summary" property on message nodes is taken from the first non-quoting section in the message body. The message body is divided into sections by blank lines. Sections where the second and all subsequent lines begin with a ">" or "|" character are considered "quoting sections". The first line of the first non-quoting section becomes the summary of the message.

Address handling

All of the addresses in the To: and Cc: headers of the incoming message are looked up among the user nodes, and the corresponding users are placed in the "recipients" property on the new "msg" node. The address in the From: header similarly determines the "author" property of the new "msg" node. The default handling for addresses that don't have corresponding users is to create new users with no passwords and a username equal to the address. (The web interface does not permit logins for users with no passwords.) If we prefer to reject mail from outside sources, we can simply register an auditor on the "user" class that prevents the creation of user nodes with no passwords.

Performing Actions

The subject line of the incoming message is examined to determine whether the message is an attempt to create a new item or to discuss an existing item. A designator enclosed in square brackets is sought as the first thing on the subject line (after skipping any "Fwd:" or "Re:" prefixes). If an item designator (class name and id number) is found there, the newly created "msg" node is added to the "messages" property for that item, and any new "file" nodes are added to the "files" property for the item. If just an item class name is found there, we attempt to create a new item of that class with its "messages" property initialized to contain the new "msg" node and its "files" property initialized to contain any new "file" nodes.

Triggers

Both cases may trigger detectors (in the first case we are calling the set() method to add the message to the item's spool; in the second case we are calling the create() method to create a new node). If an auditor raises an exception, the original message is bounced back to the sender with the explanatory message given in the exception.


Customising Roundup

Instances have the following structure:
instance_config.py Holds the basic instance configuration
dbinit.py Holds the instance schema
interfaces.py Defines the Web and E-Mail interfaces for the instance
select_db.py Selects the database back-end for the instance
db/ Holds the instance's database
db/files/ Holds the instance's upload files and messages
detectors/ Auditors and reactors for this instance
html/ Web interface templates, images and style sheets

Instance Configuration

The instance_config.py located in your instance home contains the basic configuration for the web and e-mail components of roundup's interfaces. This file is a Python module. The default instance_config.py is given below - as you can see, the MAIL_DOMAIN must be edited before any interaction with the instance is attempted.

MAIL_DOMAIN=MAILHOST=HTTP_HOST=None
HTTP_PORT=0

# roundup home is this package's directory
INSTANCE_HOME=os.path.split(__file__)[0]

# The SMTP mail host that roundup will use to send mail
if not MAILHOST:
    MAILHOST = 'localhost'

# The domain name used for email addresses.
if not MAIL_DOMAIN:
    MAIL_DOMAIN = 'fill.me.in.'

# the next two are only used for the standalone HTTP server.
if not HTTP_HOST:
    HTTP_HOST = ''
if not HTTP_PORT:
    HTTP_PORT = 9080

# This is the directory that the database is going to be stored in
DATABASE = os.path.join(INSTANCE_HOME, 'db')

# This is the directory that the HTML templates reside in
TEMPLATES = os.path.join(INSTANCE_HOME, 'html')

# The email address that mail to roundup should go to
ISSUE_TRACKER_EMAIL = 'issue_tracker@%s'%MAIL_DOMAIN

# The web address that the instance is viewable at
ISSUE_TRACKER_WEB = 'http://some.useful.url/'

# The email address that roundup will complain to if it runs into trouble
ADMIN_EMAIL = 'roundup-admin@%s'%MAIL_DOMAIN

# Somewhere for roundup to log stuff internally sent to stdout or stderr
LOG = os.path.join(INSTANCE_HOME, 'roundup.log')

Instance Schema

An instance schema defines what data is stored in the instance's database. The two schemas shipped with Roundup turn it into a typical software bug tracker (the extended schema allowing for support issues as well as bugs). Schemas are defined using Python code. The "classic" schema looks like this:

    pri = Class(db, "priority", name=String(), order=String())
    pri.setkey("name")
    pri.create(name="critical", order="1")
    pri.create(name="urgent", order="2")
    pri.create(name="bug", order="3")
    pri.create(name="feature", order="4")
    pri.create(name="wish", order="5")

    stat = Class(db, "status", name=String(), order=String())
    stat.setkey("name")
    stat.create(name="unread", order="1")
    stat.create(name="deferred", order="2")
    stat.create(name="chatting", order="3")
    stat.create(name="need-eg", order="4")
    stat.create(name="in-progress", order="5")
    stat.create(name="testing", order="6")
    stat.create(name="done-cbb", order="7")
    stat.create(name="resolved", order="8")

    keyword = Class(db, "keyword", name=String())
    keyword.setkey("name")

    user = Class(db, "user", username=String(), password=String(),
        address=String(), realname=String(), phone=String(), organisation=String())
    user.setkey("username")
    user.create(username="admin", password=adminpw, address=instance_config.ADMIN_EMAIL)

    msg = FileClass(db, "msg", author=Link("user"), recipients=Multilink("user"), 
        date=Date(), summary=String(), files=Multilink("file"))

    file = FileClass(db, "file", name=String(), type=String())

    issue = IssueClass(db, "issue", assignedto=Link("user"),
        topic=Multilink("keyword"), priority=Link("priority"), status=Link("status"))
    issue.setkey('title')

Class, FileClass, IssueClass - creating a new information store

A Class defines a particular class (or type) of data that will be stored in the database. In the instance above, we've defined 7 classes of information:
priority
Defines the possible levels of urgency for issues.
status
Defines the possible states of processing the issue may be in.
keyword
Initially empty, will hold keywords useful for searching issues.
user
Initially holding the "admin" user, will eventually have an entry for all users using roundup.
msg
Initially empty, will all e-mail messages sent to or generated by roundup.
file
Initially empty, will all files attached to issues.
issue
Initially emtyp, this is where the issue information is stored.

We define the "priority" and "status" classes to allow two things: reduction in the amount of information stored on the issue and more powerful, accurate searching of issues by priority and status. By only requiring a link on the issue (which is stored as a single number) we reduce the chance that someone mis-types a priority or status - or simply makes a new one up.

Class
Class is the basic store of information.

FileClass
FileClasses save their "content" attribute off in a separate file from the rest of the database. This reduces the number of large entries in the database, which generally makes databases more efficient, and also allows us to use command-line tools to operate on the files. They are stored in the files sub-directory of the db directory in your instance.

IssueClass
IssueClasses automatically include the "messages", "files", "nosy", and "superseder" properties.

The messages and files properties list the links to the messages and files related to the issue. The nosy property is a list of links to users who wish to be informed of changes to the issue - they get "CC'ed" e-mails when messages are sent to or generated by the issue. The nosy reactor (in the detectors directory) handles this action. The superceder link indicates an issue which has superceded this one.

They also have the dynamically generated "creation", "activity" and "creator" properties.

The value of the "creation" property is the date when an item was created, and the value of the "activity" property is the date when any property on the item was last edited (equivalently, these are the dates on the first and last records in the item's journal). The "creator" property holds a link to the user that created the issue.

setkey(property)

Select a String property of the class to be the key property. The key property muse be unique, and allows references to the items in the class by the content of the key property. That is, we can refer to users by their username, e.g. let's say that there's an issue in roundup, issue 23. There's also a user, richard who happens to be user 2. To assign an issue to him, we could do either of:

roundup-admin set issue assignedto=2

or

roundup-admin set issue assignedto=richard

Note, the same thing can be done in the web and e-mail interfaces.

create(information)

Create an item in the database. This is generally used to create items in the "definitional" classes like "priority" and "status".

Web Interface

The web interface works behind the cgi-bin/roundup.cgi or roundup-server scripts. In both cases, the scripts determine which instance is being accessed (the first part of the URL path inside the scope of the CGI handler) and pass control on to the instance interfaces.Client class which handles the rest of the access through its main() method. This means that you can do pretty much anything you want as a web interface to your instance.

Most customisation of the web view can be done by modifying the templates in the instance html directory. These are divided into index, item and newitem views. The newitem view is optional - the item view will be used if the newitem view doesn't exist.

Displaying Properties

Properties appear in the user interface in three contexts: in indices, in editors, and as filters. For each type of property, there are several display possibilities. For example, in an index view, a string property may just be printed as a plain string, but in an editor view, that property should be displayed in an editable field.

The display of a property is handled by functions in the htmltemplate module.

Displayer functions are triggered by <display> tags in templates. The call attribute of the tag provides a Python expression for calling the displayer function. The three standard arguments are inserted in front of the arguments given. For example, the occurrence of

    <display call="plain('status')">
in a template triggers a call the "plain" function. The displayer functions can accept extra arguments to further specify details about the widgets that should be generated. By defining new displayer functions, the user interface can be highly customized.

The displayer functions are
plain Display a String property directly.

Display a Date property in a specified time zone with an option to omit the time from the date stamp.

For a Link or Multilink property, display the key strings of the linked nodes (or the ids if the linked class has no key property).

Options:
escape (boolean) - HTML-escape the resulting text.

field Display a property like the plain displayer above, but in a form field to be edited. Strings, Dates and Intervals use TEXT fields, Links use SELECT fields and Multilinks use SELECT MULTIPLE fields.

Options:
size (number) - width of TEXT fields.
height (number) - number of nows in SELECT MULTIPLE tags.
showid (boolean) - true includes the id of linked items in the SELECT MULTIPLE fields.

menu For a Links and Multilinks, display the same field as would be generated using field.
link For a Link or Multilink property, display the names of the linked nodes, hyperlinked to the item views on those nodes. For other properties, link to this node with the property as the text.

Options:
property (property name) - the property to use in the second case.

count For a Multilink property, display a count of the number of links in the list.

Arguments:
property (property name) - the property to use.

reldate Display a Date property in terms of an interval relative to the current date (e.g. "+ 3w", "- 2d").

Arguments:
property (property name) - the property to use.

Options:
pretty (boolean) - display the relative date in an English form.

download Show a Link("file") or Multilink("file") property using links that allow you to download files.

Arguments:
property (property name) - the property to use.

checklist For a Link or Multilink property, display checkboxes for the available choices to permit filtering.

Arguments:
property (property name) - the property to use.

note Display the special notes field, which is a text area for entering a note to go along with a change.
list List the items specified by property using the standard index for the class.

Arguments:
property (property name) - the property to use.

history List the history of the item.
submit Add a submit button for the item.

Index Views

An index view contains two sections: a filter section and an index section. The filter section provides some widgets for selecting which items appear in the index. The index section is a table of items.

Index View Specifiers

An index view specifier (URL fragment) looks like this (whitespace has been added for clarity):

/issue?status=unread,in-progress,resolved&
        topic=security,ui&
        :group=+priority&
        :sort=-activity&
        :filters=status,topic&
        :columns=title,status,fixer

The index view is determined by two parts of the specifier: the layout part and the filter part. The layout part consists of the query parameters that begin with colons, and it determines the way that the properties of selected nodes are displayed. The filter part consists of all the other query parameters, and it determines the criteria by which nodes are selected for display.

The filter part is interactively manipulated with the form widgets displayed in the filter section. The layout part is interactively manipulated by clicking on the column headings in the table.

The filter part selects the union of the sets of items with values matching any specified Link properties and the intersection of the sets of items with values matching any specified Multilink properties.

The example specifies an index of "issue" nodes. Only items with a "status" of either "unread" or "in-progres" or "resolved" are displayed, and only items with "topic" values including both "security" and "ui" are displayed. The items are grouped by priority, arranged in ascending order; and within groups, sorted by activity, arranged in descending order. The filter section shows filters for the "status" and "topic" properties, and the table includes columns for the "title", "status", and "fixer" properties.

Associated with each item class is a default layout specifier. The layout specifier in the above example is the default layout to be provided with the default bug-tracker schema described above in section 4.4.

Filter Section

The template for a filter section provides the filtering widgets at the top of the index view. Fragments enclosed in <property>...</property> tags are included or omitted depending on whether the view specifier requests a filter for a particular property.

Here's a simple example of a filter template.

<property name=status>
    <display call="checklist('status')">
</property>
<br>
<property name=priority>
    <display call="checklist('priority')">
</property>
<br>
<property name=fixer>
    <display call="menu('fixer')">
</property>

The standard index generation code appends a section to the index pages which allows selection of the filters - from those which are defined in the filter template.

Index Section

The template for an index section describes one row of the index table. Fragments enclosed in <property>...</property> tags are included or omitted depending on whether the view specifier requests a column for a particular property. The table cells should contain <display> tags to display the values of the item's properties.

Here's a simple example of an index template.

<tr>
    <property name=title>
        <td><display call="plain('title', max=50)"></td>
    </property>
    <property name=status>
        <td><display call="plain('status')"></td>
    </property>
    <property name=fixer>
        <td><display call="plain('fixer')"></td>
    </property>
</tr>

Sorting

String and Date values are sorted in the natural way. Link properties are sorted according to the value of the "order" property on the linked nodes if it is present; or otherwise on the key string of the linked nodes; or finally on the node ids. Multilink properties are sorted according to how many links are present.

Item Views

An item view contains an editor section and a spool section. At the top of an item view, links to superseding and superseded items are always displayed.

Editor Section

The editor section is generated from a template containing <display> tags to insert the appropriate widgets for editing properties.

Here's an example of a basic editor template.

<table>
<tr>
    <td colspan=2>
        <display call="field('title', size=60)">
    </td>
</tr>
<tr>
    <td>
        <display call="field('fixer', size=30)">
    </td>
    <td>
        <display call="menu('status')>
    </td>
</tr>
<tr>
    <td>
        <display call="field('nosy', size=30)">
    </td>
    <td>
        <display call="menu('priority')>
    </td>
</tr>
<tr>
    <td colspan=2>
        <display call="note()">
    </td>
</tr>
</table>

As shown in the example, the editor template can also request the display of a "note" field, which is a text area for entering a note to go along with a change.

When a change is submitted, the system automatically generates a message describing the changed properties.

If a note is given in the "note" field, the note is appended to the description. The message is then added to the item's message spool (thus triggering the standard detector to react by sending out this message to the nosy list).

The message also displays all of the property values on the item and indicates which ones have changed. An example of such a message might be this:

Polly's taken a turn for the worse - this is now really important!
-----
title: Polly Parrot is dead
priority: critical
status: unread -> in-progress
fixer: terry
keywords: parrot,plumage,perch,nailed,dead

Spool Section

The spool section lists messages in the item's "messages" property. The index of messages displays the "date", "author", and "summary" properties on the message nodes, and selecting a message takes you to its content.

The <property> tag used in the index may also be used here - it checks to see if the nominated Multilink property has any entries. This can be used to eliminate sections of the spool section if the property has no entries.

<property name="files">
 <tr class="strong-header">
  <td><b>Files</b></td>
 </tr>

 <tr>            
  <td><display call="list('files')"></td>
 </tr>
</property>


Acknowledgements

Go Ping, you rock! Also, go Bizar Software for letting me implement this system on their time.

 


$Id: index.html,v 1.10 2001-10-08 21:49:30 richard Exp $