From: richard Date: Wed, 25 Jul 2001 01:23:07 +0000 (+0000) Subject: Added the Roundup spec to the new documentation directory. X-Git-Url: https://git.tokkee.org/?a=commitdiff_plain;h=01f46c221220f315a25ff77009e909528139dd8d;p=roundup.git Added the Roundup spec to the new documentation directory. git-svn-id: http://svn.roundup-tracker.org/svnroot/roundup/trunk@76 57a73879-2fb5-44c3-a270-3262357dd7e2 --- diff --git a/doc/images/logo-acl-medium.gif b/doc/images/logo-acl-medium.gif new file mode 100644 index 0000000..37c519a Binary files /dev/null and b/doc/images/logo-acl-medium.gif differ diff --git a/doc/images/logo-codesourcery-medium.gif b/doc/images/logo-codesourcery-medium.gif new file mode 100644 index 0000000..075cc80 Binary files /dev/null and b/doc/images/logo-codesourcery-medium.gif differ diff --git a/doc/images/logo-software-carpentry-standard.gif b/doc/images/logo-software-carpentry-standard.gif new file mode 100644 index 0000000..a2a96c2 Binary files /dev/null and b/doc/images/logo-software-carpentry-standard.gif differ diff --git a/doc/spec.html b/doc/spec.html new file mode 100644 index 0000000..e39d857 --- /dev/null +++ b/doc/spec.html @@ -0,0 +1,1544 @@ + + +Software Carpentry Track: Roundup + + + + + + + + + + + +
+[Software Carpentry logo] + + + + + +
+[ACL Logo] +
+[CodeSourcery Logo] +
+
+ +

+ +

Roundup

+

An Issue-Tracking System for Knowledge Workers

+

Ka-Ping Yee
+ping@lfw.org

+

Implementation Guide

+ +

Contents

+ +
    +
  1. Introduction +
  2. The Layer Cake +
  3. Hyperdatabase +
      +
    1. Dates and Date Arithmetic +
    2. Nodes and Classes +
    3. Identifiers and Designators +
    4. Property Names and Types +
    5. Interface Specification +
    6. Application Example +
    +
  4. Roundup Database +
      +
    1. Reserved Classes +
        +
      1. Users +
      2. Messages +
      3. Files +
      +
    2. Item Classes +
    3. Interface Specification +
    4. Default Schema +
    +
  5. Detector Interface +
      +
    1. Interface Specification +
    2. Detector Example +
    +
  6. Command Interface +
      +
    1. Interface Specification +
    2. Usage Example +
    +
  7. E-mail User Interface +
      +
    1. Message Processing +
    2. Nosy Lists +
    3. Setting Properties +
    4. Workflow Example +
    +
  8. Web User Interface +
      +
    1. Views and View Specifiers +
    2. Displaying Properties +
    3. Index Views +
        +
      1. Index View Specifiers +
      2. Filter Section +
      3. Index Section +
      4. Sorting +
      +
    4. Item Views +
        +
      1. Item View Specifiers +
      2. Editor Section +
      3. Spool Section +
      +
    +
  9. Deployment Scenarios +
  10. Acknowledgements +
+ +

+

1. Introduction

+ +

This document presents a description of the components +of the Roundup system and specifies their interfaces and +behaviour in sufficient detail to guide an implementation. +For the philosophy and rationale behind the Roundup design, +see the first-round Software Carpentry submission for Roundup. +This document fleshes out that design as well as specifying +interfaces so that the components can be developed separately. + +

+

2. The Layer Cake

+ +

Lots of software design documents come with a picture of +a cake. Everybody seems to like them. I also like cakes +(i think they are tasty). So i, too, shall include +a picture of a cake here. + +

+ + + + + + + + + + + + + + + + +
+

+E-mail Client + +

 +

+Web Browser + +

 +

+Detector Scripts + +

 +

+Shell + +

+

+E-mail User Interface + +

 +

+Web User Interface + +

 +

+Detector Interface + +

 +

+Command Interface + +

+

+Roundup Database Layer + +

+

+Hyperdatabase Layer + +

+

+Storage Layer + +

+ +

The colourful parts of the cake are part of our system; +the faint grey parts of the cake are external components. + +

I will now proceed to forgo all table manners and +eat from the bottom of the cake to the top. You may want +to stand back a bit so you don't get covered in crumbs. + +

+

3. Hyperdatabase

+ +

The lowest-level component to be implemented is the hyperdatabase. +The hyperdatabase is intended to be +a flexible data store that can hold configurable data in +records which we call nodes. + +

The hyperdatabase is implemented on top of the storage layer, +an external module for storing its data. The storage layer could +be a third-party RDBMS; for a "batteries-included" distribution, +implementing the hyperdatabase on the standard bsddb +module is suggested. + +

3.1. Dates and Date Arithmetic

+ +

Before we get into the hyperdatabase itself, we need a +way of handling dates. The hyperdatabase module provides +Timestamp objects for +representing date-and-time stamps and Interval objects for +representing date-and-time intervals. + +

As strings, date-and-time stamps are specified with +the date in international standard format +(yyyy-mm-dd) +joined to the time (hh:mm:ss) +by a period ("."). Dates in +this form can be easily compared and are fairly readable +when printed. An example of a valid stamp is +"2000-06-24.13:03:59". +We'll call this the "full date format". When Timestamp objects are +printed as strings, they appear in the full date format with +the time always given in GMT. The full date format is always +exactly 19 characters long. + +

For user input, some partial forms are also permitted: +the whole time or just the seconds may be omitted; and the whole date +may be omitted or just the year may be omitted. If the time is given, +the time is interpreted in the user's local time zone. +The Date constructor takes care of these conversions. +In the following examples, suppose that yyyy is the current year, +mm is the current month, and dd is the current +day of the month; and suppose that the user is on Eastern Standard Time. + +

+ +

Date intervals are specified using the suffixes +"y", "m", and "d". The suffix "w" (for "week") means 7 days. +Time intervals are specified in hh:mm:ss format (the seconds +may be omitted, but the hours and minutes may not). + +

+ +

The Date class should understand simple date expressions of the form +stamp + interval and stamp - interval. +When adding or subtracting intervals involving months or years, the +components are handled separately. For example, when evaluating +"2000-06-25 + 1m 10d", we first add one month to +get 2000-07-25, then add 10 days to get +2000-08-04 (rather than trying to decide whether +1m 10d means 38 or 40 or 41 days). + +

Here is an outline of the Date and Interval classes. + +

+
class Date:
+    def __init__(self, spec, offset):
+        """Construct a date given a specification and a time zone offset.
+
+        'spec' is a full date or a partial form, with an optional
+        added or subtracted interval.  'offset' is the local time
+        zone offset from GMT in hours.
+        """
+
+    def __add__(self, interval):
+        """Add an interval to this date to produce another date."""
+
+    def __sub__(self, interval):
+        """Subtract an interval from this date to produce another date."""
+
+    def __cmp__(self, other):
+        """Compare this date to another date."""
+
+    def __str__(self):
+        """Return this date as a string in the yyyy-mm-dd.hh:mm:ss format."""
+
+    def local(self, offset):
+        """Return this date as yyyy-mm-dd.hh:mm:ss in a local time zone."""
+
+class Interval:
+    def __init__(self, spec):
+        """Construct an interval given a specification."""
+
+    def __cmp__(self, other):
+        """Compare this interval to another interval."""
+        
+    def __str__(self):
+        """Return this interval as a string."""
+
+
+ +

Here are some examples of how these classes would behave in practice. +For the following examples, assume that we are on Eastern Standard +Time and the current local time is 19:34:02 on 25 June 2000. + +

>>> Date(".")
+<Date 2000-06-26.00:34:02>
+>>> _.local(-5)
+"2000-06-25.19:34:02"
+>>> Date(". + 2d")
+<Date 2000-06-28.00:34:02>
+>>> Date("1997-04-17", -5)
+<Date 1997-04-17.00:00:00>
+>>> Date("01-25", -5)
+<Date 2000-01-25.00:00:00>
+>>> Date("08-13.22:13", -5)
+<Date 2000-08-14.03:13:00>
+>>> Date("14:25", -5)
+<Date 2000-06-25.19:25:00>
+>>> Interval("  3w  1  d  2:00")
+<Interval 22d 2:00>
+>>> Date(". + 2d") - Interval("3w")
+<Date 2000-06-07.00:34:02>
+ +

3.2. Nodes and Classes

+ +

Nodes contain data in properties. To Python, these +properties are presented as the key-value pairs of a dictionary. +Each node belongs to a class which defines the names +and types of its properties. The database permits the creation +and modification of classes as well as nodes. + +

3.3. Identifiers and Designators

+ +

Each node has a numeric identifier which is unique among +nodes in its class. The nodes are numbered sequentially +within each class in order of creation, starting from 1. +The designator +for a node is a way to identify a node in the database, and +consists of the name of the node's class concatenated with +the node's numeric identifier. + +

For example, if "spam" and "eggs" are classes, the first +node created in class "spam" has id 1 and designator "spam1". +The first node created in class "eggs" also has id 1 but has +the distinct designator "eggs1". Node designators are +conventionally enclosed in square brackets when mentioned +in plain text. This permits a casual mention of, say, +"[patch37]" in an e-mail message to be turned into an active +hyperlink. + +

3.4. Property Names and Types

+ +

Property names must begin with a letter. + +

A property may be one of five basic types: + +

+ +

None is also a permitted value for any of these property +types. An attempt to store None into a String property +stores the empty string; an attempt to store None +into a Multilink property stores an empty list. + +

3.5. Interface Specification

+ +

The hyperdb module provides property objects to designate +the different kinds of properties. These objects are used when +specifying what properties belong in classes. + +

class String:
+    def __init__(self):
+        """An object designating a String property."""
+
+class Date:
+    def __init__(self):
+        """An object designating a Date property."""
+
+class Link:
+    def __init__(self, classname):
+        """An object designating a Link property that links to
+        nodes in a specified class."""
+
+class Multilink:
+    def __init__(self, classname):
+        """An object designating a Multilink property that links
+        to nodes in a specified class."""
+
+ +

Here is the interface provided by the hyperdatabase. + +

class Database:
+    """A database for storing records containing flexible data types."""
+
+    def __init__(self, storagelocator, journaltag):
+        """Open a hyperdatabase given a specifier to some storage.
+
+        The meaning of 'storagelocator' depends on the particular
+        implementation of the hyperdatabase.  It could be a file name,
+        a directory path, a socket descriptor for a connection to a
+        database over the network, etc.
+
+        The 'journaltag' is a token that will be attached to the journal
+        entries for any edits done on the database.  If 'journaltag' is
+        None, the database is opened in read-only mode: the Class.create(),
+        Class.set(), and Class.retire() methods are disabled.
+        """
+
+    def __getattr__(self, classname):
+        """A convenient way of calling self.getclass(classname)."""
+
+    def getclasses(self):
+        """Return a list of the names of all existing classes."""
+
+    def getclass(self, classname):
+        """Get the Class object representing a particular class.
+
+        If 'classname' is not a valid class name, a KeyError is raised.
+        """
+
+class Class:
+    """The handle to a particular class of nodes in a hyperdatabase."""
+
+    def __init__(self, db, classname, **properties):
+        """Create a new class with a given name and property specification.
+
+        'classname' must not collide with the name of an existing class,
+        or a ValueError is raised.  The keyword arguments in 'properties'
+        must map names to property objects, or a TypeError is raised.
+        """
+
+    # Editing nodes:
+
+    def create(self, **propvalues):
+        """Create a new node of this class and return its id.
+
+        The keyword arguments in 'propvalues' map property names to values.
+        The values of arguments must be acceptable for the types of their
+        corresponding properties or a TypeError is raised.  If this class
+        has a key property, it must be present and its value must not
+        collide with other key strings or a ValueError is raised.  Any other
+        properties on this class that are missing from the 'propvalues'
+        dictionary are set to None.  If an id in a link or multilink
+        property does not refer to a valid node, an IndexError is raised.
+        """
+
+    def get(self, nodeid, propname):
+        """Get the value of a property on an existing node of this class.
+
+        'nodeid' must be the id of an existing node of this class or an
+        IndexError is raised.  'propname' must be the name of a property
+        of this class or a KeyError is raised.
+        """
+
+    def set(self, nodeid, **propvalues):
+        """Modify a property on an existing node of this class.
+        
+        'nodeid' must be the id of an existing node of this class or an
+        IndexError is raised.  Each key in 'propvalues' must be the name
+        of a property of this class or a KeyError is raised.  All values
+        in 'propvalues' must be acceptable types for their corresponding
+        properties or a TypeError is raised.  If the value of the key
+        property is set, it must not collide with other key strings or a
+        ValueError is raised.  If the value of a Link or Multilink
+        property contains an invalid node id, a ValueError is raised.
+        """
+
+    def retire(self, nodeid):
+        """Retire a node.
+        
+        The properties on the node remain available from the get() method,
+        and the node's id is never reused.  Retired nodes are not returned
+        by the find(), list(), or lookup() methods, and other nodes may
+        reuse the values of their key properties.
+        """
+
+    def history(self, nodeid):
+        """Retrieve the journal of edits on a particular node.
+
+        'nodeid' must be the id of an existing node of this class or an
+        IndexError is raised.
+
+        The returned list contains tuples of the form
+
+            (date, tag, action, params)
+
+        'date' is a Timestamp object specifying the time of the change and
+        'tag' is the journaltag specified when the database was opened.
+        'action' may be:
+
+            'create' or 'set' -- 'params' is a dictionary of property values
+            'link' or 'unlink' -- 'params' is (classname, nodeid, propname)
+            'retire' -- 'params' is None
+        """
+
+    # Locating nodes:
+
+    def setkey(self, propname):
+        """Select a String property of this class to be the key property.
+
+        'propname' must be the name of a String property of this class or
+        None, or a TypeError is raised.  The values of the key property on
+        all existing nodes must be unique or a ValueError is raised.
+        """
+
+    def getkey(self):
+        """Return the name of the key property for this class or None."""
+
+    def lookup(self, keyvalue):
+        """Locate a particular node by its key property and return its id.
+
+        If this class has no key property, a TypeError is raised.  If the
+        'keyvalue' matches one of the values for the key property among
+        the nodes in this class, the matching node's id is returned;
+        otherwise a KeyError is raised.
+        """
+
+    def find(self, propname, nodeid):
+        """Get the ids of nodes in this class which link to a given node.
+        
+        'propname' must be the name of a property in this class, or a
+        KeyError is raised.  That property must be a Link or Multilink
+        property, or a TypeError is raised.  'nodeid' must be the id of
+        an existing node in the class linked to by the given property,
+        or an IndexError is raised.
+        """
+
+    def list(self):
+        """Return a list of the ids of the active nodes in this class."""
+
+    def count(self):
+        """Get the number of nodes in this class.
+
+        If the returned integer is 'numnodes', the ids of all the nodes
+        in this class run from 1 to numnodes, and numnodes+1 will be the
+        id of the next node to be created in this class.
+        """
+
+    # Manipulating properties:
+
+    def getprops(self):
+        """Return a dictionary mapping property names to property objects."""
+
+    def addprop(self, **properties):
+        """Add properties to this class.
+
+        The keyword arguments in 'properties' must map names to property
+        objects, or a TypeError is raised.  None of the keys in 'properties'
+        may collide with the names of existing properties, or a ValueError
+        is raised before any properties have been added.
+        """
+ +

3.6. Application Example

+ +

Here is an example of how the hyperdatabase module would work in practice. + +

>>> import hyperdb
+>>> db = hyperdb.Database("foo.db", "ping")
+>>> db
+<hyperdb.Database "foo.db" opened by "ping">
+>>> hyperdb.Class(db, "status", name=hyperdb.String())
+<hyperdb.Class "status">
+>>> _.setkey("name")
+>>> db.status.create(name="unread")
+1
+>>> db.status.create(name="in-progress")
+2
+>>> db.status.create(name="testing")
+3
+>>> db.status.create(name="resolved")
+4
+>>> db.status.count()
+4
+>>> db.status.list()
+[1, 2, 3, 4]
+>>> db.status.lookup("in-progress")
+2
+>>> db.status.retire(3)
+>>> db.status.list()
+[1, 2, 4]
+>>> hyperdb.Class(db, "issue", title=hyperdb.String(), status=hyperdb.Link("status"))
+<hyperdb.Class "issue">
+>>> db.issue.create(title="spam", status=1)
+1
+>>> db.issue.create(title="eggs", status=2)
+2
+>>> db.issue.create(title="ham", status=4)
+3
+>>> db.issue.create(title="arguments", status=2)
+4
+>>> db.issue.create(title="abuse", status=1)
+5
+>>> hyperdb.Class(db, "user", username=hyperdb.Key(), password=hyperdb.String())
+<hyperdb.Class "user">
+>>> db.issue.addprop(fixer=hyperdb.Link("user"))
+>>> db.issue.getprops()
+{"title": <hyperdb.String>, "status": <hyperdb.Link to "status">,
+ "user": <hyperdb.Link to "user">}
+>>> db.issue.set(5, status=2)
+>>> db.issue.get(5, "status")
+2
+>>> db.status.get(2, "name")
+"in-progress"
+>>> db.issue.get(5, "title")
+"abuse"
+>>> db.issue.find("status", db.status.lookup("in-progress"))
+[2, 4, 5]
+>>> db.issue.history(5)
+[(<Date 2000-06-28.19:09:43>, "ping", "create", {"title": "abuse", "status": 1}),
+ (<Date 2000-06-28.19:11:04>, "ping", "set", {"status": 2})]
+>>> db.status.history(1)
+[(<Date 2000-06-28.19:09:43>, "ping", "link", ("issue", 5, "status")),
+ (<Date 2000-06-28.19:11:04>, "ping", "unlink", ("issue", 5, "status"))]
+>>> db.status.history(2)
+[(<Date 2000-06-28.19:11:04>, "ping", "link", ("issue", 5, "status"))]
+
+ +

For the purposes of journalling, when a Multilink property is +set to a new list of nodes, the hyperdatabase compares the old +list to the new list. +The journal records "unlink" events for all the nodes that appear +in the old list but not the new list, +and "link" events for +all the nodes that appear in the new list but not in the old list. + +

+

4. Roundup Database

+ +

The Roundup database layer is implemented on top of the +hyperdatabase and mediates calls to the database. +Some of the classes in the Roundup database are considered +item classes. +The Roundup database layer adds detectors and user nodes, +and on items it provides mail spools, nosy lists, and superseders. + +

4.1. Reserved Classes

+ +

Internal to this layer we reserve three special classes +of nodes that are not items. + +

4.1.1. Users

+ +

Users are stored in the hyperdatabase as nodes of +class "user". The "user" class has the definition: + +

hyperdb.Class(db, "user", username=hyperdb.String(),
+                          password=hyperdb.String(),
+                          address=hyperdb.String())
+db.user.setkey("username")
+ +

4.1.2. Messages

+ +

E-mail messages are represented by hyperdatabase nodes of class "msg". +The actual text content of the messages is stored in separate files. +(There's no advantage to be gained by stuffing them into the +hyperdatabase, and if messages are stored in ordinary text files, +they can be grepped from the command line.) The text of a message is +saved in a file named after the message node designator (e.g. "msg23") +for the sake of the command interface (see below). Attachments are +stored separately and associated with "file" nodes. +The "msg" class has the definition: + +

hyperdb.Class(db, "msg", author=hyperdb.Link("user"),
+                         recipients=hyperdb.Multilink("user"),
+                         date=hyperdb.Date(),
+                         summary=hyperdb.String(),
+                         files=hyperdb.Multilink("file"))
+ +

The "author" property indicates the author of the message +(a "user" node must exist in the hyperdatabase for any messages +that are stored in the system). +The "summary" property contains a summary of the message for display +in a message index. + +

4.1.3. Files

+ +

Submitted files are represented by hyperdatabase +nodes of class "file". Like e-mail messages, the file content +is stored in files outside the database, +named after the file node designator (e.g. "file17"). +The "file" class has the definition: + +

hyperdb.Class(db, "file", user=hyperdb.Link("user"),
+                          name=hyperdb.String(),
+                          type=hyperdb.String())
+ +

The "user" property indicates the user who submitted the +file, the "name" property holds the original name of the file, +and the "type" property holds the MIME type of the file as received. + +

4.2. Item Classes

+ +

All items have the following standard properties: + +

title=hyperdb.String()
+messages=hyperdb.Multilink("msg")
+files=hyperdb.Multilink("file")
+nosy=hyperdb.Multilink("user")
+superseder=hyperdb.Multilink("item")
+ +

Also, two Date properties named "creation" and "activity" are +fabricated by the Roundup database layer. By "fabricated" we +mean that no such properties are actually stored in the +hyperdatabase, but when properties on items are requested, the +"creation" and "activity" properties are made available. +The value of the "creation" property is the date when an item was +created, and the value of the "activity" property is the +date when any property on the item was last edited (equivalently, +these are the dates on the first and last records in the item's journal). + +

4.3. Interface Specification

+ +

The interface to a Roundup database delegates most method +calls to the hyperdatabase, except for the following +changes and additional methods. + +

class Database:
+    # Overridden methods:
+
+    def __init__(self, storagelocator, journaltag):
+        """When the Roundup database is opened by a particular user,
+        the 'journaltag' is the id of the user's "user" node."""
+
+    def getclass(self, classname):
+        """This method now returns an instance of either Class or
+        ItemClass depending on whether an item class is specified."""
+
+    # New methods:
+
+    def getuid(self):
+        """Return the id of the "user" node associated with the user
+        that owns this connection to the hyperdatabase."""
+
+class Class:
+    # Overridden methods:
+
+    def create(self, **propvalues):
+    def set(self, **propvalues):
+    def retire(self, nodeid):
+        """These operations trigger detectors and can be vetoed.  Attempts
+        to modify the "creation" or "activity" properties cause a KeyError.
+        """
+
+    # New methods:
+
+    def audit(self, event, detector):
+    def react(self, event, detector):
+        """Register a detector (see below for more details)."""
+
+class ItemClass(Class):
+    # Overridden methods:
+
+    def __init__(self, db, classname, **properties):
+        """The newly-created class automatically includes the "messages",
+        "files", "nosy", and "superseder" properties.  If the 'properties'
+        dictionary attempts to specify any of these properties or a
+        "creation" or "activity" property, a ValueError is raised."""
+
+    def get(self, nodeid, propname):
+    def getprops(self):
+        """In addition to the actual properties on the node, these
+        methods provide the "creation" and "activity" properties."""
+
+    # New methods:
+
+    def addmessage(self, nodeid, summary, text):
+        """Add a message to an item's mail spool.
+
+        A new "msg" node is constructed using the current date, the
+        user that owns the database connection as the author, and
+        the specified summary text.  The "files" and "recipients"
+        fields are left empty.  The given text is saved as the body
+        of the message and the node is appended to the "messages"
+        field of the specified item.
+        """
+
+    def sendmessage(self, nodeid, msgid):
+        """Send a message to the members of an item's nosy list.
+
+        The message is sent only to users on the nosy list who are not
+        already on the "recipients" list for the message.  These users
+        are then added to the message's "recipients" list.
+        """
+
+ +

4.4. Default Schema

+ +

The default schema included with Roundup turns it into a +typical software bug tracker. The database is set up like this: + +

pri = Class(db, "priority", name=hyperdb.String(), order=hyperdb.String())
+pri.setkey("name")
+pri.create(name="critical", order="1")
+pri.create(name="urgent", order="2")
+pri.create(name="bug", order="3")
+pri.create(name="feature", order="4")
+pri.create(name="wish", order="5")
+
+stat = Class(db, "status", name=hyperdb.String(), order=hyperdb.String())
+stat.setkey("name")
+stat.create(name="unread", order="1")
+stat.create(name="deferred", order="2")
+stat.create(name="chatting", order="3")
+stat.create(name="need-eg", order="4")
+stat.create(name="in-progress", order="5")
+stat.create(name="testing", order="6")
+stat.create(name="done-cbb", order="7")
+stat.create(name="resolved", order="8")
+
+Class(db, "keyword", name=hyperdb.String())
+
+Class(db, "issue", fixer=hyperdb.Multilink("user"),
+                   topic=hyperdb.Multilink("keyword"),
+                   priority=hyperdb.Link("priority"),
+                   status=hyperdb.Link("status"))
+
+ +

(The "order" property hasn't been explained yet. It +gets used by the Web user interface for sorting.) + +

The above isn't as pretty-looking as the schema specification +in the first-stage submission, but it could be made just as easy +with the addition of a convenience function like Choice +for setting up the "priority" and "status" classes: + +

def Choice(name, *options):
+    cl = Class(db, name, name=hyperdb.String(), order=hyperdb.String())
+    for i in range(len(options)):
+        cl.create(name=option[i], order=i)
+    return hyperdb.Link(name)
+
+ +

+

5. Detector Interface

+ +

Detectors are Python functions that are triggered on certain +kinds of events. The definitions of the +functions live in Python modules placed in a directory set aside +for this purpose. Importing the Roundup database module also +imports all the modules in this directory, and the init() +function of each module is called when a database is opened to +provide it a chance to register its detectors. + +

There are two kinds of detectors: + +

+ +

When the Roundup database is about to perform a +create(), set(), or retire() +operation, it first calls any auditors that +have been registered for that operation on that class. +Any auditor may raise a Reject exception +to abort the operation. + +

If none of the auditors raises an exception, the database +proceeds to carry out the operation. After it's done, it +then calls all of the reactors that have been registered +for the operation. + +

5.1. Interface Specification

+ +

The audit() and react() methods +register detectors on a given class of nodes. + +

class Class:
+    def audit(self, event, detector):
+        """Register an auditor on this class.
+
+        'event' should be one of "create", "set", or "retire".
+        'detector' should be a function accepting four arguments.
+        """
+
+    def react(self, event, detector):
+        """Register a reactor on this class.
+
+        'event' should be one of "create", "set", or "retire".
+        'detector' should be a function accepting four arguments.
+        """
+
+ +

Auditors are called with the arguments: + +

audit(db, cl, nodeid, newdata)
+ +where db is the database, cl is an +instance of Class or ItemClass within the database, and newdata +is a dictionary mapping property names to values. + +For a create() +operation, the nodeid argument is None and newdata +contains all of the initial property values with which the node +is about to be created. + +For a set() operation, newdata +contains only the names and values of properties that are about +to be changed. + +For a retire() operation, newdata is None. + +

Reactors are called with the arguments: + +

react(db, cl, nodeid, olddata)
+ +where db is the database, cl is an +instance of Class or ItemClass within the database, and olddata +is a dictionary mapping property names to values. + +For a create() +operation, the nodeid argument is the id of the +newly-created node and olddata is None. + +For a set() operation, olddata +contains the names and previous values of properties that were changed. + +For a retire() operation, nodeid is the +id of the retired node and olddata is None. + +

5.2. Detector Example

+ +

Here is an example of detectors written for a hypothetical +project-management application, where users can signal approval +of a project by adding themselves to an "approvals" list, and +a project proceeds when it has three approvals. + +

# Permit users only to add themselves to the "approvals" list.
+
+def check_approvals(db, cl, id, newdata):
+    if newdata.has_key("approvals"):
+        if cl.get(id, "status") == db.status.lookup("approved"):
+            raise Reject, "You can't modify the approvals list " \
+                          "for a project that has already been approved."
+        old = cl.get(id, "approvals")
+        new = newdata["approvals"]
+        for uid in old:
+            if uid not in new and uid != db.getuid():
+                raise Reject, "You can't remove other users from the "
+                              "approvals list; you can only remove yourself."
+        for uid in new:
+            if uid not in old and uid != db.getuid():
+                raise Reject, "You can't add other users to the approvals "
+                              "list; you can only add yourself."
+
+# When three people have approved a project, change its
+# status from "pending" to "approved".
+
+def approve_project(db, cl, id, olddata):
+    if olddata.has_key("approvals") and len(cl.get(id, "approvals")) == 3:
+        if cl.get(id, "status") == db.status.lookup("pending"):
+            cl.set(id, status=db.status.lookup("approved"))
+
+def init(db):
+    db.project.audit("set", check_approval)
+    db.project.react("set", approve_project)
+ +

Here is another example of a detector that can allow or prevent +the creation of new nodes. In this scenario, patches for a software +project are submitted by sending in e-mail with an attached file, +and we want to ensure that there are text/plain attachments on +the message. The maintainer of the package can then apply the +patch by setting its status to "applied". + +

# Only accept attempts to create new patches that come with patch files.
+
+def check_new_patch(db, cl, id, newdata):
+    if not newdata["files"]:
+        raise Reject, "You can't submit a new patch without " \
+                      "attaching a patch file."
+    for fileid in newdata["files"]:
+        if db.file.get(fileid, "type") != "text/plain":
+            raise Reject, "Submitted patch files must be text/plain."
+
+# When the status is changed from "approved" to "applied", apply the patch.
+
+def apply_patch(db, cl, id, olddata):
+    if cl.get(id, "status") == db.status.lookup("applied") and \
+        olddata["status"] == db.status.lookup("approved"):
+        # ...apply the patch...
+
+def init(db):
+    db.patch.audit("create", check_new_patch)
+    db.patch.react("set", apply_patch)
+ +

+

6. Command Interface

+ +

The command interface is a very simple and minimal interface, +intended only for quick searches and checks from the shell prompt. +(Anything more interesting can simply be written in Python using +the Roundup database module.) + +

6.1. Interface Specification

+ +

A single command, roundup, provides basic access to +the hyperdatabase from the command line. + +

+ +

Property values are represented as strings in command arguments +and in the printed results: + +

+ +

When multiple nodes are specified to the +roundup get or roundup set +commands, the specified properties are retrieved or set +on all the listed nodes. + +

When multiple results are returned by the roundup get +or roundup find commands, they are printed one per +line (default) or joined by commas (with the -list) option. + +

6.2. Usage Example

+ +

To find all messages regarding in-progress issues that +contain the word "spam", for example, you could execute the +following command from the directory where the database +dumps its files: + +

shell% for issue in `roundup find issue status=in-progress`; do
+> grep -l spam `roundup get $issue messages`
+> done
+msg23
+msg49
+msg50
+msg61
+shell%
+ +

Or, using the -list option, this can be written as a single command: + +

shell% grep -l spam `roundup get \
+    \`roundup find -list issue status=in-progress\` messages`
+msg23
+msg49
+msg50
+msg61
+shell%
+ +

+

7. E-mail User Interface

+ +

The Roundup system must be assigned an e-mail address +at which to receive mail. Messages should be piped to +the Roundup mail-handling script by the mail delivery +system (e.g. using an alias beginning with "|" for sendmail). + +

7.1. Message Processing

+ +

Incoming messages are examined for multiple parts. +In a multipart/mixed message or part, each subpart is +extracted and examined. In a multipart/alternative +message or part, we look for a text/plain subpart and +ignore the other parts. The text/plain subparts are +assembled to form the textual body of the message, to +be stored in the file associated with a "msg" class node. +Any parts of other types are each stored in separate +files and given "file" class nodes that are linked to +the "msg" node. + +

The "summary" property on message nodes is taken from +the first non-quoting section in the message body. +The message body is divided into sections by blank lines. +Sections where the second and all subsequent lines begin +with a ">" or "|" character are considered "quoting +sections". The first line of the first non-quoting +section becomes the summary of the message. + +

All of the addresses in the To: and Cc: headers of the +incoming message are looked up among the user nodes, and +the corresponding users are placed in the "recipients" +property on the new "msg" node. The address in the From: +header similarly determines the "author" property of the +new "msg" node. +The default handling for +addresses that don't have corresponding users is to create +new users with no passwords and a username equal to the +address. (The web interface does not permit logins for +users with no passwords.) If we prefer to reject mail from +outside sources, we can simply register an auditor on the +"user" class that prevents the creation of user nodes with +no passwords. + +

The subject line of the incoming message is examined to +determine whether the message is an attempt to create a new +item or to discuss an existing item. A designator enclosed +in square brackets is sought as the first thing on the +subject line (after skipping any "Fwd:" or "Re:" prefixes). + +

If an item designator (class name and id number) is found +there, the newly created "msg" node is added to the "messages" +property for that item, and any new "file" nodes are added to +the "files" property for the item. + +

If just an item class name is found there, we attempt to +create a new item of that class with its "messages" property +initialized to contain the new "msg" node and its "files" +property initialized to contain any new "file" nodes. + +

Both cases may trigger detectors (in the first case we +are calling the set() method to add the message to the +item's spool; in the second case we are calling the +create() method to create a new node). If an auditor +raises an exception, the original message is bounced back to +the sender with the explanatory message given in the exception. + +

7.2. Nosy Lists

+ +

A standard detector is provided that watches for additions +to the "messages" property. When a new message is added, the +detector sends it to all the users on the "nosy" list for the +item that are not already on the "recipients" list of the +message. Those users are then appended to the "recipients" +property on the message, so multiple copies of a message +are never sent to the same user. The journal recorded by +the hyperdatabase on the "recipients" property then provides +a log of when the message was sent to whom. + +

7.3. Setting Properties

+ +

The e-mail interface also provides a simple way to set +properties on items. At the end of the subject line, +propname=value pairs can be +specified in square brackets, using the same conventions +as for the roundup set shell command. + +

+

8. Web User Interface

+ +

The web interface is provided by a CGI script that can be +run under any web server. A simple web server can easily be +built on the standard CGIHTTPServer module, and +should also be included in the distribution for quick +out-of-the-box deployment. + +

The user interface is constructed from a number of template +files containing mostly HTML. Among the HTML tags in templates +are interspersed some nonstandard tags, which we use as +placeholders to be replaced by properties and their values. + +

8.1. Views and View Specifiers

+ +

There are two main kinds of views: index views and item views. +An index view displays a list of items of a particular class, +optionally sorted and filtered as requested. An item view +presents the properties of a particular item for editing +and displays the message spool for the item. + +

A view specifier is a string that specifies +all the options needed to construct a particular view. +It goes after the URL to the Roundup CGI script or the +web server to form the complete URL to a view. When the +result of selecting a link or submitting a form takes +the user to a new view, the Web browser should be redirected +to a canonical location containing a complete view specifier +so that the view can be bookmarked. + +

8.2. Displaying Properties

+ +

Properties appear in the user interface in three contexts: +in indices, in editors, and as filters. For each type of +property, there are several display possibilities. For example, +in an index view, a string property may just be printed as +a plain string, but in an editor view, that property should +be displayed in an editable field. + +

The display of a property is handled by functions in +a displayers module. Each function accepts at +least three standard arguments -- the database, class name, +and node id -- and returns a chunk of HTML. + +

Displayer functions are triggered by <display> +tags in templates. The call attribute of the tag +provides a Python expression for calling the displayer +function. The three standard arguments are inserted in +front of the arguments given. For example, the occurrence of + +

    <display call="plain('status', max=30)">
+
+ +in a template triggers a call to + +
    plain(db, "issue", 13, "status", max=30)
+
+ +when displaying item 13 in the "issue" class. The displayer +functions can accept extra arguments to further specify +details about the widgets that should be generated. By defining new +displayer functions, the user interface can be highly customized. + +

Some of the standard displayer functions include: + +

+ +

8.3. Index Views

+ +

An index view contains two sections: a filter section +and an index section. +The filter section provides some widgets for selecting +which items appear in the index. The index section is +a table of items. + +

8.3.1. Index View Specifiers

+ +

An index view specifier looks like this (whitespace +has been added for clarity): + +

/issue?status=unread,in-progress,resolved&
+        topic=security,ui&
+        :group=+priority&
+        :sort=-activity&
+        :filters=status,topic&
+        :columns=title,status,fixer
+
+ +

The index view is determined by two parts of the +specifier: the layout part and the filter part. +The layout part consists of the query parameters that +begin with colons, and it determines the way that the +properties of selected nodes are displayed. +The filter part consists of all the other query parameters, +and it determines the criteria by which nodes +are selected for display. + +

The filter part is interactively manipulated with +the form widgets displayed in the filter section. The +layout part is interactively manipulated by clicking +on the column headings in the table. + +

The filter part selects the union of the +sets of items with values matching any specified Link +properties and the intersection of the sets +of items with values matching any specified Multilink +properties. + +

The example specifies an index of "issue" nodes. +Only items with a "status" of either +"unread" or "in-progres" or "resolved" are displayed, +and only items with "topic" values including both +"security" and "ui" are displayed. The items +are grouped by priority, arranged in ascending order; +and within groups, sorted by activity, arranged in +descending order. The filter section shows filters +for the "status" and "topic" properties, and the +table includes columns for the "title", "status", and +"fixer" properties. + +

Associated with each item class is a default +layout specifier. The layout specifier in the above +example is the default layout to be provided with +the default bug-tracker schema described above in +section 4.4. + +

8.3.2. Filter Section

+ +

The template for a filter section provides the +filtering widgets at the top of the index view. +Fragments enclosed in <property>...</property> +tags are included or omitted depending on whether the +view specifier requests a filter for a particular property. + +

Here's a simple example of a filter template. + +

<property name=status>
+    <display call="checklist('status')">
+</property>
+<br>
+<property name=priority>
+    <display call="checklist('priority')">
+</property>
+<br>
+<property name=fixer>
+    <display call="menu('fixer')">
+</property>
+ +

8.3.3. Index Section

+ +

The template for an index section describes one row of +the index table. +Fragments enclosed in <property>...</property> +tags are included or omitted depending on whether the +view specifier requests a column for a particular property. +The table cells should contain <display> tags +to display the values of the item's properties. + +

Here's a simple example of an index template. + +

<tr>
+    <property name=title>
+        <td><display call="plain('title', max=50)"></td>
+    </property>
+    <property name=status>
+        <td><display call="plain('status')"></td>
+    </property>
+    <property name=fixer>
+        <td><display call="plain('fixer')"></td>
+    </property>
+</tr>
+ +

8.3.4. Sorting

+ +

String and Date values are sorted in the natural way. +Link properties are sorted according to the value of the +"order" property on the linked nodes if it is present; or +otherwise on the key string of the linked nodes; or +finally on the node ids. Multilink properties are +sorted according to how many links are present. + +

8.4. Item Views

+ +

An item view contains an editor section and a spool section. +At the top of an item view, links to superseding and superseded +items are always displayed. + +

8.4.1. Item View Specifiers

+ +

An item view specifier is simply the item's designator: + +

/patch23
+
+ +

8.4.2. Editor Section

+ +

The editor section is generated from a template +containing <display> tags to insert +the appropriate widgets for editing properties. + +

Here's an example of a basic editor template. + +

<table>
+<tr>
+    <td colspan=2>
+        <display call="field('title', size=60)">
+    </td>
+</tr>
+<tr>
+    <td>
+        <display call="field('fixer', size=30)">
+    </td>
+    <td>
+        <display call="menu('status')>
+    </td>
+</tr>
+<tr>
+    <td>
+        <display call="field('nosy', size=30)">
+    </td>
+    <td>
+        <display call="menu('priority')>
+    </td>
+</tr>
+<tr>
+    <td colspan=2>
+        <display call="note()">
+    </td>
+</tr>
+</table>
+
+ +

As shown in the example, the editor template can also +request the display of a "note" field, which is a +text area for entering a note to go along with a change. + +

When a change is submitted, the system automatically +generates a message describing the changed properties. +The message displays all of the property values on the +item and indicates which ones have changed. +An example of such a message might be this: + +

title: Polly Parrot is dead
+priority: critical
+status: unread -> in-progress
+fixer: (none)
+keywords: parrot,plumage,perch,nailed,dead
+
+ +

If a note is given in the "note" field, the note is +appended to the description. The message is then added +to the item's message spool (thus triggering the standard +detector to react by sending out this message to the nosy list). + +

8.4.3. Spool Section

+ +

The spool section lists messages in the item's "messages" +property. The index of messages displays the "date", "author", +and "summary" properties on the message nodes, and selecting a +message takes you to its content. + +

+

9. Deployment Scenarios

+ +

The design described above should be general enough +to permit the use of Roundup for bug tracking, managing +projects, managing patches, or holding discussions. By +using nodes of multiple types, one could deploy a system +that maintains requirement specifications, catalogs bugs, +and manages submitted patches, where patches could be +linked to the bugs and requirements they address. + +

+

10. Acknowledgements

+ +

My thanks are due to Christy Heyl for +reviewing and contributing suggestions to this paper +and motivating me to get it done, and to +Jesse Vincent, Mark Miller, Christopher Simons, +Jeff Dunmall, Wayne Gramlich, and Dean Tribble for +their assistance with the first-round submission. + + + + +

+ +

+ + + + + + + + + + + + + +
   [Home]      [FAQ]      [License]      [Rules]      [Configure]      [Build]      [Test]      [Track]      [Resources]      [Archives]   
+
+ +

+
Last modified 2001/04/06 11:50:59.9063 US/Mountain
+ + diff --git a/roundup/backends/back_anydbm.py b/roundup/backends/back_anydbm.py index 23beeb0..99febb2 100644 --- a/roundup/backends/back_anydbm.py +++ b/roundup/backends/back_anydbm.py @@ -1,4 +1,4 @@ -#$Id: back_anydbm.py,v 1.2 2001-07-23 08:20:44 richard Exp $ +#$Id: back_anydbm.py,v 1.3 2001-07-25 01:23:07 richard Exp $ import anydbm, os, marshal from roundup import hyperdb, date @@ -66,7 +66,10 @@ class Database(hyperdb.Database): multiple actions ''' path = os.path.join(os.getcwd(), self.dir, 'nodes.%s'%classname) - return anydbm.open(path, mode) + if os.path.exists(path): + return anydbm.open(path, mode) + else: + return anydbm.open(path, 'n') # # Nodes @@ -100,6 +103,7 @@ class Database(hyperdb.Database): # convert the marshalled data to instances properties = self.classes[classname].properties for key in res.keys(): + if key == self.RETIRED_FLAG: continue if properties[key].isDateType: res[key] = date.Date(res[key]) elif properties[key].isIntervalType: @@ -197,4 +201,9 @@ class Database(hyperdb.Database): # #$Log: not supported by cvs2svn $ +#Revision 1.2 2001/07/23 08:20:44 richard +#Moved over to using marshal in the bsddb and anydbm backends. +#roundup-admin now has a "freshen" command that'll load/save all nodes (not +# retired - mod hyperdb.Class.list() so it lists retired nodes) +# # diff --git a/roundup/templates/extended/dbinit.py b/roundup/templates/extended/dbinit.py index ac7c654..44da2b2 100644 --- a/roundup/templates/extended/dbinit.py +++ b/roundup/templates/extended/dbinit.py @@ -1,4 +1,4 @@ -# $Id: dbinit.py,v 1.5 2001-07-23 23:20:35 richard Exp $ +# $Id: dbinit.py,v 1.6 2001-07-25 01:23:07 richard Exp $ import os @@ -25,8 +25,6 @@ class IssueClass(roundupdb.IssueClass): def open(name=None): ''' as from the roundupdb method openDB - storagelocator must be the directory the __init__.py file is in - - os.path.split(__file__)[0] gives us that I think ''' from roundup.hyperdb import String, Date, Link, Multilink @@ -96,9 +94,6 @@ def open(name=None): def init(adminpw): ''' as from the roundupdb method initDB - storagelocator must be the directory the __init__.py file is in - - os.path.split(__file__)[0] gives us that I think - Open the new database, and set up a bunch of attributes. ''' @@ -156,6 +151,9 @@ def init(adminpw): # # $Log: not supported by cvs2svn $ +# Revision 1.5 2001/07/23 23:20:35 richard +# forgot to remove the interfaces from the dbinit module ;) +# # Revision 1.4 2001/07/23 08:45:28 richard # ok, so now "./roundup-admin init" will ask questions in an attempt to get a # workable instance_home set up :) diff --git a/tests/test_db.py b/tests/test_db.py index 67b17c4..5d5511f 100644 --- a/tests/test_db.py +++ b/tests/test_db.py @@ -35,9 +35,10 @@ class DBTestCase(unittest.TestCase): self.db.issue.create(title="arguments", status='2') self.db.issue.create(title="abuse", status='1') self.db.issue.addprop(fixer=Link("user")) - self.db.issue.getprops() -#{"title": , "status": , -#"user": } + props = self.db.issue.getprops() + keys = props.keys() + keys.sort() + self.assertEqual(keys, ['title', 'status', 'user'], 'wrong prop list') self.db.issue.set('5', status=2) self.db.issue.get('5', "status") self.db.status.get('2', "name") @@ -47,6 +48,8 @@ class DBTestCase(unittest.TestCase): self.db.status.history('1') self.db.status.history('2') + def testExceptions(self): + # this tests the exceptions that should be raised def suite(): return unittest.makeSuite(DBTestCase, 'test') diff --git a/tests/test_schema.py b/tests/test_schema.py index 3620840..38318ec 100644 --- a/tests/test_schema.py +++ b/tests/test_schema.py @@ -42,9 +42,11 @@ class SchemaTestCase(unittest.TestCase): def testB_Issue(self): issue = Class(self.db, "issue", title=String(), status=Link("status")) + self.assert_(issue, 'no class object returned') def testC_User(self): user = Class(self.db, "user", username=String(), password=String()) + self.assert_(user, 'no class object returned') user.setkey("username")