1 ========================================================
2 Roundup - An Issue-Tracking System for Knowledge Workers
3 ========================================================
5 :Authors: Ka-Ping Yee (original__), Richard Jones (implementation)
7 __ spec.html
9 .. contents::
11 Introduction
12 ---------------
14 This document presents a description of the components
15 of the Roundup system and specifies their interfaces and
16 behaviour in sufficient detail to guide an implementation.
17 For the philosophy and rationale behind the Roundup design,
18 see the first-round Software Carpentry submission for Roundup.
19 This document fleshes out that design as well as specifying
20 interfaces so that the components can be developed separately.
23 The Layer Cake
24 -----------------
26 Lots of software design documents come with a picture of
27 a cake. Everybody seems to like them. I also like cakes
28 (i think they are tasty). So i, too, shall include
29 a picture of a cake here::
31 _________________________________________________________________________
32 | E-mail Client | Web Browser | Detector Scripts | Shell |
33 |------------------+-----------------+----------------------+-------------|
34 | E-mail User | Web User | Detector | Command |
35 |-------------------------------------------------------------------------|
36 | Roundup Database Layer |
37 |-------------------------------------------------------------------------|
38 | Hyperdatabase Layer |
39 |-------------------------------------------------------------------------|
40 | Storage Layer |
41 -------------------------------------------------------------------------
43 The colourful parts of the cake are part of our system;
44 the faint grey parts of the cake are external components.
46 I will now proceed to forgo all table manners and
47 eat from the bottom of the cake to the top. You may want
48 to stand back a bit so you don't get covered in crumbs.
51 Hyperdatabase
52 -------------
54 The lowest-level component to be implemented is the hyperdatabase.
55 The hyperdatabase is intended to be
56 a flexible data store that can hold configurable data in
57 records which we call nodes.
59 The hyperdatabase is implemented on top of the storage layer,
60 an external module for storing its data. The storage layer could
61 be a third-party RDBMS; for a "batteries-included" distribution,
62 implementing the hyperdatabase on the standard bsddb
63 module is suggested.
65 Dates and Date Arithmetic
66 ~~~~~~~~~~~~~~~~~~~~~~~~~
68 Before we get into the hyperdatabase itself, we need a
69 way of handling dates. The hyperdatabase module provides
70 Timestamp objects for
71 representing date-and-time stamps and Interval objects for
72 representing date-and-time intervals.
74 As strings, date-and-time stamps are specified with
75 the date in international standard format
76 (``yyyy-mm-dd``)
77 joined to the time (``hh:mm:ss``)
78 by a period "``.``". Dates in
79 this form can be easily compared and are fairly readable
80 when printed. An example of a valid stamp is
81 "``2000-06-24.13:03:59``".
82 We'll call this the "full date format". When Timestamp objects are
83 printed as strings, they appear in the full date format with
84 the time always given in GMT. The full date format is always
85 exactly 19 characters long.
87 For user input, some partial forms are also permitted:
88 the whole time or just the seconds may be omitted; and the whole date
89 may be omitted or just the year may be omitted. If the time is given,
90 the time is interpreted in the user's local time zone.
91 The Date constructor takes care of these conversions.
92 In the following examples, suppose that ``yyyy`` is the current year,
93 ``mm`` is the current month, and ``dd`` is the current
94 day of the month; and suppose that the user is on Eastern Standard Time.
96 - "2000-04-17" means <Date 2000-04-17.00:00:00>
97 - "01-25" means <Date yyyy-01-25.00:00:00>
98 - "2000-04-17.03:45" means <Date 2000-04-17.08:45:00>
99 - "08-13.22:13" means <Date yyyy-08-14.03:13:00>
100 - "11-07.09:32:43" means <Date yyyy-11-07.14:32:43>
101 - "14:25" means
102 - <Date yyyy-mm-dd.19:25:00>
103 - "8:47:11" means
104 - <Date yyyy-mm-dd.13:47:11>
105 - the special date "." means "right now"
108 Date intervals are specified using the suffixes
109 "y", "m", and "d". The suffix "w" (for "week") means 7 days.
110 Time intervals are specified in hh:mm:ss format (the seconds
111 may be omitted, but the hours and minutes may not).
113 - "3y" means three years
114 - "2y 1m" means two years and one month
115 - "1m 25d" means one month and 25 days
116 - "2w 3d" means two weeks and three days
117 - "1d 2:50" means one day, two hours, and 50 minutes
118 - "14:00" means 14 hours
119 - "0:04:33" means four minutes and 33 seconds
122 The Date class should understand simple date expressions of the form
123 *stamp* ``+`` *interval* and *stamp* ``-`` *interval*.
124 When adding or subtracting intervals involving months or years, the
125 components are handled separately. For example, when evaluating
126 "``2000-06-25 + 1m 10d``", we first add one month to
127 get 2000-07-25, then add 10 days to get
128 2000-08-04 (rather than trying to decide whether
129 1m 10d means 38 or 40 or 41 days).
131 Here is an outline of the Date and Interval classes::
133 class Date:
134 def __init__(self, spec, offset):
135 """Construct a date given a specification and a time zone offset.
137 'spec' is a full date or a partial form, with an optional
138 added or subtracted interval. 'offset' is the local time
139 zone offset from GMT in hours.
140 """
142 def __add__(self, interval):
143 """Add an interval to this date to produce another date."""
145 def __sub__(self, interval):
146 """Subtract an interval from this date to produce another date."""
148 def __cmp__(self, other):
149 """Compare this date to another date."""
151 def __str__(self):
152 """Return this date as a string in the yyyy-mm-dd.hh:mm:ss format."""
154 def local(self, offset):
155 """Return this date as yyyy-mm-dd.hh:mm:ss in a local time zone."""
157 class Interval:
158 def __init__(self, spec):
159 """Construct an interval given a specification."""
161 def __cmp__(self, other):
162 """Compare this interval to another interval."""
164 def __str__(self):
165 """Return this interval as a string."""
169 Here are some examples of how these classes would behave in practice.
170 For the following examples, assume that we are on Eastern Standard
171 Time and the current local time is 19:34:02 on 25 June 2000::
173 >>> Date(".")
174 <Date 2000-06-26.00:34:02>
175 >>> _.local(-5)
176 "2000-06-25.19:34:02"
177 >>> Date(". + 2d")
178 <Date 2000-06-28.00:34:02>
179 >>> Date("1997-04-17", -5)
180 <Date 1997-04-17.00:00:00>
181 >>> Date("01-25", -5)
182 <Date 2000-01-25.00:00:00>
183 >>> Date("08-13.22:13", -5)
184 <Date 2000-08-14.03:13:00>
185 >>> Date("14:25", -5)
186 <Date 2000-06-25.19:25:00>
187 >>> Interval(" 3w 1 d 2:00")
188 <Interval 22d 2:00>
189 >>> Date(". + 2d") - Interval("3w")
190 <Date 2000-06-07.00:34:02>
192 Nodes and Classes
193 ~~~~~~~~~~~~~~~~~
195 Nodes contain data in properties. To Python, these
196 properties are presented as the key-value pairs of a dictionary.
197 Each node belongs to a class which defines the names
198 and types of its properties. The database permits the creation
199 and modification of classes as well as nodes.
201 Identifiers and Designators
202 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
204 Each node has a numeric identifier which is unique among
205 nodes in its class. The nodes are numbered sequentially
206 within each class in order of creation, starting from 1.
207 The designator
208 for a node is a way to identify a node in the database, and
209 consists of the name of the node's class concatenated with
210 the node's numeric identifier.
212 For example, if "spam" and "eggs" are classes, the first
213 node created in class "spam" has id 1 and designator "spam1".
214 The first node created in class "eggs" also has id 1 but has
215 the distinct designator "eggs1". Node designators are
216 conventionally enclosed in square brackets when mentioned
217 in plain text. This permits a casual mention of, say,
218 "[patch37]" in an e-mail message to be turned into an active
219 hyperlink.
221 Property Names and Types
222 ~~~~~~~~~~~~~~~~~~~~~~~~
224 Property names must begin with a letter.
226 A property may be one of five basic types:
228 - String properties are for storing arbitrary-length strings.
230 - Boolean properties are for storing true/false, or yes/no values.
232 - Number properties are for storing numeric values.
234 - Date properties store date-and-time stamps.
235 Their values are Timestamp objects.
237 - A Link property refers to a single other node
238 selected from a specified class. The class is part of the property;
239 the value is an integer, the id of the chosen node.
241 - A Multilink property refers to possibly many nodes
242 in a specified class. The value is a list of integers.
244 *None* is also a permitted value for any of these property
245 types. An attempt to store None into a Multilink property stores an empty list.
247 A property that is not specified will return as None from a *get*
248 operation.
250 Hyperdb Interface Specification
251 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
253 The hyperdb module provides property objects to designate
254 the different kinds of properties. These objects are used when
255 specifying what properties belong in classes::
257 class String:
258 def __init__(self, indexme='no'):
259 """An object designating a String property."""
261 class Boolean:
262 def __init__(self):
263 """An object designating a Boolean property."""
265 class Number:
266 def __init__(self):
267 """An object designating a Number property."""
269 class Date:
270 def __init__(self):
271 """An object designating a Date property."""
273 class Link:
274 def __init__(self, classname, do_journal='yes'):
275 """An object designating a Link property that links to
276 nodes in a specified class.
278 If the do_journal argument is not 'yes' then changes to
279 the property are not journalled in the linked node.
280 """
282 class Multilink:
283 def __init__(self, classname, do_journal='yes'):
284 """An object designating a Multilink property that links
285 to nodes in a specified class.
287 If the do_journal argument is not 'yes' then changes to
288 the property are not journalled in the linked node(s).
289 """
292 Here is the interface provided by the hyperdatabase::
294 class Database:
295 """A database for storing records containing flexible data types."""
297 def __init__(self, storagelocator, journaltag):
298 """Open a hyperdatabase given a specifier to some storage.
300 The meaning of 'storagelocator' depends on the particular
301 implementation of the hyperdatabase. It could be a file name,
302 a directory path, a socket descriptor for a connection to a
303 database over the network, etc.
305 The 'journaltag' is a token that will be attached to the journal
306 entries for any edits done on the database. If 'journaltag' is
307 None, the database is opened in read-only mode: the Class.create(),
308 Class.set(), and Class.retire() methods are disabled.
309 """
311 def __getattr__(self, classname):
312 """A convenient way of calling self.getclass(classname)."""
314 def getclasses(self):
315 """Return a list of the names of all existing classes."""
317 def getclass(self, classname):
318 """Get the Class object representing a particular class.
320 If 'classname' is not a valid class name, a KeyError is raised.
321 """
323 class Class:
324 """The handle to a particular class of nodes in a hyperdatabase."""
326 def __init__(self, db, classname, **properties):
327 """Create a new class with a given name and property specification.
329 'classname' must not collide with the name of an existing class,
330 or a ValueError is raised. The keyword arguments in 'properties'
331 must map names to property objects, or a TypeError is raised.
332 """
334 # Editing nodes:
336 def create(self, **propvalues):
337 """Create a new node of this class and return its id.
339 The keyword arguments in 'propvalues' map property names to values.
340 The values of arguments must be acceptable for the types of their
341 corresponding properties or a TypeError is raised. If this class
342 has a key property, it must be present and its value must not
343 collide with other key strings or a ValueError is raised. Any other
344 properties on this class that are missing from the 'propvalues'
345 dictionary are set to None. If an id in a link or multilink
346 property does not refer to a valid node, an IndexError is raised.
347 """
349 def get(self, nodeid, propname):
350 """Get the value of a property on an existing node of this class.
352 'nodeid' must be the id of an existing node of this class or an
353 IndexError is raised. 'propname' must be the name of a property
354 of this class or a KeyError is raised.
355 """
357 def set(self, nodeid, **propvalues):
358 """Modify a property on an existing node of this class.
360 'nodeid' must be the id of an existing node of this class or an
361 IndexError is raised. Each key in 'propvalues' must be the name
362 of a property of this class or a KeyError is raised. All values
363 in 'propvalues' must be acceptable types for their corresponding
364 properties or a TypeError is raised. If the value of the key
365 property is set, it must not collide with other key strings or a
366 ValueError is raised. If the value of a Link or Multilink
367 property contains an invalid node id, a ValueError is raised.
368 """
370 def retire(self, nodeid):
371 """Retire a node.
373 The properties on the node remain available from the get() method,
374 and the node's id is never reused. Retired nodes are not returned
375 by the find(), list(), or lookup() methods, and other nodes may
376 reuse the values of their key properties.
377 """
379 def history(self, nodeid):
380 """Retrieve the journal of edits on a particular node.
382 'nodeid' must be the id of an existing node of this class or an
383 IndexError is raised.
385 The returned list contains tuples of the form
387 (date, tag, action, params)
389 'date' is a Timestamp object specifying the time of the change and
390 'tag' is the journaltag specified when the database was opened.
391 'action' may be:
393 'create' or 'set' -- 'params' is a dictionary of property values
394 'link' or 'unlink' -- 'params' is (classname, nodeid, propname)
395 'retire' -- 'params' is None
396 """
398 # Locating nodes:
400 def setkey(self, propname):
401 """Select a String property of this class to be the key property.
403 'propname' must be the name of a String property of this class or
404 None, or a TypeError is raised. The values of the key property on
405 all existing nodes must be unique or a ValueError is raised.
406 """
408 def getkey(self):
409 """Return the name of the key property for this class or None."""
411 def lookup(self, keyvalue):
412 """Locate a particular node by its key property and return its id.
414 If this class has no key property, a TypeError is raised. If the
415 'keyvalue' matches one of the values for the key property among
416 the nodes in this class, the matching node's id is returned;
417 otherwise a KeyError is raised.
418 """
420 def find(self, propname, nodeid):
421 """Get the ids of nodes in this class which link to a given node.
423 'propname' must be the name of a property in this class, or a
424 KeyError is raised. That property must be a Link or Multilink
425 property, or a TypeError is raised. 'nodeid' must be the id of
426 an existing node in the class linked to by the given property,
427 or an IndexError is raised.
428 """
430 def list(self):
431 """Return a list of the ids of the active nodes in this class."""
433 def count(self):
434 """Get the number of nodes in this class.
436 If the returned integer is 'numnodes', the ids of all the nodes
437 in this class run from 1 to numnodes, and numnodes+1 will be the
438 id of the next node to be created in this class.
439 """
441 # Manipulating properties:
443 def getprops(self):
444 """Return a dictionary mapping property names to property objects."""
446 def addprop(self, **properties):
447 """Add properties to this class.
449 The keyword arguments in 'properties' must map names to property
450 objects, or a TypeError is raised. None of the keys in 'properties'
451 may collide with the names of existing properties, or a ValueError
452 is raised before any properties have been added.
453 """
455 TODO: additional methods
457 Hyperdatabase Implementations
458 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
460 Hyperdatabase implementations exist to create the interface described in the
461 `hyperdb interface specification`_
462 over an existing storage mechanism. Examples are relational databases,
463 \*dbm key-value databases, and so on.
465 TODO: finish
468 Application Example
469 ~~~~~~~~~~~~~~~~~~~
471 Here is an example of how the hyperdatabase module would work in practice::
473 >>> import hyperdb
474 >>> db = hyperdb.Database("foo.db", "ping")
475 >>> db
476 <hyperdb.Database "foo.db" opened by "ping">
477 >>> hyperdb.Class(db, "status", name=hyperdb.String())
478 <hyperdb.Class "status">
479 >>> _.setkey("name")
480 >>> db.status.create(name="unread")
481 1
482 >>> db.status.create(name="in-progress")
483 2
484 >>> db.status.create(name="testing")
485 3
486 >>> db.status.create(name="resolved")
487 4
488 >>> db.status.count()
489 4
490 >>> db.status.list()
491 [1, 2, 3, 4]
492 >>> db.status.lookup("in-progress")
493 2
494 >>> db.status.retire(3)
495 >>> db.status.list()
496 [1, 2, 4]
497 >>> hyperdb.Class(db, "issue", title=hyperdb.String(), status=hyperdb.Link("status"))
498 <hyperdb.Class "issue">
499 >>> db.issue.create(title="spam", status=1)
500 1
501 >>> db.issue.create(title="eggs", status=2)
502 2
503 >>> db.issue.create(title="ham", status=4)
504 3
505 >>> db.issue.create(title="arguments", status=2)
506 4
507 >>> db.issue.create(title="abuse", status=1)
508 5
509 >>> hyperdb.Class(db, "user", username=hyperdb.Key(), password=hyperdb.String())
510 <hyperdb.Class "user">
511 >>> db.issue.addprop(fixer=hyperdb.Link("user"))
512 >>> db.issue.getprops()
513 {"title": <hyperdb.String>, "status": <hyperdb.Link to "status">,
514 "user": <hyperdb.Link to "user">}
515 >>> db.issue.set(5, status=2)
516 >>> db.issue.get(5, "status")
517 2
518 >>> db.status.get(2, "name")
519 "in-progress"
520 >>> db.issue.get(5, "title")
521 "abuse"
522 >>> db.issue.find("status", db.status.lookup("in-progress"))
523 [2, 4, 5]
524 >>> db.issue.history(5)
525 [(<Date 2000-06-28.19:09:43>, "ping", "create", {"title": "abuse", "status": 1}),
526 (<Date 2000-06-28.19:11:04>, "ping", "set", {"status": 2})]
527 >>> db.status.history(1)
528 [(<Date 2000-06-28.19:09:43>, "ping", "link", ("issue", 5, "status")),
529 (<Date 2000-06-28.19:11:04>, "ping", "unlink", ("issue", 5, "status"))]
530 >>> db.status.history(2)
531 [(<Date 2000-06-28.19:11:04>, "ping", "link", ("issue", 5, "status"))]
534 For the purposes of journalling, when a Multilink property is
535 set to a new list of nodes, the hyperdatabase compares the old
536 list to the new list.
537 The journal records "unlink" events for all the nodes that appear
538 in the old list but not the new list,
539 and "link" events for
540 all the nodes that appear in the new list but not in the old list.
543 Roundup Database
544 ----------------
546 The Roundup database layer is implemented on top of the
547 hyperdatabase and mediates calls to the database.
548 Some of the classes in the Roundup database are considered
549 issue classes.
550 The Roundup database layer adds detectors and user nodes,
551 and on issues it provides mail spools, nosy lists, and superseders.
553 TODO: where functionality is implemented.
555 Reserved Classes
556 ~~~~~~~~~~~~~~~~
558 Internal to this layer we reserve three special classes
559 of nodes that are not issues.
561 Users
562 """"""""""""
564 Users are stored in the hyperdatabase as nodes of
565 class "user". The "user" class has the definition::
567 hyperdb.Class(db, "user", username=hyperdb.String(),
568 password=hyperdb.String(),
569 address=hyperdb.String())
570 db.user.setkey("username")
572 Messages
573 """""""""""""""
575 E-mail messages are represented by hyperdatabase nodes of class "msg".
576 The actual text content of the messages is stored in separate files.
577 (There's no advantage to be gained by stuffing them into the
578 hyperdatabase, and if messages are stored in ordinary text files,
579 they can be grepped from the command line.) The text of a message is
580 saved in a file named after the message node designator (e.g. "msg23")
581 for the sake of the command interface (see below). Attachments are
582 stored separately and associated with "file" nodes.
583 The "msg" class has the definition::
585 hyperdb.Class(db, "msg", author=hyperdb.Link("user"),
586 recipients=hyperdb.Multilink("user"),
587 date=hyperdb.Date(),
588 summary=hyperdb.String(),
589 files=hyperdb.Multilink("file"))
591 The "author" property indicates the author of the message
592 (a "user" node must exist in the hyperdatabase for any messages
593 that are stored in the system).
594 The "summary" property contains a summary of the message for display
595 in a message index.
597 Files
598 """"""""""""
600 Submitted files are represented by hyperdatabase
601 nodes of class "file". Like e-mail messages, the file content
602 is stored in files outside the database,
603 named after the file node designator (e.g. "file17").
604 The "file" class has the definition::
606 hyperdb.Class(db, "file", user=hyperdb.Link("user"),
607 name=hyperdb.String(),
608 type=hyperdb.String())
610 The "user" property indicates the user who submitted the
611 file, the "name" property holds the original name of the file,
612 and the "type" property holds the MIME type of the file as received.
614 Issue Classes
615 ~~~~~~~~~~~~~
617 All issues have the following standard properties:
619 =========== ==========================
620 Property Definition
621 =========== ==========================
622 title hyperdb.String()
623 messages hyperdb.Multilink("msg")
624 files hyperdb.Multilink("file")
625 nosy hyperdb.Multilink("user")
626 superseder hyperdb.Multilink("issue")
627 =========== ==========================
629 Also, two Date properties named "creation" and "activity" are
630 fabricated by the Roundup database layer. By "fabricated" we
631 mean that no such properties are actually stored in the
632 hyperdatabase, but when properties on issues are requested, the
633 "creation" and "activity" properties are made available.
634 The value of the "creation" property is the date when an issue was
635 created, and the value of the "activity" property is the
636 date when any property on the issue was last edited (equivalently,
637 these are the dates on the first and last records in the issue's journal).
639 Roundupdb Interface Specification
640 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
642 The interface to a Roundup database delegates most method
643 calls to the hyperdatabase, except for the following
644 changes and additional methods::
646 class Database:
647 def getuid(self):
648 """Return the id of the "user" node associated with the user
649 that owns this connection to the hyperdatabase."""
651 class Class:
652 # Overridden methods:
654 def create(self, **propvalues):
655 def set(self, **propvalues):
656 def retire(self, nodeid):
657 """These operations trigger detectors and can be vetoed. Attempts
658 to modify the "creation" or "activity" properties cause a KeyError.
659 """
661 # New methods:
663 def audit(self, event, detector):
664 def react(self, event, detector):
665 """Register a detector (see below for more details)."""
667 class IssueClass(Class):
668 # Overridden methods:
670 def __init__(self, db, classname, **properties):
671 """The newly-created class automatically includes the "messages",
672 "files", "nosy", and "superseder" properties. If the 'properties'
673 dictionary attempts to specify any of these properties or a
674 "creation" or "activity" property, a ValueError is raised."""
676 def get(self, nodeid, propname):
677 def getprops(self):
678 """In addition to the actual properties on the node, these
679 methods provide the "creation" and "activity" properties."""
681 # New methods:
683 def addmessage(self, nodeid, summary, text):
684 """Add a message to an issue's mail spool.
686 A new "msg" node is constructed using the current date, the
687 user that owns the database connection as the author, and
688 the specified summary text. The "files" and "recipients"
689 fields are left empty. The given text is saved as the body
690 of the message and the node is appended to the "messages"
691 field of the specified issue.
692 """
694 def sendmessage(self, nodeid, msgid):
695 """Send a message to the members of an issue's nosy list.
697 The message is sent only to users on the nosy list who are not
698 already on the "recipients" list for the message. These users
699 are then added to the message's "recipients" list.
700 """
703 Default Schema
704 ~~~~~~~~~~~~~~
706 The default schema included with Roundup turns it into a
707 typical software bug tracker. The database is set up like this::
709 pri = Class(db, "priority", name=hyperdb.String(), order=hyperdb.String())
710 pri.setkey("name")
711 pri.create(name="critical", order="1")
712 pri.create(name="urgent", order="2")
713 pri.create(name="bug", order="3")
714 pri.create(name="feature", order="4")
715 pri.create(name="wish", order="5")
717 stat = Class(db, "status", name=hyperdb.String(), order=hyperdb.String())
718 stat.setkey("name")
719 stat.create(name="unread", order="1")
720 stat.create(name="deferred", order="2")
721 stat.create(name="chatting", order="3")
722 stat.create(name="need-eg", order="4")
723 stat.create(name="in-progress", order="5")
724 stat.create(name="testing", order="6")
725 stat.create(name="done-cbb", order="7")
726 stat.create(name="resolved", order="8")
728 Class(db, "keyword", name=hyperdb.String())
730 Class(db, "issue", fixer=hyperdb.Multilink("user"),
731 topic=hyperdb.Multilink("keyword"),
732 priority=hyperdb.Link("priority"),
733 status=hyperdb.Link("status"))
736 (The "order" property hasn't been explained yet. It
737 gets used by the Web user interface for sorting.)
739 The above isn't as pretty-looking as the schema specification
740 in the first-stage submission, but it could be made just as easy
741 with the addition of a convenience function like Choice
742 for setting up the "priority" and "status" classes::
744 def Choice(name, *options):
745 cl = Class(db, name, name=hyperdb.String(), order=hyperdb.String())
746 for i in range(len(options)):
747 cl.create(name=option[i], order=i)
748 return hyperdb.Link(name)
751 Detector Interface
752 ------------------
754 Detectors are Python functions that are triggered on certain
755 kinds of events. The definitions of the
756 functions live in Python modules placed in a directory set aside
757 for this purpose. Importing the Roundup database module also
758 imports all the modules in this directory, and the ``init()``
759 function of each module is called when a database is opened to
760 provide it a chance to register its detectors.
762 There are two kinds of detectors:
764 1. an auditor is triggered just before modifying an node
765 2. a reactor is triggered just after an node has been modified
767 When the Roundup database is about to perform a
768 ``create()``, ``set()``, or ``retire()``
769 operation, it first calls any *auditors* that
770 have been registered for that operation on that class.
771 Any auditor may raise a *Reject* exception
772 to abort the operation.
774 If none of the auditors raises an exception, the database
775 proceeds to carry out the operation. After it's done, it
776 then calls all of the *reactors* that have been registered
777 for the operation.
779 Detector Interface Specification
780 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
782 The ``audit()`` and ``react()`` methods
783 register detectors on a given class of nodes::
785 class Class:
786 def audit(self, event, detector):
787 """Register an auditor on this class.
789 'event' should be one of "create", "set", or "retire".
790 'detector' should be a function accepting four arguments.
791 """
793 def react(self, event, detector):
794 """Register a reactor on this class.
796 'event' should be one of "create", "set", or "retire".
797 'detector' should be a function accepting four arguments.
798 """
800 Auditors are called with the arguments::
802 audit(db, cl, nodeid, newdata)
804 where ``db`` is the database, ``cl`` is an
805 instance of Class or IssueClass within the database, and ``newdata``
806 is a dictionary mapping property names to values.
808 For a ``create()``
809 operation, the ``nodeid`` argument is None and newdata
810 contains all of the initial property values with which the node
811 is about to be created.
813 For a ``set()`` operation, newdata
814 contains only the names and values of properties that are about
815 to be changed.
817 For a ``retire()`` operation, newdata is None.
819 Reactors are called with the arguments::
821 react(db, cl, nodeid, olddata)
823 where ``db`` is the database, ``cl`` is an
824 instance of Class or IssueClass within the database, and ``olddata``
825 is a dictionary mapping property names to values.
827 For a ``create()``
828 operation, the ``nodeid`` argument is the id of the
829 newly-created node and ``olddata`` is None.
831 For a ``set()`` operation, ``olddata``
832 contains the names and previous values of properties that were changed.
834 For a ``retire()`` operation, ``nodeid`` is the
835 id of the retired node and ``olddata`` is None.
837 Detector Example
838 ~~~~~~~~~~~~~~~~
840 Here is an example of detectors written for a hypothetical
841 project-management application, where users can signal approval
842 of a project by adding themselves to an "approvals" list, and
843 a project proceeds when it has three approvals::
845 # Permit users only to add themselves to the "approvals" list.
847 def check_approvals(db, cl, id, newdata):
848 if newdata.has_key("approvals"):
849 if cl.get(id, "status") == db.status.lookup("approved"):
850 raise Reject, "You can't modify the approvals list " \
851 "for a project that has already been approved."
852 old = cl.get(id, "approvals")
853 new = newdata["approvals"]
854 for uid in old:
855 if uid not in new and uid != db.getuid():
856 raise Reject, "You can't remove other users from the "
857 "approvals list; you can only remove yourself."
858 for uid in new:
859 if uid not in old and uid != db.getuid():
860 raise Reject, "You can't add other users to the approvals "
861 "list; you can only add yourself."
863 # When three people have approved a project, change its
864 # status from "pending" to "approved".
866 def approve_project(db, cl, id, olddata):
867 if olddata.has_key("approvals") and len(cl.get(id, "approvals")) == 3:
868 if cl.get(id, "status") == db.status.lookup("pending"):
869 cl.set(id, status=db.status.lookup("approved"))
871 def init(db):
872 db.project.audit("set", check_approval)
873 db.project.react("set", approve_project)
875 Here is another example of a detector that can allow or prevent
876 the creation of new nodes. In this scenario, patches for a software
877 project are submitted by sending in e-mail with an attached file,
878 and we want to ensure that there are text/plain attachments on
879 the message. The maintainer of the package can then apply the
880 patch by setting its status to "applied"::
882 # Only accept attempts to create new patches that come with patch files.
884 def check_new_patch(db, cl, id, newdata):
885 if not newdata["files"]:
886 raise Reject, "You can't submit a new patch without " \
887 "attaching a patch file."
888 for fileid in newdata["files"]:
889 if db.file.get(fileid, "type") != "text/plain":
890 raise Reject, "Submitted patch files must be text/plain."
892 # When the status is changed from "approved" to "applied", apply the patch.
894 def apply_patch(db, cl, id, olddata):
895 if cl.get(id, "status") == db.status.lookup("applied") and \
896 olddata["status"] == db.status.lookup("approved"):
897 # ...apply the patch...
899 def init(db):
900 db.patch.audit("create", check_new_patch)
901 db.patch.react("set", apply_patch)
904 Command Interface
905 -----------------
907 The command interface is a very simple and minimal interface,
908 intended only for quick searches and checks from the shell prompt.
909 (Anything more interesting can simply be written in Python using
910 the Roundup database module.)
912 Command Interface Specification
913 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
915 A single command, roundup, provides basic access to
916 the hyperdatabase from the command line::
918 roundup get [-list] designator[, designator,...] propname
919 roundup set designator[, designator,...] propname=value ...
920 roundup find [-list] classname propname=value ...
922 TODO: more stuff here
924 Property values are represented as strings in command arguments
925 and in the printed results:
927 - Strings are, well, strings.
929 - Numbers are displayed the same as strings.
931 - Booleans are displayed as 'Yes' or 'No'.
933 - Date values are printed in the full date format in the local
934 time zone, and accepted in the full format or any of the partial
935 formats explained above.
937 - Link values are printed as node designators. When given as
938 an argument, node designators and key strings are both accepted.
940 - Multilink values are printed as lists of node designators
941 joined by commas. When given as an argument, node designators
942 and key strings are both accepted; an empty string, a single node,
943 or a list of nodes joined by commas is accepted.
945 When multiple nodes are specified to the
946 roundup get or roundup set
947 commands, the specified properties are retrieved or set
948 on all the listed nodes.
950 When multiple results are returned by the roundup get
951 or roundup find commands, they are printed one per
952 line (default) or joined by commas (with the -list) option.
954 Usage Example
955 ~~~~~~~~~~~~~
957 To find all messages regarding in-progress issues that
958 contain the word "spam", for example, you could execute the
959 following command from the directory where the database
960 dumps its files::
962 shell% for issue in `roundup find issue status=in-progress`; do
963 > grep -l spam `roundup get $issue messages`
964 > done
965 msg23
966 msg49
967 msg50
968 msg61
969 shell%
971 Or, using the -list option, this can be written as a single command::
973 shell% grep -l spam `roundup get \
974 \`roundup find -list issue status=in-progress\` messages`
975 msg23
976 msg49
977 msg50
978 msg61
979 shell%
982 E-mail User Interface
983 ---------------------
985 The Roundup system must be assigned an e-mail address
986 at which to receive mail. Messages should be piped to
987 the Roundup mail-handling script by the mail delivery
988 system (e.g. using an alias beginning with "|" for sendmail).
990 Message Processing
991 ~~~~~~~~~~~~~~~~~~
993 Incoming messages are examined for multiple parts.
994 In a multipart/mixed message or part, each subpart is
995 extracted and examined. In a multipart/alternative
996 message or part, we look for a text/plain subpart and
997 ignore the other parts. The text/plain subparts are
998 assembled to form the textual body of the message, to
999 be stored in the file associated with a "msg" class node.
1000 Any parts of other types are each stored in separate
1001 files and given "file" class nodes that are linked to
1002 the "msg" node.
1004 The "summary" property on message nodes is taken from
1005 the first non-quoting section in the message body.
1006 The message body is divided into sections by blank lines.
1007 Sections where the second and all subsequent lines begin
1008 with a ">" or "|" character are considered "quoting
1009 sections". The first line of the first non-quoting
1010 section becomes the summary of the message.
1012 All of the addresses in the To: and Cc: headers of the
1013 incoming message are looked up among the user nodes, and
1014 the corresponding users are placed in the "recipients"
1015 property on the new "msg" node. The address in the From:
1016 header similarly determines the "author" property of the
1017 new "msg" node.
1018 The default handling for
1019 addresses that don't have corresponding users is to create
1020 new users with no passwords and a username equal to the
1021 address. (The web interface does not permit logins for
1022 users with no passwords.) If we prefer to reject mail from
1023 outside sources, we can simply register an auditor on the
1024 "user" class that prevents the creation of user nodes with
1025 no passwords.
1027 The subject line of the incoming message is examined to
1028 determine whether the message is an attempt to create a new
1029 issue or to discuss an existing issue. A designator enclosed
1030 in square brackets is sought as the first thing on the
1031 subject line (after skipping any "Fwd:" or "Re:" prefixes).
1033 If an issue designator (class name and id number) is found
1034 there, the newly created "msg" node is added to the "messages"
1035 property for that issue, and any new "file" nodes are added to
1036 the "files" property for the issue.
1038 If just an issue class name is found there, we attempt to
1039 create a new issue of that class with its "messages" property
1040 initialized to contain the new "msg" node and its "files"
1041 property initialized to contain any new "file" nodes.
1043 Both cases may trigger detectors (in the first case we
1044 are calling the set() method to add the message to the
1045 issue's spool; in the second case we are calling the
1046 create() method to create a new node). If an auditor
1047 raises an exception, the original message is bounced back to
1048 the sender with the explanatory message given in the exception.
1050 Nosy Lists
1051 ~~~~~~~~~~
1053 A standard detector is provided that watches for additions
1054 to the "messages" property. When a new message is added, the
1055 detector sends it to all the users on the "nosy" list for the
1056 issue that are not already on the "recipients" list of the
1057 message. Those users are then appended to the "recipients"
1058 property on the message, so multiple copies of a message
1059 are never sent to the same user. The journal recorded by
1060 the hyperdatabase on the "recipients" property then provides
1061 a log of when the message was sent to whom.
1063 Setting Properties
1064 ~~~~~~~~~~~~~~~~~~
1066 The e-mail interface also provides a simple way to set
1067 properties on issues. At the end of the subject line,
1068 ``propname=value`` pairs can be
1069 specified in square brackets, using the same conventions
1070 as for the roundup ``set`` shell command.
1073 Web User Interface
1074 ------------------
1076 The web interface is provided by a CGI script that can be
1077 run under any web server. A simple web server can easily be
1078 built on the standard CGIHTTPServer module, and
1079 should also be included in the distribution for quick
1080 out-of-the-box deployment.
1082 The user interface is constructed from a number of template
1083 files containing mostly HTML. Among the HTML tags in templates
1084 are interspersed some nonstandard tags, which we use as
1085 placeholders to be replaced by properties and their values.
1087 Views and View Specifiers
1088 ~~~~~~~~~~~~~~~~~~~~~~~~~
1090 There are two main kinds of views: *index* views and *issue* views.
1091 An index view displays a list of issues of a particular class,
1092 optionally sorted and filtered as requested. An issue view
1093 presents the properties of a particular issue for editing
1094 and displays the message spool for the issue.
1096 A view specifier is a string that specifies
1097 all the options needed to construct a particular view.
1098 It goes after the URL to the Roundup CGI script or the
1099 web server to form the complete URL to a view. When the
1100 result of selecting a link or submitting a form takes
1101 the user to a new view, the Web browser should be redirected
1102 to a canonical location containing a complete view specifier
1103 so that the view can be bookmarked.
1105 Displaying Properties
1106 ~~~~~~~~~~~~~~~~~~~~~
1108 Properties appear in the user interface in three contexts:
1109 in indices, in editors, and as filters. For each type of
1110 property, there are several display possibilities. For example,
1111 in an index view, a string property may just be printed as
1112 a plain string, but in an editor view, that property should
1113 be displayed in an editable field.
1115 The display of a property is handled by functions in
1116 a displayers module. Each function accepts at
1117 least three standard arguments -- the database, class name,
1118 and node id -- and returns a chunk of HTML.
1120 Displayer functions are triggered by <display>
1121 tags in templates. The call attribute of the tag
1122 provides a Python expression for calling the displayer
1123 function. The three standard arguments are inserted in
1124 front of the arguments given. For example, the occurrence of::
1126 <display call="plain('status', max=30)">
1128 in a template triggers a call to::
1130 plain(db, "issue", 13, "status", max=30)
1133 when displaying issue 13 in the "issue" class. The displayer
1134 functions can accept extra arguments to further specify
1135 details about the widgets that should be generated. By defining new
1136 displayer functions, the user interface can be highly customized.
1138 Some of the standard displayer functions include:
1140 ========= ====================================================================
1141 Function Description
1142 ========= ====================================================================
1143 plain display a String property directly;
1144 display a Date property in a specified time zone with an option
1145 to omit the time from the date stamp; for a Link or Multilink
1146 property, display the key strings of the linked nodes (or the
1147 ids if the linked class has no key property)
1148 field display a property like the
1149 plain displayer above, but in a text field
1150 to be edited
1151 menu for a Link property, display
1152 a menu of the available choices
1153 link for a Link or Multilink property,
1154 display the names of the linked nodes, hyperlinked to the
1155 issue views on those nodes
1156 count for a Multilink property, display
1157 a count of the number of links in the list
1158 reldate display a Date property in terms
1159 of an interval relative to the current date (e.g. "+ 3w", "- 2d").
1160 download show a Link("file") or Multilink("file")
1161 property using links that allow you to download files
1162 checklist for a Link or Multilink property,
1163 display checkboxes for the available choices to permit filtering
1164 ========= ====================================================================
1167 Index Views
1168 ~~~~~~~~~~~
1170 An index view contains two sections: a filter section
1171 and an index section.
1172 The filter section provides some widgets for selecting
1173 which issues appear in the index. The index section is
1174 a table of issues.
1176 Index View Specifiers
1177 """""""""""""""""""""
1179 An index view specifier looks like this (whitespace
1180 has been added for clarity)::
1182 /issue?status=unread,in-progress,resolved&
1183 topic=security,ui&
1184 :group=+priority&
1185 :sort=-activity&
1186 :filters=status,topic&
1187 :columns=title,status,fixer
1190 The index view is determined by two parts of the
1191 specifier: the layout part and the filter part.
1192 The layout part consists of the query parameters that
1193 begin with colons, and it determines the way that the
1194 properties of selected nodes are displayed.
1195 The filter part consists of all the other query parameters,
1196 and it determines the criteria by which nodes
1197 are selected for display.
1199 The filter part is interactively manipulated with
1200 the form widgets displayed in the filter section. The
1201 layout part is interactively manipulated by clicking
1202 on the column headings in the table.
1204 The filter part selects the union of the
1205 sets of issues with values matching any specified Link
1206 properties and the intersection of the sets
1207 of issues with values matching any specified Multilink
1208 properties.
1210 The example specifies an index of "issue" nodes.
1211 Only issues with a "status" of either
1212 "unread" or "in-progres" or "resolved" are displayed,
1213 and only issues with "topic" values including both
1214 "security" and "ui" are displayed. The issues
1215 are grouped by priority, arranged in ascending order;
1216 and within groups, sorted by activity, arranged in
1217 descending order. The filter section shows filters
1218 for the "status" and "topic" properties, and the
1219 table includes columns for the "title", "status", and
1220 "fixer" properties.
1222 Associated with each issue class is a default
1223 layout specifier. The layout specifier in the above
1224 example is the default layout to be provided with
1225 the default bug-tracker schema described above in
1226 section 4.4.
1228 Filter Section
1229 """"""""""""""
1231 The template for a filter section provides the
1232 filtering widgets at the top of the index view.
1233 Fragments enclosed in ``<property>...</property>``
1234 tags are included or omitted depending on whether the
1235 view specifier requests a filter for a particular property.
1237 Here's a simple example of a filter template::
1239 <property name=status>
1240 <display call="checklist('status')">
1241 </property>
1242 <br>
1243 <property name=priority>
1244 <display call="checklist('priority')">
1245 </property>
1246 <br>
1247 <property name=fixer>
1248 <display call="menu('fixer')">
1249 </property>
1251 Index Section
1252 """""""""""""
1254 The template for an index section describes one row of
1255 the index table.
1256 Fragments enclosed in ``<property>...</property>``
1257 tags are included or omitted depending on whether the
1258 view specifier requests a column for a particular property.
1259 The table cells should contain <display> tags
1260 to display the values of the issue's properties.
1262 Here's a simple example of an index template::
1264 <tr>
1265 <property name=title>
1266 <td><display call="plain('title', max=50)"></td>
1267 </property>
1268 <property name=status>
1269 <td><display call="plain('status')"></td>
1270 </property>
1271 <property name=fixer>
1272 <td><display call="plain('fixer')"></td>
1273 </property>
1274 </tr>
1276 Sorting
1277 """"""""""""""
1279 String and Date values are sorted in the natural way.
1280 Link properties are sorted according to the value of the
1281 "order" property on the linked nodes if it is present; or
1282 otherwise on the key string of the linked nodes; or
1283 finally on the node ids. Multilink properties are
1284 sorted according to how many links are present.
1286 Issue Views
1287 ~~~~~~~~~~~
1289 An issue view contains an editor section and a spool section.
1290 At the top of an issue view, links to superseding and superseded
1291 issues are always displayed.
1293 Issue View Specifiers
1294 """""""""""""""""""""
1296 An issue view specifier is simply the issue's designator::
1298 /patch23
1301 Editor Section
1302 """"""""""""""
1304 The editor section is generated from a template
1305 containing <display> tags to insert
1306 the appropriate widgets for editing properties.
1308 Here's an example of a basic editor template::
1310 <table>
1311 <tr>
1312 <td colspan=2>
1313 <display call="field('title', size=60)">
1314 </td>
1315 </tr>
1316 <tr>
1317 <td>
1318 <display call="field('fixer', size=30)">
1319 </td>
1320 <td>
1321 <display call="menu('status')>
1322 </td>
1323 </tr>
1324 <tr>
1325 <td>
1326 <display call="field('nosy', size=30)">
1327 </td>
1328 <td>
1329 <display call="menu('priority')>
1330 </td>
1331 </tr>
1332 <tr>
1333 <td colspan=2>
1334 <display call="note()">
1335 </td>
1336 </tr>
1337 </table>
1339 As shown in the example, the editor template can also
1340 request the display of a "note" field, which is a
1341 text area for entering a note to go along with a change.
1343 When a change is submitted, the system automatically
1344 generates a message describing the changed properties.
1345 The message displays all of the property values on the
1346 issue and indicates which ones have changed.
1347 An example of such a message might be this::
1349 title: Polly Parrot is dead
1350 priority: critical
1351 status: unread -> in-progress
1352 fixer: (none)
1353 keywords: parrot,plumage,perch,nailed,dead
1355 If a note is given in the "note" field, the note is
1356 appended to the description. The message is then added
1357 to the issue's message spool (thus triggering the standard
1358 detector to react by sending out this message to the nosy list).
1360 Spool Section
1361 """""""""""""
1363 The spool section lists messages in the issue's "messages"
1364 property. The index of messages displays the "date", "author",
1365 and "summary" properties on the message nodes, and selecting a
1366 message takes you to its content.
1369 Deployment Scenarios
1370 --------------------
1372 The design described above should be general enough
1373 to permit the use of Roundup for bug tracking, managing
1374 projects, managing patches, or holding discussions. By
1375 using nodes of multiple types, one could deploy a system
1376 that maintains requirement specifications, catalogs bugs,
1377 and manages submitted patches, where patches could be
1378 linked to the bugs and requirements they address.
1381 Acknowledgements
1382 ----------------
1384 My thanks are due to Christy Heyl for
1385 reviewing and contributing suggestions to this paper
1386 and motivating me to get it done, and to
1387 Jesse Vincent, Mark Miller, Christopher Simons,
1388 Jeff Dunmall, Wayne Gramlich, and Dean Tribble for
1389 their assistance with the first-round submission.
1391 Changes to this document
1392 ------------------------
1394 - Added Boolean and Number types
1395 - Added section Hyperdatabase Implementations
1396 - "Item" has been renamed to "Issue" to account for the more specific nature
1397 of the Class.