======================================================= Roundup: an Issue-Tracking System for Knowledge Workers ======================================================= :Authors: Ka-Ping Yee (original_), Richard Jones (implementation) .. _original: original_overview.html .. contents:: Introduction ============ Roundup is an issue-tracking system called which will manage a number of issues (with properties such as "description", "priority", and so on) and provides the ability to: (a) submit new issues, (b) find and edit existing issues, and (c) discuss issues with other participants. Roundup facilitates communication among the participants by managing discussions and notifying interested parties when issues are edited. Background ---------- A typical software project requires the management of many tasks, usually distributed among several collaborators. In fact, any project team could use a tool for sorting out and discussing all the relevant issues. A common approach is to set up some kind of "to-do" list that people can share. However, to address the overall problem we need much more than just a shared to-do list; we need to manage a growing body of knowledge and experience to help a team collaborate effectively on a project. The issue-tracking tool becomes a nexus for communication: the Grand Central Station of the group intelligence. The primary focus of this design is to help developers work together well, not to provide a customer service interface to the developers. This is not to say that the design is to be made unsuitable for customers to use. Rather, it is assumed that many of the same qualities that are good for supporting development (see below) are also good for non-developers using the system. Additional niceties for providing a safe or simplified interface to clients are intentionally deferred for later consideration. A good issue-tracking system should have at least the following properties: **Low barrier to participation** The usefulness of the tool depends entirely on the information people contribute to it. It must be made as easy as possible to submit new issues and contribute information about existing issues. **Straightforward navigation** It should be easy for users to extract information they need from the system to direct their decisions and tasks. They should be able to get a decent overview of things as well as finding specific information when they know what they're after. **Controlled information flow** The users must have control over how much information and what information they get. A common flaw of some issue-tracking systems is that they inundate users with so much useless e-mail that people avoid the system altogether. With a nod to the time-honoured computer science tradition of "filling in the fourth quadrant", we note that there are really four kinds of information flow going on here. The three mentioned qualities really address the first three quadrants of this 2-by-2 matrix, respectively: 1. User push: a user submits information to the system. 2. User pull: a user queries for information from the system. 3. System push: the system sends information out to users. 4. System pull: the system solicits information from users. An example of the fourth kind of flow is voting. Voting isn't described in this design, but it should be noted as a potential enhancement. Guiding Principles ------------------ **Simplicity** It is a strong requirement that the tool be accessible and understandable. It should be fairly obvious what different parts of the interface do, and the inner mechanisms should operate in ways that most users can easily predict. **Efficiency** We aim to optimize for minimum effort to do the most common operations, and best use of resources like screen real estate to maximize the amount of information that we summarize and present. **Generality** We try to avoid making unnecessary assumptions that would restrict the applicability of the tool. For example, there is no reason why one might not also want to use this tool to manage a design process, non-software projects, or organizational decisions. **Persistence** We prefer hiding or reclassifying information to deleting it. This helps support the collection of statistics later. If records are never destroyed, there is little danger in providing access to a larger community, and logging yields accountability, which may encourage better behaviour. Data Model ========== Roundup stores a number of *items*, each of which can have several properties and an associated discussion. The properties can be used to classify or search for items. The discussion is a sequence of e-mail messages. Each item is identified by a unique number, and has an activity log which records the time and content of edits made on its properties. The log stays fairly small since the design intentionally provides only small data types as item properties, and encourages anything large to be attached to e-mail where it becomes part of the discussion. The next section explains how items are organized. The Hyperdatabase ----------------- Often when classifying information we are asked to select exactly one of a number of categories or to fit it into a rigid hierarchy. Yet things only sometimes fall into one category; often, a piece of information may be related to several concepts. For example, forcing each item into a single keyword category is not just suboptimal but counterproductive: seekers of that item may expect to find it in a different category and conclude that the item is not present in the database -- which has them *worse* off than if the items were not categorized at all. Some systems try to alleviate this problem by allowing items to appear at multiple locations in a tree, as with "aliases" or "symbolic links" in a filesystem, for example. This does help somewhat, but we want to be even more flexible by allowing the organization of items into sets that may freely intersect. Rather than putting each item at exactly one place in an overall "grand scheme", a item can belong to as many sets as are appropriate. If we choose to represent the sets themselves as items and set membership as a link between items, we're now ready to present the definition of a hyperdatabase. A *hyperdatabase* is a collection of *items* that may be hyperlinked to each other (hence the name "hyperdatabase"). Each item carries a collection of key-value pairs, where some of the values may be links to other items. Any item may have an arbitrary number of outgoing and incoming links. Hyperdatabases are able to efficiently answer queries such as "what items link to this item?" and "what items does this item link to?" Rationale --------- There are several reasons for building our own kind of database for Roundup rather than using an existing one. Requiring the installation of a full-blown third-party SQL database system would probably deter many potential users from attempting to set up Roundup; yet a real relational database would be too complicated to implement on our own. On the other hand, a hyperdatabase can be implemented fairly easily using one of the Python DBM modules, so we can take the "batteries-included" approach and provide it as part of the system. It's easier to build and understand than a true relational database (in accordance with our guiding principle of *simplicity*), but provides most of the query functionality we want. A hyperdatabase is well suited for finding the intersection of a number of sets in which items belong. We expect that most of the queries people want to do will be of this form, rather than complicated SQL queries. For example, a typical request might be "show me all critical items related to security". The ability to store arbitrary key-value pairs and links on items gives it more flexibility than an RDBMS. Users are not going to be making thousands of queries per second, so it makes sense to optimize for simplicity and flexibility rather than performance. .. img: images/hyperdb.png Roundup's Hyperdatabase ----------------------- For our application, we store each item as a item in a hyperdatabase. The item's properties are stored as key-value pairs on its item. Several types of properties are allowed: *string*, *number*, *boolean*, *date*, *interval, *link*, and *multlink*. Another type, *password*, is a special type of string and it's only used internally to Roundup. The *string* type is for short, free-form strings. String properties are not intended to contain large amounts of text, and it is recommended that they be presented as one-line fields to encourage brevity. A *number* is a special type of string that represents a numeric value. A *boolean* is further constrained to be a *true* or *false* value. The *date* type is for calendar dates and times. An *interval* is the time between two dates. The *link* type denotes a single selection from a number of options. A *link* property entails a link from the item possessing the property to the item representing the chosen option. The *multilink* type is for a list of links to any number of other items in the in the database. A *multilink* property, for example, can be used to refer to related items or keyword categories relevant to an item. For Roundup, all items have four properties that are not customizable: 1. a *date* property named **creation** 2. a *link* property named **creator** 3. a *date* property named **activity** These properties represent the date of the creation of the item, who created it, and when the last change was made. Further, all *issue* items have an additional four properties: 1. a *string* property named **title** 2. a *multilink* property named **nosy** 3. a *multilink* property named **messages** 4. a *multilink* property named **files** 5. a *multilink* property named **superseder** The **title** property is a short one-line description of the item. The detailed description can go in the first e-mail message of the item's messages spool. The **nosy** property contains a list of the people who are interested in an item. This mechanism is explained in the section on `Submission and Discussion`_. Each message added to the item goes in the **messages** spool - any attached files go in the **files** spool. The **superseder** property is used to support the splitting, joining, or replacing of items. When several items need to be joined into a single item, all the old items link to the new item in their **superseder** property. When an item needs to be split apart, the item references all the new items in its **superseder** propety. We can easily list all active items just by checking for an empty **superseder** property, and trace the path of an item's origins by querying the hyperdatabase for links. Users of the system are also represented by items in the hyperdatabase, containing properties like the user's e-mail address, login name, and password. The Default Schema ------------------ It is hoped that the hyperdatabase together with the specializations mentioned above for Roundup will be applicable in a variety of situations (in accordance with our guiding principle of *generality*). To address the problem at hand, we need a specific schema for items applied particularly to software development. Again, we are trying to keep the schema simple: too many options make it tougher for someone to make a good choice:: # IssueClass automatically gets these properties: # title = String() # messages = Multilink("msg") # files = Multilink("file") # nosy = Multilink("user") # superseder = Multilink("issue") # (it also gets the Class properties creation, activity and creator) issue = IssueClass(db, "issue", assignedto=Link("user"), keyword=Multilink("keyword"), priority=Link("priority"), status=Link("status")) The **assignedto** property assigns responsibility for an item to a person or a list of people. The **keyword** property places the item in an arbitrary number of relevant keyword sets (see the section on `Browsing and Searching`_). The **prority** and **status** values are initially: =========== ===================================== Priority Description =========== ===================================== "critical" panic: work is stopped! "urgent" important, but not deadly "bug" lost work or incorrect results "feature" want missing functionality "wish" avoidable bugs, missing conveniences =========== ===================================== ============= ===================================== Status Description ============= ===================================== "unread" submitted but no action yet "deferred" intentionally set aside "chatting" under review or seeking clarification "need-eg" need a reproducible example of a bug "in-progress" understood; development in progress "testing" we think it's done; others, please test "done-cbb" okay for now, but could be better "resolved" fix has been released ============= ===================================== As previously mentioned, each item gets an activity log. Whenever a property on an item is changed, the log records the time of the change, the user making the change, and the old and new values of the property. This permits the later gathering of statistics (for example, the average time from submission to resolution). We do not specify or enforce a state transition graph, since making the system rigid in that fashion is probably more trouble than it's worth. Experience has shown that there are probably two convenient automatic state transitions: 1. from **unread** to **chatting** when e-mail is written about an item 2. from **testing** to **resolved** when a new release of the software is made Beyond these, in accordance with our principle of *generality*, we allow access to the hyperdatabase API so that scripts can automate transitions themselves or be triggered by changes in the database. User Interface ============== Roundup provides its services through two main interfaces: e-mail and the Web. This division is chosen to optimize the most common tasks. E-mail is best suited for the submission of new items since most people are most comfortable with composing long messages in their own favourite e-mail client. E-mail also permits them to mention URLs or attach files relevant to their submission. Indeed, in many cases people are already used to making requests by sending e-mail to a mailing list of people; they can do exactly the same thing to use Roundup without even thinking about it. Similarly, people are already familiar with holding discussions in e-mail, and plenty of valuable usage conventions and software tools already exist for that medium. The Web, on the other hand, is best suited for summarizing and seeking information, because it can present an interactive overview of items. Since the Web has forms, it's also the best place to edit items. Submission and Discussion ------------------------- The system needs an address for receiving mail and an address that forwards mail to all participants. Each item has its own list of interested parties, known as its *nosy list*. Here's how nosy lists work: 1. New items are always submitted by sending an e-mail message to Roundup. The "Subject:" field becomes the description of the new item. The message is saved in the mail spool of the new item, and copied to the list of all participants so everyone knows that a new item has been added. The new item's nosy list initially contains the submitter. 2. All e-mail messages sent by Roundup have their "Reply-To:" field set to Roundup's address, and have the item's number in the "Subject:" field. Thus, any replies to the initial announcement and subsequent threads are all received by Roundup. Roundup notes the item number in the "Subject:" field of each incoming message and appends the message to the appropriate spool. 3. Any incoming e-mail tagged with an item number is copied to all the people on the item's nosy list, and any users found in the "From:", "To:", or "Cc:" fields are automatically added to the nosy list. Whenever a user edits an item's properties in the Web interface, they are also added to the nosy list. The effect is like each item having its own little mailing list, except that no one ever has to worry about subscribing to anything. Indicating interest in an issue is sufficient, and if you want to bring someone new into the conversation, all you need to do is "Cc:" a message to them. It turns out that no one ever has to worry about unsubscribing, either: the nosy lists are so specific in scope that the conversation tends to die down by itself when the issue is resolved or people no longer find it sufficiently important. Each nosy list is like an asynchronous chat room, lasting only a short time (typically five or ten messages) and involving a small group of people. However, that group is the *right* group of people: only those who express interest in an item in some way ever end up on the list, so no one gets spammed with mail they don't care about, and no one who *wants* to see mail about a particular item needs to be left out, for they can easily join in, and just as easily look at the mail spool on an item to catch up on any messages they might have missed. We can take this a step further and permit users to monitor particular keywords or classifications of items by allowing other kinds of items to also have their own nosy lists. For example, a manager could be on the nosy list of the priority value item for "critical", or a developer could be on the nosy list of the keyword value item for "security". The recipients are then determined by the union of the nosy lists on the item and all the items it links to. Using many small, specific mailing lists results in much more effective communication than one big list. Taking away the effort of subscribing and unsubscribing gives these lists the "feel" of being cheap and disposable. The transparent capture of the mail spool attached to each issue also yields a nice knowledge repository over time. Editing ------- Since Roundup is intended to support arbitrary user-defined schema for item properties, the editing interface must be automatically generated from the schema. The configuration for Roundup will include a template describing how to lay out the properties to present a UI for inspecting and editing items. For example::