bugger, dropping support for "+" special char

[roundup.git] / doc / customizing.txt

===================
Customising Roundup
===================

:Version: $Revision: 1.74 $

.. This document borrows from the ZopeBook section on ZPT. The original is at:
   http://www.zope.org/Documentation/Books/ZopeBook/current/ZPT.stx

.. contents::
   :depth: 1

What You Can Do
===============

Before you get too far, it's probably worth having a quick read of the Roundup
`design documentation`_.

Customisation of Roundup can take one of five forms:

1. `tracker configuration`_ file changes
2. database, or `tracker schema`_ changes
3. "definition" class `database content`_ changes
4. behavioural changes, through detectors_
5. `access controls`_
6. change the `web interface`_

The third case is special because it takes two distinctly different forms
depending upon whether the tracker has been initialised or not. The other two
may be done at any time, before or after tracker initialisation. Yes, this
includes adding or removing properties from classes.


Trackers in a Nutshell
======================

Trackers have the following structure:

=================== ========================================================
Tracker File        Description
=================== ========================================================
config.py           Holds the basic `tracker configuration`_                 
dbinit.py           Holds the `tracker schema`_                              
interfaces.py       Defines the Web and E-Mail interfaces for the tracker    
select_db.py        Selects the database back-end for the tracker            
db/                 Holds the tracker's database                             
db/files/           Holds the tracker's upload files and messages            
detectors/          Auditors and reactors for this tracker                   
html/               Web interface templates, images and style sheets         
=================== ======================================================== 

Tracker Configuration
=====================

The config.py located in your tracker home contains the basic configuration
for the web and e-mail components of roundup's interfaces. As the name
suggests, this file is a Python module. This means that any valid python
expression may be used in the file. Mostly though, you'll be setting the
configuration variables to string values. Python string values must be quoted
with either single or double quotes::

   'this is a string'
   "this is also a string - use it when you have a 'single quote' in the value"
   this is not a string - it's not quoted

Python strings may use formatting that's almost identical to C string
formatting. The ``%`` operator is used to perform the formatting, like so::

    'roundup-admin@%s'%MAIL_DOMAIN

this will create a string ``'roundup-admin@tracker.domain.example'`` if
MAIL_DOMAIN is set to ``'tracker.domain.example'``.

You'll also note some values are set to::

   os.path.join(TRACKER_HOME, 'db')

or similar. This creates a new string which holds the path to the "db"
directory in the TRACKER_HOME directory. This is just a convenience so if the
TRACKER_HOME changes you don't have to edit multiple valoues.

The configuration variables available are:

**TRACKER_HOME** - ``os.path.split(__file__)[0]``
 The tracker home directory. The above default code will automatically
 determine the tracker home for you, so you can just leave it alone.

**MAILHOST** - ``'localhost'``
 The SMTP mail host that roundup will use to send e-mail.

**MAIL_DOMAIN** - ``'tracker.domain.example'``
 The domain name used for email addresses.

**DATABASE** - ``os.path.join(TRACKER_HOME, 'db')``
 This is the directory that the database is going to be stored in. By default
 it is in the tracker home.

**TEMPLATES** - ``os.path.join(TRACKER_HOME, 'html')``
 This is the directory that the HTML templates reside in. By default they are
 in the tracker home.

**TRACKER_NAME** - ``'Roundup issue tracker'``
 A descriptive name for your roundup tracker. This is sent out in e-mails and
 appears in the heading of CGI pages.

**TRACKER_EMAIL** - ``'issue_tracker@%s'%MAIL_DOMAIN``
 The email address that e-mail sent to roundup should go to. Think of it as the
 tracker's personal e-mail address.

**TRACKER_WEB** - ``'http://tracker.example/cgi-bin/roundup.cgi/bugs/'``
 The web address that the tracker is viewable at. This will be included in
 information sent to users of the tracker. The URL **must** include the
 cgi-bin part or anything else that is required to get to the home page of
 the tracker. You **must** include a trailing '/' in the URL.

**ADMIN_EMAIL** - ``'roundup-admin@%s'%MAIL_DOMAIN``
 The email address that roundup will complain to if it runs into trouble.

**EMAIL_FROM_TAG** - ``''``
 Additional text to include in the "name" part of the ``From:`` address used
 in nosy messages. If the sending user is "Foo Bar", the ``From:`` line is
 usually::
     "Foo Bar" <issue_tracker@tracker.example>

 the EMAIL_FROM_TAG goes inside the "Foo Bar" quotes like so::

    "Foo Bar EMAIL_FROM_TAG" <issue_tracker@tracker.example>

**MESSAGES_TO_AUTHOR** - ``'yes'`` or``'no'``
 Send nosy messages to the author of the message.

**ADD_AUTHOR_TO_NOSY** - ``'new'``, ``'yes'`` or ``'no'``
 Does the author of a message get placed on the nosy list automatically?
 If ``'new'`` is used, then the author will only be added when a message
 creates a new issue. If ``'yes'``, then the author will be added on followups
 too. If ``'no'``, they're never added to the nosy.

**ADD_RECIPIENTS_TO_NOSY** - ``'new'``, ``'yes'`` or ``'no'``
 Do the recipients (To:, Cc:) of a message get placed on the nosy list?
 If ``'new'`` is used, then the recipients will only be added when a message
 creates a new issue. If ``'yes'``, then the recipients will be added on
 followups too. If ``'no'``, they're never added to the nosy.

**EMAIL_SIGNATURE_POSITION** - ``'top'``, ``'bottom'`` or ``'none'``
 Where to place the email signature in messages that Roundup generates.

**EMAIL_KEEP_QUOTED_TEXT** - ``'yes'`` or ``'no'``
 Keep email citations. Citations are the part of e-mail which the sender has
 quoted in their reply to previous e-mail.

**EMAIL_LEAVE_BODY_UNCHANGED** - ``'no'``
 Preserve the email body as is. Enabiling this will cause the entire message
 body to be stored, including all citations and signatures. It should be
 either ``'yes'`` or ``'no'``.

**MAIL_DEFAULT_CLASS** - ``'issue'`` or ``''``
 Default class to use in the mailgw if one isn't supplied in email
 subjects. To disable, comment out the variable below or leave it blank.

The default config.py is given below - as you
can see, the MAIL_DOMAIN must be edited before any interaction with the
tracker is attempted.::

    # roundup home is this package's directory
    TRACKER_HOME=os.path.split(__file__)[0]

    # The SMTP mail host that roundup will use to send mail
    MAILHOST = 'localhost'

    # The domain name used for email addresses.
    MAIL_DOMAIN = 'your.tracker.email.domain.example'

    # This is the directory that the database is going to be stored in
    DATABASE = os.path.join(TRACKER_HOME, 'db')

    # This is the directory that the HTML templates reside in
    TEMPLATES = os.path.join(TRACKER_HOME, 'html')

    # A descriptive name for your roundup tracker
    TRACKER_NAME = 'Roundup issue tracker'

    # The email address that mail to roundup should go to
    TRACKER_EMAIL = 'issue_tracker@%s'%MAIL_DOMAIN

    # The web address that the tracker is viewable at. This will be included in
    # information sent to users of the tracker. The URL MUST include the cgi-bin
    # part or anything else that is required to get to the home page of the
    # tracker. You MUST include a trailing '/' in the URL.
    TRACKER_WEB = 'http://tracker.example/cgi-bin/roundup.cgi/bugs/'

    # The email address that roundup will complain to if it runs into trouble
    ADMIN_EMAIL = 'roundup-admin@%s'%MAIL_DOMAIN

    # Additional text to include in the "name" part of the From: address used
    # in nosy messages. If the sending user is "Foo Bar", the From: line is
    # usually: "Foo Bar" <issue_tracker@tracker.example>
    # the EMAIL_FROM_TAG goes inside the "Foo Bar" quotes like so:
    #    "Foo Bar EMAIL_FROM_TAG" <issue_tracker@tracker.example>
    EMAIL_FROM_TAG = ""

    # Send nosy messages to the author of the message
    MESSAGES_TO_AUTHOR = 'no'           # either 'yes' or 'no'

    # Does the author of a message get placed on the nosy list automatically?
    # If 'new' is used, then the author will only be added when a message
    # creates a new issue. If 'yes', then the author will be added on followups
    # too. If 'no', they're never added to the nosy.
    ADD_AUTHOR_TO_NOSY = 'new'          # one of 'yes', 'no', 'new'

    # Do the recipients (To:, Cc:) of a message get placed on the nosy list?
    # If 'new' is used, then the recipients will only be added when a message
    # creates a new issue. If 'yes', then the recipients will be added on followups
    # too. If 'no', they're never added to the nosy.
    ADD_RECIPIENTS_TO_NOSY = 'new'      # either 'yes', 'no', 'new'

    # Where to place the email signature
    EMAIL_SIGNATURE_POSITION = 'bottom' # one of 'top', 'bottom', 'none'

    # Keep email citations
    EMAIL_KEEP_QUOTED_TEXT = 'no'       # either 'yes' or 'no'

    # Preserve the email body as is
    EMAIL_LEAVE_BODY_UNCHANGED = 'no'   # either 'yes' or 'no'

    # Default class to use in the mailgw if one isn't supplied in email
    # subjects. To disable, comment out the variable below or leave it blank.
    # Examples:
    MAIL_DEFAULT_CLASS = 'issue'   # use "issue" class by default
    #MAIL_DEFAULT_CLASS = ''        # disable (or just comment the var out)

    # 
    # SECURITY DEFINITIONS
    #
    # define the Roles that a user gets when they register with the tracker
    # these are a comma-separated string of role names (e.g. 'Admin,User')
    NEW_WEB_USER_ROLES = 'User'
    NEW_EMAIL_USER_ROLES = 'User'

Tracker Schema
==============

Note: if you modify the schema, you'll most likely need to edit the
      `web interface`_ HTML template files and `detectors`_ to reflect
      your changes.

A tracker schema defines what data is stored in the tracker's database.
Schemas are defined using Python code in the ``dbinit.py`` module of your
tracker. The "classic" schema looks like this::

    pri = Class(db, "priority", name=String(), order=String())
    pri.setkey("name")

    stat = Class(db, "status", name=String(), order=String())
    stat.setkey("name")

    keyword = Class(db, "keyword", name=String())
    keyword.setkey("name")

    user = Class(db, "user", username=String(), organisation=String(),
        password=String(), address=String(), realname=String(), phone=String())
    user.setkey("username")

    msg = FileClass(db, "msg", author=Link("user"), summary=String(),
        date=Date(), recipients=Multilink("user"), files=Multilink("file"))

    file = FileClass(db, "file", name=String(), type=String())

    issue = IssueClass(db, "issue", topic=Multilink("keyword"),
        status=Link("status"), assignedto=Link("user"),
        priority=Link("priority"))
    issue.setkey('title')

Classes and Properties - creating a new information store
---------------------------------------------------------

In the tracker above, we've defined 7 classes of information:

  priority
      Defines the possible levels of urgency for issues.

  status
      Defines the possible states of processing the issue may be in.

  keyword
      Initially empty, will hold keywords useful for searching issues.

  user
      Initially holding the "admin" user, will eventually have an entry for all
      users using roundup.

  msg
      Initially empty, will all e-mail messages sent to or generated by
      roundup.

  file
      Initially empty, will all files attached to issues.

  issue
      Initially empty, this is where the issue information is stored.

We define the "priority" and "status" classes to allow two things: reduction in
the amount of information stored on the issue and more powerful, accurate
searching of issues by priority and status. By only requiring a link on the
issue (which is stored as a single number) we reduce the chance that someone
mis-types a priority or status - or simply makes a new one up.

Class and Items
~~~~~~~~~~~~~~~

A Class defines a particular class (or type) of data that will be stored in the
database. A class comprises one or more properties, which given the information
about the class items.
The actual data entered into the database, using class.create() are called
items. They have a special immutable property called id. We sometimes refer to
this as the itemid.

Properties
~~~~~~~~~~

A Class is comprised of one or more properties of the following types:

* String properties are for storing arbitrary-length strings.
* Password properties are for storing encoded arbitrary-length strings. The
  default encoding is defined on the roundup.password.Password class.
* Date properties store date-and-time stamps. Their values are Timestamp
  objects.
* Number properties store numeric values.
* Boolean properties store on/off, yes/no, true/false values.
* A Link property refers to a single other item selected from a specified
  class. The class is part of the property; the value is an integer, the id
  of the chosen item.
* A Multilink property refers to possibly many items in a specified class.
  The value is a list of integers.

FileClass
~~~~~~~~~

FileClasses save their "content" attribute off in a separate file from the rest
of the database. This reduces the number of large entries in the database,
which generally makes databases more efficient, and also allows us to use
command-line tools to operate on the files. They are stored in the files sub-
directory of the db directory in your tracker.

IssueClass
~~~~~~~~~~

IssueClasses automatically include the "messages", "files", "nosy", and
"superseder" properties.
The messages and files properties list the links to the messages and files
related to the issue. The nosy property is a list of links to users who wish to
be informed of changes to the issue - they get "CC'ed" e-mails when messages
are sent to or generated by the issue. The nosy reactor (in the detectors
directory) handles this action. The superceder link indicates an issue which
has superceded this one.
They also have the dynamically generated "creation", "activity" and "creator"
properties.
The value of the "creation" property is the date when an item was created, and
the value of the "activity" property is the date when any property on the item
was last edited (equivalently, these are the dates on the first and last
records in the item's journal). The "creator" property holds a link to the user
that created the issue.

setkey(property)
~~~~~~~~~~~~~~~~

Select a String property of the class to be the key property. The key property
muse be unique, and allows references to the items in the class by the content
of the key property. That is, we can refer to users by their username, e.g.
let's say that there's an issue in roundup, issue 23. There's also a user,
richard who happens to be user 2. To assign an issue to him, we could do either
of::

     roundup-admin set issue assignedto=2

or::

     roundup-admin set issue assignedto=richard

Note, the same thing can be done in the web and e-mail interfaces.

create(information)
~~~~~~~~~~~~~~~~~~~

Create an item in the database. This is generally used to create items in the
"definitional" classes like "priority" and "status".


Examples of adding to your schema
---------------------------------

TODO


Detectors - adding behaviour to your tracker
============================================
.. _detectors:

Detectors are initialised every time you open your tracker database, so you're
free to add and remove them any time, even after the database is initliased
via the "roundup-admin initalise" command.

The detectors in your tracker fire before (*auditors*) and after (*reactors*)
changes to the contents of your database. They are Python modules that sit in
your tracker's ``detectors`` directory. You will have some installed by
default - have a look. You can write new detectors or modify the existing
ones. The existing detectors installed for you are:

**nosyreaction.py**
  This provides the automatic nosy list maintenance and email sending. The nosy
  reactor (``nosyreaction``) fires when new messages are added to issues.
  The nosy auditor (``updatenosy``) fires when issues are changed and figures
  what changes need to be made to the nosy list (like adding new authors etc)
**statusauditor.py**
  This provides the ``chatty`` auditor which changes the issue status from
  ``unread`` or ``closed`` to ``chatting`` if new messages appear. It also
  provides the ``presetunread`` auditor which pre-sets the status to
  ``unread`` on new items if the status isn't explicitly defined.

See the detectors section in the `design document`__ for details of the
interface for detectors.

__ design.html

Sample additional detectors that have been found useful will appear in the
``detectors`` directory of the Roundup distribution:

**newissuecopy.py**
  This detector sends an email to a team address whenever a new issue is
  created. The address is hard-coded into the detector, so edit it before you
  use it (look for the text 'team@team.host') or you'll get email errors!

  The detector code::

    from roundup import roundupdb

    def newissuecopy(db, cl, nodeid, oldvalues):
        ''' Copy a message about new issues to a team address.
        '''
        # so use all the messages in the create
        change_note = cl.generateCreateNote(nodeid)

        # send a copy to the nosy list
        for msgid in cl.get(nodeid, 'messages'):
            try:
                # note: last arg must be a list
                cl.send_message(nodeid, msgid, change_note, ['team@team.host'])
            except roundupdb.MessageSendError, message:
                raise roundupdb.DetectorError, message

    def init(db):
        db.issue.react('create', newissuecopy)


Database Content
================

Note: if you modify the content of definitional classes, you'll most likely
       need to edit the tracker `detectors`_ to reflect your changes.

Customisation of the special "definitional" classes (eg. status, priority,
resolution, ...) may be done either before or after the tracker is
initialised. The actual method of doing so is completely different in each
case though, so be careful to use the right one.

**Changing content before tracker initialisation**
    Edit the dbinit module in your tracker to alter the items created in using
    the create() methods.

**Changing content after tracker initialisation**
    As the "admin" user, click on the "class list" link in the web interface
    to bring up a list of all database classes. Click on the name of the class
    you wish to change the content of.

    You may also use the roundup-admin interface's create, set and retire
    methods to add, alter or remove items from the classes in question.

See "`adding a new field to the classic schema`_" for an example that requires
database content changes.


Access Controls
===============

A set of Permissions are built in to the security module by default:

- Edit (everything)
- View (everything)

The default interfaces define:

- Web Registration
- Web Access
- Web Roles
- Email Registration
- Email Access

These are hooked into the default Roles:

- Admin (Edit everything, View everything, Web Roles)
- User (Web Access, Email Access)
- Anonymous (Web Registration, Email Registration)

And finally, the "admin" user gets the "Admin" Role, and the "anonymous" user
gets the "Anonymous" assigned when the database is initialised on installation.
The two default schemas then define:

- Edit issue, View issue (both)
- Edit file, View file (both)
- Edit msg, View msg (both)
- Edit support, View support (extended only)

and assign those Permissions to the "User" Role. Put together, these settings
appear in the ``open()`` function of the tracker ``dbinit.py`` (the following
is taken from the "minimal" template ``dbinit.py``)::

    #
    # SECURITY SETTINGS
    #
    # new permissions for this schema
    for cl in ('user', ):
        db.security.addPermission(name="Edit", klass=cl,
            description="User is allowed to edit "+cl)
        db.security.addPermission(name="View", klass=cl,
            description="User is allowed to access "+cl)

    # and give the regular users access to the web and email interface
    p = db.security.getPermission('Web Access')
    db.security.addPermissionToRole('User', p)
    p = db.security.getPermission('Email Access')
    db.security.addPermissionToRole('User', p)

    # May users view other user information? Comment these lines out
    # if you don't want them to
    p = db.security.getPermission('View', 'user')
    db.security.addPermissionToRole('User', p)

    # Assign the appropriate permissions to the anonymous user's Anonymous
    # Role. Choices here are:
    # - Allow anonymous users to register through the web
    p = db.security.getPermission('Web Registration')
    db.security.addPermissionToRole('Anonymous', p)
    # - Allow anonymous (new) users to register through the email gateway
    p = db.security.getPermission('Email Registration')
    db.security.addPermissionToRole('Anonymous', p)


New User Roles
--------------

New users are assigned the Roles defined in the config file as:

- NEW_WEB_USER_ROLES
- NEW_EMAIL_USER_ROLES


Changing Access Controls
------------------------

You may alter the configuration variables to change the Role that new web or
email users get, for example to not give them access to the web interface if
they register through email. 

You may use the ``roundup-admin`` "``security``" command to display the
current Role and Permission configuration in your tracker.

Adding a new Permission
~~~~~~~~~~~~~~~~~~~~~~~

When adding a new Permission, you will need to:

1. add it to your tracker's dbinit so it is created
2. enable it for the Roles that should have it (verify with
   "``roundup-admin security``")
3. add it to the relevant HTML interface templates
4. add it to the appropriate xxxPermission methods on in your tracker
   interfaces module

Example Scenarios
~~~~~~~~~~~~~~~~~

**automatic registration of users in the e-mail gateway**
 By giving the "anonymous" user the "Email Registration" Role, any
 unidentified user will automatically be registered with the tracker (with
 no password, so they won't be able to log in through the web until an admin
 sets them a password). Note: this is the default behaviour in the tracker
 templates that ship with Roundup.

**anonymous access through the e-mail gateway**
 Give the "anonymous" user the "Email Access" and ("Edit", "issue") Roles
 but not giving them the "Email Registration" Role. This means that when an
 unknown user sends email into the tracker, they're automatically logged in
 as "anonymous". Since they don't have the "Email Registration" Role, they
 won't be automatically registered, but since "anonymous" has permission
 to use the gateway, they'll still be able to submit issues. Note that the
 Sender information - their email address - will not be available - they're
 *anonymous*.

**only developers may be assigned issues**
 Create a new Permission called "Fixer" for the "issue" class. Create a new
 Role "Developer" which has that Permission, and assign that to the
 appropriate users. Filter the list of users available in the assignedto
 list to include only those users. Enforce the Permission with an auditor. See
 the example `restricting the list of users that are assignable to a task`_.

**only managers may sign off issues as complete**
 Create a new Permission called "Closer" for the "issue" class. Create a new
 Role "Manager" which has that Permission, and assign that to the appropriate
 users. In your web interface, only display the "resolved" issue state option
 when the user has the "Closer" Permissions. Enforce the Permission with
 an auditor. This is very similar to the previous example, except that the
 web interface check would look like::

   <option tal:condition="python:request.user.hasPermission('Closer')"
           value="resolved">Resolved</option>
 
**don't give users who register through email web access**
 Create a new Role called "Email User" which has all the Permissions of the
 normal "User" Role minus the "Web Access" Permission. This will allow users
 to send in emails to the tracker, but not access the web interface.

**let some users edit the details of all users**
 Create a new Role called "User Admin" which has the Permission for editing
 users::

    db.security.addRole(name='User Admin', description='Managing users')
    p = db.security.getPermission('Edit', 'user')
    db.security.addPermissionToRole('User Admin', p)

 and assign the Role to the users who need the permission.


Web Interface
=============

.. contents::
   :local:
   :depth: 1

The web is provided by the roundup.cgi.client module and is used by
roundup.cgi, roundup-server and ZRoundup.
In all cases, we determine which tracker is being accessed
(the first part of the URL path inside the scope of the CGI handler) and pass
control on to the tracker interfaces.Client class - which uses the Client class
from roundup.cgi.client - which handles the rest of
the access through its main() method. This means that you can do pretty much
anything you want as a web interface to your tracker.

Repurcussions of changing the tracker schema
---------------------------------------------

If you choose to change the `tracker schema`_ you will need to ensure the web
interface knows about it:

1. Index, item and search pages for the relevant classes may need to have
   properties added or removed,
2. The "page" template may require links to be changed, as might the "home"
   page's content arguments.

How requests are processed
--------------------------

The basic processing of a web request proceeds as follows:

1. figure out who we are, defaulting to the "anonymous" user
2. figure out what the request is for - we call this the "context"
3. handle any requested action (item edit, search, ...)
4. render the template requested by the context, resulting in HTML output

In some situations, exceptions occur:

- HTTP Redirect  (generally raised by an action)
- SendFile       (generally raised by determine_context)
  here we serve up a FileClass "content" property
- SendStaticFile (generally raised by determine_context)
  here we serve up a file from the tracker "html" directory
- Unauthorised   (generally raised by an action)
  here the action is cancelled, the request is rendered and an error
  message is displayed indicating that permission was not
  granted for the action to take place
- NotFound       (raised wherever it needs to be)
  this exception percolates up to the CGI interface that called the client

Determining web context
-----------------------

To determine the "context" of a request, we look at the URL and the special
request variable ``:template``. The URL path after the tracker identifier
is examined. Typical URL paths look like:

1.  ``/tracker/issue``
2.  ``/tracker/issue1``
3.  ``/tracker/_file/style.css``
4.  ``/cgi-bin/roundup.cgi/tracker/file1``
5.  ``/cgi-bin/roundup.cgi/tracker/file1/kitten.png``

where the "tracker identifier" is "tracker" in the above cases. That means
we're looking at "issue", "issue1", "_file/style.css", "file1" and
"file1/kitten.png" in the cases above. The path is generally only one
entry long - longer paths are handled differently.

a. if there is no path, then we are in the "home" context.
b. if the path starts with "_file" (as in example 3,
   "/tracker/_file/style.css"), then the additional path entry,
   "style.css" specifies the filename of a static file we're to serve up
   from the tracker "html" directory. Raises a SendStaticFile
   exception.
c. if there is something in the path (as in example 1, "issue"), it identifies
   the tracker class we're to display.
d. if the path is an item designator (as in examples 2 and 4, "issue1" and
   "file1"), then we're to display a specific item.
e. if the path starts with an item designator and is longer than
   one entry (as in example 5, "file1/kitten.png"), then we're assumed
   to be handling an item of a
   FileClass, and the extra path information gives the filename
   that the client is going to label the download with (ie
   "file1/kitten.png" is nicer to download than "file1"). This
   raises a SendFile exception.

Both b. and e. stop before we bother to
determine the template we're going to use. That's because they
don't actually use templates.

The template used is specified by the ``:template`` CGI variable,
which defaults to:

- only classname suplied:          "index"
- full item designator supplied:   "item"


Performing actions in web requests
----------------------------------

When a user requests a web page, they may optionally also request for an
action to take place. As described in `how requests are processed`_, the
action is performed before the requested page is generated. Actions are
triggered by using a ``:action`` CGI variable, where the value is one of:

**login**
 Attempt to log a user in.

**logout**
 Log the user out - make them "anonymous".

**register**
 Attempt to create a new user based on the contents of the form and then log
 them in.

**edit**
 Perform an edit of an item in the database. There are some special form
 elements you may use:

 :link=designator:property and :multilink=designator:property
  The value specifies an item designator and the property on that
  item to add *this* item to as a link or multilink.
 :note
  Create a message and attach it to the current item's
  "messages" property.
 :file
  Create a file and attach it to the current item's
  "files" property. Attach the file to the message created from
  the :note if it's supplied.
 :required=property,property,...
  The named properties are required to be filled in the form.
 :remove:<propname>=id(s)
  The ids will be removed from the multilink property. You may have multiple
  :remove:<propname> form elements for a single <propname>.
 :add:<propname>=id(s)
  The ids will be added to the multilink property. You may have multiple
  :add:<propname> form elements for a single <propname>.

**new**
 Add a new item to the database. You may use the same special form elements
 as in the "edit" action.

**retire**
 Retire the item in the database.

**editCSV**
 Performs an edit of all of a class' items in one go. See also the
 *class*.csv templating method which generates the CSV data to be edited, and
 the "_generic.index" template which uses both of these features.

**search**
 Mangle some of the form variables.

 Set the form ":filter" variable based on the values of the
 filter variables - if they're set to anything other than
 "dontcare" then add them to :filter.

 Also handle the ":queryname" variable and save off the query to
 the user's query list.

Each of the actions is implemented by a corresponding *actionAction* (where
"action" is the name of the action) method on
the roundup.cgi.Client class, which also happens to be in your tracker as
interfaces.Client. So if you need to define new actions, you may add them
there (see `defining new web actions`_).

Each action also has a corresponding *actionPermission* (where
"action" is the name of the action) method which determines
whether the action is permissible given the current user. The base permission
checks are:

**login**
 Determine whether the user has permission to log in.
 Base behaviour is to check the user has "Web Access".
**logout**
 No permission checks are made.
**register**
 Determine whether the user has permission to register
 Base behaviour is to check the user has "Web Registration".
**edit**
 Determine whether the user has permission to edit this item.
 Base behaviour is to check the user can edit this class. If we're
 editing the "user" class, users are allowed to edit their own
 details. Unless it's the "roles" property, which requires the
 special Permission "Web Roles".
**new**
 Determine whether the user has permission to create (edit) this item.
 Base behaviour is to check the user can edit this class. No
 additional property checks are made. Additionally, new user items
 may be created if the user has the "Web Registration" Permission.
**editCSV**
 Determine whether the user has permission to edit this class.
 Base behaviour is to check the user can edit this class.
**search**
 Determine whether the user has permission to search this class.
 Base behaviour is to check the user can view this class.


Default templates
-----------------

Most customisation of the web view can be done by modifying the templates in
the tracker **html** directory. There are several types of files in there:

**page**
  This template usually defines the overall look of your tracker. When you
  view an issue, it appears inside this template. When you view an index, it
  also appears inside this template. This template defines a macro called
  "icing" which is used by almost all other templates as a coating for their
  content, using its "content" slot. It will also define the "head_title"
  and "body_title" slots to allow setting of the page title.
**home**
  the default page displayed when no other page is indicated by the user
**home.classlist**
  a special version of the default page that lists the classes in the tracker
**classname.item**
  displays an item of the *classname* class
**classname.index**
  displays a list of *classname* items
**classname.search**
  displays a search page for *classname* items
**_generic.index**
  used to display a list of items where there is no *classname*.index available
**_generic.help**
  used to display a "class help" page where there is no *classname*.help
**user.register**
  a special page just for the user class that renders the registration page
**style.css**
  a static file that is served up as-is

Note: Remember that you can create any template extension you want to, so 
if you just want to play around with the templating for new issues, you can
copy the current "issue.item" template to "issue.test", and then access the
test template using the ":template" URL argument::

   http://your.tracker.example/tracker/issue?:template=test

and it won't affect your users using the "issue.item" template.


How the templates work
----------------------

Basic Templating Actions
~~~~~~~~~~~~~~~~~~~~~~~~

Roundup's templates consist of special attributes on your template tags.
These attributes form the Template Attribute Language, or TAL. The basic tag
commands are:

**tal:define="variable expression; variable expression; ..."**
   Define a new variable that is local to this tag and its contents. For
   example::

      <html tal:define="title request/description">
       <head><title tal:content="title"></title></head>
      </html>

   In the example, the variable "title" is defined as being the result of the
   expression "request/description". The tal:content command inside the <html>
   tag may then use the "title" variable.

**tal:condition="expression"**
   Only keep this tag and its contents if the expression is true. For example::

     <p tal:condition="python:request.user.hasPermission('View', 'issue')">
      Display some issue information.
     </p>

   In the example, the <p> tag and its contents are only displayed if the
   user has the View permission for issues. We consider the number zero, a
   blank string, an empty list, and the built-in variable nothing to be false
   values. Nearly every other value is true, including non-zero numbers, and
   strings with anything in them (even spaces!).

**tal:repeat="variable expression"**
   Repeat this tag and its contents for each element of the sequence that the
   expression returns, defining a new local variable and a special "repeat"
   variable for each element. For example::

     <tr tal:repeat="u user/list">
      <td tal:content="u/id"></td>
      <td tal:content="u/username"></td>
      <td tal:content="u/realname"></td>
     </tr>

   The example would iterate over the sequence of users returned by
   "user/list" and define the local variable "u" for each entry.

**tal:replace="expression"**
   Replace this tag with the result of the expression. For example::

    <span tal:replace="request/user/realname"></span>

   The example would replace the <span> tag and its contents with the user's
   realname. If the user's realname was "Bruce" then the resultant output
   would be "Bruce".

**tal:content="expression"**
   Replace the contents of this tag with the result of the expression. For
   example::

    <span tal:content="request/user/realname">user's name appears here</span>

   The example would replace the contents of the <span> tag with the user's
   realname. If the user's realname was "Bruce" then the resultant output
   would be "<span>Bruce</span>".

**tal:attributes="attribute expression; attribute expression; ..."**
   Set attributes on this tag to the results of expressions. For example::

     <a tal:attributes="href string:user${request/user/id}">My Details</a>

   In the example, the "href" attribute of the <a> tag is set to the value of
   the "string:user${request/user/id}" expression, which will be something
   like "user123".

**tal:omit-tag="expression"**
   Remove this tag (but not its contents) if the expression is true. For
   example::

      <span tal:omit-tag="python:1">Hello, world!</span>

   would result in output of::

      Hello, world!

Note that the commands on a given tag are evaulated in the order above, so
*define* comes before *condition*, and so on.

Additionally, a tag is defined, tal:block, which is removed from output. Its
content is not, but the tag itself is (so don't go using any tal:attributes
commands on it). This is useful for making arbitrary blocks of HTML
conditional or repeatable (very handy for repeating multiple table rows,
which would othewise require an illegal tag placement to effect the repeat).


Templating Expressions
~~~~~~~~~~~~~~~~~~~~~~

The expressions you may use in the attibute values may be one of the following
forms:

**Path Expressions** - eg. ``item/status/checklist``
   These are object attribute / item accesses. Roughly speaking, the path
   ``item/status/checklist`` is broken into parts ``item``, ``status``
   and ``checklist``. The ``item`` part is the root of the expression.
   We then look for a ``status`` attribute on ``item``, or failing that, a
   ``status`` item (as in ``item['status']``). If that
   fails, the path expression fails. When we get to the end, the object we're
   left with is evaluated to get a string - methods are called, objects are
   stringified. Path expressions may have an optional ``path:`` prefix, though
   they are the default expression type, so it's not necessary.

   If an expression evaluates to ``default`` then the expression is
   "cancelled" - whatever HTML already exists in the template will remain
   (tag content in the case of tal:content, attributes in the case of
   tal:attributes).

   If an expression evaluates to ``nothing`` then the target of the expression
   is removed (tag content in the case of tal:content, attributes in the case
   of tal:attributes and the tag itself in the case of tal:replace).

   If an element in the path may not exist, then you can use the ``|``
   operator in the expression to provide an alternative. So, the expression
   ``request/form/foo/value | default`` would simply leave the current HTML
   in place if the "foo" form variable doesn't exist.

**String Expressions** - eg. ``string:hello ${user/name}``
   These expressions are simple string interpolations - though they can be just
   plain strings with no interpolation if you want. The expression in the
   ``${ ... }`` is just a path expression as above.

**Python Expressions** - eg. ``python: 1+1``
   These expressions give the full power of Python. All the "root level"
   variables are available, so ``python:item.status.checklist()`` would be
   equivalent to ``item/status/checklist``, assuming that ``checklist`` is
   a method.

Template Macros
~~~~~~~~~~~~~~~

Macros are used in Roundup to save us from repeating the same common page
stuctures over and over. The most common (and probably only) macro you'll use
is the "icing" macro defined in the "page" template.

Macros are generated and used inside your templates using special attributes
similar to the `basic templating actions`_. In this case though, the
attributes belong to the Macro Expansion Template Attribute Language, or
METAL. The macro commands are:

**metal:define-macro="macro name"**
  Define that the tag and its contents are now a macro that may be inserted
  into other templates using the *use-macro* command. For example::

    <html metal:define-macro="page">
     ...
    </html>

  defines a macro called "page" using the ``<html>`` tag and its contents.
  Once defined, macros are stored on the template they're defined on in the
  ``macros`` attribute. You can access them later on through the ``templates``
  variable, eg. the most common ``templates/page/macros/icing`` to access the
  "page" macro of the "page" template.

**metal:use-macro="path expression"**
  Use a macro, which is identified by the path expression (see above). This
  will replace the current tag with the identified macro contents. For
  example::

   <tal:block metal:use-macro="templates/page/macros/icing">
    ...
   </tal:block>

   will replace the tag and its contents with the "page" macro of the "page"
   template.

**metal:define-slot="slot name"** and **metal:fill-slot="slot name"**
  To define *dynamic* parts of the macro, you define "slots" which may be 
  filled when the macro is used with a *use-macro* command. For example, the
  ``templates/page/macros/icing`` macro defines a slot like so::

    <title metal:define-slot="head_title">title goes here</title>

  In your *use-macro* command, you may now use a *fill-slot* command like
  this::

    <title metal:fill-slot="head_title">My Title</title>

  where the tag that fills the slot completely replaces the one defined as
  the slot in the macro.

Note that you may not mix METAL and TAL commands on the same tag, but TAL
commands may be used freely inside METAL-using tags (so your *fill-slots*
tags may have all manner of TAL inside them).


Information available to templates
----------------------------------

Note: this is implemented by roundup.cgi.templating.RoundupPageTemplate

The following variables are available to templates.

**context**
  The current context. This is either None, a
  `hyperdb class wrapper`_ or a `hyperdb item wrapper`_
**request**
  Includes information about the current request, including:
   - the current index information (``filterspec``, ``filter`` args,
     ``properties``, etc) parsed out of the form. 
   - methods for easy filterspec link generation
   - *user*, the current user item as an HTMLItem instance
   - *form*
     The current CGI form information as a mapping of form argument
     name to value
**config**
  This variable holds all the values defined in the tracker config.py file 
  (eg. TRACKER_NAME, etc.)
**db**
  The current database, used to access arbitrary database items.
**templates**
  Access to all the tracker templates by name. Used mainly in *use-macro*
  commands.
**utils**
  This variable makes available some utility functions like batching.
**nothing**
  This is a special variable - if an expression evaluates to this, then the
  tag (in the case of a tal:replace), its contents (in the case of
  tal:content) or some attributes (in the case of tal:attributes) will not
  appear in the the output. So for example::

    <span tal:attributes="class nothing">Hello, World!</span>

  would result in::

    <span>Hello, World!</span>

**default**
  Also a special variable - if an expression evaluates to this, then the
  existing HTML in the template will not be replaced or removed, it will
  remain. So::

    <span tal:replace="default">Hello, World!</span>

  would result in::

    <span>Hello, World!</span>

The context variable
~~~~~~~~~~~~~~~~~~~~

The *context* variable is one of three things based on the current context
(see `determining web context`_ for how we figure this out):

1. if we're looking at a "home" page, then it's None
2. if we're looking at a specific hyperdb class, it's a
   `hyperdb class wrapper`_.
3. if we're looking at a specific hyperdb item, it's a
   `hyperdb item wrapper`_.

If the context is not None, we can access the properties of the class or item.
The only real difference between cases 2 and 3 above are:

1. the properties may have a real value behind them, and this will appear if
   the property is displayed through ``context/property`` or
   ``context/property/field``.
2. the context's "id" property will be a false value in the second case, but
   a real, or true value in the third. Thus we can determine whether we're
   looking at a real item from the hyperdb by testing "context/id".

Hyperdb class wrapper
:::::::::::::::::::::

Note: this is implemented by the roundup.cgi.templating.HTMLClass class.

This wrapper object provides access to a hyperb class. It is used primarily
in both index view and new item views, but it's also usable anywhere else that
you wish to access information about a class, or the items of a class, when
you don't have a specific item of that class in mind.

We allow access to properties. There will be no "id" property. The value
accessed through the property will be the current value of the same name from
the CGI form.

There are several methods available on these wrapper objects:

=========== =============================================================
Method      Description
=========== =============================================================
properties  return a `hyperdb property wrapper`_ for all of this class'
            properties.
list        lists all of the active (not retired) items in the class.
csv         return the items of this class as a chunk of CSV text.
propnames   lists the names of the properties of this class.
filter      lists of items from this class, filtered and sorted
            by the current *request* filterspec/filter/sort/group args
classhelp   display a link to a javascript popup containing this class'
            "help" template.
submit      generate a submit button (and action hidden element)
renderWith  render this class with the given template.
history     returns 'New node - no history' :)
is_edit_ok  is the user allowed to Edit the current class?
is_view_ok  is the user allowed to View the current class?
=========== =============================================================

Note that if you have a property of the same name as one of the above methods,
you'll need to access it using a python "item access" expression. For example::

   python:context['list']

will access the "list" property, rather than the list method.


Hyperdb item wrapper
::::::::::::::::::::

Note: this is implemented by the roundup.cgi.templating.HTMLItem class.

This wrapper object provides access to a hyperb item.

We allow access to properties. There will be no "id" property. The value
accessed through the property will be the current value of the same name from
the CGI form.

There are several methods available on these wrapper objects:

=============== =============================================================
Method          Description
=============== =============================================================
submit          generate a submit button (and action hidden element)
journal         return the journal of the current item (**not implemented**)
history         render the journal of the current item as HTML
renderQueryForm specific to the "query" class - render the search form for
                the query
hasPermission   specific to the "user" class - determine whether the user
                has a Permission
is_edit_ok      is the user allowed to Edit the current item?
is_view_ok      is the user allowed to View the current item?
=============== =============================================================


Note that if you have a property of the same name as one of the above methods,
you'll need to access it using a python "item access" expression. For example::

   python:context['journal']

will access the "journal" property, rather than the journal method.


Hyperdb property wrapper
::::::::::::::::::::::::

Note: this is implemented by subclasses roundup.cgi.templating.HTMLProperty
class (HTMLStringProperty, HTMLNumberProperty, and so on).

This wrapper object provides access to a single property of a class. Its
value may be either:

1. if accessed through a `hyperdb item wrapper`_, then it's a value from the
   hyperdb
2. if access through a `hyperdb class wrapper`_, then it's a value from the
   CGI form


The property wrapper has some useful attributes:

=============== =============================================================
Attribute       Description
=============== =============================================================
_name           the name of the property
_value          the value of the property if any - this is the actual value
                retrieved from the hyperdb for this property
=============== =============================================================

There are several methods available on these wrapper objects:

========= =====================================================================
Method    Description
========= =====================================================================
plain     render a "plain" representation of the property. This method may
          take two arguments:

          escape
           If true, escape the text so it is HTML safe (default: no). The
           reason this defaults to off is that text is usually escaped
           at a later stage by the TAL commands, unless the "structure"
           option is used in the template. The following are all equivalent::

            <p tal:content="structure python:msg.content.plain(escape=1)" />
            <p tal:content="python:msg.content.plain()" />
            <p tal:content="msg/content/plain" />
            <p tal:content="msg/content" />

           Usually you'll only want to use the escape option in a complex
           expression.

          hyperlink
           If true, turn URLs, email addresses and hyperdb item designators
           in the text into hyperlinks (default: no). Note that you'll need
           to use the "structure" TAL option if you want to use this::

            <p tal:content="structure python:msg.content.plain(hyperlink=1)" />

           Note also that the text is automatically HTML-escape before the
           hyperlinking transformation.

field     render an appropriate form edit field for the property - for most
          types this is a text entry box, but for Booleans it's a tri-state
          yes/no/neither selection.
stext     only on String properties - render the value of the
          property as StructuredText (requires the StructureText module
          to be installed separately)
multiline only on String properties - render a multiline form edit
          field for the property
email     only on String properties - render the value of the 
          property as an obscured email address
confirm   only on Password properties - render a second form edit field for
          the property, used for confirmation that the user typed the
          password correctly. Generates a field with name "name:confirm".
now       only on Date properties - return the current date as a new property
reldate   only on Date properties - render the interval between the
          date and now
local     only on Date properties - return this date as a new property with
          some timezone offset
pretty    only on Interval properties - render the interval in a
          pretty format (eg. "yesterday")
menu      only on Link and Multilink properties - render a form select
          list for this property
reverse   only on Multilink properties - produce a list of the linked
          items in reverse order
========= =====================================================================

The request variable
~~~~~~~~~~~~~~~~~~~~

Note: this is implemented by the roundup.cgi.templating.HTMLRequest class.

The request variable is packed with information about the current request.

.. taken from roundup.cgi.templating.HTMLRequest docstring

=========== =================================================================
Variable    Holds
=========== =================================================================
form        the CGI form as a cgi.FieldStorage
env         the CGI environment variables
base        the base URL for this tracker
user        a HTMLUser instance for this user
classname   the current classname (possibly None)
template    the current template (suffix, also possibly None)
form        the current CGI form variables in a FieldStorage
=========== =================================================================

**Index page specific variables (indexing arguments)**

=========== =================================================================
Variable    Holds
=========== =================================================================
columns     dictionary of the columns to display in an index page
show        a convenience access to columns - request/show/colname will
            be true if the columns should be displayed, false otherwise
sort        index sort column (direction, column name)
group       index grouping property (direction, column name)
filter      properties to filter the index on
filterspec  values to filter the index on
search_text text to perform a full-text search on for an index
=========== =================================================================

There are several methods available on the request variable:

=============== =============================================================
Method          Description
=============== =============================================================
description     render a description of the request - handle for the page
                title
indexargs_form  render the current index args as form elements
indexargs_url   render the current index args as a URL
base_javascript render some javascript that is used by other components of
                the templating
batch           run the current index args through a filter and return a
                list of items (see `hyperdb item wrapper`_, and
                `batching`_)
=============== =============================================================

The form variable
:::::::::::::::::

The form variable is a little special because it's actually a python
FieldStorage object. That means that you have two ways to access its
contents. For example, to look up the CGI form value for the variable
"name", use the path expression::

   request/form/name/value

or the python expression::

   python:request.form['name'].value

Note the "item" access used in the python case, and also note the explicit
"value" attribute we have to access. That's because the form variables are
stored as MiniFieldStorages. If there's more than one "name" value in
the form, then the above will break since ``request/form/name`` is actually a
*list* of MiniFieldStorages. So it's best to know beforehand what you're
dealing with.


The db variable
~~~~~~~~~~~~~~~

Note: this is implemented by the roundup.cgi.templating.HTMLDatabase class.

Allows access to all hyperdb classes as attributes of this variable. If you
want access to the "user" class, for example, you would use::

  db/user
  python:db.user

The access results in a `hyperdb class wrapper`_.

The templates variable
~~~~~~~~~~~~~~~~~~~~~~

Note: this is implemented by the roundup.cgi.templating.Templates class.

This variable doesn't have any useful methods defined. It supports being
used in expressions to access the templates, and subsequently the template
macros. You may access the templates using the following path expression::

   templates/name

or the python expression::

   templates[name]

where "name" is the name of the template you wish to access. The template you
get access to has one useful attribute, "macros". To access a specific macro
(called "macro_name"), use the path expression::

   templates/name/macros/macro_name

or the python expression::

   templates[name].macros[macro_name]


The utils variable
~~~~~~~~~~~~~~~~~~

Note: this is implemented by the roundup.cgi.templating.TemplatingUtils class,
but it may be extended as described below.

=============== =============================================================
Method          Description
=============== =============================================================
Batch           return a batch object using the supplied list
=============== =============================================================

You may add additional utility methods by writing them in your tracker
``interfaces.py`` module's ``TemplatingUtils`` class. See `adding a time log
to your issues`_ for an example. The TemplatingUtils class itself will have a
single attribute, ``client``, which may be used to access the ``client.db``
when you need to perform arbitrary database queries.

Batching
::::::::

Use Batch to turn a list of items, or item ids of a given class, into a series
of batches. Its usage is::

    python:util.Batch(sequence, size, start, end=0, orphan=0, overlap=0)

or, to get the current index batch::

    request/batch

The parameters are:

========= ==================================================================
Parameter  Usage
========= ==================================================================
sequence  a list of HTMLItems
size      how big to make the sequence.
start     where to start (0-indexed) in the sequence.
end       where to end (0-indexed) in the sequence.
orphan    if the next batch would contain less items than this
          value, then it is combined with this batch
overlap   the number of items shared between adjacent batches
========= ==================================================================

All of the parameters are assigned as attributes on the batch object. In
addition, it has several more attributes:

=============== ============================================================
Attribute       Description
=============== ============================================================
start           indicates the start index of the batch. *Note: unlike the
                argument, is a 1-based index (I know, lame)*
first           indicates the start index of the batch *as a 0-based
                index*
length          the actual number of elements in the batch
sequence_length the length of the original, unbatched, sequence.
=============== ============================================================

And several methods:

=============== ============================================================
Method          Description
=============== ============================================================
previous        returns a new Batch with the previous batch settings
next            returns a new Batch with the next batch settings
propchanged     detect if the named property changed on the current item
                when compared to the last item
=============== ============================================================

An example of batching::

 <table class="otherinfo">
  <tr><th colspan="4" class="header">Existing Keywords</th></tr>
  <tr tal:define="keywords db/keyword/list"
      tal:repeat="start python:range(0, len(keywords), 4)">
   <td tal:define="batch python:utils.Batch(keywords, 4, start)"
       tal:repeat="keyword batch" tal:content="keyword/name">keyword here</td>
  </tr>
 </table>

... which will produce a table with four columns containing the items of the
"keyword" class (well, their "name" anyway).

Displaying Properties
---------------------

Properties appear in the user interface in three contexts: in indices, in
editors, and as search arguments.
For each type of property, there are several display possibilities.
For example, in an index view, a string property may just be
printed as a plain string, but in an editor view, that property may be
displayed in an editable field.


Index Views
-----------

This is one of the class context views. It is also the default view for
classes. The template used is "*classname*.index".

Index View Specifiers
~~~~~~~~~~~~~~~~~~~~~

An index view specifier (URL fragment) looks like this (whitespace has been
added for clarity)::

     /issue?status=unread,in-progress,resolved&
            topic=security,ui&
            :group=+priority&
            :sort==activity&
            :filters=status,topic&
            :columns=title,status,fixer

The index view is determined by two parts of the specifier: the layout part and
the filter part. The layout part consists of the query parameters that begin
with colons, and it determines the way that the properties of selected items
are displayed. The filter part consists of all the other query parameters, and
it determines the criteria by which items are selected for display.
The filter part is interactively manipulated with the form widgets displayed in
the filter section. The layout part is interactively manipulated by clicking on
the column headings in the table.

The filter part selects the union of the sets of items with values matching any
specified Link properties and the intersection of the sets of items with values
matching any specified Multilink properties.

The example specifies an index of "issue" items. Only items with a "status" of
either "unread" or "in-progres" or "resolved" are displayed, and only items
with "topic" values including both "security" and "ui" are displayed. The items
are grouped by priority, arranged in ascending order; and within groups, sorted
by activity, arranged in descending order. The filter section shows filters for
the "status" and "topic" properties, and the table includes columns for the
"title", "status", and "fixer" properties.

Searching Views
---------------

Note: if you add a new column to the ``:columns`` form variable potentials
      then you will need to add the column to the appropriate `index views`_
      template so it is actually displayed.

This is one of the class context views. The template used is typically
"*classname*.search". The form on this page should have "search" as its
``:action`` variable. The "search" action:

- sets up additional filtering, as well as performing indexed text searching
- sets the ``:filter`` variable correctly
- saves the query off if ``:query_name`` is set.

The searching page should lay out any fields that you wish to allow the user
to search one. If your schema contains a large number of properties, you
should be wary of making all of those properties available for searching, as
this can cause confusion. If the additional properties are Strings, consider
having their value indexed, and then they will be searchable using the full
text indexed search. This is both faster, and more useful for the end user.

The two special form values on search pages which are handled by the "search"
action are:

:search_text
  Text to perform a search of the text index with. Results from that search
  will be used to limit the results of other filters (using an intersection
  operation)
:query_name
  If supplied, the search parameters (including :search_text) will be saved
  off as a the query item and registered against the user's queries property.
  Note that the *classic* template schema has this ability, but the *minimal*
  template schema does not.


Item Views
----------

The basic view of a hyperdb item is provided by the "*classname*.item"
template. It generally has three sections; an "editor", a "spool" and a
"history" section.



Editor Section
~~~~~~~~~~~~~~

The editor section is used to manipulate the item - it may be a
static display if the user doesn't have permission to edit the item.

Here's an example of a basic editor template (this is the default "classic"
template issue item edit form - from the "issue.item" template)::

 <table class="form">
 <tr>
  <th nowrap>Title</th>
  <td colspan=3 tal:content="structure python:context.title.field(size=60)">title</td>
 </tr>
 
 <tr>
  <th nowrap>Priority</th>
  <td tal:content="structure context/priority/menu">priority</td>
  <th nowrap>Status</th>
  <td tal:content="structure context/status/menu">status</td>
 </tr>
 
 <tr>
  <th nowrap>Superseder</th>
  <td>
   <span tal:replace="structure python:context.superseder.field(showid=1, size=20)" />
   <span tal:replace="structure python:db.issue.classhelp('id,title')" />
   <span tal:condition="context/superseder">
    <br>View: <span tal:replace="structure python:context.superseder.link(showid=1)" />
   </span>
  </td>
  <th nowrap>Nosy List</th>
  <td>
   <span tal:replace="structure context/nosy/field" />
   <span tal:replace="structure python:db.user.classhelp('username,realname,address,phone')" />
  </td>
 </tr>
 
 <tr>
  <th nowrap>Assigned To</th>
  <td tal:content="structure context/assignedto/menu">
   assignedto menu
  </td>
  <td>&nbsp;</td>
  <td>&nbsp;</td>
 </tr>
 
 <tr>
  <th nowrap>Change Note</th>
  <td colspan=3>
   <textarea name=":note" wrap="hard" rows="5" cols="60"></textarea>
  </td>
 </tr>
 
 <tr>
  <th nowrap>File</th>
  <td colspan=3><input type="file" name=":file" size="40"></td>
 </tr>
 
 <tr>
  <td>&nbsp;</td>
  <td colspan=3 tal:content="structure context/submit">
   submit button will go here
  </td>
 </tr>
 </table>


When a change is submitted, the system automatically generates a message
describing the changed properties. As shown in the example, the editor
template can use the ":note" and ":file" fields, which are added to the
standard change note message generated by Roundup.

Spool Section
~~~~~~~~~~~~~

The spool section lists related information like the messages and files of
an issue.

TODO


History Section
~~~~~~~~~~~~~~~

The final section displayed is the history of the item - its database journal.
This is generally generated with the template::

 <tal:block tal:replace="structure context/history" />

*To be done:*

*The actual history entries of the item may be accessed for manual templating
through the "journal" method of the item*::

 <tal:block tal:repeat="entry context/journal">
  a journal entry
 </tal:block>

*where each journal entry is an HTMLJournalEntry.*

Defining new web actions
------------------------

You may define new actions to be triggered by the ``:action`` form variable.
These are added to the tracker ``interfaces.py`` as methods on the ``Client``
class. 

Adding action methods takes three steps; first you `define the new action
method`_, then you `register the action method`_ with the cgi interface so
it may be triggered by the ``:action`` form variable. Finally you actually
`use the new action`_ in your HTML form.

See "`setting up a "wizard" (or "druid") for controlled adding of issues`_"
for an example.

Define the new action method
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The action methods have the following interface::

    def myActionMethod(self):
        ''' Perform some action. No return value is required.
        '''

The *self* argument is an instance of your tracker ``instance.Client`` class - 
thus it's mostly implemented by ``roundup.cgi.Client``. See the docstring of
that class for details of what it can do.

The method will typically check the ``self.form`` variable's contents. It
may then:

- add information to ``self.ok_message`` or ``self.error_message``
- change the ``self.template`` variable to alter what the user will see next
- raise Unauthorised, SendStaticFile, SendFile, NotFound or Redirect
  exceptions


Register the action method
~~~~~~~~~~~~~~~~~~~~~~~~~~

The method is now written, but isn't available to the user until you add it to
the `instance.Client`` class ``actions`` variable, like so::

    actions = client.Class.actions + (
        ('myaction', 'myActionMethod'),
    )

This maps the action name "myaction" to the action method we defined.


Use the new action
~~~~~~~~~~~~~~~~~~

In your HTML form, add a hidden form element like so::

  <input type="hidden" name=":action" value="myaction">

where "myaction" is the name you registered in the previous step.


Examples
========

.. contents::
   :local:
   :depth: 1

Adding a new field to the classic schema
----------------------------------------

This example shows how to add a new constrained property (ie. a selection of
distinct values) to your tracker.

Introduction
~~~~~~~~~~~~

To make the classic schema of roundup useful as a todo tracking system
for a group of systems administrators, it needed an extra data field
per issue: a category.

This would let sysads quickly list all todos in their particular
area of interest without having to do complex queries, and without
relying on the spelling capabilities of other sysads (a losing
proposition at best).

Adding a field to the database
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is the easiest part of the change. The category would just be a plain
string, nothing fancy. To change what is in the database you need to add
some lines to the ``open()`` function in ``dbinit.py`` under the comment::

    # add any additional database schema configuration here

add::

    category = Class(db, "category", name=String())
    category.setkey("name")

Here we are setting up a chunk of the database which we are calling
"category". It contains a string, which we are refering to as "name" for
lack of a more imaginative title. Then we are setting the key of this chunk
of the database to be that "name". This is equivalent to an index for
database types. This also means that there can only be one category with a
given name.

Adding the above lines allows us to create categories, but they're not tied
to the issues that we are going to be creating. It's just a list of categories
off on its own, which isn't much use. We need to link it in with the issues.
To do that, find the lines in the ``open()`` function in ``dbinit.py`` which
set up the "issue" class, and then add a link to the category::

    issue = IssueClass(db, "issue", ... , category=Multilink("category"), ... )

The Multilink() means that each issue can have many categories. If you were
adding something with a more one to one relationship use Link() instead.

That is all you need to do to change the schema. The rest of the effort is
fiddling around so you can actually use the new category.

Populating the new category class
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you haven't initialised the database with the roundup-admin "initialise"
command, then you can add the following to the tracker ``dbinit.py`` in the
``init()`` function under the comment::

    # add any additional database create steps here - but only if you
    # haven't initialised the database with the admin "initialise" command

add::

     category = db.getclass('category')
     category.create(name="scipy", order="1")
     category.create(name="chaco", order="2")
     category.create(name="weave", order="3")

If the database is initalised, the you need to use the roundup-admin tool::

     % roundup-admin -i <tracker home>
     Roundup <version> ready for input.
     Type "help" for help.
     roundup> create category name=scipy order=1
     1
     roundup> create category name=chaco order=1
     2
     roundup> create category name=weave order=1
     3
     roundup> exit...
     There are unsaved changes. Commit them (y/N)? y


Setting up security on the new objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

By default only the admin user can look at and change objects. This doesn't
suit us, as we want any user to be able to create new categories as
required, and obviously everyone needs to be able to view the categories of
issues for it to be useful.

We therefore need to change the security of the category objects. This is
also done in the ``open()`` function of ``dbinit.py``.

There are currently two loops which set up permissions and then assign them
to various roles. Simply add the new "category" to both lists::

    # new permissions for this schema
    for cl in 'issue', 'file', 'msg', 'user', 'category':
        db.security.addPermission(name="Edit", klass=cl,
            description="User is allowed to edit "+cl)
        db.security.addPermission(name="View", klass=cl,
            description="User is allowed to access "+cl)

    # Assign the access and edit permissions for issue, file and message
    # to regular users now
    for cl in 'issue', 'file', 'msg', 'category':
        p = db.security.getPermission('View', cl)
        db.security.addPermissionToRole('User', p)
        p = db.security.getPermission('Edit', cl)
        db.security.addPermissionToRole('User', p)

So you are in effect doing the following::

    db.security.addPermission(name="Edit", klass='category',
        description="User is allowed to edit "+'category')
    db.security.addPermission(name="View", klass='category',
        description="User is allowed to access "+'category')

which is creating two permission types; that of editing and viewing
"category" objects respectively. Then the following lines assign those new
permissions to the "User" role, so that normal users can view and edit
"category" objects::

    p = db.security.getPermission('View', 'category')
    db.security.addPermissionToRole('User', p)

    p = db.security.getPermission('Edit', 'category')
    db.security.addPermissionToRole('User', p)

This is all the work that needs to be done for the database. It will store
categories, and let users view and edit them. Now on to the interface
stuff.

Changing the web left hand frame
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We need to give the users the ability to create new categories, and the
place to put the link to this functionality is in the left hand function
bar, under the "Issues" area. The file that defines how this area looks is
``html/page``, which is what we are going to be editing next.

If you look at this file you can see that it contains a lot of "classblock"
sections which are chunks of HTML that will be included or excluded in the
output depending on whether the condition in the classblock is met. Under
the end of the classblock for issue is where we are going to add the
category code::

  <p class="classblock"
     tal:condition="python:request.user.hasPermission('View', 'category')">
   <b>Categories</b><br>
   <a tal:condition="python:request.user.hasPermission('Edit', 'category')"
      href="category?:template=item">New Category<br></a>
  </p>

The first two lines is the classblock definition, which sets up a condition
that only users who have "View" permission to the "category" object will
have this section included in their output. Next comes a plain "Categories"
header in bold. Everyone who can view categories will get that.

Next comes the link to the editing area of categories. This link will only
appear if the condition is matched: that condition being that the user has
"Edit" permissions for the "category" objects. If they do have permission
then they will get a link to another page which will let the user add new
categories.

Note that if you have permission to view but not edit categories then all
you will see is a "Categories" header with nothing underneath it. This is
obviously not very good interface design, but will do for now. I just claim
that it is so I can add more links in this section later on. However to fix
the problem you could change the condition in the classblock statement, so
that only users with "Edit" permission would see the "Categories" stuff.

Setting up a page to edit categories
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We defined code in the previous section which let users with the
appropriate permissions see a link to a page which would let them edit
conditions. Now we have to write that page.

The link was for the item template for the category object. This translates
into the system looking for a file called ``category.item`` in the ``html``
tracker directory. This is the file that we are going to write now.

First we add an info tag in a comment which doesn't affect the outcome
of the code at all but is useful for debugging. If you load a page in a
browser and look at the page source, you can see which sections come
from which files by looking for these comments::

    <!-- category.item -->

Next we need to add in the METAL macro stuff so we get the normal page
trappings::

 <tal:block metal:use-macro="templates/page/macros/icing">
  <title metal:fill-slot="head_title">Category editing</title>
  <td class="page-header-top" metal:fill-slot="body_title">
   <h2>Category editing</h2>
  </td>
  <td class="content" metal:fill-slot="content">

Next we need to setup up a standard HTML form, which is the whole
purpose of this file. We link to some handy javascript which sends the form
through only once. This is to stop users hitting the send button
multiple times when they are impatient and thus having the form sent
multiple times::

    <form method="POST" onSubmit="return submit_once()"
          enctype="multipart/form-data">

Next we define some code which sets up the minimum list of fields that we
require the user to enter. There will be only one field, that of "name", so
they user better put something in it otherwise the whole form is pointless::

    <input type="hidden" name=":required" value="name">

To get everything to line up properly we will put everything in a table,
and put a nice big header on it so the user has an idea what is happening::

    <table class="form">
     <tr><th class="header" colspan=2>Category</th></tr>

Next we need the actual field that the user is going to enter the new
category. The "context.name.field(size=60)" bit tells roundup to generate a
normal HTML field of size 60, and the contents of that field will be the
"name" variable of the current context (which is "category"). The upshot of
this is that when the user types something in to the form, a new category
will be created with that name::

    <tr>
     <th nowrap>Name</th>
     <td tal:content="structure python:context.name.field(size=60)">name</td>
    </tr>

Then a submit button so that the user can submit the new category::

    <tr>
     <td>&nbsp;</td>
     <td colspan=3 tal:content="structure context/submit">
      submit button will go here
     </td>
    </tr>

Finally we finish off the tags we used at the start to do the METAL stuff::

  </td>
 </tal:block>

So putting it all together, and closing the table and form we get::

 <!-- category.item -->
 <tal:block metal:use-macro="templates/page/macros/icing">
  <title metal:fill-slot="head_title">Category editing</title>
  <td class="page-header-top" metal:fill-slot="body_title">
   <h2>Category editing</h2>
  </td>
  <td class="content" metal:fill-slot="content">
   <form method="POST" onSubmit="return submit_once()"
         enctype="multipart/form-data">

    <input type="hidden" name=":required" value="name">

    <table class="form">
     <tr><th class="header" colspan=2>Category</th></tr>

     <tr>
      <th nowrap>Name</th>
      <td tal:content="structure python:context.name.field(size=60)">name</td>
     </tr>

     <tr>
      <td>&nbsp;</td>
      <td colspan=3 tal:content="structure context/submit">
       submit button will go here
      </td>
     </tr>
    </table>
   </form>
  </td>
 </tal:block>

This is quite a lot to just ask the user one simple question, but
there is a lot of setup for basically one line (the form line) to do
its work. To add another field to "category" would involve one more line
(well maybe a few extra to get the formatting correct).

Adding the category to the issue
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We now have the ability to create issues to our hearts content, but
that is pointless unless we can assign categories to issues.  Just like
the ``html/category.item`` file was used to define how to add a new
category, the ``html/issue.item`` is used to define how a new issue is
created.

Just like ``category.issue`` this file defines a form which has a table to lay
things out. It doesn't matter where in the table we add new stuff,
it is entirely up to your sense of aesthetics::

   <th nowrap>Category</th>
   <td><span tal:replace="structure context/category/field" />
       <span tal:replace="structure db/category/classhelp" />
   </td>

First we define a nice header so that the user knows what the next section
is, then the middle line does what we are most interested in. This
``context/category/field`` gets replaced with a field which contains the
category in the current context (the current context being the new issue).

The classhelp lines generate a link (labelled "list") to a popup window
which contains the list of currently known categories.

Searching on categories
~~~~~~~~~~~~~~~~~~~~~~~

We can add categories, and create issues with categories. The next obvious
thing that we would like to be would be to search issues based on their
category, so that any one working on the web server could look at all
issues in the category "Web" for example.

If you look in the html/page file and look for the "Search Issues" you will
see that it looks something like ``<a href="issue?:template=search">Search
Issues</a>`` which shows us that when you click on "Search Issues" it will
be looking for a ``issue.search`` file to display. So that is indeed the file
that we are going to change.

If you look at this file it should be starting to seem familiar. It is a
simple HTML form using a table to define structure. You can add the new
category search code anywhere you like within that form::

    <tr>
     <th>Category:</th>
     <td>
      <select name="category">
       <option value="">don't care</option>
       <option value="">------------</option>
       <option tal:repeat="s db/category/list" tal:attributes="value s/name"
               tal:content="s/name">category to filter on</option>
      </select>
     </td>
     <td><input type="checkbox" name=":columns" value="category" checked></td>
     <td><input type="radio" name=":sort" value="category"></td>
     <td><input type="radio" name=":group" value="category"></td>
    </tr>

Most of this is straightforward to anyone who knows HTML. It is just
setting up a select list followed by a checkbox and a couple of radio
buttons. 

The ``tal:repeat`` part repeats the tag for every item in the "category"
table and setting "s" to be each category in turn.

The ``tal:attributes`` part is setting up the ``value=`` part of the option tag
to be the name part of "s" which is the current category in the loop.

The ``tal:content`` part is setting the contents of the option tag to be the
name part of "s" again. For objects more complex than category, obviously
you would put an id in the value, and the descriptive part in the content;
but for category they are the same.

Adding category to the default view
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We can now add categories, add issues with categories, and search issues
based on categories. This is everything that we need to do, however there
is some more icing that we would like. I think the category of an issue is
important enough that it should be displayed by default when listing all
the issues.

Unfortunately, this is a bit less obvious than the previous steps. The code
defining how the issues look is in ``html/issue.index``. This is a large table
with a form down the bottom for redisplaying and so forth. 

Firstly we need to add an appropriate header to the start of the table::

    <th tal:condition="request/show/category">Category</th>

The condition part of this statement is so that if the user has selected
not to see the Category column then they won't.

The rest of the table is a loop which will go through every issue that
matches the display criteria. The loop variable is "i" - which means that
every issue gets assigned to "i" in turn.

The new part of code to display the category will look like this::

    <td tal:condition="request/show/category" tal:content="i/category"></td>

The condition is the same as above: only display the condition when the
user hasn't asked for it to be hidden. The next part is to set the content
of the cell to be the category part of "i" - the current issue.

Finally we have to edit ``html/page`` again. This time to tell it that when the
user clicks on "Unnasigned Issues" or "All Issues" that the category should
be displayed. If you scroll down the page file, you can see the links with
lots of options. The option that we are interested in is the ``:columns=`` one
which tells roundup which fields of the issue to display. Simply add
"category" to that list and it all should work.


Adding in state transition control
----------------------------------

Sometimes tracker admins want to control the states that users may move issues
to.

1. add a Multilink property to the status class::

     stat = Class(db, "status", ... , transitions=Multilink('status'), ...)

   and then edit the statuses already created either:

   a. through the web using the class list -> status class editor, or
   b. using the roundup-admin "set" command.

2. add an auditor module ``checktransition.py`` in your tracker's
   ``detectors`` directory::

     def checktransition(db, cl, nodeid, newvalues):
         ''' Check that the desired transition is valid for the "status"
             property.
         '''
         if not newvalues.has_key('status'):
             return
         current = cl.get(nodeid, 'status')
         new = newvalues['status']
         if new == current:
             return
         ok = db.status.get(current, 'transitions')
         if new not in ok:
             raise ValueError, 'Status not allowed to move from "%s" to "%s"'%(
                 db.status.get(current, 'name'), db.status.get(new, 'name'))

     def init(db):
         db.issue.audit('set', checktransition)

3. in the ``issue.item`` template, change the status editing bit from::

    <th nowrap>Status</th>
    <td tal:content="structure context/status/menu">status</td>

   to::

    <th nowrap>Status</th>
    <td>
     <select tal:condition="context/id" name="status">
      <tal:block tal:define="ok context/status/transitions"
                 tal:repeat="state db/status/list">
       <option tal:condition="python:state.id in ok"
               tal:attributes="value state/id;
                               selected python:state.id == context.status.id"
               tal:content="state/name"></option>
      </tal:block>
     </select>
     <tal:block tal:condition="not:context/id"
                tal:replace="structure context/status/menu" />
    </td>

   which displays only the allowed status to transition to.


Displaying only message summaries in the issue display
------------------------------------------------------

Alter the issue.item template section for messages to::

 <table class="messages" tal:condition="context/messages">
  <tr><th colspan=5 class="header">Messages</th></tr>
  <tr tal:repeat="msg context/messages">
   <td><a tal:attributes="href string:msg${msg/id}"
          tal:content="string:msg${msg/id}"></a></td>
   <td tal:content="msg/author">author</td>
   <td nowrap tal:content="msg/date/pretty">date</td>
   <td tal:content="msg/summary">summary</td>
   <td>
    <a tal:attributes="href string:?:remove:messages=${msg/id}&:action=edit">remove</a>
   </td>
  </tr>
 </table>

Restricting the list of users that are assignable to a task
-----------------------------------------------------------

1. In your tracker's "dbinit.py", create a new Role, say "Developer"::

     db.security.addRole(name='Developer', description='A developer')

2. Just after that, create a new Permission, say "Fixer", specific to "issue"::

     p = db.security.addPermission(name='Fixer', klass='issue',
         description='User is allowed to be assigned to fix issues')

3. Then assign the new Permission to your "Developer" Role::

     db.security.addPermissionToRole('Developer', p)

4. In the issue item edit page ("html/issue.item" in your tracker dir), use
   the new Permission in restricting the "assignedto" list::

    <select name="assignedto">
     <option value="-1">- no selection -</option>
     <tal:block tal:repeat="user db/user/list">
     <option tal:condition="python:user.hasPermission('Fixer', context.classname)"
             tal:attributes="value user/id;
                             selected python:user.id == context.assignedto"
             tal:content="user/realname"></option>
     </tal:block>
    </select>

For extra security, you may wish to set up an auditor to enforce the
Permission requirement (install this as "assignedtoFixer.py" in your tracker
"detectors" directory)::

  def assignedtoMustBeFixer(db, cl, nodeid, newvalues):
      ''' Ensure the assignedto value in newvalues is a used with the Fixer
          Permission
      '''
      if not newvalues.has_key('assignedto'):
          # don't care
          return
  
      # get the userid
      userid = newvalues['assignedto']
      if not db.security.hasPermission('Fixer', userid, cl.classname):
          raise ValueError, 'You do not have permission to edit %s'%cl.classname

  def init(db):
      db.issue.audit('set', assignedtoMustBeFixer)
      db.issue.audit('create', assignedtoMustBeFixer)

So now, if the edit attempts to set the assignedto to a user that doesn't have
the "Fixer" Permission, the error will be raised.


Setting up a "wizard" (or "druid") for controlled adding of issues
------------------------------------------------------------------

1. Set up the page templates you wish to use for data input. My wizard
   is going to be a two-step process, first figuring out what category of
   issue the user is submitting, and then getting details specific to that
   category. The first page includes a table of help, explaining what the
   category names mean, and then the core of the form::

    <form method="POST" onSubmit="return submit_once()"
          enctype="multipart/form-data">
      <input type="hidden" name=":template" value="add_page1">
      <input type="hidden" name=":action" value="page1submit">

      <strong>Category:</strong>
      <tal:block tal:replace="structure context/category/menu" />
      <input type="submit" value="Continue">
    </form>

   The next page has the usual issue entry information, with the addition of
   the following form fragments::

    <form method="POST" onSubmit="return submit_once()"
          enctype="multipart/form-data" tal:condition="context/is_edit_ok"
          tal:define="cat request/form/category/value">

      <input type="hidden" name=":template" value="add_page2">
      <input type="hidden" name=":required" value="title">
      <input type="hidden" name="category" tal:attributes="value cat">

       .
       .
       .
    </form>

   Note that later in the form, I test the value of "cat" include form
   elements that are appropriate. For example::

    <tal:block tal:condition="python:cat in '6 10 13 14 15 16 17'.split()">
     <tr>
      <th nowrap>Operating System</th>
      <td tal:content="structure context/os/field"></td>
     </tr>
     <tr>
      <th nowrap>Web Browser</th>
      <td tal:content="structure context/browser/field"></td>
     </tr>
    </tal:block>

   ... the above section will only be displayed if the category is one of 6,
   10, 13, 14, 15, 16 or 17.

3. Determine what actions need to be taken between the pages - these are
   usually to validate user choices and determine what page is next. Now
   encode those actions in methods on the interfaces Client class and insert
   hooks to those actions in the "actions" attribute on that class, like so::

    actions = client.Client.actions + (
        ('page1_submit', 'page1SubmitAction'),
    )

    def page1SubmitAction(self):
        ''' Verify that the user has selected a category, and then move on
            to page 2.
        '''
        category = self.form['category'].value
        if category == '-1':
            self.error_message.append('You must select a category of report')
            return
        # everything's ok, move on to the next page
        self.template = 'add_page2'

4. Use the usual "new" action as the :action on the final page, and you're
   done (the standard context/submit method can do this for you).


Using an external password validation source
--------------------------------------------

We have a centrally-managed password changing system for our users. This
results in a UN*X passwd-style file that we use for verification of users.
Entries in the file consist of ``name:password`` where the password is
encrypted using the standard UN*X ``crypt()`` function (see the ``crypt``
module in your Python distribution). An example entry would be::

    admin:aamrgyQfDFSHw

Each user of Roundup must still have their information stored in the Roundup
database - we just use the passwd file to check their password. To do this, we
add the following code to our ``Client`` class in the tracker home
``interfaces.py`` module::

    def verifyPassword(self, userid, password):
        # get the user's username
        username = self.db.user.get(userid, 'username')

        # the passwords are stored in the "passwd.txt" file in the tracker
        # home
        file = os.path.join(self.db.config.TRACKER_HOME, 'passwd.txt')

        # see if we can find a match
        for ent in [line.strip().split(':') for line in open(file).readlines()]:
            if ent[0] == username:
                return crypt.crypt(password, ent[1][:2]) == ent[1]

        # user doesn't exist in the file
        return 0

What this does is look through the file, line by line, looking for a name that
matches.

We also remove the redundant password fields from the ``user.item`` template.


Adding a "vacation" flag to users for stopping nosy messages
------------------------------------------------------------

When users go on vacation and set up vacation email bouncing, you'll start to
see a lot of messages come back through Roundup "Fred is on vacation". Not
very useful, and relatively easy to stop.

1. add a "vacation" flag to your users::

         user = Class(db, "user",
                    username=String(),   password=Password(),
                    address=String(),    realname=String(),
                    phone=String(),      organisation=String(),
                    alternate_addresses=String(),
                    roles=String(), queries=Multilink("query"),
                    vacation=Boolean())

2. edit your detector ``nosyreactor.py`` so that the ``nosyreaction()``
   consists of::

    def nosyreaction(db, cl, nodeid, oldvalues):
        # send a copy of all new messages to the nosy list
        for msgid in determineNewMessages(cl, nodeid, oldvalues):
            try:
                users = db.user
                messages = db.msg

                # figure the recipient ids
                sendto = []
                r = {}
                recipients = messages.get(msgid, 'recipients')
                for recipid in messages.get(msgid, 'recipients'):
                    r[recipid] = 1

                # figure the author's id, and indicate they've received the
                # message
                authid = messages.get(msgid, 'author')

                # possibly send the message to the author, as long as they aren't
                # anonymous
                if (db.config.MESSAGES_TO_AUTHOR == 'yes' and
                        users.get(authid, 'username') != 'anonymous'):
                    sendto.append(authid)
                r[authid] = 1

                # now figure the nosy people who weren't recipients
                nosy = cl.get(nodeid, 'nosy')
                for nosyid in nosy:
                    # Don't send nosy mail to the anonymous user (that user
                    # shouldn't appear in the nosy list, but just in case they
                    # do...)
                    if users.get(nosyid, 'username') == 'anonymous':
                        continue
                    # make sure they haven't seen the message already
                    if not r.has_key(nosyid):
                        # send it to them
                        sendto.append(nosyid)
                        recipients.append(nosyid)

                # generate a change note
                if oldvalues:
                    note = cl.generateChangeNote(nodeid, oldvalues)
                else:
                    note = cl.generateCreateNote(nodeid)

                # we have new recipients
                if sendto:
                    # filter out the people on vacation
                    sendto = [i for i in sendto if not users.get(i, 'vacation', 0)]

                    # map userids to addresses
                    sendto = [users.get(i, 'address') for i in sendto]

                    # update the message's recipients list
                    messages.set(msgid, recipients=recipients)

                    # send the message
                    cl.send_message(nodeid, msgid, note, sendto)
            except roundupdb.MessageSendError, message:
                raise roundupdb.DetectorError, message

   Note that this is the standard nosy reaction code, with the small addition
   of::

    # filter out the people on vacation
    sendto = [i for i in sendto if not users.get(i, 'vacation', 0)]

   which filters out the users that have the vacation flag set to true.


Adding a time log to your issues
--------------------------------

We want to log the dates and amount of time spent working on issues, and be
able to give a summary of the total time spent on a particular issue.

1. Add a new class to your tracker ``dbinit.py``::

    # storage for time logging
    timelog = Class(db, "timelog", period=Interval())

   Note that we automatically get the date of the time log entry creation
   through the standard property "creation".

2. Link to the new class from your issue class (again, in ``dbinit.py``)::

    issue = IssueClass(db, "issue", 
                    assignedto=Link("user"), topic=Multilink("keyword"),
                    priority=Link("priority"), status=Link("status"),
                    times=Multilink("timelog"))

   the "times" property is the new link to the "timelog" class.

3. We'll need to let people add in times to the issue, so in the web interface
   we'll have a new entry field, just below the change note box::

    <tr>
     <th nowrap>Time Log</th>
     <td colspan=3><input name=":timelog">
      (enter as "3y 1m 4d 2:40:02" or parts thereof)
     </td>
    </tr>

   Note that we've made up a new form variable, but since we place a colon ":"
   in front of it, it won't clash with any existing property variables. The
   names you *can't* use are ``:note``, ``:file``, ``:action``, ``:required``
   and ``:template``. These variables are described in the section
   `performing actions in web requests`_.

4. We also need to handle this new field in the CGI interface - the way to
   do this is through implementing a new form action (see `Setting up a
   "wizard" (or "druid") for controlled adding of issues`_ for another example
   where we implemented a new CGI form action).

   In this case, we'll want our action to:

   1. create a new "timelog" entry,
   2. fake that the issue's "times" property has been edited, and then
   3. call the normal CGI edit action handler.

   The code to do this is::

    class Client(client.Client): 
        ''' derives basic CGI implementation from the standard module, 
            with any specific extensions 
        ''' 
        actions = client.Client.actions + (
            ('edit_with_timelog', 'timelogEditAction'),
            ('new_with_timelog', 'timelogEditAction'),
        )

        def timelogEditAction(self):
            ''' Handle the creation of a new time log entry if necessary.

                If we create a new entry, fake up a CGI form value for the
                altered "times" property of the issue being edited.
   
                Punt to the regular edit action when we're done.
            '''
            # if there's a timelog value specified, create an entry
            if self.form.has_key(':timelog') and \
                    self.form[':timelog'].value.strip():
                period = Interval(self.form[':timelog'].value)
                # create it
                newid = self.db.timelog.create(period=period)

                # if we're editing an existing item, get the old timelog value
                if self.nodeid:
                    l = self.db.issue.get(self.nodeid, 'times')
                    l.append(newid)
                else:
                    l = [newid]

                # now make the fake CGI form values
                for entry in l:
                    self.form.list.append(MiniFieldStorage('times', entry))

            # punt to the normal edit action
            if self.nodeid:
                return self.editItemAction()
            else:
                return self.newItemAction()
   
   you add this code to your Client class in your tracker's ``interfaces.py``
   file. Locate the section that looks like::

    class Client:
        ''' derives basic CGI implementation from the standard module, 
            with any specific extensions 
        ''' 
        pass

   and insert this code in place of the ``pass`` statement.

5. You'll also need to modify your ``issue.item`` form submit action so it
   calls the time logging action we just created. The current template will
   look like this::

    <tr>
     <td>&nbsp;</td>
     <td colspan=3 tal:content="structure context/submit">
      submit button will go here
     </td>
    </tr>

   replace it with this::

    <tr>
     <td>&nbsp;</td>
     <td colspan=3>
      <tal:block tal:condition="context/id">
        <input type="hidden" name=":action" value="edit_with_timelog">
        <input type="submit" name="submit" value="Submit Changes">
      </tal:block>
      <tal:block tal:condition="not:context/id">
        <input type="hidden" name=":action" value="new_with_timelog">
        <input type="submit" name="submit" value="Submit New Issue">
      </tal:block>
     </td>
    </tr>

   The important change is setting the action to "edit_with_timelog" for
   edit operations (where the item exists) and "new_with_timelog" for
   creations operations.

6. We want to display a total of the time log times that have been accumulated
   for an issue. To do this, we'll need to actually write some Python code,
   since it's beyond the scope of PageTemplates to perform such calculations.
   We do this by adding a method to the TemplatingUtils class in our tracker
   ``interfaces.py`` module::

    class TemplatingUtils:
        ''' Methods implemented on this class will be available to HTML
            templates through the 'utils' variable.
        '''
        def totalTimeSpent(self, times):
            ''' Call me with a list of timelog items (which have an Interval 
                "period" property)
            '''
            total = Interval('')
            for time in times:
                total += time.period._value
            return total

   Replace the ``pass`` line as we did in step 4 above with the Client class.
   As indicated in the docstrings, we will be able to access the
   ``totalTimeSpent`` method via the ``utils`` variable in our templates.

7. Display the time log for an issue::

     <table class="otherinfo" tal:condition="context/times">
      <tr><th colspan="3" class="header">Time Log
       <tal:block tal:replace="python:utils.totalTimeSpent(context.times)" />
      </th></tr>
      <tr><th>Date</th><th>Period</th><th>Logged By</th></tr>
      <tr tal:repeat="time context/times">
       <td tal:content="time/creation"></td>
       <td tal:content="time/period"></td>
       <td tal:content="time/creator"></td>
      </tr>
     </table>

   I put this just above the Messages log in my issue display. Note our use
   of the ``totalTimeSpent`` method which will total up the times for the
   issue and return a new Interval. That will be automatically displayed in
   the template as text like "+ 1y 2:40" (1 year, 2 hours and 40 minutes).

8. If you're using a persistent web server - roundup-server or mod_python for
   example - then you'll need to restart that to pick up the code changes.
   When that's done, you'll be able to use the new time logging interface.

Using a UN*X passwd file as the user database
---------------------------------------------

On some systems the primary store of users is the UN*X passwd file. It holds
information on users such as their username, real name, password and primary
user group.

Roundup can use this store as its primary source of user information, but it
needs additional information too - email address(es), roundup Roles, vacation
flags, roundup hyperdb item ids, etc. Also, "retired" users must still exist
in the user database, unlike some passwd files in which the users are removed
when they no longer have access to a system.

To make use of the passwd file, we therefore synchronise between the two user
stores. We also use the passwd file to validate the user logins, as described
in the previous example, `using an external password validation source`_. We
keep the users lists in sync using a fairly simple script that runs once a
day, or several times an hour if more immediate access is needed. In short, it:

1. parses the passwd file, finding usernames, passwords and real names,
2. compares that list to the current roundup user list:

   a. entries no longer in the passwd file are *retired*
   b. entries with mismatching real names are *updated*
   c. entries only exist in the passwd file are *created*

3. send an email to administrators to let them know what's been done.

The retiring and updating are simple operations, requiring only a call to
``retire()`` or ``set()``. The creation operation requires more information
though - the user's email address and their roundup Roles. We're going to
assume that the user's email address is the same as their login name, so we
just append the domain name to that. The Roles are determined using the
passwd group identifier - mapping their UN*X group to an appropriate set of
Roles.

The script to perform all this, broken up into its main components, is as
follows. Firstly, we import the necessary modules and open the tracker we're
to work on::

    import sys, os, smtplib
    from roundup import instance, date

    # open the tracker
    tracker_home = sys.argv[1]
    tracker = instance.open(tracker_home)

Next we read in the *passwd* file from the tracker home::

    # read in the users
    file = os.path.join(tracker_home, 'users.passwd')
    users = [x.strip().split(':') for x in open(file).readlines()]

Handle special users (those to ignore in the file, and those who don't appear
in the file)::

    # users to not keep ever, pre-load with the users I know aren't
    # "real" users
    ignore = ['ekmmon', 'bfast', 'csrmail']

    # users to keep - pre-load with the roundup-specific users
    keep = ['comment_pool', 'network_pool', 'admin', 'dev-team', 'cs_pool',
        'anonymous', 'system_pool', 'automated']

Now we map the UN*X group numbers to the Roles that users should have::

    roles = {
     '501': 'User,Tech',  # tech
     '502': 'User',       # finance
     '503': 'User,CSR',   # customer service reps
     '504': 'User',       # sales
     '505': 'User',       # marketing
    }

Now we do all the work. Note that the body of the script (where we have the
tracker database open) is wrapped in a ``try`` / ``finally`` clause, so that
we always close the database cleanly when we're finished. So, we now do all
the work::

    # open the database
    db = tracker.open('admin')
    try:
        # store away messages to send to the tracker admins
        msg = []

        # loop over the users list read in from the passwd file
        for user,passw,uid,gid,real,home,shell in users:
            if user in ignore:
                # this user shouldn't appear in our tracker
                continue
            keep.append(user)
            try:
                # see if the user exists in the tracker
                uid = db.user.lookup(user)

                # yes, they do - now check the real name for correctness
                if real != db.user.get(uid, 'realname'):
                    db.user.set(uid, realname=real)
                    msg.append('FIX %s - %s'%(user, real))
            except KeyError:
                # nope, the user doesn't exist
                db.user.create(username=user, realname=real,
                    address='%s@ekit-inc.com'%user, roles=roles[gid])
                msg.append('ADD %s - %s (%s)'%(user, real, roles[gid]))

        # now check that all the users in the tracker are also in our "keep"
        # list - retire those who aren't
        for uid in db.user.list():
            user = db.user.get(uid, 'username')
            if user not in keep:
                db.user.retire(uid)
                msg.append('RET %s'%user)

        # if we did work, then send email to the tracker admins
        if msg:
            # create the email
            msg = '''Subject: %s user database maintenance

            %s
            '''%(db.config.TRACKER_NAME, '\n'.join(msg))

            # send the email
            smtp = smtplib.SMTP(db.config.MAILHOST)
            addr = db.config.ADMIN_EMAIL
            smtp.sendmail(addr, addr, msg)

        # now we're done - commit the changes
        db.commit()
    finally:
        # always close the database cleanly
        db.close()

And that's it!


Enabling display of either message summaries or the entire messages
-------------------------------------------------------------------

This is pretty simple - all we need to do is copy the code from the example
`displaying only message summaries in the issue display`_ into our template
alongside the summary display, and then introduce a switch that shows either
one or the other. We'll use a new form variable, ``:whole_messages`` to
achieve this::

 <table class="messages" tal:condition="context/messages">
  <tal:block tal:condition="not:request/form/:whole_messages/value | python:0">
   <tr><th colspan=3 class="header">Messages</th>
       <th colspan=2 class="header">
         <a href="?:whole_messages=yes">show entire messages</a>
       </th>
   </tr>
   <tr tal:repeat="msg context/messages">
    <td><a tal:attributes="href string:msg${msg/id}"
           tal:content="string:msg${msg/id}"></a></td>
    <td tal:content="msg/author">author</td>
    <td nowrap tal:content="msg/date/pretty">date</td>
    <td tal:content="msg/summary">summary</td>
    <td>
     <a tal:attributes="href string:?:remove:messages=${msg/id}&:action=edit">remove</a>
    </td>
   </tr>
  </tal:block>

  <tal:block tal:condition="request/form/:whole_messages/value | python:0">
   <tr><th colspan=2 class="header">Messages</th>
       <th class="header"><a href="?:whole_messages=">show only summaries</a></th>
   </tr>
   <tal:block tal:repeat="msg context/messages">
    <tr>
     <th tal:content="msg/author">author</th>
     <th nowrap tal:content="msg/date/pretty">date</th>
     <th style="text-align: right">
      (<a tal:attributes="href string:?:remove:messages=${msg/id}&:action=edit">remove</a>)
     </th>
    </tr>
    <tr><td colspan=3 tal:content="msg/content"></td></tr>
   </tal:block>
  </tal:block>
 </table>


-------------------

Back to `Table of Contents`_

.. _`Table of Contents`: index.html
.. _`design documentation`: design.html