Sorry for the huge checkin message - I was only intending to implement #496356

but I found a number of places where things had been broken by transactions:
 . modified ROUNDUPDBSENDMAILDEBUG to be SENDMAILDEBUG and hold a filename
   for _all_ roundup-generated smtp messages to be sent to.
 . the transaction cache had broken the roundupdb.Class set() reactors
 . newly-created author users in the mailgw weren't being committed to the db

Stuff that made it into CHANGES.txt (ie. the stuff I was actually working
on when I found that stuff :):
 . #496356 ] Use threading in messages
 . detectors were being registered multiple times
 . added tests for mailgw
 . much better attaching of erroneous messages in the mail gateway

git-svn-id: http://svn.roundup-tracker.org/svnroot/roundup/trunk@487 57a73879-2fb5-44c3-a270-3262357dd7e2

diff --git a/CHANGES.txt b/CHANGES.txt
index 021b8bf3870727c5cd67dd8775e94b2f82b0694d..46d224441c765419e0e080ddec0b1d38b0b231d2 100644 (file)
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,7 +1,7 @@
 This file contains the changes to the Roundup system over time. The entries
 are given with the most recent entry first.
 
-2001-12-?? - 0.3.1b1
+2001-12-?? - 0.4.0b1
 Feature:
  . Added INSTANCE_NAME to configuration - used in web and email to identify
    the instance.
@@ -25,6 +25,11 @@ Feature:
  . Added a Zope frontend for roundup.
  . Centralised the python version check code, bumped version to 2.1.1 (really
    needs to be 2.1.2, but that isn't released yet :)
+ . much better attaching of erroneous messages in the mail gateway
+ . #496356 ] Use threading in messages
+   This adds the tracking of messages by message-id and allows threading
+   using in-reply-to. Most e-mail clients support threading using this
+   feature, and we hope to add support for it to the web gateway.
 
 Fixed:
  . Lots of bugs, thanks Roché and others on the devel mailing list!
@@ -47,7 +52,12 @@ Fixed:
  . envelope-from is now set to the roundup-admin and not roundup itself so
    delivery reports aren't sent to roundup (thanks Patrick Ohly)
  . #495400 ] entering blanks
+   Values with spaces are now accepted in roundup-admin - check the long help
+   for details.
  . #496360 ] table width does not work
+ . detectors were being registered multiple times
+ . added tests for mailgw
+
 
 2001-11-23 - 0.3.0 
 Feature:

diff --git a/MIGRATION.txt b/MIGRATION.txt
index 7288755c7fbb8bf8e2ef3721344cbe758c48af5c..c27230a151d7e6fe9217e4912023d4722845b3d9 100644 (file)
--- a/MIGRATION.txt
+++ b/MIGRATION.txt
@@ -1,13 +1,54 @@
 Migrating to newer versions of Roundup
 ======================================
 
+Please read each section carefully and edit your instance home files
+accordingly.
 
-Migrating from 0.2.x to 0.3.x
+This file contains information for users upgrading from:
+  0.3.x -> 0.4.x
+  0.2.x -> 0.3.x
+
+
+Migrating from 0.3.x to 0.4.x
 =============================
 
-Please read each section carefully and edit your instance home files
-accordingly.
+Message-ID and In-Reply-To addition
+-----------------------------------
+0.4.0 adds the tracking of messages by message-id and allows threading
+using in-reply-to. Most e-mail clients support threading using this
+feature, and we hope to add support for it to the web gateway. If you
+have not edited the dbinit.py file in your instance home directory, you may
+simply copy the new dbinit.py file from the core code. If you used the
+classic schema, the interfaces file is in:
 
+ <roundup source>/roundup/templates/classic/dbinit.py
+
+If you used the extended schema, the file is in:
+
+ <roundup source>/roundup/templates/extended/dbinit.pybinit.py needs updating from the original. 
+
+If you have modified your dbinit.py file, you may use encoded passwords:
+
+ 1. Edit the dbinit.py file in your instance home directory. Find the lines
+ which define the msg class:
+
+    msg = FileClass(db, "msg",
+                    author=Link("user"), recipients=Multilink("user"),
+                    date=Date(),         summary=String(),
+                    files=Multilink("file"))
+
+ and add the messageid and inreplyto properties like so:
+
+    msg = FileClass(db, "msg",
+                    author=Link("user"), recipients=Multilink("user"),
+                    date=Date(),         summary=String(),
+                    files=Multilink("file"),
+                    messageid=String(),  inreplyto=String())
+
+
+
+Migrating from 0.2.x to 0.3.x
+=============================
 
 Cookie Authentication changes
 -----------------------------

diff --git a/roundup/backends/back_anydbm.py b/roundup/backends/back_anydbm.py
index b6226aac7de490ad1ef17b067a53638f0bf0ef9a..6071940a540e376e10016dcae4b6980eb74bbd1b 100644 (file)
--- a/roundup/backends/back_anydbm.py
+++ b/roundup/backends/back_anydbm.py
@@ -15,7 +15,7 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 # 
-#$Id: back_anydbm.py,v 1.20 2001-12-18 15:30:34 rochecompaan Exp $
+#$Id: back_anydbm.py,v 1.21 2002-01-02 02:31:38 richard Exp $
 '''
 This module defines a backend that saves the hyperdatabase in a database
 chosen by anydbm. It is guaranteed to always be available in python
@@ -187,23 +187,25 @@ class Database(hyperdb.Database):
             print 'savenode', (self, classname, nodeid, node)
         self.transactions.append((self._doSaveNode, (classname, nodeid, node)))
 
-    def getnode(self, classname, nodeid, db=None):
+    def getnode(self, classname, nodeid, db=None, cache=1):
         ''' get a node from the database
         '''
         if DEBUG:
             print 'getnode', (self, classname, nodeid, cldb)
-        # try the cache
-        cache = self.cache.setdefault(classname, {})
-        if cache.has_key(nodeid):
-            return cache[nodeid]
+        if cache:
+            # try the cache
+            cache = self.cache.setdefault(classname, {})
+            if cache.has_key(nodeid):
+                return cache[nodeid]
 
         # get from the database and save in the cache
         if db is None:
             db = self.getclassdb(classname)
         if not db.has_key(nodeid):
-            raise IndexError, nodeid
+            raise IndexError, "no such %s %s"%(classname, nodeid)
         res = marshal.loads(db[nodeid])
-        cache[nodeid] = res
+        if cache:
+            cache[nodeid] = res
         return res
 
     def hasnode(self, classname, nodeid, db=None):
@@ -402,6 +404,15 @@ class Database(hyperdb.Database):
 
 #
 #$Log: not supported by cvs2svn $
+#Revision 1.20  2001/12/18 15:30:34  rochecompaan
+#Fixed bugs:
+# .  Fixed file creation and retrieval in same transaction in anydbm
+#    backend
+# .  Cgi interface now renders new issue after issue creation
+# .  Could not set issue status to resolved through cgi interface
+# .  Mail gateway was changing status back to 'chatting' if status was
+#    omitted as an argument
+#
 #Revision 1.19  2001/12/17 03:52:48  richard
 #Implemented file store rollback. As a bonus, the hyperdb is now capable of
 #storing more than one file per node - if a property name is supplied,

diff --git a/roundup/cgi_client.py b/roundup/cgi_client.py
index b43102a597c2cb1b7bd47f0d9348e9e2dcbdd827..e3191810fd32563f0687c6fa664db5ff806fa777 100644 (file)
--- a/roundup/cgi_client.py
+++ b/roundup/cgi_client.py
@@ -15,14 +15,14 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 # 
-# $Id: cgi_client.py,v 1.87 2001-12-23 23:18:49 richard Exp $
+# $Id: cgi_client.py,v 1.88 2002-01-02 02:31:38 richard Exp $
 
 __doc__ = """
 WWW request handler (also used in the stand-alone server).
 """
 
 import os, cgi, pprint, StringIO, urlparse, re, traceback, mimetypes
-import binascii, Cookie, time
+import binascii, Cookie, time, random
 
 import roundupdb, htmltemplate, date, hyperdb, password
 from roundup.i18n import _
@@ -456,11 +456,16 @@ class Client:
             # don't generate a useless message
             return None, files
 
+        # handle the messageid
+        # TODO: handle inreplyto
+        messageid = "%s.%s.%s%s-%s"%(time.time(), random.random(),
+            classname, nodeid, self.MAIL_DOMAIN)
+
         # now create the message, attaching the files
         content = '\n'.join(m)
         message_id = self.db.msg.create(author=self.getuid(),
             recipients=[], date=date.Date('.'), summary=summary,
-            content=content, files=files)
+            content=content, files=files, messageid=messageid)
 
         # update the messages property
         return message_id, files
@@ -1163,6 +1168,10 @@ def parsePropsFromForm(db, cl, form, nodeid=0):
 
 #
 # $Log: not supported by cvs2svn $
+# Revision 1.87  2001/12/23 23:18:49  richard
+# We already had an admin-specific section of the web heading, no need to add
+# another one :)
+#
 # Revision 1.86  2001/12/20 15:43:01  rochecompaan
 # Features added:
 #  .  Multilink properties are now displayed as comma separated values in

diff --git a/roundup/hyperdb.py b/roundup/hyperdb.py
index 26d324aa4f8e2dd722f244f979bfa59b35c0fe3c..ea4817479b29002fdd3cfb0964754fea0c455b71 100644 (file)
--- a/roundup/hyperdb.py
+++ b/roundup/hyperdb.py
@@ -15,7 +15,7 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 # 
-# $Id: hyperdb.py,v 1.43 2001-12-20 06:13:24 rochecompaan Exp $
+# $Id: hyperdb.py,v 1.44 2002-01-02 02:31:38 richard Exp $
 
 __doc__ = """
 Hyperdatabase implementation, especially field types.
@@ -240,18 +240,23 @@ class Class:
         self.db.addjournal(self.classname, newid, 'create', propvalues)
         return newid
 
-    def get(self, nodeid, propname, default=_marker):
+    def get(self, nodeid, propname, default=_marker, cache=1):
         """Get the value of a property on an existing node of this class.
 
         'nodeid' must be the id of an existing node of this class or an
         IndexError is raised.  'propname' must be the name of a property
         of this class or a KeyError is raised.
+
+        'cache' indicates whether the transaction cache should be queried
+        for the node. If the node has been modified and you need to
+        determine what its values prior to modification are, you need to
+        set cache=0.
         """
         if propname == 'id':
             return nodeid
 
         # get the node's dict
-        d = self.db.getnode(self.classname, nodeid)
+        d = self.db.getnode(self.classname, nodeid, cache=cache)
         if not d.has_key(propname) and default is not _marker:
             return default
 
@@ -271,10 +276,18 @@ class Class:
         return d[propname]
 
     # XXX not in spec
-    def getnode(self, nodeid):
-        ''' Return a convenience wrapper for the node
+    def getnode(self, nodeid, cache=1):
+        ''' Return a convenience wrapper for the node.
+
+        'nodeid' must be the id of an existing node of this class or an
+        IndexError is raised.
+
+        'cache' indicates whether the transaction cache should be queried
+        for the node. If the node has been modified and you need to
+        determine what its values prior to modification are, you need to
+        set cache=0.
         '''
-        return Node(self, nodeid)
+        return Node(self, nodeid, cache=cache)
 
     def set(self, nodeid, **propvalues):
         """Modify a property on an existing node of this class.
@@ -824,20 +837,21 @@ class Class:
 class Node:
     ''' A convenience wrapper for the given node
     '''
-    def __init__(self, cl, nodeid):
+    def __init__(self, cl, nodeid, cache=1):
         self.__dict__['cl'] = cl
         self.__dict__['nodeid'] = nodeid
+        self.cache = cache
     def keys(self, protected=1):
         return self.cl.getprops(protected=protected).keys()
     def values(self, protected=1):
         l = []
         for name in self.cl.getprops(protected=protected).keys():
-            l.append(self.cl.get(self.nodeid, name))
+            l.append(self.cl.get(self.nodeid, name, cache=self.cache))
         return l
     def items(self, protected=1):
         l = []
         for name in self.cl.getprops(protected=protected).keys():
-            l.append((name, self.cl.get(self.nodeid, name)))
+            l.append((name, self.cl.get(self.nodeid, name, cache=self.cache)))
         return l
     def has_key(self, name):
         return self.cl.getprops().has_key(name)
@@ -845,7 +859,7 @@ class Node:
         if self.__dict__.has_key(name):
             return self.__dict__[name]
         try:
-            return self.cl.get(self.nodeid, name)
+            return self.cl.get(self.nodeid, name, cache=self.cache)
         except KeyError, value:
             # we trap this but re-raise it as AttributeError - all other
             # exceptions should pass through untrapped
@@ -853,7 +867,7 @@ class Node:
         # nope, no such attribute
         raise AttributeError, str(value)
     def __getitem__(self, name):
-        return self.cl.get(self.nodeid, name)
+        return self.cl.get(self.nodeid, name, cache=self.cache)
     def __setattr__(self, name, value):
         try:
             return self.cl.set(self.nodeid, **{name: value})
@@ -875,6 +889,16 @@ def Choice(name, *options):
 
 #
 # $Log: not supported by cvs2svn $
+# Revision 1.43  2001/12/20 06:13:24  rochecompaan
+# Bugs fixed:
+#   . Exception handling in hyperdb for strings-that-look-like numbers got
+#     lost somewhere
+#   . Internet Explorer submits full path for filename - we now strip away
+#     the path
+# Features added:
+#   . Link and multilink properties are now displayed sorted in the cgi
+#     interface
+#
 # Revision 1.42  2001/12/16 10:53:37  richard
 # take a copy of the node dict so that the subsequent set
 # operation doesn't modify the oldvalues structure

diff --git a/roundup/mailgw.py b/roundup/mailgw.py
index 9b3c1411deba5bcf8935fa6aa1b2439f23dab116..c250c623666ff29c8325317bf4d6bf9712ba7c4f 100644 (file)
--- a/roundup/mailgw.py
+++ b/roundup/mailgw.py
@@ -73,14 +73,17 @@ are calling the create() method to create a new node). If an auditor raises
 an exception, the original message is bounced back to the sender with the
 explanatory message given in the exception. 
 
-$Id: mailgw.py,v 1.45 2001-12-20 15:43:01 rochecompaan Exp $
+$Id: mailgw.py,v 1.46 2002-01-02 02:31:38 richard Exp $
 '''
 
 
 import string, re, os, mimetools, cStringIO, smtplib, socket, binascii, quopri
+import time, random
 import traceback, MimeWriter
 import hyperdb, date, password
 
+SENDMAILDEBUG = os.environ.get('SENDMAILDEBUG', '')
+
 class MailGWError(ValueError):
     pass
 
@@ -180,14 +183,18 @@ class MailGW:
                 subject='Badly formed message from mail gateway')
 
         # now send the message
-        try:
-            smtp = smtplib.SMTP(self.MAILHOST)
-            smtp.sendmail(self.ADMIN_EMAIL, sendto, m.getvalue())
-        except socket.error, value:
-            raise MailGWError, "Couldn't send confirmation email: "\
-                "mailhost %s"%value
-        except smtplib.SMTPException, value:
-            raise MailGWError, "Couldn't send confirmation email: %s"%value
+        if SENDMAILDEBUG:
+            open(SENDMAILDEBUG, 'w').write('From: %s\nTo: %s\n%s\n'%(
+                self.ADMIN_EMAIL, ', '.join(sendto), m.getvalue()))
+        else:
+            try:
+                smtp = smtplib.SMTP(self.MAILHOST)
+                smtp.sendmail(self.ADMIN_EMAIL, sendto, m.getvalue())
+            except socket.error, value:
+                raise MailGWError, "Couldn't send error email: "\
+                    "mailhost %s"%value
+            except smtplib.SMTPException, value:
+                raise MailGWError, "Couldn't send error email: %s"%value
 
     def bounce_message(self, message, sendto, error,
             subject='Failed issue tracker submission'):
@@ -210,20 +217,24 @@ class MailGW:
         # reconstruct the original message
         m = cStringIO.StringIO()
         w = MimeWriter.MimeWriter(m)
+        # default the content_type, just in case...
+        content_type = 'text/plain'
+        # add the headers except the content-type
         for header in message.headers:
             header_name = header.split(':')[0]
-            if message.getheader(header_name):
-                w.addheader(header_name,message.getheader(header_name))
-        body = w.startbody('text/plain')
-        try:
-            message.fp.seek(0)
-        except:
-            pass
+            if header_name.lower() == 'content-type':
+                content_type = message.getheader(header_name)
+            elif message.getheader(header_name):
+                w.addheader(header_name, message.getheader(header_name))
+        # now attach the message body
+        body = w.startbody(content_type)
+        message.rewindbody()
         body.write(message.fp.read())
 
         # attach the original message to the returned message
         part = writer.nextpart()
         part.addheader('Content-Disposition','attachment')
+        part.addheader('Content-Description','Message that caused the error')
         part.addheader('Content-Transfer-Encoding', '7bit')
         body = part.startbody('message/rfc822')
         body.write(m.getvalue())
@@ -371,12 +382,11 @@ Subject was: "%s"
                         else:
                             props[key] = [v]
 
-
         #
         # handle the users
         #
 
-        # Don't create users if ANONYMOUS_ACCESS is denied
+        # Don't create users if ANONYMOUS_REGISTER is denied
         if self.ANONYMOUS_ACCESS == 'deny':
             create = 0
         else:
@@ -389,6 +399,10 @@ You are not a registered user.
 
 Unknown address: %s
 '''%message.getaddrlist('from')[0][1]
+
+        # the author may have been created - make sure the change is
+        # committed before we reopen the database
+        self.db.commit()
             
         # reopen the database as the author
         username = self.db.user.get(author, 'username')
@@ -401,11 +415,24 @@ Unknown address: %s
         recipients = []
         tracker_email = self.ISSUE_TRACKER_EMAIL.lower()
         for recipient in message.getaddrlist('to') + message.getaddrlist('cc'):
-            if recipient[1].strip().lower() == tracker_email:
+            r = recipient[1].strip().lower()
+            if r == tracker_email or not r:
                 continue
             recipients.append(self.db.uidFromAddress(recipient))
 
+        #
+        # handle message-id and in-reply-to
+        #
+        messageid = message.getheader('message-id')
+        inreplyto = message.getheader('in-reply-to') or ''
+        # generate a messageid if there isn't one
+        if not messageid:
+            messageid = "%s.%s.%s%s-%s"%(time.time(), random.random(),
+                classname, nodeid, self.MAIL_DOMAIN)
+
+        #
         # now handle the body - find the message
+        #
         content_type =  message.gettype()
         attachments = []
         if content_type == 'multipart/mixed':
@@ -487,13 +514,17 @@ not find a text/plain part to use.
 
         summary, content = parseContent(content)
 
-        # handle the files
+        # 
+        # handle the attachments
+        #
         files = []
         for (name, mime_type, data) in attachments:
             files.append(self.db.file.create(type=mime_type, name=name,
                 content=data))
 
+        #
         # now handle the db stuff
+        #
         if nodeid:
             # If an item designator (class name and id number) is found there,
             # the newly created "msg" node is added to the "messages" property
@@ -536,10 +567,11 @@ not find a text/plain part to use.
                     props['nosy'].append(assignedto)
             except:
                 pass
-                
+
             message_id = self.db.msg.create(author=author,
                 recipients=recipients, date=date.Date('.'), summary=summary,
-                content=content, files=files)
+                content=content, files=files, messageid=messageid,
+                inreplyto=inreplyto)
             try:
                 messages = cl.get(nodeid, 'messages')
             except IndexError:
@@ -569,7 +601,8 @@ There was a problem with the message you sent:
             # contain any new "file" nodes. 
             message_id = self.db.msg.create(author=author,
                 recipients=recipients, date=date.Date('.'), summary=summary,
-                content=content, files=files)
+                content=content, files=files, messageid=messageid,
+                inreplyto=inreplyto)
 
             # pre-set the issue to unread
             if properties.has_key('status') and not props.has_key('status'):
@@ -585,8 +618,10 @@ There was a problem with the message you sent:
             if properties.has_key('title') and not props.has_key('title'):
                 props['title'] = title
 
-            # pre-load the messages list and nosy list
+            # pre-load the messages list
             props['messages'] = [message_id]
+
+            # set up (clean) the nosy list
             nosy = props.get('nosy', [])
             n = {}
             for value in nosy:
@@ -596,18 +631,30 @@ There was a problem with the message you sent:
                     continue
                 if n.has_key(nid): continue
                 n[nid] = 1
-            props['nosy'] = n.keys() + recipients
+            props['nosy'] = n.keys()
+            # add on the recipients of the message
+            for recipient in recipients:
+                if not n.has_key(recipient):
+                    props['nosy'].append(recipient)
+                    n[recipient] = 1
+
             # add the author to the nosy list
             if not n.has_key(author):
                 props['nosy'].append(author)
                 n[author] = 1
+
             # add assignedto to the nosy list
-            try:
-                assignedto = self.db.user.lookup(props['assignedto'])
+            if properties.has_key('assignedto') and props.has_key('assignedto'):
+                try:
+                    assignedto = self.db.user.lookup(props['assignedto'])
+                except KeyError:
+                    raise MailUsageError, '''
+There was a problem with the message you sent:
+   Assignedto user '%s' doesn't exist
+'''%props['assignedto']
                 if not n.has_key(assignedto):
                     props['nosy'].append(assignedto)
-            except:
-                pass
+                    n[assignedto] = 1
 
             # and attempt to create the new node
             try:
@@ -661,6 +708,14 @@ def parseContent(content, blank_line=re.compile(r'[\r\n]+\s*[\r\n]+'),
 
 #
 # $Log: not supported by cvs2svn $
+# Revision 1.45  2001/12/20 15:43:01  rochecompaan
+# Features added:
+#  .  Multilink properties are now displayed as comma separated values in
+#     a textbox
+#  .  The add user link is now only visible to the admin user
+#  .  Modified the mail gateway to reject submissions from unknown
+#     addresses if ANONYMOUS_ACCESS is denied
+#
 # Revision 1.44  2001/12/18 15:30:34  rochecompaan
 # Fixed bugs:
 #  .  Fixed file creation and retrieval in same transaction in anydbm

diff --git a/roundup/roundupdb.py b/roundup/roundupdb.py
index 603e608ed35f13f0d2a1733f149f1a9172c1f4b2..bd1933eefc5a52282fa78c658de1dcd595eb46a4 100644 (file)
--- a/roundup/roundupdb.py
+++ b/roundup/roundupdb.py
@@ -15,20 +15,21 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 # 
-# $Id: roundupdb.py,v 1.35 2001-12-20 15:43:01 rochecompaan Exp $
+# $Id: roundupdb.py,v 1.36 2002-01-02 02:31:38 richard Exp $
 
 __doc__ = """
 Extending hyperdb with types specific to issue-tracking.
 """
 
-import re, os, smtplib, socket, copy
+import re, os, smtplib, socket, copy, time, random
 import mimetools, MimeWriter, cStringIO
 import base64, mimetypes
 
 import hyperdb, date
 
 # set to indicate to roundup not to actually _send_ email
-ROUNDUPDBSENDMAILDEBUG = os.environ.get('ROUNDUPDBSENDMAILDEBUG', '')
+# this var must contain a file to write the mail to
+SENDMAILDEBUG = os.environ.get('SENDMAILDEBUG', '')
 
 class DesignatorError(ValueError):
     pass
@@ -72,6 +73,7 @@ class Database:
 
         # couldn't match address or username, so create a new user
         if create:
+            print 'CREATING USER', address
             return self.user.create(username=address, address=address,
                 realname=realname)
         else:
@@ -110,9 +112,16 @@ class Class(hyperdb.Class):
             raise KeyError, '"creation" and "activity" are reserved'
         for audit in self.auditors['set']:
             audit(self.db, self, nodeid, propvalues)
-        # take a copy of the node dict so that the subsequent set
-        # operation doesn't modify the oldvalues structure
-        oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid))
+        # Take a copy of the node dict so that the subsequent set
+        # operation doesn't modify the oldvalues structure.
+        try:
+            # try not using the cache initially
+            oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid,
+                cache=0))
+        except IndexError:
+            # this will be needed if somone does a create() and set()
+            # with no intervening commit()
+            oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid))
         hyperdb.Class.set(self, nodeid, **propvalues)
         for react in self.reactors['set']:
             react(self.db, self, nodeid, oldvalues)
@@ -127,7 +136,7 @@ class Class(hyperdb.Class):
         for react in self.reactors['retire']:
             react(self.db, self, nodeid, None)
 
-    def get(self, nodeid, propname, default=_marker):
+    def get(self, nodeid, propname, default=_marker, cache=1):
         """Attempts to get the "creation" or "activity" properties should
         do the right thing.
         """
@@ -153,9 +162,10 @@ class Class(hyperdb.Class):
                 return None
             return self.db.user.lookup(name)
         if default is not _marker:
-            return hyperdb.Class.get(self, nodeid, propname, default)
+            return hyperdb.Class.get(self, nodeid, propname, default,
+                cache=cache)
         else:
-            return hyperdb.Class.get(self, nodeid, propname)
+            return hyperdb.Class.get(self, nodeid, propname, cache=cache)
 
     def getprops(self, protected=1):
         """In addition to the actual properties on the node, these
@@ -176,12 +186,16 @@ class Class(hyperdb.Class):
     def audit(self, event, detector):
         """Register a detector
         """
-        self.auditors[event].append(detector)
+        l = self.auditors[event]
+        if detector not in l:
+            self.auditors[event].append(detector)
 
     def react(self, event, detector):
         """Register a detector
         """
-        self.reactors[event].append(detector)
+        l = self.reactors[event]
+        if detector not in l:
+            self.reactors[event].append(detector)
 
 
 class FileClass(Class):
@@ -194,15 +208,15 @@ class FileClass(Class):
         self.db.storefile(self.classname, newid, None, content)
         return newid
 
-    def get(self, nodeid, propname, default=_marker):
+    def get(self, nodeid, propname, default=_marker, cache=1):
         ''' trap the content propname and get it from the file
         '''
         if propname == 'content':
             return self.db.getfile(self.classname, nodeid, None)
         if default is not _marker:
-            return Class.get(self, nodeid, propname, default)
+            return Class.get(self, nodeid, propname, default, cache=cache)
         else:
-            return Class.get(self, nodeid, propname)
+            return Class.get(self, nodeid, propname, cache=cache)
 
     def getprops(self, protected=1):
         ''' In addition to the actual properties on the node, these methods
@@ -270,25 +284,28 @@ class IssueClass(Class):
         
         These users are then added to the message's "recipients" list.
         """
+        users = self.db.user
+        messages = self.db.msg
+        files = self.db.file
+
         # figure the recipient ids
-        recipients = self.db.msg.get(msgid, 'recipients')
+        sendto = []
         r = {}
-        for recipid in recipients:
+        recipients = messages.get(msgid, 'recipients')
+        for recipid in messages.get(msgid, 'recipients'):
             r[recipid] = 1
-        rlen = len(recipients)
 
         # figure the author's id, and indicate they've received the message
-        authid = self.db.msg.get(msgid, 'author')
+        authid = messages.get(msgid, 'author')
 
         # get the current nosy list, we'll need it
         nosy = self.get(nodeid, 'nosy')
 
-        # ... but duplicate the message to the author as long as it's not
-        # the anonymous user
+        # possibly send the message to the author, as long as they aren't
+        # anonymous
         if (self.MESSAGES_TO_AUTHOR == 'yes' and
-                self.db.user.get(authid, 'username') != 'anonymous'):
-            if not r.has_key(authid):
-                recipients.append(authid)
+                users.get(authid, 'username') != 'anonymous'):
+            sendto.append(authid)
         r[authid] = 1
 
         # now figure the nosy people who weren't recipients
@@ -296,26 +313,40 @@ class IssueClass(Class):
             # Don't send nosy mail to the anonymous user (that user
             # shouldn't appear in the nosy list, but just in case they
             # do...)
-            if self.db.user.get(nosyid, 'username') == 'anonymous': continue
+            if users.get(nosyid, 'username') == 'anonymous':
+                continue
+            # make sure they haven't seen the message already
             if not r.has_key(nosyid):
+                # send it to them
+                sendto.append(nosyid)
                 recipients.append(nosyid)
 
         # no new recipients
-        if rlen == len(recipients):
+        if not sendto:
             return
 
+        # determine the messageid and inreplyto of the message
+        inreplyto = messages.get(msgid, 'inreplyto')
+        messageid = messages.get(msgid, 'messageid')
+        if not messageid:
+            # this is an old message that didn't get a messageid, so
+            # create one
+            messageid = "%s.%s.%s%s-%s"%(time.time(), random.random(),
+                self.classname, nodeid, self.MAIL_DOMAIN)
+            messages.set(msgid, messageid=messageid)
+
         # update the message's recipients list
-        self.db.msg.set(msgid, recipients=recipients)
+        messages.set(msgid, recipients=recipients)
 
         # send an email to the people who missed out
-        sendto = [self.db.user.get(i, 'address') for i in recipients]
+        sendto = [users.get(i, 'address') for i in sendto]
         cn = self.classname
         title = self.get(nodeid, 'title') or '%s message copy'%cn
         # figure author information
-        authname = self.db.user.get(authid, 'realname')
+        authname = users.get(authid, 'realname')
         if not authname:
-            authname = self.db.user.get(authid, 'username')
-        authaddr = self.db.user.get(authid, 'address')
+            authname = users.get(authid, 'username')
+        authaddr = users.get(authid, 'address')
         if authaddr:
             authaddr = ' <%s>'%authaddr
         else:
@@ -336,7 +367,7 @@ class IssueClass(Class):
         m.append('')
 
         # add the content
-        m.append(self.db.msg.get(msgid, 'content'))
+        m.append(messages.get(msgid, 'content'))
 
         # add the change note
         if change_note:
@@ -347,7 +378,7 @@ class IssueClass(Class):
             m.append(self.email_signature(nodeid, msgid))
 
         # get the files for this message
-        files = self.db.msg.get(msgid, 'files')
+        files = messages.get(msgid, 'files')
 
         # create the message
         message = cStringIO.StringIO()
@@ -358,6 +389,10 @@ class IssueClass(Class):
         writer.addheader('Reply-To', '%s <%s>'%(self.INSTANCE_NAME,
             self.ISSUE_TRACKER_EMAIL))
         writer.addheader('MIME-Version', '1.0')
+        if messageid:
+            writer.addheader('Message-Id', messageid)
+        if inreplyto:
+            writer.addheader('In-Reply-To', inreplyto)
 
         # attach files
         if files:
@@ -366,9 +401,9 @@ class IssueClass(Class):
             body = part.startbody('text/plain')
             body.write('\n'.join(m))
             for fileid in files:
-                name = self.db.file.get(fileid, 'name')
-                mime_type = self.db.file.get(fileid, 'type')
-                content = self.db.file.get(fileid, 'content')
+                name = files.get(fileid, 'name')
+                mime_type = files.get(fileid, 'type')
+                content = files.get(fileid, 'content')
                 part = writer.nextpart()
                 if mime_type == 'text/plain':
                     part.addheader('Content-Disposition',
@@ -394,21 +429,21 @@ class IssueClass(Class):
             body.write('\n'.join(m))
 
         # now try to send the message
-        try:
-            if ROUNDUPDBSENDMAILDEBUG:
-                print 'From: %s\nTo: %s\n%s\n=-=-=-=-=-=-=-='%(
-                    self.ADMIN_EMAIL, sendto, message.getvalue())
-            else:
+        if SENDMAILDEBUG:
+            open(SENDMAILDEBUG, 'w').write('FROM: %s\nTO: %s\n%s\n'%(
+                self.ADMIN_EMAIL, ', '.join(sendto), message.getvalue()))
+        else:
+            try:
+                # send the message as admin so bounces are sent there
+                # instead of to roundup
                 smtp = smtplib.SMTP(self.MAILHOST)
-                # send the message as admin so bounces are sent there instead
-                # of to roundup
                 smtp.sendmail(self.ADMIN_EMAIL, sendto, message.getvalue())
-        except socket.error, value:
-            raise MessageSendError, \
-                "Couldn't send confirmation email: mailhost %s"%value
-        except smtplib.SMTPException, value:
-            raise MessageSendError, \
-                "Couldn't send confirmation email: %s"%value
+            except socket.error, value:
+                raise MessageSendError, \
+                    "Couldn't send confirmation email: mailhost %s"%value
+            except smtplib.SMTPException, value:
+                raise MessageSendError, \
+                    "Couldn't send confirmation email: %s"%value
 
     def email_signature(self, nodeid, msgid):
         ''' Add a signature to the e-mail with some useful information
@@ -495,6 +530,14 @@ class IssueClass(Class):
 
 #
 # $Log: not supported by cvs2svn $
+# Revision 1.35  2001/12/20 15:43:01  rochecompaan
+# Features added:
+#  .  Multilink properties are now displayed as comma separated values in
+#     a textbox
+#  .  The add user link is now only visible to the admin user
+#  .  Modified the mail gateway to reject submissions from unknown
+#     addresses if ANONYMOUS_ACCESS is denied
+#
 # Revision 1.34  2001/12/17 03:52:48  richard
 # Implemented file store rollback. As a bonus, the hyperdb is now capable of
 # storing more than one file per node - if a property name is supplied,

diff --git a/roundup/templates/classic/dbinit.py b/roundup/templates/classic/dbinit.py
index 865ee757b5aa9aa79e2bd26a8ff4291065f55a70..0ea40ddc7c8da7f6af341084b6e8e741d86beef9 100644 (file)
--- a/roundup/templates/classic/dbinit.py
+++ b/roundup/templates/classic/dbinit.py
@@ -15,7 +15,7 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 # 
-# $Id: dbinit.py,v 1.12 2001-12-02 05:06:16 richard Exp $
+# $Id: dbinit.py,v 1.13 2002-01-02 02:31:38 richard Exp $
 
 import os
 
@@ -75,7 +75,8 @@ def open(name=None):
     msg = FileClass(db, "msg", 
                     author=Link("user"), recipients=Multilink("user"), 
                     date=Date(),         summary=String(), 
-                    files=Multilink("file"))
+                    files=Multilink("file"),
+                    messageid=String(),  inreplyto=String())
 
     file = FileClass(db, "file", 
                     name=String(),       type=String())
@@ -127,6 +128,20 @@ def init(adminpw):
 
 #
 # $Log: not supported by cvs2svn $
+# Revision 1.12  2001/12/02 05:06:16  richard
+# . We now use weakrefs in the Classes to keep the database reference, so
+#   the close() method on the database is no longer needed.
+#   I bumped the minimum python requirement up to 2.1 accordingly.
+# . #487480 ] roundup-server
+# . #487476 ] INSTALL.txt
+#
+# I also cleaned up the change message / post-edit stuff in the cgi client.
+# There's now a clearly marked "TODO: append the change note" where I believe
+# the change note should be added there. The "changes" list will obviously
+# have to be modified to be a dict of the changes, or somesuch.
+#
+# More testing needed.
+#
 # Revision 1.11  2001/12/01 07:17:50  richard
 # . We now have basic transaction support! Information is only written to
 #   the database when the commit() method is called. Only the anydbm

diff --git a/roundup/templates/extended/dbinit.py b/roundup/templates/extended/dbinit.py
index 57cb45c48ce58573453c5060504f923cb48827bb..f50f0d3ffed147374164f1e6085a06ba3ae802dd 100644 (file)
--- a/roundup/templates/extended/dbinit.py
+++ b/roundup/templates/extended/dbinit.py
@@ -15,7 +15,7 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 # 
-# $Id: dbinit.py,v 1.17 2001-12-02 05:06:16 richard Exp $
+# $Id: dbinit.py,v 1.18 2002-01-02 02:31:38 richard Exp $
 
 import os
 
@@ -75,7 +75,8 @@ def open(name=None):
     msg = FileClass(db, "msg", 
                     author=Link("user"), recipients=Multilink("user"), 
                     date=Date(),         summary=String(), 
-                    files=Multilink("file"))
+                    files=Multilink("file"),
+                    messageid=String(),  inreplyto=String())
 
     file = FileClass(db, "file", 
                     name=String(),       type=String())
@@ -178,6 +179,20 @@ def init(adminpw):
 
 #
 # $Log: not supported by cvs2svn $
+# Revision 1.17  2001/12/02 05:06:16  richard
+# . We now use weakrefs in the Classes to keep the database reference, so
+#   the close() method on the database is no longer needed.
+#   I bumped the minimum python requirement up to 2.1 accordingly.
+# . #487480 ] roundup-server
+# . #487476 ] INSTALL.txt
+#
+# I also cleaned up the change message / post-edit stuff in the cgi client.
+# There's now a clearly marked "TODO: append the change note" where I believe
+# the change note should be added there. The "changes" list will obviously
+# have to be modified to be a dict of the changes, or somesuch.
+#
+# More testing needed.
+#
 # Revision 1.16  2001/12/01 07:17:50  richard
 # . We now have basic transaction support! Information is only written to
 #   the database when the commit() method is called. Only the anydbm

diff --git a/roundup/token.py b/roundup/token.py
index c51a8ab92e2a0060de6dcec1d705cc270de54474..d90d8b65a82622f3cf2fa3605b738a9b91e2d5d2 100644 (file)
--- a/roundup/token.py
+++ b/roundup/token.py
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2001 Richard Jones.
+# Copyright (c) 2001 Richard Jones, richard@bofh.asn.au.
 # This module is free software, and you may redistribute it and/or modify
 # under the same terms as Python, so long as this copyright message and
 # disclaimer are retained in their original form.
@@ -8,7 +8,7 @@
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 # 
-# $Id: token.py,v 1.1 2001-12-31 05:09:20 richard Exp $
+# $Id: token.py,v 1.2 2002-01-02 02:31:38 richard Exp $
 #
 
 __doc__ = """
@@ -113,6 +113,10 @@ def token_split(s, whitespace=' \r\n\t', quotes='\'"',
 
 #
 # $Log: not supported by cvs2svn $
+# Revision 1.1  2001/12/31 05:09:20  richard
+# Added better tokenising to roundup-admin - handles spaces and stuff. Can
+# use quoting or backslashes. See the roundup.token pydoc.
+#
 #
 #
 # vim: set filetype=python ts=4 sw=4 et si

diff --git a/test/__init__.py b/test/__init__.py
index 419d6ae8458022c0dff17a9fb45b8aa84299ea0e..5d821a26037e3abeb4493266000d27389729bcad 100644 (file)
--- a/test/__init__.py
+++ b/test/__init__.py
@@ -15,12 +15,14 @@
 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
 # 
-# $Id: __init__.py,v 1.8 2001-12-31 05:09:20 richard Exp $
+# $Id: __init__.py,v 1.9 2002-01-02 02:31:38 richard Exp $
 
 import unittest
+import os, tempfile
+os.environ['SENDMAILDEBUG'] = tempfile.mktemp()
 
 import test_dates, test_schema, test_db, test_multipart, test_mailsplit
-import test_init, test_token
+import test_init, test_token, test_mailgw
 
 def go():
     suite = unittest.TestSuite((
@@ -30,6 +32,7 @@ def go():
         test_init.suite(),
         test_multipart.suite(),
         test_mailsplit.suite(),
+        test_mailgw.suite(),
         test_token.suite(),
     ))
     runner = unittest.TextTestRunner()
@@ -37,6 +40,10 @@ def go():
 
 #
 # $Log: not supported by cvs2svn $
+# Revision 1.8  2001/12/31 05:09:20  richard
+# Added better tokenising to roundup-admin - handles spaces and stuff. Can
+# use quoting or backslashes. See the roundup.token pydoc.
+#
 # Revision 1.7  2001/08/07 00:24:43  richard
 # stupid typo
 #

diff --git a/test/test_mailgw.py b/test/test_mailgw.py
new file mode 100644 (file)
index 0000000..12dbfe2
--- /dev/null
+++ b/test/test_mailgw.py
@@ -0,0 +1,152 @@
+#
+# Copyright (c) 2001 Richard Jones, richard@bofh.asn.au.
+# This module is free software, and you may redistribute it and/or modify
+# under the same terms as Python, so long as this copyright message and
+# disclaimer are retained in their original form.
+#
+# This module is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# $Id: test_mailgw.py,v 1.1 2002-01-02 02:31:38 richard Exp $
+
+import unittest, cStringIO, tempfile, os, shutil, errno, imp, sys
+
+from roundup.mailgw import MailGW
+from roundup import init, instance
+
+class MailgwTestCase(unittest.TestCase):
+    count = 0
+    schema = 'classic'
+    def setUp(self):
+        MailgwTestCase.count = MailgwTestCase.count + 1
+        self.dirname = '_test_%s'%self.count
+        try:
+            shutil.rmtree(self.dirname)
+        except OSError, error:
+            if error.errno not in (errno.ENOENT, errno.ESRCH): raise
+        # create the instance
+        init.init(self.dirname, self.schema, 'anydbm', 'sekrit')
+        # check we can load the package
+        self.instance = instance.open(self.dirname)
+        # and open the database
+        self.db = self.instance.open('sekrit')
+        self.db.user.create(username='Chef', address='chef@bork.bork.bork')
+        self.db.user.create(username='richard', address='richard@test')
+
+    def tearDown(self):
+        if os.path.exists(os.environ['SENDMAILDEBUG']):
+            os.remove(os.environ['SENDMAILDEBUG'])
+        try:
+            shutil.rmtree(self.dirname)
+        except OSError, error:
+            if error.errno not in (errno.ENOENT, errno.ESRCH): raise
+
+    def testNewIssue(self):
+        message = cStringIO.StringIO('''Content-Type: text/plain;
+  charset="iso-8859-1"
+From: Chef <chef@bork.bork.bork
+To: issue_tracker@fill.me.in.
+Cc: richard@test
+Message-Id: <dummy_test_message_id>
+Subject: [issue] Testing...
+
+This is a test submission of a new issue.
+''')
+        handler = self.instance.MailGW(self.instance, self.db)
+        handler.main(message)
+        if os.path.exists(os.environ['SENDMAILDEBUG']):
+            error = open(os.environ['SENDMAILDEBUG']).read()
+            self.assertEqual('no error', error)
+
+    def testNewIssueAuthMsg(self):
+        message = cStringIO.StringIO('''Content-Type: text/plain;
+  charset="iso-8859-1"
+From: Chef <chef@bork.bork.bork
+To: issue_tracker@fill.me.in.
+Message-Id: <dummy_test_message_id>
+Subject: [issue] Testing...
+
+This is a test submission of a new issue.
+''')
+        handler = self.instance.MailGW(self.instance, self.db)
+        # TODO: fix the damn config - this is apalling
+        self.instance.IssueClass.MESSAGES_TO_AUTHOR = 'yes'
+        handler.main(message)
+
+        self.assertEqual(open(os.environ['SENDMAILDEBUG']).read(),
+'''FROM: roundup-admin@fill.me.in.
+TO: chef@bork.bork.bork
+Content-Type: text/plain
+Subject: [issue1] Testing...
+To: chef@bork.bork.bork
+From: Chef <issue_tracker@fill.me.in.>
+Reply-To: Roundup issue tracker <issue_tracker@fill.me.in.>
+MIME-Version: 1.0
+Message-Id: <dummy_test_message_id>
+
+
+New submission from Chef <chef@bork.bork.bork>:
+
+This is a test submission of a new issue.
+
+___________________________________________________
+"Roundup issue tracker" <issue_tracker@fill.me.in.>
+http://some.useful.url/issue1
+___________________________________________________
+''', 'Generated message not correct')
+
+    def testFollowup(self):
+        self.testNewIssue()
+        message = cStringIO.StringIO('''Content-Type: text/plain;
+  charset="iso-8859-1"
+From: richard <richard@test>
+To: issue_tracker@fill.me.in.
+Message-Id: <followup_dummy_id>
+In-Reply-To: <dummy_test_message_id>
+Subject: [issue1] Testing...
+
+This is a followup
+''')
+        handler = self.instance.MailGW(self.instance, self.db)
+        # TODO: fix the damn config - this is apalling
+        handler.main(message)
+
+        self.assertEqual(open(os.environ['SENDMAILDEBUG']).read(),
+'''FROM: roundup-admin@fill.me.in.
+TO: chef@bork.bork.bork
+Content-Type: text/plain
+Subject: [issue1] Testing...
+To: chef@bork.bork.bork
+From: richard <issue_tracker@fill.me.in.>
+Reply-To: Roundup issue tracker <issue_tracker@fill.me.in.>
+MIME-Version: 1.0
+Message-Id: <followup_dummy_id>
+In-Reply-To: <dummy_test_message_id>
+
+
+richard <richard@test> added the comment:
+
+This is a followup
+
+___________________________________________________
+"Roundup issue tracker" <issue_tracker@fill.me.in.>
+http://some.useful.url/issue1
+___________________________________________________
+''', 'Generated message not correct')
+
+class ExtMailgwTestCase(MailgwTestCase):
+    schema = 'extended'
+
+def suite():
+    l = [unittest.makeSuite(MailgwTestCase, 'test'),
+        unittest.makeSuite(ExtMailgwTestCase, 'test')]
+    return unittest.TestSuite(l)
+
+
+#
+# $Log: not supported by cvs2svn $
+#
+#
+#
+# vim: set filetype=python ts=4 sw=4 et si