From: richard Date: Wed, 2 Jan 2002 02:31:38 +0000 (+0000) Subject: Sorry for the huge checkin message - I was only intending to implement #496356 X-Git-Url: https://git.tokkee.org/?a=commitdiff_plain;h=aaf075600ba4813c7a74ef7fb55f5bdbfb2ad831;p=roundup.git Sorry for the huge checkin message - I was only intending to implement #496356 but I found a number of places where things had been broken by transactions: . modified ROUNDUPDBSENDMAILDEBUG to be SENDMAILDEBUG and hold a filename for _all_ roundup-generated smtp messages to be sent to. . the transaction cache had broken the roundupdb.Class set() reactors . newly-created author users in the mailgw weren't being committed to the db Stuff that made it into CHANGES.txt (ie. the stuff I was actually working on when I found that stuff :): . #496356 ] Use threading in messages . detectors were being registered multiple times . added tests for mailgw . much better attaching of erroneous messages in the mail gateway git-svn-id: http://svn.roundup-tracker.org/svnroot/roundup/trunk@487 57a73879-2fb5-44c3-a270-3262357dd7e2 --- diff --git a/CHANGES.txt b/CHANGES.txt index 021b8bf..46d2244 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,7 +1,7 @@ This file contains the changes to the Roundup system over time. The entries are given with the most recent entry first. -2001-12-?? - 0.3.1b1 +2001-12-?? - 0.4.0b1 Feature: . Added INSTANCE_NAME to configuration - used in web and email to identify the instance. @@ -25,6 +25,11 @@ Feature: . Added a Zope frontend for roundup. . Centralised the python version check code, bumped version to 2.1.1 (really needs to be 2.1.2, but that isn't released yet :) + . much better attaching of erroneous messages in the mail gateway + . #496356 ] Use threading in messages + This adds the tracking of messages by message-id and allows threading + using in-reply-to. Most e-mail clients support threading using this + feature, and we hope to add support for it to the web gateway. Fixed: . Lots of bugs, thanks Roché and others on the devel mailing list! @@ -47,7 +52,12 @@ Fixed: . envelope-from is now set to the roundup-admin and not roundup itself so delivery reports aren't sent to roundup (thanks Patrick Ohly) . #495400 ] entering blanks + Values with spaces are now accepted in roundup-admin - check the long help + for details. . #496360 ] table width does not work + . detectors were being registered multiple times + . added tests for mailgw + 2001-11-23 - 0.3.0 Feature: diff --git a/MIGRATION.txt b/MIGRATION.txt index 7288755..c27230a 100644 --- a/MIGRATION.txt +++ b/MIGRATION.txt @@ -1,13 +1,54 @@ Migrating to newer versions of Roundup ====================================== +Please read each section carefully and edit your instance home files +accordingly. -Migrating from 0.2.x to 0.3.x +This file contains information for users upgrading from: + 0.3.x -> 0.4.x + 0.2.x -> 0.3.x + + +Migrating from 0.3.x to 0.4.x ============================= -Please read each section carefully and edit your instance home files -accordingly. +Message-ID and In-Reply-To addition +----------------------------------- +0.4.0 adds the tracking of messages by message-id and allows threading +using in-reply-to. Most e-mail clients support threading using this +feature, and we hope to add support for it to the web gateway. If you +have not edited the dbinit.py file in your instance home directory, you may +simply copy the new dbinit.py file from the core code. If you used the +classic schema, the interfaces file is in: + /roundup/templates/classic/dbinit.py + +If you used the extended schema, the file is in: + + /roundup/templates/extended/dbinit.pybinit.py needs updating from the original. + +If you have modified your dbinit.py file, you may use encoded passwords: + + 1. Edit the dbinit.py file in your instance home directory. Find the lines + which define the msg class: + + msg = FileClass(db, "msg", + author=Link("user"), recipients=Multilink("user"), + date=Date(), summary=String(), + files=Multilink("file")) + + and add the messageid and inreplyto properties like so: + + msg = FileClass(db, "msg", + author=Link("user"), recipients=Multilink("user"), + date=Date(), summary=String(), + files=Multilink("file"), + messageid=String(), inreplyto=String()) + + + +Migrating from 0.2.x to 0.3.x +============================= Cookie Authentication changes ----------------------------- diff --git a/roundup/backends/back_anydbm.py b/roundup/backends/back_anydbm.py index b6226aa..6071940 100644 --- a/roundup/backends/back_anydbm.py +++ b/roundup/backends/back_anydbm.py @@ -15,7 +15,7 @@ # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. # -#$Id: back_anydbm.py,v 1.20 2001-12-18 15:30:34 rochecompaan Exp $ +#$Id: back_anydbm.py,v 1.21 2002-01-02 02:31:38 richard Exp $ ''' This module defines a backend that saves the hyperdatabase in a database chosen by anydbm. It is guaranteed to always be available in python @@ -187,23 +187,25 @@ class Database(hyperdb.Database): print 'savenode', (self, classname, nodeid, node) self.transactions.append((self._doSaveNode, (classname, nodeid, node))) - def getnode(self, classname, nodeid, db=None): + def getnode(self, classname, nodeid, db=None, cache=1): ''' get a node from the database ''' if DEBUG: print 'getnode', (self, classname, nodeid, cldb) - # try the cache - cache = self.cache.setdefault(classname, {}) - if cache.has_key(nodeid): - return cache[nodeid] + if cache: + # try the cache + cache = self.cache.setdefault(classname, {}) + if cache.has_key(nodeid): + return cache[nodeid] # get from the database and save in the cache if db is None: db = self.getclassdb(classname) if not db.has_key(nodeid): - raise IndexError, nodeid + raise IndexError, "no such %s %s"%(classname, nodeid) res = marshal.loads(db[nodeid]) - cache[nodeid] = res + if cache: + cache[nodeid] = res return res def hasnode(self, classname, nodeid, db=None): @@ -402,6 +404,15 @@ class Database(hyperdb.Database): # #$Log: not supported by cvs2svn $ +#Revision 1.20 2001/12/18 15:30:34 rochecompaan +#Fixed bugs: +# . Fixed file creation and retrieval in same transaction in anydbm +# backend +# . Cgi interface now renders new issue after issue creation +# . Could not set issue status to resolved through cgi interface +# . Mail gateway was changing status back to 'chatting' if status was +# omitted as an argument +# #Revision 1.19 2001/12/17 03:52:48 richard #Implemented file store rollback. As a bonus, the hyperdb is now capable of #storing more than one file per node - if a property name is supplied, diff --git a/roundup/cgi_client.py b/roundup/cgi_client.py index b43102a..e319181 100644 --- a/roundup/cgi_client.py +++ b/roundup/cgi_client.py @@ -15,14 +15,14 @@ # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. # -# $Id: cgi_client.py,v 1.87 2001-12-23 23:18:49 richard Exp $ +# $Id: cgi_client.py,v 1.88 2002-01-02 02:31:38 richard Exp $ __doc__ = """ WWW request handler (also used in the stand-alone server). """ import os, cgi, pprint, StringIO, urlparse, re, traceback, mimetypes -import binascii, Cookie, time +import binascii, Cookie, time, random import roundupdb, htmltemplate, date, hyperdb, password from roundup.i18n import _ @@ -456,11 +456,16 @@ class Client: # don't generate a useless message return None, files + # handle the messageid + # TODO: handle inreplyto + messageid = "%s.%s.%s%s-%s"%(time.time(), random.random(), + classname, nodeid, self.MAIL_DOMAIN) + # now create the message, attaching the files content = '\n'.join(m) message_id = self.db.msg.create(author=self.getuid(), recipients=[], date=date.Date('.'), summary=summary, - content=content, files=files) + content=content, files=files, messageid=messageid) # update the messages property return message_id, files @@ -1163,6 +1168,10 @@ def parsePropsFromForm(db, cl, form, nodeid=0): # # $Log: not supported by cvs2svn $ +# Revision 1.87 2001/12/23 23:18:49 richard +# We already had an admin-specific section of the web heading, no need to add +# another one :) +# # Revision 1.86 2001/12/20 15:43:01 rochecompaan # Features added: # . Multilink properties are now displayed as comma separated values in diff --git a/roundup/hyperdb.py b/roundup/hyperdb.py index 26d324a..ea48174 100644 --- a/roundup/hyperdb.py +++ b/roundup/hyperdb.py @@ -15,7 +15,7 @@ # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. # -# $Id: hyperdb.py,v 1.43 2001-12-20 06:13:24 rochecompaan Exp $ +# $Id: hyperdb.py,v 1.44 2002-01-02 02:31:38 richard Exp $ __doc__ = """ Hyperdatabase implementation, especially field types. @@ -240,18 +240,23 @@ class Class: self.db.addjournal(self.classname, newid, 'create', propvalues) return newid - def get(self, nodeid, propname, default=_marker): + def get(self, nodeid, propname, default=_marker, cache=1): """Get the value of a property on an existing node of this class. 'nodeid' must be the id of an existing node of this class or an IndexError is raised. 'propname' must be the name of a property of this class or a KeyError is raised. + + 'cache' indicates whether the transaction cache should be queried + for the node. If the node has been modified and you need to + determine what its values prior to modification are, you need to + set cache=0. """ if propname == 'id': return nodeid # get the node's dict - d = self.db.getnode(self.classname, nodeid) + d = self.db.getnode(self.classname, nodeid, cache=cache) if not d.has_key(propname) and default is not _marker: return default @@ -271,10 +276,18 @@ class Class: return d[propname] # XXX not in spec - def getnode(self, nodeid): - ''' Return a convenience wrapper for the node + def getnode(self, nodeid, cache=1): + ''' Return a convenience wrapper for the node. + + 'nodeid' must be the id of an existing node of this class or an + IndexError is raised. + + 'cache' indicates whether the transaction cache should be queried + for the node. If the node has been modified and you need to + determine what its values prior to modification are, you need to + set cache=0. ''' - return Node(self, nodeid) + return Node(self, nodeid, cache=cache) def set(self, nodeid, **propvalues): """Modify a property on an existing node of this class. @@ -824,20 +837,21 @@ class Class: class Node: ''' A convenience wrapper for the given node ''' - def __init__(self, cl, nodeid): + def __init__(self, cl, nodeid, cache=1): self.__dict__['cl'] = cl self.__dict__['nodeid'] = nodeid + self.cache = cache def keys(self, protected=1): return self.cl.getprops(protected=protected).keys() def values(self, protected=1): l = [] for name in self.cl.getprops(protected=protected).keys(): - l.append(self.cl.get(self.nodeid, name)) + l.append(self.cl.get(self.nodeid, name, cache=self.cache)) return l def items(self, protected=1): l = [] for name in self.cl.getprops(protected=protected).keys(): - l.append((name, self.cl.get(self.nodeid, name))) + l.append((name, self.cl.get(self.nodeid, name, cache=self.cache))) return l def has_key(self, name): return self.cl.getprops().has_key(name) @@ -845,7 +859,7 @@ class Node: if self.__dict__.has_key(name): return self.__dict__[name] try: - return self.cl.get(self.nodeid, name) + return self.cl.get(self.nodeid, name, cache=self.cache) except KeyError, value: # we trap this but re-raise it as AttributeError - all other # exceptions should pass through untrapped @@ -853,7 +867,7 @@ class Node: # nope, no such attribute raise AttributeError, str(value) def __getitem__(self, name): - return self.cl.get(self.nodeid, name) + return self.cl.get(self.nodeid, name, cache=self.cache) def __setattr__(self, name, value): try: return self.cl.set(self.nodeid, **{name: value}) @@ -875,6 +889,16 @@ def Choice(name, *options): # # $Log: not supported by cvs2svn $ +# Revision 1.43 2001/12/20 06:13:24 rochecompaan +# Bugs fixed: +# . Exception handling in hyperdb for strings-that-look-like numbers got +# lost somewhere +# . Internet Explorer submits full path for filename - we now strip away +# the path +# Features added: +# . Link and multilink properties are now displayed sorted in the cgi +# interface +# # Revision 1.42 2001/12/16 10:53:37 richard # take a copy of the node dict so that the subsequent set # operation doesn't modify the oldvalues structure diff --git a/roundup/mailgw.py b/roundup/mailgw.py index 9b3c141..c250c62 100644 --- a/roundup/mailgw.py +++ b/roundup/mailgw.py @@ -73,14 +73,17 @@ are calling the create() method to create a new node). If an auditor raises an exception, the original message is bounced back to the sender with the explanatory message given in the exception. -$Id: mailgw.py,v 1.45 2001-12-20 15:43:01 rochecompaan Exp $ +$Id: mailgw.py,v 1.46 2002-01-02 02:31:38 richard Exp $ ''' import string, re, os, mimetools, cStringIO, smtplib, socket, binascii, quopri +import time, random import traceback, MimeWriter import hyperdb, date, password +SENDMAILDEBUG = os.environ.get('SENDMAILDEBUG', '') + class MailGWError(ValueError): pass @@ -180,14 +183,18 @@ class MailGW: subject='Badly formed message from mail gateway') # now send the message - try: - smtp = smtplib.SMTP(self.MAILHOST) - smtp.sendmail(self.ADMIN_EMAIL, sendto, m.getvalue()) - except socket.error, value: - raise MailGWError, "Couldn't send confirmation email: "\ - "mailhost %s"%value - except smtplib.SMTPException, value: - raise MailGWError, "Couldn't send confirmation email: %s"%value + if SENDMAILDEBUG: + open(SENDMAILDEBUG, 'w').write('From: %s\nTo: %s\n%s\n'%( + self.ADMIN_EMAIL, ', '.join(sendto), m.getvalue())) + else: + try: + smtp = smtplib.SMTP(self.MAILHOST) + smtp.sendmail(self.ADMIN_EMAIL, sendto, m.getvalue()) + except socket.error, value: + raise MailGWError, "Couldn't send error email: "\ + "mailhost %s"%value + except smtplib.SMTPException, value: + raise MailGWError, "Couldn't send error email: %s"%value def bounce_message(self, message, sendto, error, subject='Failed issue tracker submission'): @@ -210,20 +217,24 @@ class MailGW: # reconstruct the original message m = cStringIO.StringIO() w = MimeWriter.MimeWriter(m) + # default the content_type, just in case... + content_type = 'text/plain' + # add the headers except the content-type for header in message.headers: header_name = header.split(':')[0] - if message.getheader(header_name): - w.addheader(header_name,message.getheader(header_name)) - body = w.startbody('text/plain') - try: - message.fp.seek(0) - except: - pass + if header_name.lower() == 'content-type': + content_type = message.getheader(header_name) + elif message.getheader(header_name): + w.addheader(header_name, message.getheader(header_name)) + # now attach the message body + body = w.startbody(content_type) + message.rewindbody() body.write(message.fp.read()) # attach the original message to the returned message part = writer.nextpart() part.addheader('Content-Disposition','attachment') + part.addheader('Content-Description','Message that caused the error') part.addheader('Content-Transfer-Encoding', '7bit') body = part.startbody('message/rfc822') body.write(m.getvalue()) @@ -371,12 +382,11 @@ Subject was: "%s" else: props[key] = [v] - # # handle the users # - # Don't create users if ANONYMOUS_ACCESS is denied + # Don't create users if ANONYMOUS_REGISTER is denied if self.ANONYMOUS_ACCESS == 'deny': create = 0 else: @@ -389,6 +399,10 @@ You are not a registered user. Unknown address: %s '''%message.getaddrlist('from')[0][1] + + # the author may have been created - make sure the change is + # committed before we reopen the database + self.db.commit() # reopen the database as the author username = self.db.user.get(author, 'username') @@ -401,11 +415,24 @@ Unknown address: %s recipients = [] tracker_email = self.ISSUE_TRACKER_EMAIL.lower() for recipient in message.getaddrlist('to') + message.getaddrlist('cc'): - if recipient[1].strip().lower() == tracker_email: + r = recipient[1].strip().lower() + if r == tracker_email or not r: continue recipients.append(self.db.uidFromAddress(recipient)) + # + # handle message-id and in-reply-to + # + messageid = message.getheader('message-id') + inreplyto = message.getheader('in-reply-to') or '' + # generate a messageid if there isn't one + if not messageid: + messageid = "%s.%s.%s%s-%s"%(time.time(), random.random(), + classname, nodeid, self.MAIL_DOMAIN) + + # # now handle the body - find the message + # content_type = message.gettype() attachments = [] if content_type == 'multipart/mixed': @@ -487,13 +514,17 @@ not find a text/plain part to use. summary, content = parseContent(content) - # handle the files + # + # handle the attachments + # files = [] for (name, mime_type, data) in attachments: files.append(self.db.file.create(type=mime_type, name=name, content=data)) + # # now handle the db stuff + # if nodeid: # If an item designator (class name and id number) is found there, # the newly created "msg" node is added to the "messages" property @@ -536,10 +567,11 @@ not find a text/plain part to use. props['nosy'].append(assignedto) except: pass - + message_id = self.db.msg.create(author=author, recipients=recipients, date=date.Date('.'), summary=summary, - content=content, files=files) + content=content, files=files, messageid=messageid, + inreplyto=inreplyto) try: messages = cl.get(nodeid, 'messages') except IndexError: @@ -569,7 +601,8 @@ There was a problem with the message you sent: # contain any new "file" nodes. message_id = self.db.msg.create(author=author, recipients=recipients, date=date.Date('.'), summary=summary, - content=content, files=files) + content=content, files=files, messageid=messageid, + inreplyto=inreplyto) # pre-set the issue to unread if properties.has_key('status') and not props.has_key('status'): @@ -585,8 +618,10 @@ There was a problem with the message you sent: if properties.has_key('title') and not props.has_key('title'): props['title'] = title - # pre-load the messages list and nosy list + # pre-load the messages list props['messages'] = [message_id] + + # set up (clean) the nosy list nosy = props.get('nosy', []) n = {} for value in nosy: @@ -596,18 +631,30 @@ There was a problem with the message you sent: continue if n.has_key(nid): continue n[nid] = 1 - props['nosy'] = n.keys() + recipients + props['nosy'] = n.keys() + # add on the recipients of the message + for recipient in recipients: + if not n.has_key(recipient): + props['nosy'].append(recipient) + n[recipient] = 1 + # add the author to the nosy list if not n.has_key(author): props['nosy'].append(author) n[author] = 1 + # add assignedto to the nosy list - try: - assignedto = self.db.user.lookup(props['assignedto']) + if properties.has_key('assignedto') and props.has_key('assignedto'): + try: + assignedto = self.db.user.lookup(props['assignedto']) + except KeyError: + raise MailUsageError, ''' +There was a problem with the message you sent: + Assignedto user '%s' doesn't exist +'''%props['assignedto'] if not n.has_key(assignedto): props['nosy'].append(assignedto) - except: - pass + n[assignedto] = 1 # and attempt to create the new node try: @@ -661,6 +708,14 @@ def parseContent(content, blank_line=re.compile(r'[\r\n]+\s*[\r\n]+'), # # $Log: not supported by cvs2svn $ +# Revision 1.45 2001/12/20 15:43:01 rochecompaan +# Features added: +# . Multilink properties are now displayed as comma separated values in +# a textbox +# . The add user link is now only visible to the admin user +# . Modified the mail gateway to reject submissions from unknown +# addresses if ANONYMOUS_ACCESS is denied +# # Revision 1.44 2001/12/18 15:30:34 rochecompaan # Fixed bugs: # . Fixed file creation and retrieval in same transaction in anydbm diff --git a/roundup/roundupdb.py b/roundup/roundupdb.py index 603e608..bd1933e 100644 --- a/roundup/roundupdb.py +++ b/roundup/roundupdb.py @@ -15,20 +15,21 @@ # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. # -# $Id: roundupdb.py,v 1.35 2001-12-20 15:43:01 rochecompaan Exp $ +# $Id: roundupdb.py,v 1.36 2002-01-02 02:31:38 richard Exp $ __doc__ = """ Extending hyperdb with types specific to issue-tracking. """ -import re, os, smtplib, socket, copy +import re, os, smtplib, socket, copy, time, random import mimetools, MimeWriter, cStringIO import base64, mimetypes import hyperdb, date # set to indicate to roundup not to actually _send_ email -ROUNDUPDBSENDMAILDEBUG = os.environ.get('ROUNDUPDBSENDMAILDEBUG', '') +# this var must contain a file to write the mail to +SENDMAILDEBUG = os.environ.get('SENDMAILDEBUG', '') class DesignatorError(ValueError): pass @@ -72,6 +73,7 @@ class Database: # couldn't match address or username, so create a new user if create: + print 'CREATING USER', address return self.user.create(username=address, address=address, realname=realname) else: @@ -110,9 +112,16 @@ class Class(hyperdb.Class): raise KeyError, '"creation" and "activity" are reserved' for audit in self.auditors['set']: audit(self.db, self, nodeid, propvalues) - # take a copy of the node dict so that the subsequent set - # operation doesn't modify the oldvalues structure - oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid)) + # Take a copy of the node dict so that the subsequent set + # operation doesn't modify the oldvalues structure. + try: + # try not using the cache initially + oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid, + cache=0)) + except IndexError: + # this will be needed if somone does a create() and set() + # with no intervening commit() + oldvalues = copy.deepcopy(self.db.getnode(self.classname, nodeid)) hyperdb.Class.set(self, nodeid, **propvalues) for react in self.reactors['set']: react(self.db, self, nodeid, oldvalues) @@ -127,7 +136,7 @@ class Class(hyperdb.Class): for react in self.reactors['retire']: react(self.db, self, nodeid, None) - def get(self, nodeid, propname, default=_marker): + def get(self, nodeid, propname, default=_marker, cache=1): """Attempts to get the "creation" or "activity" properties should do the right thing. """ @@ -153,9 +162,10 @@ class Class(hyperdb.Class): return None return self.db.user.lookup(name) if default is not _marker: - return hyperdb.Class.get(self, nodeid, propname, default) + return hyperdb.Class.get(self, nodeid, propname, default, + cache=cache) else: - return hyperdb.Class.get(self, nodeid, propname) + return hyperdb.Class.get(self, nodeid, propname, cache=cache) def getprops(self, protected=1): """In addition to the actual properties on the node, these @@ -176,12 +186,16 @@ class Class(hyperdb.Class): def audit(self, event, detector): """Register a detector """ - self.auditors[event].append(detector) + l = self.auditors[event] + if detector not in l: + self.auditors[event].append(detector) def react(self, event, detector): """Register a detector """ - self.reactors[event].append(detector) + l = self.reactors[event] + if detector not in l: + self.reactors[event].append(detector) class FileClass(Class): @@ -194,15 +208,15 @@ class FileClass(Class): self.db.storefile(self.classname, newid, None, content) return newid - def get(self, nodeid, propname, default=_marker): + def get(self, nodeid, propname, default=_marker, cache=1): ''' trap the content propname and get it from the file ''' if propname == 'content': return self.db.getfile(self.classname, nodeid, None) if default is not _marker: - return Class.get(self, nodeid, propname, default) + return Class.get(self, nodeid, propname, default, cache=cache) else: - return Class.get(self, nodeid, propname) + return Class.get(self, nodeid, propname, cache=cache) def getprops(self, protected=1): ''' In addition to the actual properties on the node, these methods @@ -270,25 +284,28 @@ class IssueClass(Class): These users are then added to the message's "recipients" list. """ + users = self.db.user + messages = self.db.msg + files = self.db.file + # figure the recipient ids - recipients = self.db.msg.get(msgid, 'recipients') + sendto = [] r = {} - for recipid in recipients: + recipients = messages.get(msgid, 'recipients') + for recipid in messages.get(msgid, 'recipients'): r[recipid] = 1 - rlen = len(recipients) # figure the author's id, and indicate they've received the message - authid = self.db.msg.get(msgid, 'author') + authid = messages.get(msgid, 'author') # get the current nosy list, we'll need it nosy = self.get(nodeid, 'nosy') - # ... but duplicate the message to the author as long as it's not - # the anonymous user + # possibly send the message to the author, as long as they aren't + # anonymous if (self.MESSAGES_TO_AUTHOR == 'yes' and - self.db.user.get(authid, 'username') != 'anonymous'): - if not r.has_key(authid): - recipients.append(authid) + users.get(authid, 'username') != 'anonymous'): + sendto.append(authid) r[authid] = 1 # now figure the nosy people who weren't recipients @@ -296,26 +313,40 @@ class IssueClass(Class): # Don't send nosy mail to the anonymous user (that user # shouldn't appear in the nosy list, but just in case they # do...) - if self.db.user.get(nosyid, 'username') == 'anonymous': continue + if users.get(nosyid, 'username') == 'anonymous': + continue + # make sure they haven't seen the message already if not r.has_key(nosyid): + # send it to them + sendto.append(nosyid) recipients.append(nosyid) # no new recipients - if rlen == len(recipients): + if not sendto: return + # determine the messageid and inreplyto of the message + inreplyto = messages.get(msgid, 'inreplyto') + messageid = messages.get(msgid, 'messageid') + if not messageid: + # this is an old message that didn't get a messageid, so + # create one + messageid = "%s.%s.%s%s-%s"%(time.time(), random.random(), + self.classname, nodeid, self.MAIL_DOMAIN) + messages.set(msgid, messageid=messageid) + # update the message's recipients list - self.db.msg.set(msgid, recipients=recipients) + messages.set(msgid, recipients=recipients) # send an email to the people who missed out - sendto = [self.db.user.get(i, 'address') for i in recipients] + sendto = [users.get(i, 'address') for i in sendto] cn = self.classname title = self.get(nodeid, 'title') or '%s message copy'%cn # figure author information - authname = self.db.user.get(authid, 'realname') + authname = users.get(authid, 'realname') if not authname: - authname = self.db.user.get(authid, 'username') - authaddr = self.db.user.get(authid, 'address') + authname = users.get(authid, 'username') + authaddr = users.get(authid, 'address') if authaddr: authaddr = ' <%s>'%authaddr else: @@ -336,7 +367,7 @@ class IssueClass(Class): m.append('') # add the content - m.append(self.db.msg.get(msgid, 'content')) + m.append(messages.get(msgid, 'content')) # add the change note if change_note: @@ -347,7 +378,7 @@ class IssueClass(Class): m.append(self.email_signature(nodeid, msgid)) # get the files for this message - files = self.db.msg.get(msgid, 'files') + files = messages.get(msgid, 'files') # create the message message = cStringIO.StringIO() @@ -358,6 +389,10 @@ class IssueClass(Class): writer.addheader('Reply-To', '%s <%s>'%(self.INSTANCE_NAME, self.ISSUE_TRACKER_EMAIL)) writer.addheader('MIME-Version', '1.0') + if messageid: + writer.addheader('Message-Id', messageid) + if inreplyto: + writer.addheader('In-Reply-To', inreplyto) # attach files if files: @@ -366,9 +401,9 @@ class IssueClass(Class): body = part.startbody('text/plain') body.write('\n'.join(m)) for fileid in files: - name = self.db.file.get(fileid, 'name') - mime_type = self.db.file.get(fileid, 'type') - content = self.db.file.get(fileid, 'content') + name = files.get(fileid, 'name') + mime_type = files.get(fileid, 'type') + content = files.get(fileid, 'content') part = writer.nextpart() if mime_type == 'text/plain': part.addheader('Content-Disposition', @@ -394,21 +429,21 @@ class IssueClass(Class): body.write('\n'.join(m)) # now try to send the message - try: - if ROUNDUPDBSENDMAILDEBUG: - print 'From: %s\nTo: %s\n%s\n=-=-=-=-=-=-=-='%( - self.ADMIN_EMAIL, sendto, message.getvalue()) - else: + if SENDMAILDEBUG: + open(SENDMAILDEBUG, 'w').write('FROM: %s\nTO: %s\n%s\n'%( + self.ADMIN_EMAIL, ', '.join(sendto), message.getvalue())) + else: + try: + # send the message as admin so bounces are sent there + # instead of to roundup smtp = smtplib.SMTP(self.MAILHOST) - # send the message as admin so bounces are sent there instead - # of to roundup smtp.sendmail(self.ADMIN_EMAIL, sendto, message.getvalue()) - except socket.error, value: - raise MessageSendError, \ - "Couldn't send confirmation email: mailhost %s"%value - except smtplib.SMTPException, value: - raise MessageSendError, \ - "Couldn't send confirmation email: %s"%value + except socket.error, value: + raise MessageSendError, \ + "Couldn't send confirmation email: mailhost %s"%value + except smtplib.SMTPException, value: + raise MessageSendError, \ + "Couldn't send confirmation email: %s"%value def email_signature(self, nodeid, msgid): ''' Add a signature to the e-mail with some useful information @@ -495,6 +530,14 @@ class IssueClass(Class): # # $Log: not supported by cvs2svn $ +# Revision 1.35 2001/12/20 15:43:01 rochecompaan +# Features added: +# . Multilink properties are now displayed as comma separated values in +# a textbox +# . The add user link is now only visible to the admin user +# . Modified the mail gateway to reject submissions from unknown +# addresses if ANONYMOUS_ACCESS is denied +# # Revision 1.34 2001/12/17 03:52:48 richard # Implemented file store rollback. As a bonus, the hyperdb is now capable of # storing more than one file per node - if a property name is supplied, diff --git a/roundup/templates/classic/dbinit.py b/roundup/templates/classic/dbinit.py index 865ee75..0ea40dd 100644 --- a/roundup/templates/classic/dbinit.py +++ b/roundup/templates/classic/dbinit.py @@ -15,7 +15,7 @@ # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. # -# $Id: dbinit.py,v 1.12 2001-12-02 05:06:16 richard Exp $ +# $Id: dbinit.py,v 1.13 2002-01-02 02:31:38 richard Exp $ import os @@ -75,7 +75,8 @@ def open(name=None): msg = FileClass(db, "msg", author=Link("user"), recipients=Multilink("user"), date=Date(), summary=String(), - files=Multilink("file")) + files=Multilink("file"), + messageid=String(), inreplyto=String()) file = FileClass(db, "file", name=String(), type=String()) @@ -127,6 +128,20 @@ def init(adminpw): # # $Log: not supported by cvs2svn $ +# Revision 1.12 2001/12/02 05:06:16 richard +# . We now use weakrefs in the Classes to keep the database reference, so +# the close() method on the database is no longer needed. +# I bumped the minimum python requirement up to 2.1 accordingly. +# . #487480 ] roundup-server +# . #487476 ] INSTALL.txt +# +# I also cleaned up the change message / post-edit stuff in the cgi client. +# There's now a clearly marked "TODO: append the change note" where I believe +# the change note should be added there. The "changes" list will obviously +# have to be modified to be a dict of the changes, or somesuch. +# +# More testing needed. +# # Revision 1.11 2001/12/01 07:17:50 richard # . We now have basic transaction support! Information is only written to # the database when the commit() method is called. Only the anydbm diff --git a/roundup/templates/extended/dbinit.py b/roundup/templates/extended/dbinit.py index 57cb45c..f50f0d3 100644 --- a/roundup/templates/extended/dbinit.py +++ b/roundup/templates/extended/dbinit.py @@ -15,7 +15,7 @@ # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. # -# $Id: dbinit.py,v 1.17 2001-12-02 05:06:16 richard Exp $ +# $Id: dbinit.py,v 1.18 2002-01-02 02:31:38 richard Exp $ import os @@ -75,7 +75,8 @@ def open(name=None): msg = FileClass(db, "msg", author=Link("user"), recipients=Multilink("user"), date=Date(), summary=String(), - files=Multilink("file")) + files=Multilink("file"), + messageid=String(), inreplyto=String()) file = FileClass(db, "file", name=String(), type=String()) @@ -178,6 +179,20 @@ def init(adminpw): # # $Log: not supported by cvs2svn $ +# Revision 1.17 2001/12/02 05:06:16 richard +# . We now use weakrefs in the Classes to keep the database reference, so +# the close() method on the database is no longer needed. +# I bumped the minimum python requirement up to 2.1 accordingly. +# . #487480 ] roundup-server +# . #487476 ] INSTALL.txt +# +# I also cleaned up the change message / post-edit stuff in the cgi client. +# There's now a clearly marked "TODO: append the change note" where I believe +# the change note should be added there. The "changes" list will obviously +# have to be modified to be a dict of the changes, or somesuch. +# +# More testing needed. +# # Revision 1.16 2001/12/01 07:17:50 richard # . We now have basic transaction support! Information is only written to # the database when the commit() method is called. Only the anydbm diff --git a/roundup/token.py b/roundup/token.py index c51a8ab..d90d8b6 100644 --- a/roundup/token.py +++ b/roundup/token.py @@ -1,5 +1,5 @@ # -# Copyright (c) 2001 Richard Jones. +# Copyright (c) 2001 Richard Jones, richard@bofh.asn.au. # This module is free software, and you may redistribute it and/or modify # under the same terms as Python, so long as this copyright message and # disclaimer are retained in their original form. @@ -8,7 +8,7 @@ # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. # -# $Id: token.py,v 1.1 2001-12-31 05:09:20 richard Exp $ +# $Id: token.py,v 1.2 2002-01-02 02:31:38 richard Exp $ # __doc__ = """ @@ -113,6 +113,10 @@ def token_split(s, whitespace=' \r\n\t', quotes='\'"', # # $Log: not supported by cvs2svn $ +# Revision 1.1 2001/12/31 05:09:20 richard +# Added better tokenising to roundup-admin - handles spaces and stuff. Can +# use quoting or backslashes. See the roundup.token pydoc. +# # # # vim: set filetype=python ts=4 sw=4 et si diff --git a/test/__init__.py b/test/__init__.py index 419d6ae..5d821a2 100644 --- a/test/__init__.py +++ b/test/__init__.py @@ -15,12 +15,14 @@ # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. # -# $Id: __init__.py,v 1.8 2001-12-31 05:09:20 richard Exp $ +# $Id: __init__.py,v 1.9 2002-01-02 02:31:38 richard Exp $ import unittest +import os, tempfile +os.environ['SENDMAILDEBUG'] = tempfile.mktemp() import test_dates, test_schema, test_db, test_multipart, test_mailsplit -import test_init, test_token +import test_init, test_token, test_mailgw def go(): suite = unittest.TestSuite(( @@ -30,6 +32,7 @@ def go(): test_init.suite(), test_multipart.suite(), test_mailsplit.suite(), + test_mailgw.suite(), test_token.suite(), )) runner = unittest.TextTestRunner() @@ -37,6 +40,10 @@ def go(): # # $Log: not supported by cvs2svn $ +# Revision 1.8 2001/12/31 05:09:20 richard +# Added better tokenising to roundup-admin - handles spaces and stuff. Can +# use quoting or backslashes. See the roundup.token pydoc. +# # Revision 1.7 2001/08/07 00:24:43 richard # stupid typo # diff --git a/test/test_mailgw.py b/test/test_mailgw.py new file mode 100644 index 0000000..12dbfe2 --- /dev/null +++ b/test/test_mailgw.py @@ -0,0 +1,152 @@ +# +# Copyright (c) 2001 Richard Jones, richard@bofh.asn.au. +# This module is free software, and you may redistribute it and/or modify +# under the same terms as Python, so long as this copyright message and +# disclaimer are retained in their original form. +# +# This module is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. +# +# $Id: test_mailgw.py,v 1.1 2002-01-02 02:31:38 richard Exp $ + +import unittest, cStringIO, tempfile, os, shutil, errno, imp, sys + +from roundup.mailgw import MailGW +from roundup import init, instance + +class MailgwTestCase(unittest.TestCase): + count = 0 + schema = 'classic' + def setUp(self): + MailgwTestCase.count = MailgwTestCase.count + 1 + self.dirname = '_test_%s'%self.count + try: + shutil.rmtree(self.dirname) + except OSError, error: + if error.errno not in (errno.ENOENT, errno.ESRCH): raise + # create the instance + init.init(self.dirname, self.schema, 'anydbm', 'sekrit') + # check we can load the package + self.instance = instance.open(self.dirname) + # and open the database + self.db = self.instance.open('sekrit') + self.db.user.create(username='Chef', address='chef@bork.bork.bork') + self.db.user.create(username='richard', address='richard@test') + + def tearDown(self): + if os.path.exists(os.environ['SENDMAILDEBUG']): + os.remove(os.environ['SENDMAILDEBUG']) + try: + shutil.rmtree(self.dirname) + except OSError, error: + if error.errno not in (errno.ENOENT, errno.ESRCH): raise + + def testNewIssue(self): + message = cStringIO.StringIO('''Content-Type: text/plain; + charset="iso-8859-1" +From: Chef +Subject: [issue] Testing... + +This is a test submission of a new issue. +''') + handler = self.instance.MailGW(self.instance, self.db) + handler.main(message) + if os.path.exists(os.environ['SENDMAILDEBUG']): + error = open(os.environ['SENDMAILDEBUG']).read() + self.assertEqual('no error', error) + + def testNewIssueAuthMsg(self): + message = cStringIO.StringIO('''Content-Type: text/plain; + charset="iso-8859-1" +From: Chef +Subject: [issue] Testing... + +This is a test submission of a new issue. +''') + handler = self.instance.MailGW(self.instance, self.db) + # TODO: fix the damn config - this is apalling + self.instance.IssueClass.MESSAGES_TO_AUTHOR = 'yes' + handler.main(message) + + self.assertEqual(open(os.environ['SENDMAILDEBUG']).read(), +'''FROM: roundup-admin@fill.me.in. +TO: chef@bork.bork.bork +Content-Type: text/plain +Subject: [issue1] Testing... +To: chef@bork.bork.bork +From: Chef +Reply-To: Roundup issue tracker +MIME-Version: 1.0 +Message-Id: + + +New submission from Chef : + +This is a test submission of a new issue. + +___________________________________________________ +"Roundup issue tracker" +http://some.useful.url/issue1 +___________________________________________________ +''', 'Generated message not correct') + + def testFollowup(self): + self.testNewIssue() + message = cStringIO.StringIO('''Content-Type: text/plain; + charset="iso-8859-1" +From: richard +To: issue_tracker@fill.me.in. +Message-Id: +In-Reply-To: +Subject: [issue1] Testing... + +This is a followup +''') + handler = self.instance.MailGW(self.instance, self.db) + # TODO: fix the damn config - this is apalling + handler.main(message) + + self.assertEqual(open(os.environ['SENDMAILDEBUG']).read(), +'''FROM: roundup-admin@fill.me.in. +TO: chef@bork.bork.bork +Content-Type: text/plain +Subject: [issue1] Testing... +To: chef@bork.bork.bork +From: richard +Reply-To: Roundup issue tracker +MIME-Version: 1.0 +Message-Id: +In-Reply-To: + + +richard added the comment: + +This is a followup + +___________________________________________________ +"Roundup issue tracker" +http://some.useful.url/issue1 +___________________________________________________ +''', 'Generated message not correct') + +class ExtMailgwTestCase(MailgwTestCase): + schema = 'extended' + +def suite(): + l = [unittest.makeSuite(MailgwTestCase, 'test'), + unittest.makeSuite(ExtMailgwTestCase, 'test')] + return unittest.TestSuite(l) + + +# +# $Log: not supported by cvs2svn $ +# +# +# +# vim: set filetype=python ts=4 sw=4 et si