1 #
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
3 # This module is free software, and you may redistribute it and/or modify
4 # under the same terms as Python, so long as this copyright message and
5 # disclaimer are retained in their original form.
6 #
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
10 # POSSIBILITY OF SUCH DAMAGE.
11 #
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
17 #
18 '''
19 An e-mail gateway for Roundup.
21 Incoming messages are examined for multiple parts:
22 . In a multipart/mixed message or part, each subpart is extracted and
23 examined. The text/plain subparts are assembled to form the textual
24 body of the message, to be stored in the file associated with a "msg"
25 class node. Any parts of other types are each stored in separate files
26 and given "file" class nodes that are linked to the "msg" node.
27 . In a multipart/alternative message or part, we look for a text/plain
28 subpart and ignore the other parts.
30 Summary
31 -------
32 The "summary" property on message nodes is taken from the first non-quoting
33 section in the message body. The message body is divided into sections by
34 blank lines. Sections where the second and all subsequent lines begin with
35 a ">" or "|" character are considered "quoting sections". The first line of
36 the first non-quoting section becomes the summary of the message.
38 Addresses
39 ---------
40 All of the addresses in the To: and Cc: headers of the incoming message are
41 looked up among the user nodes, and the corresponding users are placed in
42 the "recipients" property on the new "msg" node. The address in the From:
43 header similarly determines the "author" property of the new "msg"
44 node. The default handling for addresses that don't have corresponding
45 users is to create new users with no passwords and a username equal to the
46 address. (The web interface does not permit logins for users with no
47 passwords.) If we prefer to reject mail from outside sources, we can simply
48 register an auditor on the "user" class that prevents the creation of user
49 nodes with no passwords.
51 Actions
52 -------
53 The subject line of the incoming message is examined to determine whether
54 the message is an attempt to create a new item or to discuss an existing
55 item. A designator enclosed in square brackets is sought as the first thing
56 on the subject line (after skipping any "Fwd:" or "Re:" prefixes).
58 If an item designator (class name and id number) is found there, the newly
59 created "msg" node is added to the "messages" property for that item, and
60 any new "file" nodes are added to the "files" property for the item.
62 If just an item class name is found there, we attempt to create a new item
63 of that class with its "messages" property initialized to contain the new
64 "msg" node and its "files" property initialized to contain any new "file"
65 nodes.
67 Triggers
68 --------
69 Both cases may trigger detectors (in the first case we are calling the
70 set() method to add the message to the item's spool; in the second case we
71 are calling the create() method to create a new node). If an auditor raises
72 an exception, the original message is bounced back to the sender with the
73 explanatory message given in the exception.
75 $Id: mailgw.py,v 1.34 2001-11-15 10:24:27 richard Exp $
76 '''
79 import string, re, os, mimetools, cStringIO, smtplib, socket, binascii, quopri
80 import traceback
81 import hyperdb, date, password
83 class MailGWError(ValueError):
84 pass
86 class MailUsageError(ValueError):
87 pass
89 class Message(mimetools.Message):
90 ''' subclass mimetools.Message so we can retrieve the parts of the
91 message...
92 '''
93 def getPart(self):
94 ''' Get a single part of a multipart message and return it as a new
95 Message instance.
96 '''
97 boundary = self.getparam('boundary')
98 mid, end = '--'+boundary, '--'+boundary+'--'
99 s = cStringIO.StringIO()
100 while 1:
101 line = self.fp.readline()
102 if not line:
103 break
104 if line.strip() in (mid, end):
105 break
106 s.write(line)
107 if not s.getvalue().strip():
108 return None
109 s.seek(0)
110 return Message(s)
112 subject_re = re.compile(r'(?P<refwd>\s*\W?\s*(fwd|re)\s*\W?\s*)*'
113 r'\s*(\[(?P<classname>[^\d]+)(?P<nodeid>\d+)?\])'
114 r'\s*(?P<title>[^\[]+)(\[(?P<args>.+?)\])?', re.I)
116 class MailGW:
117 def __init__(self, instance, db):
118 self.instance = instance
119 self.db = db
121 def main(self, fp):
122 ''' fp - the file from which to read the Message.
123 '''
124 self.handle_Message(Message(fp))
126 def handle_Message(self, message):
127 '''Handle an RFC822 Message
129 Handle the Message object by calling handle_message() and then cope
130 with any errors raised by handle_message.
131 This method's job is to make that call and handle any
132 errors in a sane manner. It should be replaced if you wish to
133 handle errors in a different manner.
134 '''
135 m = []
136 # in some rare cases, a particularly stuffed-up e-mail will make
137 # its way into here... try to handle it gracefully
138 sendto = message.getaddrlist('from')
139 if sendto:
140 try:
141 self.handle_message(message)
142 return
143 except MailUsageError, value:
144 # bounce the message back to the sender with the usage message
145 fulldoc = '\n'.join(string.split(__doc__, '\n')[2:])
146 sendto = [sendto[0][1]]
147 m = ['Subject: Failed issue tracker submission', '']
148 m.append(str(value))
149 m.append('\n\nMail Gateway Help\n=================')
150 m.append(fulldoc)
151 except:
152 # bounce the message back to the sender with the error message
153 sendto = [sendto[0][1]]
154 m = ['Subject: failed issue tracker submission', '']
155 # TODO as attachments?
156 m.append('---- traceback of failure ----')
157 s = cStringIO.StringIO()
158 import traceback
159 traceback.print_exc(None, s)
160 m.append(s.getvalue())
161 m.append('---- failed message follows ----')
162 try:
163 message.fp.seek(0)
164 except:
165 pass
166 m.append(message.fp.read())
167 else:
168 # very bad-looking message - we don't even know who sent it
169 sendto = [self.ADMIN_EMAIL]
170 m = ['Subject: badly formed message from mail gateway']
171 m.append('')
172 m.append('The mail gateway retrieved a message which has no From:')
173 m.append('line, indicating that it is corrupt. Please check your')
174 m.append('mail gateway source.')
175 m.append('')
176 m.append('---- failed message follows ----')
177 try:
178 message.fp.seek(0)
179 except:
180 pass
181 m.append(message.fp.read())
183 # now send the message
184 try:
185 smtp = smtplib.SMTP(self.MAILHOST)
186 smtp.sendmail(self.ADMIN_EMAIL, sendto, '\n'.join(m))
187 except socket.error, value:
188 raise MailGWError, "Couldn't send confirmation email: "\
189 "mailhost %s"%value
190 except smtplib.SMTPException, value:
191 raise MailGWError, "Couldn't send confirmation email: %s"%value
193 def handle_message(self, message):
194 ''' message - a Message instance
196 Parse the message as per the module docstring.
197 '''
198 # handle the subject line
199 subject = message.getheader('subject', '')
200 m = subject_re.match(subject)
201 if not m:
202 raise MailUsageError, '''
203 The message you sent to roundup did not contain a properly formed subject
204 line. The subject must contain a class name or designator to indicate the
205 "topic" of the message. For example:
206 Subject: [issue] This is a new issue
207 - this will create a new issue in the tracker with the title "This is
208 a new issue".
209 Subject: [issue1234] This is a followup to issue 1234
210 - this will append the message's contents to the existing issue 1234
211 in the tracker.
213 Subject was: "%s"
214 '''%subject
215 classname = m.group('classname')
216 nodeid = m.group('nodeid')
217 title = m.group('title').strip()
218 subject_args = m.group('args')
219 try:
220 cl = self.db.getclass(classname)
221 except KeyError:
222 raise MailUsageError, '''
223 The class name you identified in the subject line ("%s") does not exist in the
224 database.
226 Valid class names are: %s
227 Subject was: "%s"
228 '''%(classname, ', '.join(self.db.getclasses()), subject)
230 # If there's no nodeid, check to see if this is a followup and
231 # maybe someone's responded to the initial mail that created an
232 # entry. Try to find the matching nodes with the same title, and
233 # use the _last_ one matched (since that'll _usually_ be the most
234 # recent...)
235 if not nodeid and m.group('refwd'):
236 l = cl.stringFind(title=title)
237 if l:
238 nodeid = l[-1]
240 # start of the props
241 properties = cl.getprops()
242 props = {}
244 # handle the args
245 args = m.group('args')
246 if args:
247 for prop in string.split(args, ';'):
248 try:
249 key, value = prop.split('=')
250 except ValueError, message:
251 raise MailUsageError, '''
252 Subject argument list not of form [arg=value,value,...;arg=value,value...]
253 (specific exception message was "%s")
255 Subject was: "%s"
256 '''%(message, subject)
257 try:
258 type = properties[key]
259 except KeyError:
260 raise MailUsageError, '''
261 Subject argument list refers to an invalid property: "%s"
263 Subject was: "%s"
264 '''%(key, subject)
265 if isinstance(type, hyperdb.String):
266 props[key] = value
267 if isinstance(type, hyperdb.Password):
268 props[key] = password.Password(value)
269 elif isinstance(type, hyperdb.Date):
270 try:
271 props[key] = date.Date(value)
272 except ValueError, message:
273 raise UsageError, '''
274 Subject argument list contains an invalid date for %s.
276 Error was: %s
277 Subject was: "%s"
278 '''%(key, message, subject)
279 elif isinstance(type, hyperdb.Interval):
280 try:
281 props[key] = date.Interval(value)
282 except ValueError, message:
283 raise UsageError, '''
284 Subject argument list contains an invalid date interval for %s.
286 Error was: %s
287 Subject was: "%s"
288 '''%(key, message, subject)
289 elif isinstance(type, hyperdb.Link):
290 props[key] = value
291 elif isinstance(type, hyperdb.Multilink):
292 props[key] = value.split(',')
294 #
295 # handle the users
296 #
297 author = self.db.uidFromAddress(message.getaddrlist('from')[0])
298 # reopen the database as the author
299 username = self.db.user.get(author, 'username')
300 self.db.close()
301 self.db = self.instance.open(username)
302 # now update the recipients list
303 recipients = []
304 tracker_email = self.ISSUE_TRACKER_EMAIL.lower()
305 for recipient in message.getaddrlist('to') + message.getaddrlist('cc'):
306 if recipient[1].strip().lower() == tracker_email:
307 continue
308 recipients.append(self.db.uidFromAddress(recipient))
310 # now handle the body - find the message
311 content_type = message.gettype()
312 attachments = []
313 if content_type == 'multipart/mixed':
314 # skip over the intro to the first boundary
315 part = message.getPart()
316 content = None
317 while 1:
318 # get the next part
319 part = message.getPart()
320 if part is None:
321 break
322 # parse it
323 subtype = part.gettype()
324 if subtype == 'text/plain' and not content:
325 # add all text/plain parts to the message content
326 if content is None:
327 content = part.fp.read()
328 else:
329 content = content + part.fp.read()
331 elif subtype == 'message/rfc822':
332 # handle message/rfc822 specially - the name should be
333 # the subject of the actual e-mail embedded here
334 i = part.fp.tell()
335 mailmess = Message(part.fp)
336 name = mailmess.getheader('subject')
337 part.fp.seek(i)
338 attachments.append((name, 'message/rfc822', part.fp.read()))
340 else:
341 # try name on Content-Type
342 name = part.getparam('name')
343 # this is just an attachment
344 encoding = part.getencoding()
345 if encoding == 'base64':
346 data = binascii.a2b_base64(part.fp.read())
347 elif encoding == 'quoted-printable':
348 # the quopri module wants to work with files
349 decoded = cStringIO.StringIO()
350 quopri.decode(part.fp, decoded)
351 data = decoded.getvalue()
352 elif encoding == 'uuencoded':
353 data = binascii.a2b_uu(part.fp.read())
354 attachments.append((name, part.gettype(), data))
356 if content is None:
357 raise MailUsageError, '''
358 Roundup requires the submission to be plain text. The message parser could
359 not find a text/plain part to use.
360 '''
362 elif content_type[:10] == 'multipart/':
363 # skip over the intro to the first boundary
364 message.getPart()
365 content = None
366 while 1:
367 # get the next part
368 part = message.getPart()
369 if part is None:
370 break
371 # parse it
372 if part.gettype() == 'text/plain' and not content:
373 # this one's our content
374 content = part.fp.read()
375 if content is None:
376 raise MailUsageError, '''
377 Roundup requires the submission to be plain text. The message parser could
378 not find a text/plain part to use.
379 '''
381 elif content_type != 'text/plain':
382 raise MailUsageError, '''
383 Roundup requires the submission to be plain text. The message parser could
384 not find a text/plain part to use.
385 '''
387 else:
388 content = message.fp.read()
390 summary, content = parseContent(content)
392 # handle the files
393 files = []
394 for (name, type, data) in attachments:
395 files.append(self.db.file.create(type=type, name=name,
396 content=data))
398 # now handle the db stuff
399 if nodeid:
400 # If an item designator (class name and id number) is found there,
401 # the newly created "msg" node is added to the "messages" property
402 # for that item, and any new "file" nodes are added to the "files"
403 # property for the item.
404 message_id = self.db.msg.create(author=author,
405 recipients=recipients, date=date.Date('.'), summary=summary,
406 content=content, files=files)
407 try:
408 messages = cl.get(nodeid, 'messages')
409 except IndexError:
410 raise MailUsageError, '''
411 The node specified by the designator in the subject of your message ("%s")
412 does not exist.
414 Subject was: "%s"
415 '''%(nodeid, subject)
416 messages.append(message_id)
417 props['messages'] = messages
419 # if the message is currently 'unread' or 'resolved', then set
420 # it to 'chatting'
421 if properties.has_key('status'):
422 try:
423 # determine the id of 'unread', 'resolved' and 'chatting'
424 unread_id = self.db.status.lookup('unread')
425 resolved_id = self.db.status.lookup('resolved')
426 chatting_id = self.db.status.lookup('chatting')
427 except KeyError:
428 pass
429 else:
430 if (not props.has_key('status') or
431 props['status'] == unread_id or
432 props['status'] == resolved_id):
433 props['status'] = chatting_id
435 try:
436 cl.set(nodeid, **props)
437 except (TypeError, IndexError, ValueError), message:
438 raise MailUsageError, '''
439 There was a problem with the message you sent:
440 %s
441 '''%message
442 else:
443 # If just an item class name is found there, we attempt to create a
444 # new item of that class with its "messages" property initialized to
445 # contain the new "msg" node and its "files" property initialized to
446 # contain any new "file" nodes.
447 message_id = self.db.msg.create(author=author,
448 recipients=recipients, date=date.Date('.'), summary=summary,
449 content=content, files=files)
450 # fill out the properties with defaults where required
451 if properties.has_key('assignedto') and \
452 not props.has_key('assignedto'):
453 props['assignedto'] = '1' # "admin"
455 # pre-set the issue to unread
456 if properties.has_key('status') and not props.has_key('status'):
457 try:
458 # determine the id of 'unread'
459 unread_id = self.db.status.lookup('unread')
460 except KeyError:
461 pass
462 else:
463 props['status'] = '1'
465 # set the title to the subject
466 if properties.has_key('title') and not props.has_key('title'):
467 props['title'] = title
469 # pre-load the messages list and nosy list
470 props['messages'] = [message_id]
471 props['nosy'] = props.get('nosy', []) + recipients
472 props['nosy'].append(author)
473 props['nosy'].sort()
475 # and attempt to create the new node
476 try:
477 nodeid = cl.create(**props)
478 except (TypeError, IndexError, ValueError), message:
479 raise MailUsageError, '''
480 There was a problem with the message you sent:
481 %s
482 '''%message
484 def parseContent(content, blank_line=re.compile(r'[\r\n]+\s*[\r\n]+'),
485 eol=re.compile(r'[\r\n]+'), signature=re.compile(r'^[>|\s]*[-_]+\s*$')):
486 ''' The message body is divided into sections by blank lines.
487 Sections where the second and all subsequent lines begin with a ">" or "|"
488 character are considered "quoting sections". The first line of the first
489 non-quoting section becomes the summary of the message.
490 '''
491 # strip off leading carriage-returns / newlines
492 i = 0
493 for i in range(len(content)):
494 if content[i] not in '\r\n':
495 break
496 if i > 0:
497 sections = blank_line.split(content[i:])
498 else:
499 sections = blank_line.split(content)
501 # extract out the summary from the message
502 summary = ''
503 l = []
504 for section in sections:
505 #section = section.strip()
506 if not section:
507 continue
508 lines = eol.split(section)
509 if lines[0] and lines[0][0] in '>|':
510 continue
511 if len(lines) > 1 and lines[1] and lines[1][0] in '>|':
512 continue
513 if not summary:
514 summary = lines[0]
515 l.append(section)
516 continue
517 if signature.match(lines[0]):
518 break
519 l.append(section)
520 return summary, '\n\n'.join(l)
522 #
523 # $Log: not supported by cvs2svn $
524 # Revision 1.33 2001/11/13 21:44:44 richard
525 # . re-open the database as the author in mail handling
526 #
527 # Revision 1.32 2001/11/12 22:04:29 richard
528 # oops, left debug in there
529 #
530 # Revision 1.31 2001/11/12 22:01:06 richard
531 # Fixed issues with nosy reaction and author copies.
532 #
533 # Revision 1.30 2001/11/09 22:33:28 richard
534 # More error handling fixes.
535 #
536 # Revision 1.29 2001/11/07 05:29:26 richard
537 # Modified roundup-mailgw so it can read e-mails from a local mail spool
538 # file. Truncates the spool file after parsing.
539 # Fixed a couple of small bugs introduced in roundup.mailgw when I started
540 # the popgw.
541 #
542 # Revision 1.28 2001/11/01 22:04:37 richard
543 # Started work on supporting a pop3-fetching server
544 # Fixed bugs:
545 # . bug #477104 ] HTML tag error in roundup-server
546 # . bug #477107 ] HTTP header problem
547 #
548 # Revision 1.27 2001/10/30 11:26:10 richard
549 # Case-insensitive match for ISSUE_TRACKER_EMAIL in address in e-mail.
550 #
551 # Revision 1.26 2001/10/30 00:54:45 richard
552 # Features:
553 # . #467129 ] Lossage when username=e-mail-address
554 # . #473123 ] Change message generation for author
555 # . MailGW now moves 'resolved' to 'chatting' on receiving e-mail for an issue.
556 #
557 # Revision 1.25 2001/10/28 23:22:28 richard
558 # fixed bug #474749 ] Indentations lost
559 #
560 # Revision 1.24 2001/10/23 22:57:52 richard
561 # Fix unread->chatting auto transition, thanks Roch'e
562 #
563 # Revision 1.23 2001/10/21 04:00:20 richard
564 # MailGW now moves 'unread' to 'chatting' on receiving e-mail for an issue.
565 #
566 # Revision 1.22 2001/10/21 03:35:13 richard
567 # bug #473125: Paragraph in e-mails
568 #
569 # Revision 1.21 2001/10/21 00:53:42 richard
570 # bug #473130: Nosy list not set correctly
571 #
572 # Revision 1.20 2001/10/17 23:13:19 richard
573 # Did a fair bit of work on the admin tool. Now has an extra command "table"
574 # which displays node information in a tabular format. Also fixed import and
575 # export so they work. Removed freshen.
576 # Fixed quopri usage in mailgw from bug reports.
577 #
578 # Revision 1.19 2001/10/11 23:43:04 richard
579 # Implemented the comma-separated printing option in the admin tool.
580 # Fixed a typo (more of a vim-o actually :) in mailgw.
581 #
582 # Revision 1.18 2001/10/11 06:38:57 richard
583 # Initial cut at trying to handle people responding to CC'ed messages that
584 # create an issue.
585 #
586 # Revision 1.17 2001/10/09 07:25:59 richard
587 # Added the Password property type. See "pydoc roundup.password" for
588 # implementation details. Have updated some of the documentation too.
589 #
590 # Revision 1.16 2001/10/05 02:23:24 richard
591 # . roundup-admin create now prompts for property info if none is supplied
592 # on the command-line.
593 # . hyperdb Class getprops() method may now return only the mutable
594 # properties.
595 # . Login now uses cookies, which makes it a whole lot more flexible. We can
596 # now support anonymous user access (read-only, unless there's an
597 # "anonymous" user, in which case write access is permitted). Login
598 # handling has been moved into cgi_client.Client.main()
599 # . The "extended" schema is now the default in roundup init.
600 # . The schemas have had their page headings modified to cope with the new
601 # login handling. Existing installations should copy the interfaces.py
602 # file from the roundup lib directory to their instance home.
603 # . Incorrectly had a Bizar Software copyright on the cgitb.py module from
604 # Ping - has been removed.
605 # . Fixed a whole bunch of places in the CGI interface where we should have
606 # been returning Not Found instead of throwing an exception.
607 # . Fixed a deviation from the spec: trying to modify the 'id' property of
608 # an item now throws an exception.
609 #
610 # Revision 1.15 2001/08/30 06:01:17 richard
611 # Fixed missing import in mailgw :(
612 #
613 # Revision 1.14 2001/08/13 23:02:54 richard
614 # Make the mail parser a little more robust.
615 #
616 # Revision 1.13 2001/08/12 06:32:36 richard
617 # using isinstance(blah, Foo) now instead of isFooType
618 #
619 # Revision 1.12 2001/08/08 01:27:00 richard
620 # Added better error handling to mailgw.
621 #
622 # Revision 1.11 2001/08/08 00:08:03 richard
623 # oops ;)
624 #
625 # Revision 1.10 2001/08/07 00:24:42 richard
626 # stupid typo
627 #
628 # Revision 1.9 2001/08/07 00:15:51 richard
629 # Added the copyright/license notice to (nearly) all files at request of
630 # Bizar Software.
631 #
632 # Revision 1.8 2001/08/05 07:06:07 richard
633 # removed some print statements
634 #
635 # Revision 1.7 2001/08/03 07:18:22 richard
636 # Implemented correct mail splitting (was taking a shortcut). Added unit
637 # tests. Also snips signatures now too.
638 #
639 # Revision 1.6 2001/08/01 04:24:21 richard
640 # mailgw was assuming certain properties existed on the issues being created.
641 #
642 # Revision 1.5 2001/07/29 07:01:39 richard
643 # Added vim command to all source so that we don't get no steenkin' tabs :)
644 #
645 # Revision 1.4 2001/07/28 06:43:02 richard
646 # Multipart message class has the getPart method now. Added some tests for it.
647 #
648 # Revision 1.3 2001/07/28 00:34:34 richard
649 # Fixed some non-string node ids.
650 #
651 # Revision 1.2 2001/07/22 12:09:32 richard
652 # Final commit of Grande Splite
653 #
654 #
655 # vim: set filetype=python ts=4 sw=4 et si