17f62f7e42287d2c1cf94afa72977029be1f9847
1 #
2 # Copyright (c) 2001 Bizar Software Pty Ltd (http://www.bizarsoftware.com.au/)
3 # This module is free software, and you may redistribute it and/or modify
4 # under the same terms as Python, so long as this copyright message and
5 # disclaimer are retained in their original form.
6 #
7 # IN NO EVENT SHALL BIZAR SOFTWARE PTY LTD BE LIABLE TO ANY PARTY FOR
8 # DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING
9 # OUT OF THE USE OF THIS CODE, EVEN IF THE AUTHOR HAS BEEN ADVISED OF THE
10 # POSSIBILITY OF SUCH DAMAGE.
11 #
12 # BIZAR SOFTWARE PTY LTD SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING,
13 # BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
14 # FOR A PARTICULAR PURPOSE. THE CODE PROVIDED HEREUNDER IS ON AN "AS IS"
15 # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE,
16 # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
17 #
18 '''
19 An e-mail gateway for Roundup.
21 Incoming messages are examined for multiple parts:
22 . In a multipart/mixed message or part, each subpart is extracted and
23 examined. The text/plain subparts are assembled to form the textual
24 body of the message, to be stored in the file associated with a "msg"
25 class node. Any parts of other types are each stored in separate files
26 and given "file" class nodes that are linked to the "msg" node.
27 . In a multipart/alternative message or part, we look for a text/plain
28 subpart and ignore the other parts.
30 Summary
31 -------
32 The "summary" property on message nodes is taken from the first non-quoting
33 section in the message body. The message body is divided into sections by
34 blank lines. Sections where the second and all subsequent lines begin with
35 a ">" or "|" character are considered "quoting sections". The first line of
36 the first non-quoting section becomes the summary of the message.
38 Addresses
39 ---------
40 All of the addresses in the To: and Cc: headers of the incoming message are
41 looked up among the user nodes, and the corresponding users are placed in
42 the "recipients" property on the new "msg" node. The address in the From:
43 header similarly determines the "author" property of the new "msg"
44 node. The default handling for addresses that don't have corresponding
45 users is to create new users with no passwords and a username equal to the
46 address. (The web interface does not permit logins for users with no
47 passwords.) If we prefer to reject mail from outside sources, we can simply
48 register an auditor on the "user" class that prevents the creation of user
49 nodes with no passwords.
51 Actions
52 -------
53 The subject line of the incoming message is examined to determine whether
54 the message is an attempt to create a new item or to discuss an existing
55 item. A designator enclosed in square brackets is sought as the first thing
56 on the subject line (after skipping any "Fwd:" or "Re:" prefixes).
58 If an item designator (class name and id number) is found there, the newly
59 created "msg" node is added to the "messages" property for that item, and
60 any new "file" nodes are added to the "files" property for the item.
62 If just an item class name is found there, we attempt to create a new item
63 of that class with its "messages" property initialized to contain the new
64 "msg" node and its "files" property initialized to contain any new "file"
65 nodes.
67 Triggers
68 --------
69 Both cases may trigger detectors (in the first case we are calling the
70 set() method to add the message to the item's spool; in the second case we
71 are calling the create() method to create a new node). If an auditor raises
72 an exception, the original message is bounced back to the sender with the
73 explanatory message given in the exception.
75 $Id: mailgw.py,v 1.31 2001-11-12 22:01:06 richard Exp $
76 '''
79 import string, re, os, mimetools, cStringIO, smtplib, socket, binascii, quopri
80 import traceback
81 import hyperdb, date, password
83 class MailGWError(ValueError):
84 pass
86 class MailUsageError(ValueError):
87 pass
89 class Message(mimetools.Message):
90 ''' subclass mimetools.Message so we can retrieve the parts of the
91 message...
92 '''
93 def getPart(self):
94 ''' Get a single part of a multipart message and return it as a new
95 Message instance.
96 '''
97 boundary = self.getparam('boundary')
98 mid, end = '--'+boundary, '--'+boundary+'--'
99 s = cStringIO.StringIO()
100 while 1:
101 line = self.fp.readline()
102 if not line:
103 break
104 if line.strip() in (mid, end):
105 break
106 s.write(line)
107 if not s.getvalue().strip():
108 return None
109 s.seek(0)
110 return Message(s)
112 subject_re = re.compile(r'(?P<refwd>\s*\W?\s*(fwd|re)\s*\W?\s*)*'
113 r'\s*(\[(?P<classname>[^\d]+)(?P<nodeid>\d+)?\])'
114 r'\s*(?P<title>[^\[]+)(\[(?P<args>.+?)\])?', re.I)
116 class MailGW:
117 def __init__(self, db):
118 self.db = db
120 def main(self, fp):
121 ''' fp - the file from which to read the Message.
122 '''
123 self.handle_Message(Message(fp))
125 def handle_Message(self, message):
126 '''Handle an RFC822 Message
128 Hanle the Message object by calling handle_message() and then cope
129 with any errors raised by handle_message.
130 This method's job is to make that call and handle any
131 errors in a sane manner. It should be replaced if you wish to
132 handle errors in a different manner.
133 '''
134 m = []
135 # in some rare cases, a particularly stuffed-up e-mail will make
136 # its way into here... try to handle it gracefully
137 sendto = message.getaddrlist('from')
138 if sendto:
139 try:
140 self.handle_message(message)
141 sendto = [sendto[0][1]]
142 m = ['Subject: Well, it seemed to work', '', 'hi, mum!']
143 except MailUsageError, value:
144 # bounce the message back to the sender with the usage message
145 fulldoc = '\n'.join(string.split(__doc__, '\n')[2:])
146 sendto = [sendto[0][1]]
147 m = ['Subject: Failed issue tracker submission', '']
148 m.append(str(value))
149 m.append('\n\nMail Gateway Help\n=================')
150 m.append(fulldoc)
151 except:
152 # bounce the message back to the sender with the error message
153 sendto = [sendto[0][1]]
154 m = ['Subject: failed issue tracker submission', '']
155 # TODO as attachments?
156 m.append('---- traceback of failure ----')
157 s = cStringIO.StringIO()
158 import traceback
159 traceback.print_exc(None, s)
160 m.append(s.getvalue())
161 m.append('---- failed message follows ----')
162 try:
163 message.fp.seek(0)
164 except:
165 pass
166 m.append(message.fp.read())
167 else:
168 # very bad-looking message - we don't even know who sent it
169 sendto = [self.ADMIN_EMAIL]
170 m = ['Subject: badly formed message from mail gateway']
171 m.append('')
172 m.append('The mail gateway retrieved a message which has no From:')
173 m.append('line, indicating that it is corrupt. Please check your')
174 m.append('mail gateway source.')
175 m.append('')
176 m.append('---- failed message follows ----')
177 try:
178 message.fp.seek(0)
179 except:
180 pass
181 m.append(message.fp.read())
183 # now send the message
184 try:
185 smtp = smtplib.SMTP(self.MAILHOST)
186 smtp.sendmail(self.ADMIN_EMAIL, sendto, '\n'.join(m))
187 except socket.error, value:
188 raise MailGWError, "Couldn't send confirmation email: "\
189 "mailhost %s"%value
190 except smtplib.SMTPException, value:
191 raise MailGWError, "Couldn't send confirmation email: %s"%value
193 def handle_message(self, message):
194 ''' message - a Message instance
196 Parse the message as per the module docstring.
197 '''
198 # handle the subject line
199 subject = message.getheader('subject', '')
200 m = subject_re.match(subject)
201 if not m:
202 raise MailUsageError, '''
203 The message you sent to roundup did not contain a properly formed subject
204 line. The subject must contain a class name or designator to indicate the
205 "topic" of the message. For example:
206 Subject: [issue] This is a new issue
207 - this will create a new issue in the tracker with the title "This is
208 a new issue".
209 Subject: [issue1234] This is a followup to issue 1234
210 - this will append the message's contents to the existing issue 1234
211 in the tracker.
213 Subject was: "%s"
214 '''%subject
215 classname = m.group('classname')
216 nodeid = m.group('nodeid')
217 title = m.group('title').strip()
218 subject_args = m.group('args')
219 try:
220 cl = self.db.getclass(classname)
221 except KeyError:
222 raise MailUsageError, '''
223 The class name you identified in the subject line ("%s") does not exist in the
224 database.
226 Valid class names are: %s
227 Subject was: "%s"
228 '''%(classname, ', '.join(self.db.getclasses()), subject)
230 # If there's no nodeid, check to see if this is a followup and
231 # maybe someone's responded to the initial mail that created an
232 # entry. Try to find the matching nodes with the same title, and
233 # use the _last_ one matched (since that'll _usually_ be the most
234 # recent...)
235 if not nodeid and m.group('refwd'):
236 l = cl.stringFind(title=title)
237 if l:
238 nodeid = l[-1]
240 # start of the props
241 properties = cl.getprops()
242 props = {}
244 # handle the args
245 args = m.group('args')
246 if args:
247 for prop in string.split(args, ';'):
248 try:
249 key, value = prop.split('=')
250 except ValueError, message:
251 raise MailUsageError, '''
252 Subject argument list not of form [arg=value,value,...;arg=value,value...]
253 (specific exception message was "%s")
255 Subject was: "%s"
256 '''%(message, subject)
257 try:
258 type = properties[key]
259 except KeyError:
260 raise MailUsageError, '''
261 Subject argument list refers to an invalid property: "%s"
263 Subject was: "%s"
264 '''%(key, subject)
265 if isinstance(type, hyperdb.String):
266 props[key] = value
267 if isinstance(type, hyperdb.Password):
268 props[key] = password.Password(value)
269 elif isinstance(type, hyperdb.Date):
270 try:
271 props[key] = date.Date(value)
272 except ValueError, message:
273 raise UsageError, '''
274 Subject argument list contains an invalid date for %s.
276 Error was: %s
277 Subject was: "%s"
278 '''%(key, message, subject)
279 elif isinstance(type, hyperdb.Interval):
280 try:
281 props[key] = date.Interval(value)
282 except ValueError, message:
283 raise UsageError, '''
284 Subject argument list contains an invalid date interval for %s.
286 Error was: %s
287 Subject was: "%s"
288 '''%(key, message, subject)
289 elif isinstance(type, hyperdb.Link):
290 props[key] = value
291 elif isinstance(type, hyperdb.Multilink):
292 props[key] = value.split(',')
294 #
295 # handle the users
296 #
297 author = self.db.uidFromAddress(message.getaddrlist('from')[0])
298 recipients = []
299 tracker_email = self.ISSUE_TRACKER_EMAIL.lower()
300 for recipient in message.getaddrlist('to') + message.getaddrlist('cc'):
301 if recipient[1].strip().lower() == tracker_email:
302 continue
303 recipients.append(self.db.uidFromAddress(recipient))
305 # now handle the body - find the message
306 content_type = message.gettype()
307 attachments = []
308 if content_type == 'multipart/mixed':
309 # skip over the intro to the first boundary
310 part = message.getPart()
311 content = None
312 while 1:
313 # get the next part
314 part = message.getPart()
315 if part is None:
316 break
317 # parse it
318 subtype = part.gettype()
319 if subtype == 'text/plain' and not content:
320 # add all text/plain parts to the message content
321 if content is None:
322 content = part.fp.read()
323 else:
324 content = content + part.fp.read()
326 elif subtype == 'message/rfc822':
327 # handle message/rfc822 specially - the name should be
328 # the subject of the actual e-mail embedded here
329 i = part.fp.tell()
330 mailmess = Message(part.fp)
331 name = mailmess.getheader('subject')
332 part.fp.seek(i)
333 attachments.append((name, 'message/rfc822', part.fp.read()))
335 else:
336 # try name on Content-Type
337 name = part.getparam('name')
338 # this is just an attachment
339 encoding = part.getencoding()
340 if encoding == 'base64':
341 data = binascii.a2b_base64(part.fp.read())
342 elif encoding == 'quoted-printable':
343 # the quopri module wants to work with files
344 decoded = cStringIO.StringIO()
345 quopri.decode(part.fp, decoded)
346 data = decoded.getvalue()
347 elif encoding == 'uuencoded':
348 data = binascii.a2b_uu(part.fp.read())
349 attachments.append((name, part.gettype(), data))
351 if content is None:
352 raise MailUsageError, '''
353 Roundup requires the submission to be plain text. The message parser could
354 not find a text/plain part to use.
355 '''
357 elif content_type[:10] == 'multipart/':
358 # skip over the intro to the first boundary
359 message.getPart()
360 content = None
361 while 1:
362 # get the next part
363 part = message.getPart()
364 if part is None:
365 break
366 # parse it
367 if part.gettype() == 'text/plain' and not content:
368 # this one's our content
369 content = part.fp.read()
370 if content is None:
371 raise MailUsageError, '''
372 Roundup requires the submission to be plain text. The message parser could
373 not find a text/plain part to use.
374 '''
376 elif content_type != 'text/plain':
377 raise MailUsageError, '''
378 Roundup requires the submission to be plain text. The message parser could
379 not find a text/plain part to use.
380 '''
382 else:
383 content = message.fp.read()
385 summary, content = parseContent(content)
387 # handle the files
388 files = []
389 for (name, type, data) in attachments:
390 files.append(self.db.file.create(type=type, name=name,
391 content=data))
393 # now handle the db stuff
394 if nodeid:
395 # If an item designator (class name and id number) is found there,
396 # the newly created "msg" node is added to the "messages" property
397 # for that item, and any new "file" nodes are added to the "files"
398 # property for the item.
399 message_id = self.db.msg.create(author=author,
400 recipients=recipients, date=date.Date('.'), summary=summary,
401 content=content, files=files)
402 try:
403 messages = cl.get(nodeid, 'messages')
404 except IndexError:
405 raise MailUsageError, '''
406 The node specified by the designator in the subject of your message ("%s")
407 does not exist.
409 Subject was: "%s"
410 '''%(nodeid, subject)
411 messages.append(message_id)
412 props['messages'] = messages
414 # if the message is currently 'unread' or 'resolved', then set
415 # it to 'chatting'
416 if properties.has_key('status'):
417 try:
418 # determine the id of 'unread', 'resolved' and 'chatting'
419 unread_id = self.db.status.lookup('unread')
420 resolved_id = self.db.status.lookup('resolved')
421 chatting_id = self.db.status.lookup('chatting')
422 except KeyError:
423 pass
424 else:
425 if (not props.has_key('status') or
426 props['status'] == unread_id or
427 props['status'] == resolved_id):
428 props['status'] = chatting_id
430 try:
431 cl.set(nodeid, **props)
432 except (TypeError, IndexError, ValueError), message:
433 raise MailUsageError, '''
434 There was a problem with the message you sent:
435 %s
436 '''%message
437 else:
438 # If just an item class name is found there, we attempt to create a
439 # new item of that class with its "messages" property initialized to
440 # contain the new "msg" node and its "files" property initialized to
441 # contain any new "file" nodes.
442 message_id = self.db.msg.create(author=author,
443 recipients=recipients, date=date.Date('.'), summary=summary,
444 content=content, files=files)
445 # fill out the properties with defaults where required
446 if properties.has_key('assignedto') and \
447 not props.has_key('assignedto'):
448 props['assignedto'] = '1' # "admin"
450 # pre-set the issue to unread
451 if properties.has_key('status') and not props.has_key('status'):
452 try:
453 # determine the id of 'unread'
454 unread_id = self.db.status.lookup('unread')
455 except KeyError:
456 pass
457 else:
458 props['status'] = '1'
460 # set the title to the subject
461 if properties.has_key('title') and not props.has_key('title'):
462 props['title'] = title
464 # pre-load the messages list and nosy list
465 props['messages'] = [message_id]
466 props['nosy'] = props.get('nosy', []) + recipients
467 props['nosy'].append(author)
468 props['nosy'].sort()
470 # and attempt to create the new node
471 try:
472 nodeid = cl.create(**props)
473 except (TypeError, IndexError, ValueError), message:
474 raise MailUsageError, '''
475 There was a problem with the message you sent:
476 %s
477 '''%message
479 def parseContent(content, blank_line=re.compile(r'[\r\n]+\s*[\r\n]+'),
480 eol=re.compile(r'[\r\n]+'), signature=re.compile(r'^[>|\s]*[-_]+\s*$')):
481 ''' The message body is divided into sections by blank lines.
482 Sections where the second and all subsequent lines begin with a ">" or "|"
483 character are considered "quoting sections". The first line of the first
484 non-quoting section becomes the summary of the message.
485 '''
486 # strip off leading carriage-returns / newlines
487 i = 0
488 for i in range(len(content)):
489 if content[i] not in '\r\n':
490 break
491 if i > 0:
492 sections = blank_line.split(content[i:])
493 else:
494 sections = blank_line.split(content)
496 # extract out the summary from the message
497 summary = ''
498 l = []
499 for section in sections:
500 #section = section.strip()
501 if not section:
502 continue
503 lines = eol.split(section)
504 if lines[0] and lines[0][0] in '>|':
505 continue
506 if len(lines) > 1 and lines[1] and lines[1][0] in '>|':
507 continue
508 if not summary:
509 summary = lines[0]
510 l.append(section)
511 continue
512 if signature.match(lines[0]):
513 break
514 l.append(section)
515 return summary, '\n\n'.join(l)
517 #
518 # $Log: not supported by cvs2svn $
519 # Revision 1.30 2001/11/09 22:33:28 richard
520 # More error handling fixes.
521 #
522 # Revision 1.29 2001/11/07 05:29:26 richard
523 # Modified roundup-mailgw so it can read e-mails from a local mail spool
524 # file. Truncates the spool file after parsing.
525 # Fixed a couple of small bugs introduced in roundup.mailgw when I started
526 # the popgw.
527 #
528 # Revision 1.28 2001/11/01 22:04:37 richard
529 # Started work on supporting a pop3-fetching server
530 # Fixed bugs:
531 # . bug #477104 ] HTML tag error in roundup-server
532 # . bug #477107 ] HTTP header problem
533 #
534 # Revision 1.27 2001/10/30 11:26:10 richard
535 # Case-insensitive match for ISSUE_TRACKER_EMAIL in address in e-mail.
536 #
537 # Revision 1.26 2001/10/30 00:54:45 richard
538 # Features:
539 # . #467129 ] Lossage when username=e-mail-address
540 # . #473123 ] Change message generation for author
541 # . MailGW now moves 'resolved' to 'chatting' on receiving e-mail for an issue.
542 #
543 # Revision 1.25 2001/10/28 23:22:28 richard
544 # fixed bug #474749 ] Indentations lost
545 #
546 # Revision 1.24 2001/10/23 22:57:52 richard
547 # Fix unread->chatting auto transition, thanks Roch'e
548 #
549 # Revision 1.23 2001/10/21 04:00:20 richard
550 # MailGW now moves 'unread' to 'chatting' on receiving e-mail for an issue.
551 #
552 # Revision 1.22 2001/10/21 03:35:13 richard
553 # bug #473125: Paragraph in e-mails
554 #
555 # Revision 1.21 2001/10/21 00:53:42 richard
556 # bug #473130: Nosy list not set correctly
557 #
558 # Revision 1.20 2001/10/17 23:13:19 richard
559 # Did a fair bit of work on the admin tool. Now has an extra command "table"
560 # which displays node information in a tabular format. Also fixed import and
561 # export so they work. Removed freshen.
562 # Fixed quopri usage in mailgw from bug reports.
563 #
564 # Revision 1.19 2001/10/11 23:43:04 richard
565 # Implemented the comma-separated printing option in the admin tool.
566 # Fixed a typo (more of a vim-o actually :) in mailgw.
567 #
568 # Revision 1.18 2001/10/11 06:38:57 richard
569 # Initial cut at trying to handle people responding to CC'ed messages that
570 # create an issue.
571 #
572 # Revision 1.17 2001/10/09 07:25:59 richard
573 # Added the Password property type. See "pydoc roundup.password" for
574 # implementation details. Have updated some of the documentation too.
575 #
576 # Revision 1.16 2001/10/05 02:23:24 richard
577 # . roundup-admin create now prompts for property info if none is supplied
578 # on the command-line.
579 # . hyperdb Class getprops() method may now return only the mutable
580 # properties.
581 # . Login now uses cookies, which makes it a whole lot more flexible. We can
582 # now support anonymous user access (read-only, unless there's an
583 # "anonymous" user, in which case write access is permitted). Login
584 # handling has been moved into cgi_client.Client.main()
585 # . The "extended" schema is now the default in roundup init.
586 # . The schemas have had their page headings modified to cope with the new
587 # login handling. Existing installations should copy the interfaces.py
588 # file from the roundup lib directory to their instance home.
589 # . Incorrectly had a Bizar Software copyright on the cgitb.py module from
590 # Ping - has been removed.
591 # . Fixed a whole bunch of places in the CGI interface where we should have
592 # been returning Not Found instead of throwing an exception.
593 # . Fixed a deviation from the spec: trying to modify the 'id' property of
594 # an item now throws an exception.
595 #
596 # Revision 1.15 2001/08/30 06:01:17 richard
597 # Fixed missing import in mailgw :(
598 #
599 # Revision 1.14 2001/08/13 23:02:54 richard
600 # Make the mail parser a little more robust.
601 #
602 # Revision 1.13 2001/08/12 06:32:36 richard
603 # using isinstance(blah, Foo) now instead of isFooType
604 #
605 # Revision 1.12 2001/08/08 01:27:00 richard
606 # Added better error handling to mailgw.
607 #
608 # Revision 1.11 2001/08/08 00:08:03 richard
609 # oops ;)
610 #
611 # Revision 1.10 2001/08/07 00:24:42 richard
612 # stupid typo
613 #
614 # Revision 1.9 2001/08/07 00:15:51 richard
615 # Added the copyright/license notice to (nearly) all files at request of
616 # Bizar Software.
617 #
618 # Revision 1.8 2001/08/05 07:06:07 richard
619 # removed some print statements
620 #
621 # Revision 1.7 2001/08/03 07:18:22 richard
622 # Implemented correct mail splitting (was taking a shortcut). Added unit
623 # tests. Also snips signatures now too.
624 #
625 # Revision 1.6 2001/08/01 04:24:21 richard
626 # mailgw was assuming certain properties existed on the issues being created.
627 #
628 # Revision 1.5 2001/07/29 07:01:39 richard
629 # Added vim command to all source so that we don't get no steenkin' tabs :)
630 #
631 # Revision 1.4 2001/07/28 06:43:02 richard
632 # Multipart message class has the getPart method now. Added some tests for it.
633 #
634 # Revision 1.3 2001/07/28 00:34:34 richard
635 # Fixed some non-string node ids.
636 #
637 # Revision 1.2 2001/07/22 12:09:32 richard
638 # Final commit of Grande Splite
639 #
640 #
641 # vim: set filetype=python ts=4 sw=4 et si