From: richard Date: Mon, 17 Dec 2001 03:52:48 +0000 (+0000) Subject: Implemented file store rollback. As a bonus, the hyperdb is now capable of X-Git-Url: https://git.tokkee.org/?a=commitdiff_plain;h=1fa4095c2850f546bddd49f171785fe6a3cbeba2;p=roundup.git Implemented file store rollback. As a bonus, the hyperdb is now capable of storing more than one file per node - if a property name is supplied, the file is called designator.property. I decided not to migrate the existing files stored over to the new naming scheme - the FileClass just doesn't specify the property name. git-svn-id: http://svn.roundup-tracker.org/svnroot/roundup/trunk@473 57a73879-2fb5-44c3-a270-3262357dd7e2 --- diff --git a/CHANGES.txt b/CHANGES.txt index 7d44745..e247897 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,7 +1,7 @@ This file contains the changes to the Roundup system over time. The entries are given with the most recent entry first. -2001-11-?? - 0.3.1 +2001-12-?? - 0.3.1b1 Feature: . Added INSTANCE_NAME to configuration - used in web and email to identify the instance. diff --git a/doc/announcement.txt b/doc/announcement.txt index f1ca352..cdba406 100644 --- a/doc/announcement.txt +++ b/doc/announcement.txt @@ -1,16 +1,16 @@ - Roundup 0.3.0 - an issue tracking system + Roundup 0.3.1b1 - an issue tracking system -This release contains several new features which will require migration, so -please read MIGRATION.txt! +If you are upgrading from pre-0.3.0, please read MIGRATION.txt. + +Roundup requires python 2.1.1 for correct operation. Support for dumbdbm has +been disabled until python 2.1.2 and 2.2 are released. Big stuff in this release: - - lots of bug fixes, thanks to all users for their great feedback! - - much more flexible administration tool - - much better handling of errors - - more configuration options - - CGI login uses cookies instead of basic auth - - passwords are encoded in the database - - much, much more: see the CHANGES file for details. + - Use of transactions to prevent partial data commits + - Zope Product front-end + - Nicer, more consistent change message generation + - Several bug fixes + - Much, much more: see the CHANGES file for details. Source and documentation is available at the website: http://roundup.sourceforge.net/ diff --git a/doc/index.html b/doc/index.html index 59d2f35..b53b2b6 100644 --- a/doc/index.html +++ b/doc/index.html @@ -2,7 +2,7 @@ Roundup: an Issue-Tracking System for Knowledge Workers -

Roundup (0.3.0)

+

Roundup (0.3.1)

An Issue-Tracking System for Knowledge Workers

Contents

@@ -46,12 +46,14 @@

Prerequisites

-
-
Python 2.1.1 is required for the correct operation of roundup -
+

+Python 2.1.1 is required for the correct operation of roundup. +

+

Download the latest version from http://www.python.org/. +

@@ -342,96 +344,176 @@ Usage:

- +
- - - + + + + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + + + + - - + + + - - - - - - - - - - - + + + + + + + - + - + - - - - - - - + + - + + + + + - -
Command Help
historyhistory designator

- Lists the journal entries for the node identified by the designator. -

commitUsage: commit

+

+The changes made during an interactive session are not
+automatically written to the database - they must be committed
+using this command.
+
+One-off commands on the command-line are automatically committed if
+they are successful.
+
+
createUsage: create classname property=value ...

+

+This creates a new entry of the given class using the property
+name=value arguments provided on the command line after the "create"
+command.
+
+
displayUsage: display designator

+

+This lists the properties and their associated values for the given
+node.
+
+
exportUsage: export class[,class] destination_dir

+

+This action exports the current data from the database into
+tab-separated-value files that are placed in the nominated destination
+directory. The journals are not exported.
+
+
findfind classname propname=value ...

- Find the nodes of the given class with a given property value. The - value may be either the nodeid of the linked node, or its key value. -

Usage: find classname propname=value ...

+

+Find the nodes of the given class with a given link property value. The
+value may be either the nodeid of the linked node, or its key value.
+
+
getUsage: get property designator[,designator]*

+

+Retrieves the property value of the nodes specified by the designators.
+
+
helpUsage: help topic

+

+commands  -- list commands
+ -- help specific to a command
+initopts  -- init command options
+all       -- all available help
+
+
historyUsage: history designator

+

+Lists the journal entries for the node identified by the designator.
+
+
importUsage: import class file

+

+The file must define the same properties as the class (including having
+a "header" line with those property names.) The new nodes are added to
+the existing database - if you want to create a new database using the
+imported data, then create a new database (or, tediously, retire all
+the old data.)
+
+
initialiseUsage: initialise [template [backend [admin password]]]

+

+The command will prompt for the instance home directory (if not supplied
+through INSTANCE_HOME or the -i option. The template, backend and admin
+password may be specified on the command-line as arguments, in that
+order.
+
+See also initopts help.
+
+
listlist classname [property]

- Lists all instances of the given class along. If the property is not - specified, the "label" property is used. The label property is tried - in order: the key, "name", "title" and then the first property, - alphabetically. -

Usage: list classname [property]

+

+Lists all instances of the given class. If the property is not
+specified, the  "label" property is used. The label property is tried
+in order: the key, "name", "title" and then the first property,
+alphabetically.
+
+
retireretire designator[,designator]*

- This action indicates that a particular node is not to be retrieved by - the list or find commands, and its key value may be re-used. -

createcreate classname property=value ...

- This creates a new entry of the given class using the property - name=value arguments provided on the command line after the "create" - command. -

getget property designator[,designator]*

- Retrieves the property value of the nodes specified by the designators. -

specspec classname

- This lists the properties for a given class. -

Usage: retire designator[,designator]*

+

+This action indicates that a particular node is not to be retrieved by
+the list or find commands, and its key value may be re-used.
+
+
rollbackUsage: rollback

+

+The changes made during an interactive session are not
+automatically written to the database - they must be committed
+manually. This command undoes all those changes, so a commit
+immediately after would make no changes to the database.
+
+
setset designator[,designator]* propname=value ...

- Sets the property to the value for all designators given. -

Usage: set designator[,designator]* propname=value ...

+

+Sets the property to the value for all designators given.
 
-
initinit [template [backend [admin password]]]

- The command will prompt for the instance home directory (if not supplied - through INSTANCE_HOME or the -i option. The template, backend and admin - password may be specified on the command-line as arguments, in that order. -

exportexport class[,class] destination_dir

- This action exports the current data from the database into - comma-separated files that are placed in the nominated destination - directory. The journals are not exported. -

importimport class file

- The file must define the same properties as the class (including having - a "header" line with those property names.) -

freshenfreshen

- **DO NOT USE** -

- This currently kills databases!!!! -

- This action should generally not be used. It reads in an instance - database and writes it again. In the future, is may also update - instance code to account for changes in templates. It's probably wise - not to use it anyway. Until we're sure it won't break things... -

specificationUsage: specification classname

+

+This lists the properties for a given class.
 
-
helphelp [command]

- Short help about roundup-admin or the specific command. -

tableUsage: table classname [property[,property]*]

+

+Lists all instances of the given class. If the properties are not
+specified, all properties are displayed. By default, the column widths
+are the width of the property names. The width may be explicitly defined
+by defining the property as "name:width". For example::
+  roundup> table priority id,name:10
+  Id Name
+  1  fatal-bug 
+  2  bug       
+  3  usability 
+  4  feature   
+
+
morehelpmorehelp

- All available help from the roundup-admin tool. -

@@ -1197,7 +1279,7 @@ system on their time.

 

-$Id: index.html,v 1.21 2001-12-13 00:20:01 richard Exp $ +$Id: index.html,v 1.22 2001-12-17 03:52:47 richard Exp $

 

diff --git a/roundup-admin b/roundup-admin index 2a4162e..6a1d5c0 100755 --- a/roundup-admin +++ b/roundup-admin @@ -16,7 +16,7 @@ # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. # -# $Id: roundup-admin,v 1.54 2001-12-15 23:09:23 richard Exp $ +# $Id: roundup-admin,v 1.55 2001-12-17 03:52:47 richard Exp $ # python version check from roundup import version_check @@ -93,6 +93,28 @@ Options: print '\n'.join(commands) print + def help_commands_html(self, indent_re=re.compile(r'^(\s+)\S+')): + commands = self.commands.values() + def sortfun(a, b): + return cmp(a.__name__, b.__name__) + commands.sort(sortfun) + for command in commands: + h = command.__doc__.split('\n') + name = command.__name__[3:] + usage = h[0] + print ''' +%(name)s + %(usage)s

+

'''%locals()
+            indent = indent_re.match(h[3])
+            if indent: indent = len(indent.group(1))
+            for line in h[3:]:
+                if indent:
+                    print line[indent:]
+                else:
+                    print line
+            print '
\n' + def help_all(self): print ''' All commands (except help) require an instance specifier. This is just the path @@ -978,6 +1000,9 @@ if __name__ == '__main__': # # $Log: not supported by cvs2svn $ +# Revision 1.54 2001/12/15 23:09:23 richard +# Some cleanups in roundup-admin, also made it work again... +# # Revision 1.53 2001/12/13 00:20:00 richard # . Centralised the python version check code, bumped version to 2.1.1 (really # needs to be 2.1.2, but that isn't released yet :) diff --git a/roundup/backends/back_anydbm.py b/roundup/backends/back_anydbm.py index b11a461..0d52750 100644 --- a/roundup/backends/back_anydbm.py +++ b/roundup/backends/back_anydbm.py @@ -15,7 +15,7 @@ # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. # -#$Id: back_anydbm.py,v 1.18 2001-12-16 10:53:38 richard Exp $ +#$Id: back_anydbm.py,v 1.19 2001-12-17 03:52:48 richard Exp $ ''' This module defines a backend that saves the hyperdatabase in a database chosen by anydbm. It is guaranteed to always be available in python @@ -245,6 +245,36 @@ class Database(hyperdb.Database): res = res + db.keys() return res + + # + # Files - special node properties + # + def filename(self, classname, nodeid, property=None): + '''Determine what the filename for the given node and optionally property is. + ''' + # TODO: split into multiple files directories + if property: + return os.path.join(self.dir, 'files', '%s%s.%s'%(classname, + nodeid, property)) + else: + # roundupdb.FileClass never specified the property name, so don't include it + return os.path.join(self.dir, 'files', '%s%s'%(classname, + nodeid)) + + def storefile(self, classname, nodeid, property, content): + '''Store the content of the file in the database. The property may be None, in + which case the filename does not indicate which property is being saved. + ''' + name = self.filename(classname, nodeid, property) + open(name + '.tmp', 'wb').write(content) + self.transactions.append((self._doStoreFile, (name, ))) + + def getfile(self, classname, nodeid, property): + '''Store the content of the file in the database. + ''' + return open(self.filename(classname, nodeid, property), 'rb').read() + + # # Journal # @@ -291,11 +321,20 @@ class Database(hyperdb.Database): ''' if DEBUG: print 'commit', (self,) - # lock the DB + # TODO: lock the DB + + # keep a handle to all the database files opened + self.databases = {} + + # now, do all the transactions for method, args in self.transactions: - # TODO: optimise this, duh! method(*args) - # unlock the DB + + # now close all the database files + for db in self.databases.values(): + db.close() + del self.databases + # TODO: unlock the DB # all transactions committed, back to normal self.cache = {} @@ -306,17 +345,31 @@ class Database(hyperdb.Database): def _doSaveNode(self, classname, nodeid, node): if DEBUG: print '_doSaveNode', (self, classname, nodeid, node) - db = self.getclassdb(classname, 'c') + + # get the database handle + db_name = 'nodes.%s'%classname + if self.databases.has_key(db_name): + db = self.databases[db_name] + else: + db = self.databases[db_name] = self.getclassdb(classname, 'c') + # now save the marshalled data db[nodeid] = marshal.dumps(node) - db.close() def _doSaveJournal(self, classname, nodeid, action, params): if DEBUG: print '_doSaveJournal', (self, classname, nodeid, action, params) entry = (nodeid, date.Date().get_tuple(), self.journaltag, action, params) - db = self._opendb('journals.%s'%classname, 'c') + + # get the database handle + db_name = 'journals.%s'%classname + if self.databases.has_key(db_name): + db = self.databases[db_name] + else: + db = self.databases[db_name] = self._opendb(db_name, 'c') + + # now insert the journal entry if db.has_key(nodeid): s = db[nodeid] l = marshal.loads(db[nodeid]) @@ -324,13 +377,20 @@ class Database(hyperdb.Database): else: l = [entry] db[nodeid] = marshal.dumps(l) - db.close() + + def _doStoreFile(self, name, **databases): + # the file is currently ".tmp" - move it to its real name to commit + os.rename(name+".tmp", name) def rollback(self): ''' Reverse all actions from the current transaction. ''' if DEBUG: print 'rollback', (self, ) + for method, args in self.transactions: + # delete temporary files + if method == self._doStoreFile: + os.remove(args[0]+".tmp") self.cache = {} self.dirtynodes = {} self.newnodes = {} @@ -338,6 +398,10 @@ class Database(hyperdb.Database): # #$Log: not supported by cvs2svn $ +#Revision 1.18 2001/12/16 10:53:38 richard +#take a copy of the node dict so that the subsequent set +#operation doesn't modify the oldvalues structure +# #Revision 1.17 2001/12/14 23:42:57 richard #yuck, a gdbm instance tests false :( #I've left the debugging code in - it should be removed one day if we're ever diff --git a/roundup/roundupdb.py b/roundup/roundupdb.py index f57139e..50cd90c 100644 --- a/roundup/roundupdb.py +++ b/roundup/roundupdb.py @@ -15,7 +15,7 @@ # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. # -# $Id: roundupdb.py,v 1.33 2001-12-16 10:53:37 richard Exp $ +# $Id: roundupdb.py,v 1.34 2001-12-17 03:52:48 richard Exp $ __doc__ = """ Extending hyperdb with types specific to issue-tracking. @@ -188,28 +188,14 @@ class FileClass(Class): content = propvalues['content'] del propvalues['content'] newid = Class.create(self, **propvalues) - self.setcontent(self.classname, newid, content) + self.db.storefile(self.classname, newid, None, content) return newid - def filename(self, classname, nodeid): - # TODO: split into multiple files directories - return os.path.join(self.db.dir, 'files', '%s%s'%(classname, nodeid)) - - def setcontent(self, classname, nodeid, content): - ''' set the content file for this file - ''' - open(self.filename(classname, nodeid), 'wb').write(content) - - def getcontent(self, classname, nodeid): - ''' get the content file for this file - ''' - return open(self.filename(classname, nodeid), 'rb').read() - def get(self, nodeid, propname, default=_marker): ''' trap the content propname and get it from the file ''' if propname == 'content': - return self.getcontent(self.classname, nodeid) + return self.db.getfile(self.classname, nodeid, None) if default is not _marker: return Class.get(self, nodeid, propname, default) else: @@ -506,6 +492,10 @@ class IssueClass(Class): # # $Log: not supported by cvs2svn $ +# Revision 1.33 2001/12/16 10:53:37 richard +# take a copy of the node dict so that the subsequent set +# operation doesn't modify the oldvalues structure +# # Revision 1.32 2001/12/15 23:48:35 richard # Added ROUNDUPDBSENDMAILDEBUG so one can test the sendmail method without # actually sending mail :) diff --git a/test/test_db.py b/test/test_db.py index 08b918a..c0919f5 100644 --- a/test/test_db.py +++ b/test/test_db.py @@ -15,12 +15,13 @@ # BASIS, AND THERE IS NO OBLIGATION WHATSOEVER TO PROVIDE MAINTENANCE, # SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. # -# $Id: test_db.py,v 1.11 2001-12-10 23:17:20 richard Exp $ +# $Id: test_db.py,v 1.12 2001-12-17 03:52:48 richard Exp $ import unittest, os, shutil from roundup.hyperdb import String, Password, Link, Multilink, Date, \ Interval, Class, DatabaseError +from roundup.roundupdb import FileClass def setupSchema(db, create): status = Class(db, "status", name=String()) @@ -33,6 +34,7 @@ def setupSchema(db, create): Class(db, "user", username=String(), password=Password()) Class(db, "issue", title=String(), status=Link("status"), nosy=Multilink("user")) + FileClass(db, "file", name=String(), type=String()) db.commit() class MyTestCase(unittest.TestCase): @@ -46,7 +48,7 @@ class anydbmDBTestCase(MyTestCase): # remove previous test, ignore errors if os.path.exists('_test_dir'): shutil.rmtree('_test_dir') - os.mkdir('_test_dir') + os.makedirs('_test_dir/files') self.db = anydbm.Database('_test_dir', 'test') setupSchema(self.db, 1) @@ -73,6 +75,11 @@ class anydbmDBTestCase(MyTestCase): def testTransactions(self): num_issues = len(self.db.issue.list()) + files_dir = os.path.join('_test_dir', 'files') + if os.path.exists(files_dir): + num_files = len(os.listdir(files_dir)) + else: + num_files = 0 self.db.issue.create(title="don't commit me!", status='1') self.assertNotEqual(num_issues, len(self.db.issue.list())) self.db.rollback() @@ -83,6 +90,18 @@ class anydbmDBTestCase(MyTestCase): self.assertNotEqual(num_issues, len(self.db.issue.list())) self.db.rollback() self.assertNotEqual(num_issues, len(self.db.issue.list())) + self.db.file.create(name="test", type="text/plain", content="hi") + self.db.rollback() + self.assertEqual(num_files, len(os.listdir(files_dir))) + self.db.file.create(name="test", type="text/plain", content="hi") + self.db.commit() + self.assertNotEqual(num_files, len(os.listdir(files_dir))) + num_files2 = len(os.listdir(files_dir)) + self.db.file.create(name="test", type="text/plain", content="hi") + self.db.rollback() + self.assertNotEqual(num_files, len(os.listdir(files_dir))) + self.assertEqual(num_files2, len(os.listdir(files_dir))) + def testExceptions(self): # this tests the exceptions that should be raised @@ -156,7 +175,7 @@ class anydbmReadOnlyDBTestCase(MyTestCase): # remove previous test, ignore errors if os.path.exists('_test_dir'): shutil.rmtree('_test_dir') - os.mkdir('_test_dir') + os.makedirs('_test_dir/files') db = anydbm.Database('_test_dir', 'test') setupSchema(db, 1) self.db = anydbm.Database('_test_dir') @@ -178,7 +197,7 @@ class bsddbDBTestCase(anydbmDBTestCase): # remove previous test, ignore errors if os.path.exists('_test_dir'): shutil.rmtree('_test_dir') - os.mkdir('_test_dir') + os.makedirs('_test_dir/files') self.db = bsddb.Database('_test_dir', 'test') setupSchema(self.db, 1) @@ -188,7 +207,7 @@ class bsddbReadOnlyDBTestCase(anydbmReadOnlyDBTestCase): # remove previous test, ignore errors if os.path.exists('_test_dir'): shutil.rmtree('_test_dir') - os.mkdir('_test_dir') + os.makedirs('_test_dir/files') db = bsddb.Database('_test_dir', 'test') setupSchema(db, 1) self.db = bsddb.Database('_test_dir') @@ -201,7 +220,7 @@ class bsddb3DBTestCase(anydbmDBTestCase): # remove previous test, ignore errors if os.path.exists('_test_dir'): shutil.rmtree('_test_dir') - os.mkdir('_test_dir') + os.makedirs('_test_dir/files') self.db = bsddb3.Database('_test_dir', 'test') setupSchema(self.db, 1) @@ -211,7 +230,7 @@ class bsddb3ReadOnlyDBTestCase(anydbmReadOnlyDBTestCase): # remove previous test, ignore errors if os.path.exists('_test_dir'): shutil.rmtree('_test_dir') - os.mkdir('_test_dir') + os.makedirs('_test_dir/files') db = bsddb3.Database('_test_dir', 'test') setupSchema(db, 1) self.db = bsddb3.Database('_test_dir') @@ -241,6 +260,9 @@ def suite(): # # $Log: not supported by cvs2svn $ +# Revision 1.11 2001/12/10 23:17:20 richard +# Added transaction tests to test_db +# # Revision 1.10 2001/12/03 21:33:39 richard # Fixes so the tests use commit and not close #