Introduction
pgasync is a Twisted-based PostgreSQL client library which fully conforms to the DB API 2.0 specification, and provides adbapi compatibility. It's designed to run quickly and scale well. It provides connection pooling and persistence, database types, and safe conversions.
For the uninitiated: If you're into Twisted, this project is potentially very interesting to you. If you don't use Twisted, or have never heard of it, you probably won't care.
Requirements
- Python 2.3+: Needs the datetime module.
- Twisted 1.3
- PostgreSQL 7.4: Uses protocol version 3.0.
Download
Releases:
- [3/14/2005]: Version 2.01 is released.
(download)
[svn co svn://svn.jamwt.com/pgasync/tags/pgasync-release-2.01]Oops! runQuery() and runInteraction were broken by the removal of ConnectionPool._error. Fixed now. - [3/14/2005]: Version 2.0 is released.
(download)
[svn co svn://svn.jamwt.com/pgasync/tags/pgasync-release-2.0]Errback handling fixes. Removed useless explicit conversion of integers to NUMBER when no special formatting is done on them. Exposed __int__, __float__ and friends in NUMBER just in case a NUMBER instance is tossed to format(). You can now do %03d style formatting in format() and a trailing semicolon can be included on queries. Added boolean support (thanks: Andrea Arcangeli). Experimental Unicode Support (thanks Matt Goodall). Unix socket support added (thanks Stephen Early). convertBinary() function to get stored binary back out of the database. - [1/25/2005]: Version 1.15b is released.
(download)
[svn co svn://svn.jamwt.com/pgasync/tags/pgasync-release-1.15b]Circular references and memory leaks were resolved, an error that occured when the same errback was called twice from runOperation was fixed. - [1/19/2005]: Version 1.14b is released.
(download)
[svn co svn://svn.jamwt.com/pgasync/tags/pgasync-release-1.14b]Fixed a typo in runOperation, improved errback handling. Fixed dropped connection handling and a bug in format() error display. - [1/19/2005]: Version 1.13b is released.
(download)
[svn co svn://svn.jamwt.com/pgasync/tags/pgasync-release-1.13b]Pool minimum/maximum size controls were implemented. - [1/18/2005]: Version 1.12b is released.
(download)
[svn co svn://svn.jamwt.com/pgasync/tags/pgasync-release-1.12b]Oops: connect() method added to ConnectionPool class. - [1/18/2005]: Version 1.11b is released.
(download)
[svn co svn://svn.jamwt.com/pgasync/tags/pgasync-release-1.11b]Transaction and ConnectionPool (adbapi compat) were added. A bug in the protocol layer which caused a stale transactional state flag during callback was fixed. - [1/14/2005]: Version 1.10b is released.
(download)
[svn co svn://svn.jamwt.com/pgasync/tags/pgasync-release-1.10b]format() and execute() now take all pyformat argument types, not just dicts. Closed connections now handle failed query execution more gracefully. - [1/13/2005]: Version 1.9b is released.
(download)
[svn co svn://svn.jamwt.com/pgasync/tags/pgasync-release-1.9b]connection.cursor() is no longer deferred. Pre-connection queries are now queued. Transaction awareness was implemented to avoid unnecessary BEGINs and ROLLBACKs. connection.exFetch was added for convenience. - [1/12/2005]: Version 1.8b is released.
(download)
[svn co svn://svn.jamwt.com/pgasync/tags/pgasync-release-1.8b]str, int, float, True/False, datetime instances returned from fetch() queries instead of pgtypes. date/time/timestamp handling corrected. None passed to execute/exFetch results in NULL insertion. Premature cursor release issues resolved; removed automatic cursor release. Simplified pgtypes for output-only usage; removed stack inspection. - [1/5/2005]: Version 1.7b is released.
(download)
[svn co svn://svn.jamwt.com/pgasync/tags/pgasync-release-1.7b]fetchone() was corrected to return values consistent with the PEP. (credit: Matt Goodall - mg) - [1/3/2005]: Version 1.6b is released.
(download)
[svn co svn://svn.jamwt.com/pgasync/tags/pgasync-release-1.6b]STRING and BINARY types now inherit from str, not types.StringType. (credit: Valentino Volonghi - dialtone) - [1/1/2005]: Version 1.5b is released.
(download)
[svn co svn://svn.jamwt.com/pgasync/tags/pgasync-release-1.5b]Execute() params were fixed to be optional, queries are now whitespace-stripped when looking for 'SELECT' to create a cursor, docstrings were corrected to reflect the new param style. (credit: Matt Goodall - mg) - [12/31/2004]: Version 1.4b is released.
(download)
pyformat arguments now takes a dict instead of keyword arguments in compliance with the DB API convention.
- [12/30/2004]: Version 1.3b is released.
(download)
Proper deferred error handling was added, the ability to manually release() back into the pool was implemented, format arguments can now take a python datetime object, and a simple nevow/wiki example is now included in the distribution.
- [12/22/2004]: Version 1.2b is released.
(download)
Old examples which did not work correctly with the 1.1b changes were accidentally distributed.
- [12/19/2004]: Version 1.1b is released. (download)
Query queues added (you don't need to worry about command completion before issuing another query. So only addCallback when you really care about the outcome); setup.py Pyrex dependency removed (credit: Bob Ippolito); adbapi compatibility removed.
- [12/18/2004]: Version 1.0b is released. (download)
Installation
pgasync uses distutils. To install, simply execute the following as root in the unpacked distribution directory:
# python setup.py install
This will usually "just work", but there are two places that you may run into problems here: first, you may encounter a compilation error on cache.c. If that's the case, you can try to use Pyrex to regenerate cache.c from the cache.pyx file in pgasync/. See the README for more detail.
The other problem that may occur is that compilation of convert.c fails or gives warnings about "implicit declaration" of the funcions htons and/or htonl. If this does happen, I'd appreciate an email about what platform/os you're on so I can track down the proper include file. Linux and FreeBSD should work.
Usage
from pgasync import ConnectionPool
pool = ConnectionPool("pgasync",
dbname="mydb",user="someone",password="something")
# simple query usage where you don't care about the cursor
# (description, rowcount, etc)
def printUsers(results):
for row in results:
# do something with row
pool.runQuery("select * from users where uid = %s",13).addCallback(printUsers)
# simple, one-pass operation
pool.runOperation("insert into users values (%s,%s,%s)",(1,2,3))
# transactions, more complex db-api style things
connection = pool.connect()
cursor = connection.cursor()
cursor.execute("delete from users where uid = %(uid)s", {"uid" : 13})
cursor.execute("update users set status = 'confirmed' where uid = %s", 13)
cursor.execute("select * from users").addCallback(printUsers)
connection.commit()
cursor.release()
# pool control
pool.min = 3
pool.max = 12
print pool.running
Notes
If you observe these notes, you'll have a good understanding of the couple of places that it differentiates from standard sync db api or adbapi libraries, the bit of special treatment/thought it needs.
- Keep a ConnectionPool object global if you'd like, perhaps using site.remember in Nevow. Execute run* functions using this pool, and create Connection objects for transactional work using pool.connect() when you need to. *Don't* try to keep a global Connection object.
- runIteration is there, but it's usage is not recommended due to several limitations. pgasync's runIteration will not call your passed function in a thread, so don't block! Also, you will need to add callbacks to any SELECT queries, so existing runIteration functions that assume a sync-like db api will not work!
- pgasync uses query queues. This means that you can say execute("query") in a tight loop 100 times without worrying about blocking, or waiting for a callback before the next query gets executed. The protocol layer will queue outgoing queries (and callbacks) and process them when the backend is ready. This means saying execute("query"); fetchone() without making fetchone in execute's callback is no problem whatsoever. Only utilize callbacks when you really care about the timing, completion, or results of a query.
- By default, pgasync will close unused connections that are idle for a certain number of seconds. It will obey ConnectionPool's min and max attribute. Values of zero mean "no minimum" and "no maximum." The default is no minimum, with a maximum of 20 concurrent backend connections.
- Always do cursor.release() when you're done using a cursor. If you don't, you won't reuse connections! release() is queued, so you can call it even when you still have queries pending.
- In pgasync, a connection can only have one cursor at a time. This means the recommended way to utilize pgasync is to create a cursor from a fresh connection every time. The ConnectionPool object will do this for you via runOperation, runQuery, runInteraction. The "Connection" object is very, very light and inexpensive in pgasync--it's just an abstaction of the pool. In fact, connect() doesn't even block. Cursor creation is the key. So always connect() or pool.connect(), then conn.cursor().
Subversion
Public subversion read access is available:
svn co svn://svn.jamwt.com/pgasync/trunk/pgasync
Author
pgasync was written by Jamie Turner. I'd appreciate hearing about any bugs or suggestions.
I'm often on #twisted.web on irc.freenode.net during working hours, pacific time. You can subscribe to the mailing list using the address below.
<this project>-subscribe at j a mwt dot (com)
Omit '-subscribe' to send the list a message. Mailing list archives.