[review] [PyKDE] large tables problem
jbublitz at nwinternet.com
Wed Jun 23 07:18:01 BST 2004
On Tuesday 22 June 2004 11:50, Vicent Mas wrote:
> I'm working on a viewer for large tabular datasets. The underlying
> application from which my viewer read data is a hierarchical database that
> can deal with really large datasets (tipically too large to be managed with
> relational databases). Following the advises of Qt documentation and the Qt
> Quarterly article http://doc.trolltech.com/qq/qq07-big-tables.html I've
> written my own table class by deriving QTable class and overwriting several
> functions. In particular the resizeData() function has been reimplemented
> to do nothing. In addition, I use buffers so just a small part of the
> displayed dataset is in memory at a given time.
> However the amount of memory allocated by my application is still too big
> when I create large tables (you can see a summary at the bottom of the
> message). Does anybody know what else can I do in order to reduce the
> amount of allocated memory? Help would be really appreciated.
> Thanks in advance.
> Some useful info:
> Debian box (kernel 2.6.2)
> python 2.3.4
> qt 3.2.3
> pyqt 3.12
> The following table is based on the output of command <ps u -C python>.
> ROWS %CPU %MEM VSZ(KB) RSS TIME COMMAND
> 0 9.7 4.0 40652 20796 0:53 python main.py
> 10**5 1.2 4.5 43332 23412 0:03 python main.py
> 10**6 1.3 9.5 69168 49208 0:04 python main.py
> 10**7 7.4 59.4 326512 306584 0:35 python main.py
> Of course main.py is the script that launches my application and the ROWS
> column gives the number of rows of the opened table :). For testing
> purposes columns have been fixed to 4 and cells contain strings of 1
> character length.
If your application allows it, you can operate the table with fixed number of
cells/fixed amount of memory, reload the cells as necessary (don't
destroy/construct new cells), and overload the navigation methods (cursor
up/down, page up/down, home, end, and the equivalent mouse/scrollbar/button
operations) to cause a "page fault" and reload the table when navigation is
about to go outside the range of rows currently displayed.
There are probably some ways to minimize data movement, but the limiting
factor for speed is probably redraws. The memory used should then be
independent of the table size.
I used to support the display of a 3-5M row database that way, but it was all
custom code (DB, table widget, "page fault" mgmt) written in C++. I think
Delphi worked basically the same way (paging). I'm not sure QTable makes this
especially easy though, but I haven't looked into it in a long time.
More information about the PyQt