<div dir="ltr">I took your suggestion and tried out sqlite3 and also apsw. Using either of these modules the unpacking time is reduced to 1-2 seconds.<br><br>Furthermore just cycling through the results takes 1-1.5 seconds, meaning that adding the results to a list is much quicker than using QSqlQuery ( less than 1 second ).<br>
<br>The query time is basically unaffected.<br><br>So it looks like the major problem I was having is in how Qt (or PyQT) handles the query extraction (I suspect that part of it has to do with Qt converting everything into QVariants).<br>
<br>With sqlite3 or apsw things are much more tolerable, though if any one has ideas on how to make the extraction even faster I would love to here them. What I am doing for now is to use:<br><br> results = [row for row in self.cursor]<br>
<div class="gmail_quote"><br>I think though you are right that if I want my app to feel fast I need to do the query/extraction in the background and return stuff in chunks. I'll have to see how well the gui handles that sort of thing.<br>
<br><br>-- amicitas<br><br><br><br>On Wed, Oct 8, 2008 at 1:20 PM, Aaron Digulla <span dir="ltr"><<a href="mailto:digulla@hepe.com">digulla@hepe.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div><div class="Wj3C7c"><br>
<br>
</div></div>It seems that loading 10'000 records out of the sqlite DB just takes a<br>
couple of seconds. sqlite isn't very fast when you have a lot of data<br>
(and yeah, 10'000 isn't that much ...).<br>
<br>
You must find a way to split the operation into smaller chunks. Also try<br>
the python sqlite3 module to get a feeling if Python is slow or if it's<br>
Qt or PyQt.<br>
<br>
In my app, I'm running sqlite operations in a background thread so they<br>
won't block the user from working. It takes a bit of effort but it was<br>
much more simple than I expected.<br>
<br>
Regards,<br>
<font color="#888888"><br>
--<br>
Aaron "Optimizer" Digulla a.k.a. Philmann Dark<br>
"It's not the universe that's limited, it's our imagination.<br>
Follow me and I'll show you something beyond the limits."<br>
<a href="http://darkviews.blogspot.com/" target="_blank">http://darkviews.blogspot.com/</a> <a href="http://www.pdark.de/" target="_blank">http://www.pdark.de/</a><br>
</font></blockquote></div><br></div>