2008/09/19

Anyway, you should stop finding bugs!

The title of this post is the phrase that Paolo (my brother) told me after the bug I discovered was actually confirmed on the Zope mailing list.

The bug I've found is something which has haunted me for many years; I'm not sure why I've experienced this bug, and other people didn't; I'm surprised about it, but what I know is that software usually is known to work in a sound way in a known field. Outside of that field, you can't be sure about what it happens. This has nothing to do with closed or open source software: software is complex, and you learn how it works by experience.

After this rant, let me show a small piece of software which can show the bug in action:

import os
import sys
import unittest

if __name__ == '__main__':
execfile(os.path.join(sys.path[0], '../framework.py'))

from Testing import ZopeTestCase

from OFS import Image

from Products.ZPsycopgDA.DA import manage_addZPsycopgConnection
from Products.ZSQLMethods import SQL


class DoubleTransactionTest(ZopeTestCase.ZopeTestCase):

def _add_big_image(self, value, data):
Image.manage_addFile(self.app, "f%06s" % value, data , "a title")

def test_showdouble(self):
manage_addZPsycopgConnection(self.app, "db_connection", "", "host=localhost user=postgres dbname=template1")
self.app._setObject('sql', SQL.SQL("sql", "", "db_connection", "", "select * from pg_tables"))
self.app.sql()
data = "*" * (1 << 20)
for x in range(1000):
self._add_big_image(x, data)
print "Added %s " % x
self.app.sql()

if __name__ == '__main__':
unittest.main()


This small piece of code (which is actually a ZopeTestCase) does the following things:




  1. creates a new database connection, using the ZPsycopg Database Adapter;

  2. creates a SQL method (named, with a lack of fantasy, "sql");

  3. runs the SQL method;

  4. adds 1000 "big" images;

  5. runs again the same SQL method.




In order to understand what is the problem, you should enable the logging of queries; you would end with something like this:

SET DATESTYLE TO 'ISO'
BEGIN; SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
statement: select * from pg_tables
SET DATESTYLE TO 'ISO'
BEGIN; SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
select * from pg_tables
ABORT
ABORT

Can you see the problem? There are two database transactions! For one transaction in Zope! Indeed, one run of the test can be considered as just one transaction of Zope.

How can it be?

Well, actually I've been able to dig this problem, even though, at the moment, I've not found a "real" solution.

The bug is triggered by the size of the file (the size had been chosen so that the bug is shown, of course): when a big image is loaded, the following path of code is executed inside the OFS.Image.File._read_data method:

      if size <= 2*n:
seek(0)
if size < n: return read(size), size
return Pdata(read(size)), size

# Make sure we have an _p_jar, even if we are a new object, by
# doing a sub-transaction commit.
transaction.savepoint(optimistic=True)


So, for a large enough file (actually, greater than 128k) the transaction.savepoint is called, which, in turns causes the ZODB.Connection.savepoint method to be called. Which, just before the end, does the following:

        # While the interface doesn't guarantee this, savepoints are
# sometimes used just to "break up" very long transactions, and as
# a pragmatic matter this is a good time to reduce the cache
# memory burden.
self.cacheGC()
return result

So, just before ending, it cause the cache to be minimized; in particular, this causes the database connection to be ghostified, i.e. to be removed from the memory; but this also means that its "volatile" attributes are lost; and what is among its volatile attributes? The actual connection to the database! So, just after, when we call again the SQL method, the object is restored in memory; since it hasn't a good connection anymore, it creates a new one, thus beginning a new transaction for the same Zope transaction!

Removing the self.cacheGC() calls removes the problem, but I'm not sure about the consequences; therefore, I'm avoiding it, at the moment. But I'll see what I should do about it.

2008/09/15

What have I to test?

I'm currently working on a software who has to create a PDF document from an ODT template. Of course, I'm doing in python, and I want to test it.

Actually, I already have the methods to create the PDF document.

At the moment, however, the name of the template is hard coded in a class; the name leads to a file on the file system, in a specific directory, which contains the template.

My next story requires that the name of the template to use bs different according to the configuration of an object.

So, the first step could be to askthe name of the template to the object. In a first implementation, the method returns a fixed string; later, I can make it provide different names according to different configurations.

Of course, this part is quite easy to check; also, I don't deal with actual templates, but just with file names.

There is another part, however, which is not so easy: the other object which uses this path to actually build the PDF. In this case, my problem is that I've to get the file name and actually build the object. Or I could use a fake, for this special case, and just check if the path required is passed to me.

This is not so good. I'm checking the implementation. Rather, I want to check the results of this method.

To be able to check the results, I've first to define what are the results I want to check.

I could say that the results are the PDF document (i.e. the file) produced by my method. Even though this can be fine, up to a point, it is not worthy. A generated file can differ from the file used as a compare because of stupid reasons, like the date of creation, or the path of the document which could be embedded in the document itself, or anything else. In short, even though the "content" could be the same, the binary of the two files could be quite different. This is no good, and it should be avoided.

So, I think that in the end I will unit test whatever I can test, mocking and checking that the interactions are actually performed how I expect them to be performed, and then I will make a real test with real openoffice in the acceptance tests.

2008/09/14

Automated testing

Here is what I will watch today while I eat my pasta :-)

2008/09/13

YouTube - Campfire One: Introducing Google App Engine (pt. 1)

Ok, this is the latest "cool" technology I'm watching, right now.



And, of course, I've already created my own sample application: gflow.appstore.com, which at the moment is a simple guestbook application, but, who knows ;) ?

2008/09/12

Plotting the execution times of tests

After a message in the extreme programming mailing list, I sent the execution times of PAFlow unit tests to Kent Beck.

He used google charts to plot the results, and here are the data :-)



Just to explain what is going on: the x are the execution times, and the y are the number of tests which have that execution times. Both scales are logarithmic, with the y which is 1 at start and 1000 at top. 4096 actually means 4096 millisec.

2008/09/11

The fastest wi-fi network... in Pisa!

A Pisa la rete wi-fi più veloce del mondo - Corriere della Sera

Optimizing "janitor"

Janitor is my laptop; I've used it for I think more than two years; even though you might consider it "old", I'm quite comfortable with it.

Janitor is a Dell Inspiron 1300 with 2 GB of memory; even though it hasn't a top performing processor, its wide screen makes it very handy for my developing activites.

As I already told, in the recent past I had some troubles, caused by the internal fan which cheased working all of sudden.

After a lot of work, I finally made the thing I should have done in the first place: call the Dell assistance; they provided me with the obvious suggestion (i.e. to try cleaning the fan with compressed air) and it worked very well.

In the mean time, however, I tried everything else. So, I found a number of optimizations for the Inspiron and for the graphic card (which was the one showing most problems when the fan was not working).

In the end, I configured the device section of my /etc/X11/xorg.conf as follows:


Section "Device"
Identifier "Intel Corporation Mobile 915GM/GMS/910GML Express Graphics Controller"
Driver "intel"
BusID "PCI:0:2:0"
Option "DRI" "true"
Option "AccelMethod" "exa"
Option "MigrationHeuristic" "greedy"
Option "ExaNoComposite" "false"
EndSection


and I added the INTEL_BATCH="1" setting to /etc/environment.

After a reboot, I'd been able to notice a great improvement in the perfomance of my computer. Gmail is lightning fast, and everything seems a lot faster. My unittest, which last time required something like 600 seconds to be run, now ask me just for 260 seconds. That's an improvement, indeed :-)

2008/09/10

Beautiful mornings (II)

And today, after I cleaned the fan of my computer a bottle of compressed air, my notebook is happily running again :-)

Moreover, since I tryied to optmize all that I could, it seems to me that it is running much faster than before (gmail being notably faster).

2008/09/07

Beautiful mornings

You start some of your days with your best intentions; you wake up early, wash yourself, shave, get a breakfast and are ready in thirty minutes to work, even if it is Sunday.

Then, you go to the computer and wait for your online pairer to show up. You're early, so it is not a problem if he's not there; in the meantime, you browse some online newspaper. Your computer seems a little more laggy than usual.

You check, and see that the lightweight browser named Firefox is eating 300 MB of your memory, and 90% of your CPU. "Ehi!" you think, "it must be that effect once I read about Firefox not being too much happy when you resume from suspend mode. No problem, I will kill him and restart."

But your computer still behaves like someone poured glue on your keyboard during the night.

You're wondering where your online pairer is; now he's late, but after all you agreed to meet at 6:30 AM; maybe he just slept a little more than he should, and will be online soon.

Then, all of sudden, your computer turns itself off. Not a warning message, not a dialogue, nothing.

You check the power cable: sometimes you forget to put it when the little beast complains about its batteries, but this is not the case.

Checking the temperature, you discover that either your computer got the flu or the fan that you had to change at least two months ago, and which whined its pain during the night, has eventually reached the peace, and it is broken.

So, since you know you have to work, you grab your home fan, and put it near your computer. In that way, it remains cool enough.

Trying to work again, you find that your computer is still as fast as a turtle in its sleepy days; so, since yesterday your computer was working, you try to remember what you did yesterday, besides working as a mad on your project.

Desperate, you try to install some upgrade on the available packages, hoping that they will help you in fixing your problem. But to no avail.

So, in the end, since you can't use all your morning for it, you decide to live with it. And you try to post it on your blog.

The only problem is that your blog does not accept posts, today.

2008/09/06

Why are they so happy?

L'Università di Pisa batte tutti su Internet - intoscana.it - il Portale ufficiale della Toscana


L’ultima graduatoria, pubblicata ad agosto e consultabile all’indirizzo www.webometrics.info, vede l’Università di Pisa al 207° posto nel mondo e seconda in Italia solo all’Università di Bologna, che occupa il 95° posto mondiale


Basically, they are saying that the University of Pisa is number 207 in this particular list. I do not know the list. I do not know how accurate is. But exactly, what is so nice in being number 207 in a list where being the number 1 is better?

2008/09/05

Slow running unit tests...

I've a problem, and it can be easily described:


python2.3 test/runalltests.py
(lots of work)
Ran 2002 tests in 557.207s


I work in cycles of 25 minutes, followed by 5 minute of rest, then 25 minutes of work again, and so on. If I think that I need to spend almost half of that time in running my tests, I cannot write tests anymore.

This needs to be fixed.

2008/09/01

He did it again...

Another book Paolo gave me to read...


Image of Getting Things Done

2008/08/31

Good luck, Paolo

Tomorrow Paolo Bizzarri will leave Pisa to work on a project in Rome. He will be away, more or less, for a month.

It will be the longest time we will not work togheter since... I don't even remember since when!

What can I say, then?

Good luck, Paolo.

Goobuntu

After such a news, I think I'm going to switch to Ubuntu in the next few days.

2008/08/30

Always learning...

It should not happen. I mean, writing python programs is something I did, more or less, in the past four years, as one of my daily activity.

It should not happen that I discover that the recommended way to do import is:

import module

and not

from module import MyClass

It should not happen; and that's all.

2008/03/08

User Stories (really) Applied

I don't remember the first book I read on Extreme Programming, but I clearly remember two sensations: enthusiasm and confusion. Indeed, I felt there was something great in it, but I was unable to grasp many of the real practices they were talking about; it was like I was looking at them through a thick fog; user stories was one of those practices.

Many years have passed from that day, five, maybe more; my understanding of user stories expanded from a shape in the fog to a paper card. However, as many things you learn on the Net, some parts of the card were very clear, while other were still difficult to discern: for example I was not sure how big the card should have been or how many details I should have put on it. Less confusion and a little more enthusiasm.

Then my twin brother brought me this book, and told me "read it". And as I always do with everything he gives me to read, I put in on the shelf, and I forgot it. You may wonder why I do it; you're welcome: I wonder too.

Anyway, the book remained on the shelf for many months, then, I don't remember why, I took it again and started reading it; I've stopped and started many times, then, yesterday, thanks to a trip in train, I finished it.

Indeed, Mike Cohn did a very good job at explaining user stories; many of the questions I had about user stories found their answers here, and I think I'm better at using them, now; at least, Mike gave me all the tools I needed to improve, and not only in the user stories, but in the release and iteration planning as well, two practices intertwined with the former. There was even a nice example which gave me more insights on all the process.

As a matter of fact, I don't like how the book is written: each chapter is divided in an introduction, a core, a general summary, a summary of developer responsibilities, a summary of customer responsibilities, and some questions; this structure makes hard to read the book from the start to the end, as I like to do.

Nonetheless, I like when Mike tells us not what we should do, but what he did, and this happens most of time; he shows us real life examples, tells short and long anecdotes to explain and expand what he said just a paragraph before; in short, he does what is often forgotten: he shows as the theory of user stories becomes the practice of user stories. And I thank him for this.

2008/03/03

At least, On Writing Well...

Image of On Writing Well, 25th Anniversary

When I was in high school, my grades in written italian were bad. Not so bad to become a problem, but bad nonetheless. The strange thing is that I loved to write. It was my favourite way to spend my free time.

But I was not able to write in good italian when I started high school. And I was still bad when I finished high school.

I've always wondered why. After reading this book, I understood: no one taught me. Ever.

Writing is a craft, with its rules and its tools, and this book shows you them; you still need to practice them: just because you know what you should do to drive does not make you a good driver; but knowing it allows you to practice, to acknowledge your mistakes, and to fix them.

This is a book for us all. We all have to write, be it an entry in a blog, a report for a manager, a mail to a friend, or a recipe of our favourite dish. Or, even, an howto on installing Linux on the latest notebook from Alienware. We all need to write.

Better then if we can learn to Write Well.