AT91SAM7S256, SPI and PDC

I have a problem using DMA with SPI. I am supposed to start an SPI 
transaction via DMA, wait for an interrupt to trigger once the DMA is 
done with, and finally start over. I am using the ENDRX flag in SPI_SR 
to trigger the interrupt. The transactiocs are 1-6 bytes long.

The problem is that the interrupt gets triggered as soon as I start the 
transaction. It seems that the ENDRX flag gets set immediately and when 
I then exit the interrupt, I will get back to it again and again, until 
I disable it.

In the case that revealed this problem, I am trying to receive three 
bytes. I start that transaction in the interrupt and the interrupt 
finishes when 18 bits out of 24 have been received. Then there is only 
some constant delay between the interrupts and the same interrupt gets 
triggered again when there are four more bits to be received. My 
oscilloscope reveals that those remaining bytes get clocked in but the 
never end up in the buffer.

This is how I start the DMA transaction:
SPI_RPR = pointer_to_rx_buffer;
SPI_TPR = pointer_to_tx_buffer;
SPI_RCR = amount_of_bytes;
SPI_TCR = amount_of_bytes;
SPI_PTCR = SPI_PTCR_RXTEN | SPI_PTCR_TXTEN;

I have tried adding
SPI_PTCR = SPI_PTCR_RXTDIS | SPI_PTCR_TXTDIS;
in the beginning of the interrupt but that didn't help.

Any ideas what I might be doing wrong?