Was imo by far the best film from 2003. But I need to see it again, two years is quite a time for an evolving cineaste. 

Regarding your question, I think the effect would be lost very much. The device is used so the viewer gets to know the troubled relationship better as the film unfolds. So our feelings towards the protagonists can be reflected upon in a new light each time. The MAIN focus is on the last days before death, and only through these flashbacks can, imo the dramatic effect be kept to the fullest. It is like a kaleidoscope with a center but the emotions and colors changing with every new insight.