Flashback database impact

Ran into this interesting problem not too long ago. A procedure doing an insert and a merge ran much slower in one environment compared to another similar environment. After tracing in the slow environment and profiling the trace, this came up:



That much time spend on "db file sequential read" for a straightforward "insert into ... values (....)"? Weird...
A bit more investigation showed that lots of those waits were for the undo tablespace:
p5skew --name="db file sequential read" --group=$p1 xxxxx_ora_13766882.trc



So what are those files?

select file_id, tablespace_name from dba_data_files where file_id in (116,179,124,2,115,187,244);

FILE_ID TABLESPACE_NAME
244     UNDO2
179     xxxxx_DAT_IOT
187     UNDO2
124     UNDO2
116     UNDO2
115     UNDO2
2       UNDO2

So mostly undo (83.6%)! Why?

Then I started thinking about one major difference between the two databases, the one with the slow performance had flashback database enabled. A quick search on Metalink gave me Note:565535.1 "Flashback Database Best Practices & Performance". This note tells us that "Insert intensive batch jobs may be impacted by as much as 30% with flashback database on."
So we disabled flashback database, reran the proc and performance improved dramatically. So Oracle was absolutely telling us the truth in that note. So beware of this in Oracle 10g, when using this very cool feature!

PS. Screenshots are from Hotsos profiler output.