I'd try to narrow down where exactly in glReadPixels the delay is incurred. Either using some profiling / tracing tool, or just by adding printfs with timestamps in strategic places.