Extremly high random I/O in query overlaping API for CRAM

[karlestira]

CRAM query overlaping has very high random io(over 20% CPU time and over 50% real time cost in java RandomAccessFile). And this does not happen in BAM file(only less than 1% cost in the same filesystem, and BAM shows much faster than CRAM).

I am using GPFS filesystem, which has poor support for linux pagecache(which means the cache is very poor). The I/O time will not very high in local filesystem(such as btrfs or xfs). I think a buffer/cache may need to cover the I/O time?

I tried to wrap SeekableFileStream by the SeekableBufferedStream in CRAMFileReader::getSeekableStreamOrFailWithRTE(), line 397, and the CRAM input process became much faster(60min -> 25min).

Modified code:

CRAMFileReader.java: line 397 for HTSJDK version 4.2.0
old code:

seekableStream = new SeekableFileStream(cramFile);

new code:

seekableStream = new SeekableBufferedStream(new SeekableFileStream(cramFile));

The overlaping size is about 150bp in my testing, WES sample, calling SNPs in a inner-use program.

In my test, this has no benefit(and also no bad effect) on local filesystem(xfs, btrfs, ext4 or so on), but has great benefit on remote filesystem(GPFS, zfs(low ARC cache), nfs(maybe)).

Maybe we can give a option in SamReaderFactory to let user control the buffer reader behaviour?