trouble with files bigger than 4GB

[Raimo Niskanen]

Comments inserted below...

Daniel Solaz wrote:
> On Thursday 17 April 2003 10:14 Raimo Niskanen wrote:
> 
>>There is some way to compile a program for solaris so that fopen()
>>means fopen64(), off_t becomes 64 bit, and so on. If this is done,
>>the code should work. It is perhaps even so that for an Erlang/OTP
>>system built for Solaris8 it works, while not for a system built for
>>Solaris2.5.1.
> 
> Yes, this is described in the lfcompile(5) man page.
> I'll try to hack the appropriate Makefile(s) and recompile so I can 
> check whether this is enough to get the code working, where working 
> means "like in FreeBSD".
> 
> 
>>Since the size of off_t is configured when building open source
>>Erlang, on a system with 64 bits off_t - large file operations should
>>work. Have a look in $ERL_TOP/erts/$PLATFORM/config.h for
>>SIZEOF_OFF_T and see if it is 4 (bytes). $PLATFORM is the system name
>>$ERL_TOP/erts/autoconf/config.guess reports when executed. $ERL_TOP
>>is your open source build root.
> 
> You're right, off_t is reported to be 4 bytes long.
> 
> 
>>Please try some more file operations on your large file. Try
>>file:open, file:position, file:read, and see if it seems to work on
>>FreeBSD. I am really curious.
>>I have browsed the source and think I see a bug in reading the size
>>field for file:read_file_info - the high word of the size field is
>>ignored, so if other file operations work, it is a small bug fix and
>>can be done for R9C. If no other file operations work, it is a bigger
>>problem and will have to wait. Please let me know.
> 
> Reading the large file buffer by buffer all the way from the beginning 
> works fine; file:position(F, {bof, 4294967296}) returns {ok, 0} BUT the 
> file pointer is actually set to byte 4294967296, as does
> file:position(F, {eof, 0}) on a 4 GB file.
> So it seems the problem lies in the code that reports sizes and offsets 
> ignoring the high word of the file size or position.
> 
> -Daniel

Well, I will fix the simple bug i mentioned above, but if off_t is 4 
bytes, there is not much to do on such a system. There is no high word 
to make use of. File size (or absolute position) 4294967296 is 
indistinguishable from 0. Absolute position 4294967295 is the same as 
error return value -1 from lseek().

So file sizes up to and including 4GByte-2Byte should work, but for 
larger files you need a system with 8 byte off_t. And the bug fix for 
file:read_file_info/1 to come in R9C.

/ Raimo Niskanen, Erlang/OTP, Ericsson AB