Untimely garbage collection

[Per Bergqvist]

Your strategy should be working, I use more or less the same in       
several drivers and it is the most efficient scheme I have figured    
out so far.                                                           
                                                                      
1) the driver have two logical threads a) the vm itself and b) a      
worker thread.                                                        
2) in the outputv thread a) make a copy of ErlIOVec,SysIOVec AND the  
ErlDrvBinary vector in ErlIOVec. NOTE that it is simply the array of  
pointers to the real ErlDrvBinaries. Store this in the drivers        
context.                                                              
3) For each element in the ErlDrvBinaries update refc.                
4) signal is some way (i.e. use pipes) to thread b) to do the actual  
processing and return from outputv. The vm may now do the gc as it    
likes since the refc's are correct.                                   
5) when thread b) is done with the chunk it signal over the pipe to   
thread a).                                                            
6) When ready_input is called in thread a) decrement the refc's for   
all elements in the ErlDrvBinaries vector.                            
Deallocate the ErlDrvBinary vector, ErlIOVec and SysIOVec.            
                                                                      
For a simple implementation use the the set_busy_port in step 2) and  
6). The more ambitious approach is to allow multiple outstanding      
outputv requests.                                                     
                                                                      
/Per                                                                  
                                                                      
> I know this isn't my problem.  If I got the 30+ binaries, each      
> over 128K apiece back in my shell, it'd choke.                      
>                                                                     
> I'd dreged through most of the processes in my Erlang node, it      
> doesn't look like they are stuck in anybody's message queue         
> either.                                                             
>                                                                     
> I had figured my strategy would work well, as it had been           
> at one point about two weeks ago.  I also checked most of           
> the Erts source code, and in R8B-1 the binary should                
> get collected and its refc updated immediately during               
> a call to erlang:garbage_collect().                                 
>                                                                     
> I recently concluded about two days ago that I may not              
> have all of the cases worked out on interaction between             
> my background pthread and Erts.  I think I am going to              
> revisit them and see what might be missing.                         
>                                                                     
> As an intersting side note, the refc's are messed up                
> unless I use fprintf(stderr, "test!\r\n") in my                     
> driver.  That causes the refc's to be correct.  Cute.               
> Clearly something is wrong.                                         
>                                                                     
> And I think Sean Hinde is right, what I am trying to                
> do should just work, providing my thread communication is           
> correct.                                                            
>                                                                     
> Sean Hinde <Sean.Hinde@REDACTED> scrawled:                    
> > > Suddenly this has stopped working.  My C drivers are            
> > > seizing when they run out of binaries, and all of the           
> > > binaries have a refc of 3:  one for the driver that             
> > > "owns" the binary, and one for each gen_server process.         
> >                                                                   
> > I had this problem once when developing my (now abandoned - mnesia
got even                                                              
> > better :) ) SQL driver. The solution turned out to be sipmly that 
I was                                                                 
> > testing from the shell which also retained the binaries in it's   
result                                                                
> > history!                                                          
> >                                                                   
> > Otherwise binaries got GC'd very quickly and I could rely on the  
refc no                                                               
> > problem.                                                          
>                                                                     
> --                                                                  
> Shawn.                                                              
>                                                                     
> Why do I like Perl?  Because ``in accordance with Unix tradition    
Perl                                                                  
> gives you enough rope to hang yourself with.''                      
>                                                                     
> Why do I dislike Java? Because ``the class ROPE that should contain 
the                                                                   
> method HANG to do the hanging doesn't exist because there is too    
much                                                                  
> 'security' built into the base language.''                          
>                                                                     
=========================================================             
Per Bergqvist                                                         
Synapse Systems AB                                                    
Phone: +46 709 686 685                                                
Email: per@REDACTED