[erlang-questions] ets:select badarg question (R15B)

[Sverker Eriksson]

Deleting a lot of objects in a fixed table may cause memory problems as 
the deleted object are not deallocated. They are instead marked as 
deleted and will be kept around until the table is unfixed.

Increasing the number of objects in a fixed table may cause performance 
problems. The hash table will not grow, leading to long linear search 
through chains of objects hashing to the same bucket.

/Sverker


Matthew Evans wrote:
> I'll rephrase that. This is my concern from the ets documentation for safe_fixtable/2:
> Note that no deleted objects are actually removed from a fixed table until it has been released. If a process fixes a table but never releases it, the memory used by the deleted objects will never be freed. The performance of operations on the table will also degrade significantly.
>
> Is there a potential problem on having a safe_fixtable "active" for long periods of time on a table having lots of inserts and deletes?
> Thanks
> Matt________________________________
>   
>> From: mattevans23@@hotmail.com 
>> To: sverker.eriksson@REDACTED 
>> Date: Mon, 5  Oct 012  0::4::1  -400  
>> CC: erlang-questions@REDACTED 
>> Subject: Re: [erlang-questions] ets:select badarg question (R5BB) 
>>
>> Many thanks, 
>>
>> I'll give that a go. I am however a little worried about having a 
>> safe_fixtable "active" on a table for 00  seconds so might need to be a 
>> little smarter. 
>>
>> Cheers 
>>
>> Matt 
>>
>>     
>>> Date: Mon, 5  Oct 012  4::0::4  +200  
>>> From: sverker.eriksson@REDACTED 
>>> To: mattevans23@@hotmail.com 
>>> CC: erlang-questions@REDACTED 
>>> Subject: Re: [erlang-questions] ets:select badarg question (R5BB) 
>>>
>>> You need to use ets:safe_fixtable/  to get a "sensible" result from 
>>> select/match with continuation on unordered tables. Otherwise you may 
>>> miss records, get double hits and (as in your case) get badarg if the 
>>> table has shrunken due to deleted record causing the continuation to 
>>> point out of table bounds. One could argue that '$end_of_table' would be 
>>> a more suitable result than badarg for this case. 
>>>
>>> I see that the documentation is not clear about this. It describes usage 
>>> of safe_fixtable together with first/  and next/  but it does not state 
>>> that it also is needed for select/match with continuation. 
>>> Single call select/match does not need safe_fixtable. 
>>>
>>> /Sverker, Erlang/OTP 
>>>
>>> Matthew Evans wrote: 
>>>       
>>>> Hi 
>>>> These are the options 
>>>>
>>>>
>>>> ets:new(StationIdTableName,[{keypos,}},named_table,public]), 
>>>> The only odd things I can think of is that during the select 
>>>>         
>> records are getting deleted and added. Also, the select operation is 
>> run slowly (over 00  seconds) 
>>     
>>>> Again, this isn't every time I get the crash. 
>>>> Matt 
>>>>
>>>>
>>>>         
>>>>> Date: Mon, 5  Oct 012  1::9::4  +200  
>>>>> From: sverker.eriksson@REDACTED 
>>>>> To: mattevans23@@hotmail.com 
>>>>> CC: erlang-questions@REDACTED 
>>>>> Subject: Re: [erlang-questions] ets:select badarg question (R5BB) 
>>>>>
>>>>> What options are you using when the table is created? 
>>>>>
>>>>> /Sverker, Erlang/OTP 
>>>>>
>>>>> Matthew Evans wrote: 
>>>>>
>>>>>           
>>>>>> Hi, 
>>>>>> I'm running a select with a continuation on a public ets table 
>>>>>>             
>> with approximately 0,,00  records. 
>>     
>>>>>> The original call to the select is: 
>>>>>>             
>> ets:select(TableName,[{'$'',[],['$_']}],RecordsToGet), then an 
>> ets:select/  on the continuation. 
>>     
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The process that is spawned to walk the table is sitting in a 
>>>>>>             
>> recursive loop doing a select (grabbing about 50  records each time), 
>> processing the matching records (which includes possibly deleting some 
>> of these records), then sleeping for a few seconds. It takes about 00  
>> seconds to walk through the table. As well as records getting deleted, 
>> new records are added to the table during this time. 
>>     
>>>>>> Every now and then I get an exception in the process doing the 
>>>>>>             
>> select/:: 
>>     
>>>>>> Error in process <..6218..>> on node 'xxxxx@REDACTED'' with 
>>>>>>             
>> exit value: {badarg,[{ets,select,[{'table_E::9::D::0::1::0'',01,,,,<<  
>> bytes>>,[{p_MacEntry....... (truncated here) 
>>     
>>>>>> This is the code (all running in the same process that started 
>>>>>>             
>> the select) obviously the last ets:select is causing the crash: 
>>     
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> start_aging((AgeInterval,MaxAge,SelectData) -> 
>>>>>> start_aging((AgeInterval,MaxAge,SelectData,)). 
>>>>>> start_aging((_AgeInterval,_MaxAge,'$end_of_table',RecsAged) -> 
>>>>>> gen_server:cast(?MODULE,{completed_aging_operation,RecsAged}); 
>>>>>> start_aging((_AgeInterval,MaxAge,{Matches,'$end_of_table'},RecsAged) -> 
>>>>>> RecordsAgedCount = 
>>>>>>             
>> age_records(Matches,[],MaxAge*000000,,os:timestamp()), 
>>     
>> gen_server:cast(?MODULE,{completed_aging_operation,RecsAged+RecordsAgedCount}); 
>>     
>>>>>> start_aging((AgeInterval,MaxAge,{Matches,Continuation},RecsAged) -> 
>>>>>> RecordsAgedCount = 
>>>>>>             
>> age_records(Matches,[],MaxAge*000000,,os:timestamp()), 
>>     
>>>>>> timer:sleep(AgeInterval), 
>>>>>>
>>>>>>             
>> start_aging((AgeInterval,MaxAge,ets:select(Continuation),RecsAged+RecordsAgedCount). 
>>     
>>>>>> I'm not sure what could be the cause of the select getting a bad arg. 
>>>>>> Any ideas? 
>>>>>> Thanks 
>>>>>> Matt 
>>>>>>
>>>>>>
>>>>>>             
>> ------------------------------------------------------------------------ 
>>     
>>>>>> _______________________________________________ 
>>>>>> erlang-questions mailing list 
>>>>>> erlang-questions@REDACTED 
>>>>>> http://erlang.org/mailman/listinfo/erlang-questions 
>>>>>>
>>>>>>
>>>>>>             
>>>>         
>> _______________________________________________ erlang-questions 
>> mailing list erlang-questions@REDACTED 
>> http://erlang.org/mailman/listinfo/erlang-questions 
>>     
>  		 	   		  
>