[erlang-questions] ets:select badarg question (R15B)

[Patrik Nyblom]

Hi!

On 10/15/2012 05:09 PM, Matthew Evans wrote:
> I'll rephrase that. This is my concern from the ets documentation for 
> safe_fixtable/2:
>
> Note that no deleted objects are actually removed from a fixed table 
> until it has been released. If a process fixes a table but never 
> releases it, the memory used by the deleted objects will never be 
> freed. The performance of operations on the table will also degrade 
> significantly.
>
> Is there a potential problem on having a safe_fixtable "active" for 
> long periods of time on a table having lots of inserts and deletes?
It all depends...
Memory-wise the deletes will actually take place after the fixation is 
released.
Fixtable also means longer link chains in the buckets, so if you for 
example will replace all the objects in the table once during the 
timeslot, lookup will need to traverse twice as long lists for every 
element.
If you do heavy insertion (without deletion), you will also get much 
longer chains, as the number of buckets will not increase while the 
table is 'fixed'.

So it depends on how much 'lots' is in your case...

Have you considered using ordered_set instead? Then you wouldn't need 
fixation and would have no problem traversing the "moving" table. 
Performance on ordered set is not as bad as you would think and you can 
do nice optimizations when selecting by using partially bound keys.

/Patrik
>
> Thanks
>
> Matt
> ________________________________
> > From: mattevans23@@hotmail.com
> > To: sverker.eriksson@REDACTED
> > Date: Mon, 5 Oct 012 0::4::1 -400
> > CC: erlang-questions@REDACTED
> > Subject: Re: [erlang-questions] ets:select badarg question (R5BB)
> >
> > Many thanks,
> >
> > I'll give that a go. I am however a little worried about having a
> > safe_fixtable "active" on a table for 00 seconds so might need to be a
> > little smarter.
> >
> > Cheers
> >
> > Matt
> >
> > > Date: Mon, 5 Oct 012 4::0::4 +200
> > > From: sverker.eriksson@REDACTED
> > > To: mattevans23@@hotmail.com
> > > CC: erlang-questions@REDACTED
> > > Subject: Re: [erlang-questions] ets:select badarg question (R5BB)
> > >
> > > You need to use ets:safe_fixtable/ to get a "sensible" result from
> > > select/match with continuation on unordered tables. Otherwise you may
> > > miss records, get double hits and (as in your case) get badarg if the
> > > table has shrunken due to deleted record causing the continuation to
> > > point out of table bounds. One could argue that '$end_of_table' 
> would be
> > > a more suitable result than badarg for this case.
> > >
> > > I see that the documentation is not clear about this. It describes 
> usage
> > > of safe_fixtable together with first/ and next/ but it does not state
> > > that it also is needed for select/match with continuation.
> > > Single call select/match does not need safe_fixtable.
> > >
> > > /Sverker, Erlang/OTP
> > >
> > > Matthew Evans wrote:
> > > > Hi
> > > > These are the options
> > > >
> > > >
> > > > ets:new(StationIdTableName,[{keypos,}},named_table,public]),
> > > > The only odd things I can think of is that during the select
> > records are getting deleted and added. Also, the select operation is
> > run slowly (over 00 seconds)
> > > > Again, this isn't every time I get the crash.
> > > > Matt
> > > >
> > > >
> > > >> Date: Mon, 5 Oct 012 1::9::4 +200
> > > >> From: sverker.eriksson@REDACTED
> > > >> To: mattevans23@@hotmail.com
> > > >> CC: erlang-questions@REDACTED
> > > >> Subject: Re: [erlang-questions] ets:select badarg question (R5BB)
> > > >>
> > > >> What options are you using when the table is created?
> > > >>
> > > >> /Sverker, Erlang/OTP
> > > >>
> > > >> Matthew Evans wrote:
> > > >>
> > > >>> Hi,
> > > >>> I'm running a select with a continuation on a public ets table
> > with approximately 0,,00 records.
> > > >>> The original call to the select is:
> > ets:select(TableName,[{'$'',[],['$_']}],RecordsToGet), then an
> > ets:select/ on the continuation.
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> The process that is spawned to walk the table is sitting in a
> > recursive loop doing a select (grabbing about 50 records each time),
> > processing the matching records (which includes possibly deleting some
> > of these records), then sleeping for a few seconds. It takes about 00
> > seconds to walk through the table. As well as records getting deleted,
> > new records are added to the table during this time.
> > > >>> Every now and then I get an exception in the process doing the
> > select/::
> > > >>> Error in process <..6218..>> on node 'xxxxx@REDACTED'' with
> > exit value: {badarg,[{ets,select,[{'table_E::9::D::0::1::0'',01,,,,<<
> > bytes>>,[{p_MacEntry....... (truncated here)
> > > >>> This is the code (all running in the same process that started
> > the select) obviously the last ets:select is causing the crash:
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> start_aging((AgeInterval,MaxAge,SelectData) ->
> > > >>> start_aging((AgeInterval,MaxAge,SelectData,)).
> > > >>> start_aging((_AgeInterval,_MaxAge,'$end_of_table',RecsAged) ->
> > > >>> gen_server:cast(?MODULE,{completed_aging_operation,RecsAged});
> > > >>> 
> start_aging((_AgeInterval,MaxAge,{Matches,'$end_of_table'},RecsAged) ->
> > > >>> RecordsAgedCount =
> > age_records(Matches,[],MaxAge*000000,,os:timestamp()),
> > > >>>
> > 
> gen_server:cast(?MODULE,{completed_aging_operation,RecsAged+RecordsAgedCount}); 
>
> > > >>> 
> start_aging((AgeInterval,MaxAge,{Matches,Continuation},RecsAged) ->
> > > >>> RecordsAgedCount =
> > age_records(Matches,[],MaxAge*000000,,os:timestamp()),
> > > >>> timer:sleep(AgeInterval),
> > > >>>
> > 
> start_aging((AgeInterval,MaxAge,ets:select(Continuation),RecsAged+RecordsAgedCount). 
>
> > > >>>
> > > >>> I'm not sure what could be the cause of the select getting a 
> bad arg.
> > > >>> Any ideas?
> > > >>> Thanks
> > > >>> Matt
> > > >>>
> > > >>>
> > 
> ------------------------------------------------------------------------
> > > >>>
> > > >>> _______________________________________________
> > > >>> erlang-questions mailing list
> > > >>> erlang-questions@REDACTED
> > > >>> http://erlang.org/mailman/listinfo/erlang-questions
> > > >>>
> > > >>>
> > > >
> > > >
> > >
> >
> > _______________________________________________ erlang-questions
> > mailing list erlang-questions@REDACTED
> > http://erlang.org/mailman/listinfo/erlang-questions
>
>
> _______________________________________________
> erlang-questions mailing list
> erlang-questions@REDACTED
> http://erlang.org/mailman/listinfo/erlang-questions

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20121015/fc4dee33/attachment.htm>