[erlang-questions] Erlang overload/bandwidth limitation (JOBS?)

[Bob Ippolito]

I'm not familiar with JOBS but your first case is easily solved with
just serializing that work to a single process, e.g. a gen_server. The
mailbox of that process is your queue.

Without seeing the code I don't have anything very specific to suggest
for the second case, but you'll probably want to look at using ets and
ets:update_counter/3.

On Tue, Nov 23, 2010 at 7:20 PM, Jesper Louis Andersen
<jesper.louis.andersen@REDACTED> wrote:
> Hi List,
>
> The etorrent application, a bittorrent client for erlang, is in need
> of some load balancing in two ways, which I describe below, and I need
> a tool to do some load balancing in the two cases. Are there any
> suggestions besides using the Erlang Solutions JOBS application for
> the task?
>
> Case 1: The client can handle multiple torrents at once. When you
> start off on a torrent and you find a file existing in the download
> directory matching the name and size of what you want, you have to
> check it. This check is running SHA1 checksums over the whole file and
> thus puts quite a burden on the disk IO subsystem of the computer. In
> the lucky case that we already did some checking and the file passes a
> sanity check, we can look up a lot of the checking result in a
> persistent table - but in general we have to go over the file. To not
> trash the disk we only want to allow a single torrent to be checked at
> a time.
>
> The current solution is ugly. A token is passed among the torrents who
> wants checking and he who has the token may check. There are some
> monitoring going on to ensure that if we loose the token, we reinject
> a new one in the system. As such, there is some reminiscence of a
> token ring configuration.
>
> JOBS seems like a good fit for this case: Add a job queue with a
> severe limitation of 1. Fire off a job to the queue (which you make
> large enough that it can hold the pending jobs) which is the actual
> checking code. Done.
>
> Case 2: Upstream bandwidth limitation. TCP will by default eat all of
> a lines upstream bandwidth, which unfortunately includes ACK-packets
> needed for the downstream. The usual trick is to put some limitation
> to how much you send out from the client to crudely steer the
> connection. A very big problem here is that most internet connections
> from homes have cheap routers in front of them and these have a small
> buffer of, say, 8 packets. Any more packets arriving are simply
> dropped from the router. This makes TCP sad and erratic such that it
> cannot really do its job too well. There is an UDP based protocol
> extension, μTP, which attempt to battle this problem directly but that
> protocol is a bit out in the future, hence the bandwidth limitation.
>
> The question is, can JOBS or some other overload framework be easily
> adapted to this scenario?
>
> Discussion: Basically, you have a lot of willing senders contending
> for a global upstream bandwidth resource. You allow them access to the
> resource but you don't want one sender to claim all of the resource.
> Hence you can round-robin schedule the bandwidth in smaller chunks or
> use RED - random early detection; in which the greatest hoarder runs a
> larger risk of being denied bandwidth. I don't know if the JOBS
> framework can do this in full or in part.
>
>
> --
> J.
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe@REDACTED
>
>