[erlang-questions] Beginner Screen Scaping, and Auto-login/input on 3rd party web app?

[Alain O'Dea]

On 2010-12-24, at 11:08, JETkoten <jetkoten@REDACTED> wrote:

> Hi Everyone,
> 
> I'm (very) new to Erlang, and hoping to get some basic experience with it.
> 
> I've made some initial steps in learning Ruby, and was in the bookstore the other day and thought I'd look at what's new in Ruby books. They were listed in "Other Languages" and the bookstore didn't have anything I hadn't seen before for Ruby so I thought since it was such a small selection I'd just grab and browse all the books for the languages I wasn't familiar with.  After reading a little bit about Haskell, Common Lisp and Scala, I found Joe Armstrong's Erlang book and was very interested. I've since viewed hours of Erlang videos online, and am reading as many general tutorials as I can find.
> 
> I really learn best by doing something I'm interested in. I have a "pet project" that I've been working on in Ruby that I would also like to implement now in Erlang.
> 
> Here it is:
> 
> I have a large personal library of books and find that I don't need many of them anymore. I'd like to create a program that will help me manage my online sales on a marketplace site, by automatically checking competing sellers' prices at a set time interval an then logging into their website and adjusting my prices according to a formula I'd set based on the other prices.
> 
> I started it in Ruby already, and got the basic price lookup functionality working for one title at a time only by manually inputting the URL and then using Nokogiri to screen scrape the price info from parsed CSS3 tags on the page, and then Mechanize to login and change my prices for me.
> 
> I did a Google search on Erlang "screen scraping" and saw some options:
> 
> www_tools, Yaws parser, xmerl, mochiweb
> 
> However, none of the posts that suggest those are less than 2 years old... which is the best/easiest way, and/or are there newer, better options now?
> 
> Approaching Erlang, I have absolutely no idea how to get started at getting the same functionality working as I have in Ruby. I am hoping that someone here can point me in the right direction. If possible, I think it'd be neat to do it all concurrently too, but I don't want to make it too hard on myself if that'd bring up too much complexity for a beginner like me.
> 
> Any ideas?
> 
> Thanks in advance,
> Jack

Hi Jack:

I would do the coordination/timing in Erlang and invoke utility scripts written in Ruby with os:cmd/1 as a starting point.  Gradually it probably makes sense to switch to native Erlang utilities if you find them to perform or integrate better.

The advantage Erlang gives you for this is how radically easy it would make running pricing agents for many books.  Just spawn a new one for each book :)

Eventually it makes sense to use OTP and supervisors to consistently handle agent crashes.  If you find yourself writing a lot of try/catch logic, then stop and refactor to OTP.  Erlang and OTP in Action http://manning.com/logan is the best book for this.

Cheers and Merry Christmas,
Alain