OO, FP and XP training (extremely long)

[Jay Nelson]

[The meat about Functional Programming is at the bottom, you need to 
read the set up to understand how I arrived at my conclusions.]

At work we are mainly a C++ / C# shop now with most development being 
geared towards .NET.  We previously were a UNIX / C++ shop with 
excellent scaleability, reliability and redundancy.  Needless to say, 
scale is being challenged by the move to interactive GUI-based 
applications on Windows XP coming from UNIX servers.

As part of this multi-year transition we have embraced the principles of 
eXtreme Programming (XP) and agile development.  Our team has used an 
external consultant in the last few weeks to coach us, although we have 
been using the Test Driven Development (TDD) approach for a year now.

One of the demos that they gave was a live process of developing a 
bowling game scoring application to show how refactoring works in 
practical terms.  The lecturer ran through the typical OO design anyone 
would come up with which involved 4 or 5 classes and inheritance, and 
then held it to the side while he developed the code with prompting from 
an audience using TDD.  The end result was a for loop with two if 
statements and no classes.  This caused a little discussion as to why he 
ended up with the result he did (TDD forced him to write only the 
necessary code) and his reflection on why "the power of OO was not 
needed in this case".  After being challenged on this in the past, he 
wrote the full OO implementation and ended up with over 400 lines of 
code, confirming his suspicion that no OO was a simpler implementation.  
However, still convinced that OO was a good thing, he believed for more 
complicated programs it was warranted and aided in development*.

In our actual pairs programming with the coach observing, he often 
wanted to simplify the code we have built up over the last year.  One of 
the things that surprised me was that he wanted to eliminate Getters and 
Setters and just use public member variables.  Another difference was 
that all member functions be declared public since that made them much 
easier to test without proxy methods.  He also advocated removing 
inheritance in many cases and replacing it with object factories, 
cooperating classes and interfaces that exposed only the required public 
methods.


-----
*My immediate question was, "If this is so obvious with a toy problem, 
why isn't the complexity of the code multiplied many times when scaled 
up to a larger problem?".
-----


It was enlightening to see the change that TDD is bringing about from 
the standpoint of the experts in the field (our coaching organization 
has been a long advocate and publisher of some seminal texts on the 
subject).  Here is my take from the exercise:

1) IDEs are very important in changing the way you program
    - Verbosity is great, type a few letters and allow completion
    - Refactoring, renaming is much easier
    - Don't ever worry about overhead, avoid all duplicate code
    - Rely heavily on the editor to point out errors without compiling
    - Declare types so that the IDE signatures are more readable, rather 
than type safety

2) TDD changes the way you code
    - Test first means most functions are best made public
    - Constant refactoring makes the code units smaller
    - Make sure you refactor the tests just as diligently
    - Freely modify anything, the tests protect you
    - Classes are essential to override external behavior

3) Interfaces are much more important than classes
    - Use object factories and only access via interfaces
    - Use interfaces as replaced facades when testing
    - Mock objects allows dynamic swapping of functionality

4) All meetings and planning are evil


In general I shuddered at a lot of the "best practices" that violated my 
old training in OO about protection of data and encapsulation.  This was 
after all one of the big advantages of OO.  I came to the following 
general conclusions (feel free to challenge or augment them):


A) TDD subverts staticly typed languages

The main purpose of static typing is to allow the compiler to notify the 
programmer of errors before allowing the execution of code.  If you have 
a bunch of tests that break, it can be more thorough than just a static 
type check of variables and signatures.  This equates in my mind to 
preference for run-time validation with real data.

The second advantage of static typing is compiler optimization, but with 
XP, programmer optimization is more important.  Performance is only 
considered if the acceptance test that specifies a performance level 
does not pass.


B) Engineering principles and experience hinder development

TDD and XP are all about making the team more effective -- faster with 
fewer software errors.  The methods used boost the worst performers a 
lot and hinder the best performers to a lesser degree.  (I suspect the 
very best programmers are slowed down by more than the boost of the 
worst, but there are so few world-class programmers that most companies 
have never met one.)   Any anticipation based on previous experience is 
eschewed for fear it results in over-engineering when a simple solution 
works.  Start with the simplest case that passes the test and gradually 
refine it -- the process of refinement incurs refactoring which will 
produce the desired solution with the simplest design.

While this works in practice for most problems -- it is very easy to 
refactor code to speed it up -- architectural problems do not fall out 
so simply.  Refactoring a data flow that involves multiple systems which 
did not consider performance (or, say, hot code loading) cannot just be 
refactored to introduce a solution.  Deep thought and insight are needed 
to produce an innovative solution.  But a fast, reproducible, straight 
forward approach gets you into the marketplace competing with 
like-minded companies quickly.


C) No one understands the system

Everyone works in pairs on every part of the code.  Writing code is a 
flurry of making a test case fail, writing a simple solution, increasing 
the test case and refactoring.  The tools simplify this so that extra 
arguments can be added to a function in one place and it will be updated 
everywhere -- even if you don't know where those changes happen.  I 
stepped away after hours of coding  several times without understanding 
anything we coded, but knowing that it passed the tests and therefore 
was functionally correct.

Everyone is interchangeable.  Mindless code generation and copious unit 
test studying produce a localized understanding quickly that leads to 
enhancements and bug fixes being done rapidly.  Ask anyone a deep 
question about the philosophy of the system, the general performance 
characteristics or whether there is a clean approach to expanding its 
range of scope and you will be met with blank stares and requests for 
specific test descriptions.


D) XP developers are coding Functionally, not using OO

Gradually the constraints of class encapsulation are being removed.  All 
methods and member variables become public to ease testing, interfaces 
are used to make the IDE work again rather than give a meaningless list 
of every data item in the system (not to restrict code access, since the 
unit tests prove which functions are accessed).  Global variables and 
global functions harken back to the days of APL, one of the first 
functional languages.  The use of cooperating classes rather than 
inheritance is concurrency oriented and relies on function modules 
(interfaces) to access the ghost classes.

When XP developers believe they are taking advantage of Mock Objects or 
proxy classes, they are really doing late binding of function 
interfaces.  When unit tests run, they are simulating the dynamic scope 
of execution and use run-time validation.  When acceptance tests are 
executing they have built an entire scaffolding that should be part of 
the language and always enforced during execution.  UBF-B is a general 
solution that doesn't require all the test instances to be written.


My bold predictions:

1) Agile development will lead to OO languages being replaced by FP 
languages.

2) Agile development will hit a wall that will require a "new innovative 
development process" when average programmers use it for concurrent 
programming without a language like erlang.

3) Businesses will quickly compete in markets and with a level of 
software complexity that they previously couldn't, but will find 
themselves stuck with a product and user base that is not understood 
well enough for strategic planning purposes.

4) "Lone wolf" teams and programmers will be more innovative, but they 
will be copied more quickly by average teams.  The sheer volume of code 
needed to compete will overwhelm the small organizations and force 
innovation out of the market.

5) Big paradigm shifts will be necessary to rebalance industry 
competition; mundane evolution of versions will be possible by all 
companies.

6) IDEs will hide the complexity and level of insight equally, making 
"hand coded" programming a specialized black art that is equated to 
hotrod enthusiast engine tinkerers.

7) Big businesses will continue to make money touting large complex 
systems worked by a team of programmers; startups will continue to find 
niches that they can subvert with new approaches that they sell to the 
big businesses.


jay