Thursday, December 13, 2007

BGGA closures: The end of many Java careers

Before you begin, I'd encourage you to read Neal Gafter's latest rebuttal to Josh Bloch's explanation of why BGGA can do more harm than good to Java. My two cents (as a Java developer) after going through both arguments:
  1. First, Neal Gafter states that Josh's examples were carefully chosen from "twisted" compiler test cases written for the BGGA closure implementation. Why did the test cases have to be "twisted" beyond comprehension? Well, its perhaps because complicated features beget complicated test cases. While I agree that test cases are generally more heavyweight (primarily due to the boiler plate and to cover edge cases) compared to the feature being tested, you will rarely find that they are an order of magnitude complicated than the feature itself. You won't find unrealistic test cases either. You will only need gnarled test cases for convoluted, counter-intuitive "features". The fact that Josh didn't have to look very far to come up with a "hard" example says a lot about the BGGA proposal itself.

  2. Let's look at Item 1 of Chapter 2, Effective Java: Programming Language Guide. It says:
    "One advantage of static factory methods is that, unlike constructors, they have names. If the parameters to a constructor do not, in and of themselves, describe the object being
    returned, a static factory with a well-chosen name can make a class easier to use and the resulting client code easier to read. For example, the constructor BigInteger(int, int,
    Random), which returns a BigInteger that is probably prime, would have been better expressed as a static factory method named BigInteger.probablePrime".

    Let's also refer to section 1.3 of one of the fundamental text books on programming, Structure and Interpretation of Computer Programs. In the section preceding higher order procedures (not to be confused with functions), the authors say:
    "Our programs would be able to compute cubes, but our language would lack the ability to express the concept of cubing. One of the things we should demand from a powerful programming language is the ability to build abstractions by assigning names to common patterns and then to work in terms of the abstractions directly"
    (Before you jump to the conclusion that this chapter favors the BGGA approach, let me remind you that it merely explains one programming paradigm and does not advocate introducing functions or procedures on a whim to a strongly typed, mature, Object Oriented language.)

    Finally, here are couple of quotes from Martin Fowler's bliki -
    "...this principle comes with an important consequence - that it's important that programmers put in the effort to make sure that this code is clear and readable. " and
    "...the first step to clear code is to accept that code is documentation, and then put the effort in to make it be clear. I think this comes down to what was taught to most programmers when they began to program... We as a whole industry need to put much more emphasis on valuing the clarity of code."

    So, having read the above excerpts (and assuming that seasoned Java developers get used to the ugly syntax in the near future), what can be said about a very simple BGGA construct like {int, int, Random => BigInteger} ??
    Is it readable? Intuitive? Self explanatory?
    And yes, while BGGA doesn't mandate such a coding style, it certainly provides plenty more scope for it than the Java language does in its current state. As Josh Bloch mentions, its not the syntax, but the semantics (like that of the ugly non local returns) that make BGGA complicated.

  3. It would be a mistake to evaluate the complexity of BGGA in isolation. Indeed, the 427 page generics FAQ, for instance, doesn't talk about generics in isolation. In reality, it is the interaction of generics with other language features that makes generics so hard to master. Its a curious coincidence that Angelika Langer on the generics FAQ thanks 3 of the 4 BGGA authors for patiently answering "countless questions posed" regarding generics. You'd expect that the BGGA authors' first initiative would be to simplify generics and make its behavior & interaction with other language features more predictable, instead of attempting to make changes that further convolute the type system and add cognitive load to the language for little benefit. (I'd highly recommend reading Alex Buckley's excellent write up on the role of complexity in language design and ways to evaluate the complexity.)

  4. From Sun's rebuttal to Microsoft's Delegates proposal -

    "Many of the advantages claimed for Visual J++ delegates -- type safety, object orientation, and ease of component interconnection -- are simply consequences of the security or flexibility of the Java object model. These advantages are routinely enjoyed by all programmers using the Java language, without resorting to non-standard language syntax or platform-specific VM extensions.

    Bound method references are simply unnecessary. They are not part of the Java programming language, and are thus not accepted by compliant compilers. Moreover, they detract from the simplicity and unity of the Java language. Bound method references are not the right path for future language evolution."

    Why were delegates or bound method references considered antithetical to the Java language? From the Java language specification,
    "The Java programming language is a general-purpose, concurrent, class-based, object-oriented language. It is designed to be simple enough that many programmers can achieve fluency in the language. The Java programming language is related to C and C++ but is organized rather differently, with a number of aspects of C and C++ omitted and a few ideas from other languages included. It is intended to be a production language, not a research language, and so, as C. A.R. Hoare suggested in his classic paper on language design, the design has avoided including new and untested features."

    Firstly, it is meant to be object oriented. Why would you wince if you saw C-style procedural code written in Java? Because Java is not meant to be used in such a style of programming. Each style of programming has its place (and has a dedicated set of languages for it). While I'm not against functional programming, I certainly wouldn't be using Java if I needed to program in such a style. I'd probably use Erlang, (like Amazon Simple DB supposedly does), Lisp or Scheme. It's horses for courses.
    Saying "me too" to every new language and trying to cram multiple programming styles into an evolved language like Java is nothing short of catastrophic, to say the least.
    If a language theorist wants to try that, he shouldn't attempt to force changes into a language that millions of programmers earn their livelihood from. (Hence the title of this post.) Remember, its a blue collar language. And its ubiquity is attributed to the ease with which a newbie can write reasonably reliable, readable Java code after relatively less training. Besides, as Josh mentioned in the video interview, he'd have probably included support for closures in a new language built ground-up for that purpose. The danger lies in continuously adding seemingly minor, self-contained features to a language that is more than a decade old, has a multi-billion dollar industry relying on it and is deployed in mission critical environments. If developers still seek functional programming with strong typing and a binding to the JVM, there's an excellent implementation in the form of Scala.

  5. To rephrase the latter half of the previous point, the practical (read economic) consequences of making sweeping changes to the foundation of a language or even a runtime cannot be ignored. For the same reason, Sun has always been (rightly) obsessed with maintaining binary compatibility of the Java platform. Also, you don't see every over zealous JUG or committee proposing a VM spec change with every release of Java. But does the same rule not apply to the Java language? Does it imply that the language can change swiftly in any manner subject to the whims of a few experts (or even a few JUGs)?

  6. As James Gosling stated (about GridBagLayout), "with great power comes great complexity". Assuming we need the "power" of BGGA closures, what cost are we paying for it? Were the costs even considered while making this proposal? Speaking of which, what (ideally) differentiates an engineer from a theorist is the former's ability to evaluate various tradeoffs with a certain pragmatism. Apart from the the use-cases mentioned by Bloch - fine grained concurrency and resource management which can be solved more elegantly with CICE - there's hardly a compelling use case that can't be addressed by present day Java. Do you want to use closures for event handling? Tom Ball, one of the initial members of the Swing/AWT engineering team, should know a thing or two about it? Read Charles Nutter's comment to that blog post and you'll see that BGGA closures aren't a significant improvement over anonymous inner classes and in fact perform poorly as proven by Josh. So, are we getting drastically better capabilities in return for the conundrum?

  7. Josh's presentation makes a passing reference to "dialects" being encouraged by BGGA. Why should we worry about dialects, you might ask. Firstly, it is important to realize that code lives on forever. A recent java.net poll found more than 50% of all running code being more than 6 -10 years old. Its highly unlikely that the author of the code is still sitting around maintaining that code. To give you more evidence, during my short career (working on telecom software, data center management software and the server side of a payments engine), I've had to read and modify existing code far more than write shiny new components. And despite such varied fields of application and diverse use cases, I've always found Java code behaving reasonably predictably with respect to how it reads. (What you read is what you observe in production, barring bugs and deployment goof-ups. No magic there. ) And with minimal effort (despite poor documentation in some cases), I was able to get along with my work within a few weeks (or a couple of months at most) of joining the new company. Does any of this sound surprising or astonishing to any seasoned Java developer? No. It is something you come to expect. And imagine how much money the employer saves by having the developer be productive within a few weeks of joining . Now, add BGGA's library/programmer defined control structures to this scenario: at each company that a developer moves to (or each new project he/she works on), he will have to learn a new "style" of Java programming (as against a new API). Knowing the language alone will be even more futile, and a new engineer will potentially have to go through months of unlearning an old library construct (because, you see, its a new flavor of Java altogether) and then learning a new style of Java programming.

  8. Looking at the above argument slightly differently, why would anyone choose Java for large scale (and more importantly, long lived) business applications when you can (theoretically) write hand-tuned platform specific code efficiently in C/C++/Assembly or what have you? In addition to platform independence, it is primarily due to the ease of reading and writing code in Java. What one engineer writes can be read by another. Assuming (wrongly) that Java is slower than C++ for long running server applications, why would anyone still use it? Because hardware is cheap. Maintenance isn't.

  9. Speaking of the CICE and ARM proposals, it'd be great to see more meat and concrete illustrations added to these proposals (maybe even a kitchen sink implementation - anyone willing to give it a shot?) to highlight their merits over other proposals. IMHO, the TODOs must be completed asap. While I'm unequivocally in favor of these two proposals, the truth remains that a sizeable chunk of the developer community is more easily convinced with specifics and tangibles.

  10. Lastly, I'm relieved that Josh finally spoke out and expressed his view point and the rationale behind his stand. His arguments were based on examples obtained from the BGGA implementation itself and technical publications from Java's formative days. Most importantly, his statements were steeped in pragmatism. I find it rather amusing that the voting trend at JavaPolis reversed soon after his talk. Until then, for nearly a year, the proponents of BGGA conveniently interpreted the community's demand for closures (and relief from the boiler plate of inner classes) as support for the BGGA closures.
    Having said that, no language feature should be determined on the basis of a poll or vote. Nor should we rush into such JSRs. Its taken a long time for Java to reach its current stature in the developer community, and the least we owe the language is a lot of thought, consideration, debate and deliberation before changing it significantly. We should also refrain from portraying the closures debate as a war between two rival camps. Our energies would be better spent by simply focusing on what the language needs (and doesn't need) to thrive in the next decade.

    Update: A few readers seem to believe that the title is a bit far fetched. Here's the explanation:
    You add more complexity to the language and make it harder for newbies or existing programmers to use it -> Programmers look for saner/more intuitive languages -> Java adoption plummets -> "Java specialists" have no option but to choose alternative careers.
    Is that too far fetched an extrapolation? Am I exaggerating? Trying to spread panic? Think again. Look at the battles that the Java platform itself is waging against Flex and other rival runtime environments, for instance. Would we rather stabilize the language and focus on strengthening the platform, or make the language harder to understand, and as a result further alienate the developer community?

16 comments:

kodeninja said...

"Remember, its a blue collar language"... I like that :)

studdugie said...

Thank you for putting this post together. For the first time since the closure debate began there is a glimmer of hope that it may not be forced down my throat.

gcomnz said...

The end of Java careers is when Java isn't used because programmers who like closures have moved on to other languages that provide them, which is pretty much ALL OTHER LANGUAGES.

If Java 7 comes out without closures, then I will be migrating my career to another language. No problem there, but a little sad since Java is a nice language and would be nicer for supporting the important tool of closures that every white collar programmer should have in his or her toolbox. I'll leave behind non-closure languages for those who like working in steel mills, oil wells, and salt mines.

Bharath said...

gcomnz:

The argument is equivalent to saying, "I'll move out of Java if it doesn't support reified Generics". Sure, it can very well be done. Its also perfectly possible to have functional programming in an object oriented language.
However, the problem in the former case lies in changing the VM spec to have additional type information at runtime. In the latter case, the danger lies (as I mentioned repeatedly) in adding such a flavor of closures to a language that already has a complicated type system and will succumb under the burden of its own complexity if sufficient caution isn't exercised while introducing more language changes.
Java has been able to reach a certain level of popularity within the developer community and has a thriving industry built around it for a certain reason: it is indeed easy for (in your words) lumberjacks, coal mine workers, fishermen and lock smiths (not to mention the scientists working at JPL, NASA) to get jump-started coding in Java and be reasonably productive. It makes practical sense for it to stay that way. As for fancy new language constructs, there's nothing stopping "white collar" developers from adding new languages (like Groovy, Scala, etc with JVM bindings) to their toolbox and have these languages co-exist with Java. The choice exists even today.

Rohan said...

Let's add a "class" keyword to C, C is an evolved language, and OOP support is a very nice to have feature, too sad it isn't there in C, I will move on to C++. Oh, but let's make C++ platform independent by introducing a VM for it. C++ is also an evolved language and platform independence is nice to have. Or else, let's add everything to every language. Let's go all out and break things apart.

Unknown said...

About point number 7: This is why it took more than 3 years before Java developers started to actually read (and sometimes use) Generics and Annotations.
Java will not be what it is today without Annotations and Generics and their powerful use found in JPA, EJB3, JAXWS, and coming WebBeans.
Java 7 language features are for frameworks that will be widely used in 2012. Think about it!

gcomnz said...

In response, no, it's not like saying that about Generics. Not at all. Because I happen to understand that erasure is the lesser evil of two choices for the time.

I also happen to understand that Java 7 is a significant change to the JVM and my bet is it won't have bytecode compatibility with previous versions. This is why closures can be done right now. (And why erasure can be solved).

Generics made library improvements, despite the many complainers who should go back to writing blogs rather than code, since that seems to be what they are better at.

Nikolay said...

Java must evolve. If it doesn't, if it's fails to make our life easier and easier, we'll migrate to C#, which was brave enough to incorporate so many new and revolutionary features, Scala with its amazing Lift framework, Ruby on Rails, or Python's Django. "Readable code" often is equivalent to "less code" and this is what the negatively sounding "syntactic sugar" gives us. "Sugar" is a bad word, so, I like to call it "syntactic honey". "Honey" is good!

It's sad that the Java community is afraid of change. It's sad that the Java community sees the syntactic sugar as helping the code typing (i.e. it something that the IDE should do instead of us) and not its reading. But it's usually "code once read many times", right?

Bharath said...

Folks,

1) For comments on generics , I needn't say more, really. As far as generics being library improvements is concerned, what can you say about this simple code, which compiles merrily on pre-b58 of Java SE 5 and fails to build in later versions:

Map<String, String> props = (Map<String, String>) new Properties();

A simple case of generifying a library class and adding compiler support for it evidently turned out to be error-prone . The engineers working on the library and javac are obviously supremely talented. When they could slip (even if in a small way) with generics, what about the millions of ordinary programmers? They should all give up programming and take to blogging for a living, perhaps?

2)BGGA-style closures might indeed help frameworks, but that's exactly the danger that Josh was referring to: each framework (and accompanying library) can easily promote its own flavor of the Java language. The problem of framework-fueled fragmentation already exists today (Java EE versus Spring for instance). We would only be exacerbating it.

3)For obvious reasons, I wouldn't compare annotations to function types.

4)I'm not against solving the problem that BGGA closures attempt to address. I favor an approach like CICE & ARM that doesn't complicate the type system.
[Try explaining to a newbie programmer that null is no more just a reference or a value as in -
Class<java.lang.Null> clazz = null.class;
]

As Tim Bray mentions , its better if a solution is tested out in the real world before being forced upon the developer community.

Paulo Faria said...

Python is moving away from higher order functions (that use closures and method pointers, normally with yield inside eyech) to Generators (that evaluate lazily by default, probably with a hidden implicit yield. All that whoopla about the nature of return inside a closure is not more than a hidden yield (with the exceptions complications added). If you want to make custom for constructs i'd HIGHLY, HIGHLY advocate that we look at python generators.

Do you know why they are moving away from higher order functions. Because they get hard to read... Now add Generics and types like in java. The result is not pretty for me.

ashishwave said...

your args against implementation of generics using the example of difficulty by newbie programmer in undetstanding that why this will fail:
class <java.lang.Null> clazz = null.class;

my question is that if an idiot like a person who can't RTFM atleast even at this level for understanding generics which has been for years in C++ and other languages, can also have problem in understanding - what inheritance is , or why we need interfaces? why don't we just plain write the functions there in class itself, rather than implementing interface, why learn/use a new notion of "exception" just why not return some "errorcode" from a function.
Hence due to a newbie programmer , we should drop try,catch,implements, interface etc. keywords from java; and then Java will become super-friendly to the newbie programmer.
Generics are a duly required feature of java or any good OO language, and i do not see merit in the argument that as a so called newbie will have difficulty in understanding this , so we should not implement this very much feature in java. That level of newbie programmer can very well be happy with php,c,... etc

bye :-)
Ashish Ranjan
ashishwave@yahoo.com

Praveen said...

Thanks for coming up with supereb analysis. Java has to evolve. Joel, Paul Graham have been arguing from quite long time the necessity of closures for any langauage, Now with Martin Fowler & Thoughtworks siding Ruby way, difficult days are ahead for Java.

I think we should stop putting anymore pressure on java & adopt Scala/Groovy/JRuby for DSL stuff. With the amount of java code I guess it takes decade to move away from Java.

Unknown said...

I've been programming with Java for more than a decade, and I'm not sure how long its been since generics came out, but you don't have to USE them: there are not that many core APIs (or libraries) that mandate the use of generics.

Same goes for any other language feature (in fact, any language feature in ANY language): barring libraries that REQUIRE the use of the feature, the option of declining to use that feature is always there. Just don't touch it if you don't need it: so long as they don't "upgrade" the old libraries to use the new features, and the implementation of the new features doesn't change the behavior of existing features, its no harm, no foul. I have a ton of code that ran in 1.3, and with a simple recompile (with SuppressWarnings turned on) is humming along quite fine in 1.6.

On the other hand, asking people for restraint in application of unnecessary techniques may be hubris of the worst kind. However, that's up to them: if you hand an ostenibly informed soul a gun and they put it in their mouth of their own volition and pull the trigger, they deserve to die.

Mario Fusco said...

I really don't see the point against closure in Java and honestly to say that it'll add complexity to the language is an extremely poor justification.

My point of view is: let's add closure to java and developers or teams who don't like them won't just use them. It's simple, isn't it?

The same happened with generics in Java 5. I like them and I feel comfortable with them, but, hey, people who don't like (or more probably don't understand) them can just ignore them and carry on programming in the old java 1.4 style.

In a word my suggestion is: let's add closure to java and let's each programmer free to decide if they like (and understand) them or not.

Joel said...

The big problems with closures will pop up if you team used closures and moved away... And the new team doesn't know how to use closures, or even how to read the cyphered code 'enclosured'... As far as I can see these guys are creating another universe in the java multiverse...

Joel said...

A very nice post, I'm personally concerned about the increasing java complexity, this post gave me more things to think about...