Tuesday, May 19, 2009

Eight Scala or Scala related talks in the coming JavaOne

There will be eight Scala or Scala related talks in the coming ScalaOne!
Oops, JavaOne


//Could be done nicer but (_ + _) wanted to be included
(0 /: (JavaOneTalks map {case ScalaRelated => 1; case _ => 0}))(_ + _)
  1. Actor-Based Concurrency in Scala
  2. The Feel of Scala
  3. Lift: The Best Way to Create Rich Internet Applications with Scala
  4. State: You're Doing It Wrong -- Alternative Concurrency Paradigms on the JVM™ Machine
  5. Performance Comparisons of Dynamic Languages on the Java™ Virtual Machine
  6. Alternative Languages on the JVM™ Machine
  7. Toward a Renaissance VM
  8. Script Bowl 2009: A Scripting Languages Shootout
res0: Int = 8
The weekend after JavaOne there is also the Scala Lift Off on Saturday, 6 June 2009, San Francisco

And not too far ahead there is the Silicon Valley Code Camp (10/3-10/4) with at least three Scala talks (I guess there will be more).

Monday, May 11, 2009

Scala and XML - Part 1

As you know, Scala has a great syntax for XML. Here are some lessons from working with it. Here is a yet another Scala XML sample which creates html.

import scala.xml.XML._
import scala.xml.NodeBuffer
import scala.xml.dtd.{DocType, PublicID}
object ScalaHtml {
val words = List("one", "two", "three")
val url = "http://www.scala-lang.org/"
def main(args : Array[String]) : Unit = {
val page =
<html>
<a href={url}>Scala</a>
<h1>My words</h1>
<ul>{listOfWords(words)}</ul>
</html>;
save("save.html", page)

val doctype = DocType("html",
PublicID("-//W3C//DTD XHTML 1.0 Strict//EN",
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"),
Nil)
saveFull("save.full.html", page, false, doctype)
}
def listOfWords(words: List[String]) = {
val result = new NodeBuffer
for(word <- words) {result &+ (<li>{word}</li>)}
result
}
}

The output is
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<a href="http://www.scala-lang.org/">Scala</a>
<h1>My words</h1>
<ul><li>one</li><li>two</li><li>three</li></ul>
</html>
Where the first line (with the DOCTYPE) will be generated when using the second save option.
Notes:
[1] When using a variable as an attribute value (see the <a> tag) you must not wrap it with double quotes. The underlying code is doing something like this:
scala.xml.MetaData $md = new UnprefixedAttribute("href", url(), $md);
Object _tmp1 = null;
NodeBuffer $buf = new NodeBuffer();
$buf.$amp$plus(new Elem(null, "a", $md, Predef$.MODULE$.$scope(), $buf));

[2] You can't create a partial node. I.e. you can't create the begin tag of a node in one command, then do some other stuff and at the end close it. Scala node creation is atomic and immutable, if you wish to embed on the fly date then it must be simple text or Node* (for example a single Node or NodeBuffer). You can't have any statements int the curly brackets though you may have a call to a method that does that in its own block.
Addendum [3] An emphasis on the string as a value in the curly brackets. Some (like myself) placed there an Int or other primitives expecting something like string concatenation. Well, it won't work and you'll get this error:
error: overloaded method constructor UnprefixedAttribute with alternatives    [scalac](String,Option[Seq[scala.xml.Node]],scala.xml.MetaData)scala.xml.UnprefixedAttribute  (String,String,scala.xml.MetaData)scala.xml.UnprefixedAttribute  (String,Seq[scala.xml.Node],scala.xml.MetaData)scala.xml.UnprefixedAttribute cannot be applied to (java.lang.String,Int,scala.xml.MetaData)
[scalac] Cluster #{summary.id}
[scalac] ^

Saturday, May 09, 2009

Iterating over Map with Scala

Here is a summary of the iteration over a Scala Map discussion, started by Ikai at the Scala mailing list.
Here is the Map we wish to iterate over.

scala> val m1 = Map[Int, Int](1->2, 2->3, 4->50)                
m1: scala.collection.immutable.Map[Int,Int] = Map(1 -> 2, 2 -> 3, 4 -> 50)
The options are:
Accessing the Tuple2 object structure.
scala> m1 foreach ( (t2) => println (t2._1 + "-->" + t2._2))
1-->2
2-->3
4-->50
Using case on the Tuple2
scala> m1 foreach {case (key, value) => println (key + "-->" + value)}
1-->2
2-->3
4-->50
Using a for loop
scala> for ((key, value) <- m1) println (key + "-->" + value) 
1-->2
2-->3
4-->50
Advanced: using unapply to expand the Tuple and use it in the match:
scala> object -> { def unapply[A,B](x : (A,B)) : Option[(A,B)] = Some(x) }
defined module $minus$greater

scala> m1 foreach {case k->v => println (k + "-->" + v)}
1-->2
2-->3
4-->50
I don't know why the next one does not work, any idea?
scala> m1 foreach (println (_._1 + "-->" + _._2))           
:6: error: missing parameter type for expanded function ((x$1, x$2) => x$1._1.$plus("-->").$plus(x$2._2))
m1 foreach (println (_._1 + "-->" + _._2))
^
If you know of other options, please add.

Monday, May 04, 2009

Beware of Scala’s type inference !

Scala's type inference could be unpleasant, creating problems that did not exist in Java (no inference) or dynamically typed languages.

To be clear, Scala's type inference is awesome! Yet with great powers come great responsibility and the library writer, and to some extent the consumer too, should know what to beware of the rough edges.

Examine this library method which returns a Map

class MyLib {
def getMap = Map("a"->"b")
}
And the client code that reads the map
class MyClient {
def useLib {
val lib = new MyLib
val map = lib.getMap
useMap(map)
}
def useMap(map: Map[String, String]) = println("useMap " + map)
}
Looks nice and definitely works. After a while the library is using a mutable HashMap for some imperative reason
import scala.collection.mutable.HashMap
class MyLib {
def getMap = {
var map = new HashMap[String, String]()
map += ("a"->"b")
map += ("c"->"d")
map
}
}
Looks harmless enough, but when we'll run the client we'll get
java.lang.NoSuchMethodError: MyLib.getMap()Lscala/collection/immutable/Map;
MyClient.useLib(MyClient.scala)
What happened? The client used the Map trait but the code compiled with a dependency on the underlying implementation. Here is the library java representation using javap
public class MyLib extends java.lang.Object implements scala.ScalaObject{
public test1.MyLib();
public scala.collection.mutable.HashMap getMap();
public int $tag() throws java.rmi.RemoteException;
}
While if we'll decompiling with JAD gives us
import java.rmi.RemoteException;
import scala.*;
import scala.collection.Map;
public class MyClient implements ScalaObject{
public MyClient(){}
public void useMap(Map map){
Predef$.MODULE$.println((new StringBuilder()).append("useMap ").append(map).toString());
}
public void useLib() {
MyLib lib = new MyLib();
scala.collection.immutable.Map map = lib.getMap();
useMap(map);
}
public int $tag() throws RemoteException{
return scala.ScalaObject.class.$tag(this);
}
}
So although the useMap method use scala.collection.Map the compiler in useLib references scala.collection.immutable.Map. It seems that it it could use scala.collection.Map and by that dodging some of the problem.
If we'll change the library to
import scala.collection.mutable.HashMap
import scala.collection.Map
class MyLib {
def getMap: Map[String, String] = {
var map = new HashMap[String, String]()
map += ("a"->"b")
map += ("c"->"d")
map
}
}
Which provides the interface (using javap)
public class test1.MyLib extends java.lang.Object implements scala.ScalaObject{
public test1.MyLib();
public scala.collection.Map getMap();
public int $tag() throws java.rmi.RemoteException;
}
which gives the library writer flexibility to change the Map implementation without breaking the client's code. Another solution could be a defensive client
import scala.collection.Map
class MyClient {
def useLib {
val lib = new MyLib
val map : Map[String, String] = lib.getMap
useMap(map)
}
def useMap(map: Map[String, String]) = println("useMap " + map)
}
It looks ugly but it would not break the client when the library changes the map implementation.

Coding conventions should mandate explicit return value for external API. Obviously it's also a good practice for readability and self documented code. This issue is especially relevant for large projects that break up to binary dependent modules.

Addendum

In view of the comments I need to add a clarification:
Java will also fail at build time (and runtime). To be more clear, if in Java you change the source one class it does not changes the compilation of other classes (though it could break their compilation). I don't know much about dynamic languages, but I believe the case is similar there (please correct me if I'm wrong). On the other hand, if in Scala you use implicit all around then changing one the source code of a library class does change the compilation output of the consumer class and this is the dangerous part.

In many projects there are binary dependencies between libraries / modules. Actually most projects are depending on some sort of external library, linking to its jar. Compiled versions of the modules are kept in a repository and unless you change their code they are will not be rebuilt. Lets say I wish to upgrade a version of a module v1.0 to v2.0 used by a client v1.0. I will build it with all the others and run some tests, and if all goes well then I assume that the module is backwards compatible and add it to the binary repository. Not the folks that build the product are taking together all the jar files and they break on runtime!

Now, this is not a big deal as itself. The bigger deal is that the module is not really backward compatible and you may not have all the sources handy to rebuild them from scratch as in the unfortunate case of using not open source library, but even if you do have the sources its a pain.

Saturday, May 02, 2009

Scala and Ant JUnit Batchtest

When absorbing Scala into an existing large project at work, you would like to play it nicely with the build and tooling system (and engineers).
Many of the common testing tools out there are based on Ant, JUnit and some test manager, at LinkedIn we use Hudson (great tool by the way). Writing your tests in JUnit and Scala is not a problem, and you might want to check out some of the Scala testing specific libraries which plays nicely with JUnit.
A small problem you may encounter is with the JUnit task batchtest tag since we used it like this:

<attribute name="file-pattern" default="**/Test*.java, **/Test*.scala"/>
<junit fork="..." forkmode="..." dir="...">
...
<batchtest fork="..." todir="...">
...
<fileset dir="@{test-src-dir}" includes="${test.package.path}@{file-pattern}"/>
</batchtest>
</junit>
And it does not work for Scala sources (only for Java) because the tag "generates a test class name for each resource that ends in .java or .class.". Actually it might be just as well since unlike Java, Scala does not force classes names to match the file names they are declared in. The solution was to match against the classes in the build path since both Scala and Java compile into the same build directory into *.class files. Note we added the exclude pattern *$*.class that counts out inner classes.

<attribute name="file-pattern" default="**/Test*.class"/>
<attribute name="excludes" default="**/*$*.class"/>
<junit fork="..." forkmode="..." dir="...">
...
<batchtest fork="..." todir="...">
...
<fileset dir="@{test-build-dir}" includes="${test.package.path}@{file-pattern}" excludes="@{excludes}"/>
</batchtest>
</junit>

Friday, May 01, 2009

Scala & Java interoperability: statics

Scala and Java interoperability is great. In most cases its stemless and its a great way to introduce Scala into existing code base. Actually, it has great benefits as I'm slowly absorbing Scala to one of the modules at LinkedIn. Artifacts can inherit and call each other in the same compilation unit in a transparent way. Well... almost

There is a small surprising factor when you get to statics and it has two parts.
Part 1: Scala <- Java
A Java class that extends another Java class that contains static artifacts can use those statics since they are inherited with the super class. A classic example is JUnit TestCase by the most popular Java test framework. Since I am trying to use the existing JUnit based test framework my Scala test classes are extending TestCase, and like in the Java tests I wish to use the assert* methods in Assert. But here comes the surprise, Scala will not recognize those.
Consider the following example. Here is a simple Java class with two methods, one of them is static:

package test1.java;
public class SuperTest{
protected String superMethod() {return "super";}
public static String superStatic() {return "super static";}
}
And a Scala class extending it
package test1
import test1.java.SuperTest
class Test extends SuperTest{
def useSuper = println(superMethod)
def useSuperStatic = println(superStatic)
}
Which gives us
.../src/test1/Test.scala:7: error: not found: value superStatic
def useSuperStatic = println(superStatic)
Note that scalac didn't have problem with using superMethod, its only superStatic who had the problem. Adding this import would solve the problem
import test1.java.SuperTest._


Part 2: Scala -> Java
There are no statics in Scala's syntax. The closest thing to static is Scala's Object which is a singleton. Actually, Scala compiles the object's artifacts to statics so from Java's point of few a Scala object is a final class (i.e. can't extend) with static members and methods. For example
package test1
object Test{
def scalaStatic = "scala static"
}
gives us
javap bin/test1/Test
Compiled from "Test.scala"
public final class test1.Test extends java.lang.Object{
public static final java.lang.String scalaStatic();
public static final int $tag() throws java.rmi.RemoteException;
}
javap bin/test1/Test$
Compiled from "Test.scala"
public final class test1.Test$ extends java.lang.Object implements scala.ScalaObject{
public static final test1.Test$ MODULE$;
public static {};
public test1.Test$();
public java.lang.String scalaStatic();
public int $tag() throws java.rmi.RemoteException;
}
So we can do the following from Java
package test1.java;
import test1.Test;
public class JavaTest{
public String usingScalaStatic() {return Test.scalaStatic();}
}
It works fine and is a good showcase to how to integrate nicely between the languages. But now assume that someone decides to add a Scala class named Test alongside the Test object
package test1
class Test{
def hi = println("hi")
}

object Test{
def scalaStatic = "scala static"
}
Scala is very happy about it, our Test class has a companion object. But for some reason Java breaks!
src/test1/java/JavaTest.java:5: cannot find symbol
symbol : method scalaStatic()
location: class test1.Test
public String usingScalaStatic() {return Test.scalaStatic();}
But we didn't change the object, only added the class. As we learned at the previous post, the companion class messes things a bit.
javap bin/test1/Test$ bin/test1/Test 
Compiled from "Test.scala"
public final class test1.Test$ extends java.lang.Object implements scala.ScalaObject{
public static final test1.Test$ MODULE$;
public static {};
public test1.Test$();
public java.lang.String scalaStatic();
public int $tag() throws java.rmi.RemoteException;
}

Compiled from "Test.scala"
public class test1.Test extends java.lang.Object implements scala.ScalaObject{
public test1.Test();
public void hi();
public int $tag() throws java.rmi.RemoteException;
}
so now we no longer have a static scalaStatic method. Scala code does not care about it, but it does matter when we want to integrate with Java. A solution could be something like
package test1.java;
import test1.Test$;
public class JavaTest{
public String usingScalaStatic() {return (new Test$()).scalaStatic();}
}
But its ugly and even worse, we create another instance of the object Test which is supposed to be a singleton! I wonder why the constructor is not private. Another, maybe better but not less ugly is
public class JavaTest{
public String usingScalaStatic() {return Test$.MODULE$.scalaStatic();}
}
Which works as well since MODULE$ is the member that keeps the static reference to the singleton. Needless to say, this is a nasty side effect. Hopefully it will fix at Scala 2.8 with the main bug.

Scala's main issue

From Chris Gioran: For those arriving here from googling up this issue, I would like to point out that it has been fixed @2.8.0
Now main methods defined in companion objects are generated static and will work as expected. The ticket has also been closed as fixed.

I love Scala, I really do and I think its the next big thing. My main issue with the language is its unclear and sometimes misleading compiler errors.

My working practice in learning new technology, and I think its a common case, is to read a bit about it and then try it out. If something goes wrong then google about it using the error message, read a bit more, and master it as I go. It means error messages are actually as important as (and maybe more then) the documentation, especially for beginners.

Since Scala is a relatively young and very rich language, there are not enough writing on the web about its rough spots. In that light it is even more important to provide clear documentation about these spots. To be sure, Scala has its share of rough spots and I don't mean by that that other languages has less of them. To paraphrase on Robert Glass "Scala is a very bad language, but all the others are so much worse".

Now to the point. I was trying to write this very simple Scala app with a main method:
package test1
import test1.java.SuperTest

class Test extends SuperTest{
def useSuper {
print(superMethod())
}
}

object Test{
def main(args: Array[String]): Unit = {
val t = new Test
print(t.useSuper)
}
}
And tried to run the main method on it but got
java.lang.NoSuchMethodException: test1.Test.main([Ljava.lang.String;)
. Ok, I said, I know that the companion object is compiled to Test$ and the main method
is defined in the object, not in the class. So I tried to run Test$ but then I got
main() method in test1.Test$ is not declared static
. Checked the class and found that its true:
$ javap build/test1/Test$
Compiled from "Test.scala"
public final class test1.Test$ extends java.lang.Object implements scala.ScalaObject{
public static final test1.Test$ MODULE$;
public static {};
public test1.Test$();
public void main(java.lang.String[]);
public int $tag()       throws java.rmi.RemoteException;
}
Wow! what's going on here? All the code example I found claim that that's the way to go. After banging my head a bit I removed the class definition and left the object to be a plain object (not a companion object anymore), and it worked! It means that there is an undocumented side effect to having a companion (as all of us married folks know). After figuring out the solution I googled a bit more and found that its a bug and it might be solved in Scala 2.8.0. But since its open for more then 16 months now, and the compiler knows its an edge case I think this kind of issue should have a special compiler warning.

Creative Commons License This work by Eishay Smith is licensed under a Creative Commons Attribution 3.0 Unported License.