noodling towards a functional brain

Thursday, November 29, 2012

F-Bounded Type Polymorphism Considered Tricky

This post is intended to address a question that I seem to see come up every few months on the Scala mailing
list or the #scala IRC channel on freenode. The question is often posed in terms of the phrase
"recursive type" and usually involves someone having some difficulty or other (and many arise) with
a construction like this:

trait Account[T <: Account[T]] {
  def addFunds(amount: BigDecimal): T
}

class BrokerageAccount(total: BigDecimal) extends Account[BrokerageAccount] {
  def addFunds(amount: BigDecimal) = new BrokerageAccount(total + amount)
}

class SavingsAccount(total: BigDecimal) extends Account[SavingsAccount] {
  def addFunds(amount: BigDecimal) = new SavingsAccount(total + amount)
}

This sort of self-referential type constraint is known formally as F-bounded type polymorphism and is usually attempted when someone is
trying to solve a common problem of abstraction in object-oriented languages; how to define a polymorphic
function that, though defined in terms of a supertype, will when passed a value of some subtype will always
return a value of the same subtype as its argument.

It's an interesting construct, but there are some subtleties that can trip up the unwary user, and which make
naive or incautious use of the pattern problematic. The first issue that must be dealt with is how to
properly refer to the abstract supertype, rather than some specific subtype. Let's begin with a simple
example; let's say that we need a function that, given an account, will apply a transaction fee for adding
funds below a certain threshold.

object Account {
  val feePercentage = BigDecimal("0.02")
  val feeThreshold = BigDecimal("10000.00")

  def deposit[T <: Account[T]](amount: BigDecimal, account: T): T = {
    if (amount < feeThreshold) account.addFunds(amount - (amount * feePercentage))
    else account.addFunds(amount)
  }
}

This is straightforward; the type bound is enforced via polymorphism at the call site. You'll notice that the
type ascribed to the "account" argument is T, and not Account[T] - the bound on T gives us all the constraints
that we want. This does what we want it to do when we're talking about working with one account at a time.
But, what if we want instead to perform some action with a collection of accounts of varying types; suppose
we need a method to debit all of a customer's accounts for a maintenance fee? We can expect our type bounds
to hold, but things begin to get a little complicated; we're forced to use bounded existential types.
Here is the correct way to do so:

object Account {
  def debitAll(amount: BigDecimal, accounts: List[T forSome { type T <: Account[T] }]): List[T forSome { type T <: Account[T] }] = {
    accounts map { _.addFunds(-amount) }
  }
}

The important thing to notice here is that the type of individual members of the list are existentially
bounded, rather than the list being existentially bounded as a whole. This is important, because it means
that the type of elements may vary, rather than something like "List[T] forSome { type T <: Account[T] }"
which states that the values of the list are of some consistent subtype of T.

So, this is a bit of an issue, but not a terrible one. The existential types clutter up our codebase and
sometimes give the type inferencer headaches, but it's not intractable. The ability to state these existential
type bounds does, however, showcase one advantage that Scala's existentials have over Java's wildcard types,
which cannot express this same construct accurately.

The most subtle point about F-bounded types that is important to grasp is that the type bound is *not*
as tight as one would ideally want it to be; instead of stating that a subtype must be eventually
parameterized by itself, it simply states that a subtype must be parameterized by some (potentially
other) subtype
. Here's an example.

class MalignantAccount extends Account[SavingsAccount] {
  def addFunds(amount: BigDecimal) = new SavingsAccount(-amount)
}

This will compile without error, and presents a bit of a pitfall. Fortunately, the type bounds that we
were required to declare at the use sites will prevent many of the failure scenarios that we might be
concerned about:

object Test {
  def main(argv: Array[String]): Unit = {
    Account.deposit(BigDecimal("10.00"), new MalignantAccount)
  }
}

nuttycom@crash: ~/tmp/scala/f-bounds $ scalac Test.scala
Test.scala:27: error: inferred type arguments [MalignantAccount] do not conform to method deposit's type parameter bounds [T <: Account[T]]
    deposit(BigDecimal("10.00"), new MalignantAccount)
    ^
one error found

but it's a little disconcerting to realize that this level of strictness is *only* available at the use site,
and cannot be readily enforced (except perhaps by implicit type witness trickery, which I've not yet tried)
in the declaration of the supertype, which is what we were really trying to do in the
first place.

Finally, it's important to note that F-bounded type polymorphism in Scala falls on its face when you start
talking about higher-kinded types. Suppose that one desired to state a supertype for Scala's structurally
monadic types (those which can be used in for-comprehensions):

trait Monadic[M[+_] <: ({ type λ[+α] = Monadic[M, α] })#λ, +A] {
  def map[B](f: A => B): M[B]
  def flatMap[B](f: A => M[B]): M[B]
}

This fails to compile outright, complaining about cyclic references in the type constructor M.

In conclusion, my experience has been that F-bounded type polymorphism is tricky to get right and causes typing clutter in the codebase. That isn't to say that it's without value, but I think it's best to consider very carefully whether it is actually necessary to your particular application before you step into the rabbit hole. Most of the time, there's no wonderland at the bottom.

EDIT:

Heiko asks below why not use the following definition of debitAll:

object Account {                                                                                                   
  def debitAll2[T <: Account[T]](amount: BigDecimal, accounts: List[T]): List[T] = {                            
    accounts map { _.addFunds(-amount) }                                                                           
  } 
}

The problem here is that this has a subtly different meaning; this says that for debitAll2, all the members
of the list must be of the *same* subtype of Account. This becomes apparent when we actually try to use the
method with a list where the subtype varies. In both constructions, actually, you end up having to explicitly
ascribe the type of the list, but I've not been able to find a variant for the debitAll2 where the use site
will actually compile with such a variant-membered list.

object Test {
  def main(argv: Array[String]): Unit = {                                                                          
    // compiles, though requires type ascription of the list; this is where the inferencer breaks down. 
    Account.debitAll(BigDecimal("10.00"), List[T forSome { type T <: Account[T] }](new SavingsAccount(BigDecimal("0")), new BrokerageAccount(BigDecimal("0"))))

    // doesn't compile                                                                                             
    // Account.debitAll2(BigDecimal("10.00"), new SavingsAccount(BigDecimal("0")) :: new BrokerageAccount(BigDecimal("0")) :: Nil)
    
    // doesn't compile
    // Account.debitAll2(BigDecimal("10.00"), List[T forSome { type T <: Account[T] }](new SavingsAccount(BigDecimal("0")), new BrokerageAccount(BigDecimal("0"))))
  }                                                                                                                
}

Saturday, February 18, 2012

Correcting the Visitor pattern.

I'm writing this post because I want to address a problem that I see time and time again: people who are trying to figure out how to encode algebraic data types in languages that do not support them come up with all kinds of crazy solutions. But, there is a simple and effective encoding of algebraic data types that everyone knows about, but had just been doing wrong: the Visitor pattern. At this point, I expect many devotees of functional programming to start screaming "but... mutability!!!" And, yes, the mutability required by the textbook definition of the Visitor pattern is indeed a major problem - the problem that I intend to show how to correct right here. The solution is remarkably simple. Here's a tree traversal using the Visitor pattern, implemented *correctly* in Java:
public interface Tree<A> {
  public <B> B accept(TreeVisitor<A, B> v);
}

public class Empty<A> implements Tree<A> {
  public <B> B accept(TreeVisitor<A, B> v) {
    return v.visitEmpty();
  }
}

public class Leaf<A> implements Tree<A> {
  public final A value;

  public Leaf(A value) {
    this.value = value;
  }

  public <B> B accept(TreeVisitor<A, B> v) {
    return v.visitLeaf(this);
  }
}

public class Node<A> implements Tree<A> {
  public final Tree<A> left;
  public final Tree<A> right;

  public Node(Tree<A> left, Tree<A> right) {
    this.left = left;
    this.right = right;
  }

  public <B> B accept(TreeVisitor<A, B> v) {
    return v.visitNode(this);
  }
}

public interface TreeVisitor<A, B> {
  public B visitEmpty();
  public B visitLeaf(Leaf<A> t);
  public B visitNode(Node<A> t);
}
This is exactly the traditional Visitor pattern, with one minor variation: the 'accept' method is generic (parametrically polymorphic), returning whatever return type is defined by a particular TreeVisitor instance instead of void. This is a vitally important distinction; by making accept polymorphic and non-void returning, it allows you to escape the curse of being forced to rely on mutability to accumulate a result. Here's an example of the implementation of the 'depth' method from the previously mentioned blog post, and an example of its use. You'll note that no mutable variables were harmed (or indeed used) in the creation of this example:
public class TreeUtil {
  public static final <A> TreeVisitor<A, Integer> depth() {
    return new TreeVisitor<A, Integer>() {
      public Integer visitEmpty() {
        return 0;
      }

      public Integer visitLeaf(Leaf<A> t) {
        return 1;
      }

      public Integer visitNode(Node<A> t) {
        int leftDepth = t.left.accept(this);
        int rightDepth = t.right.accept(this);
        return (leftDepth > rightDepth) ? leftDepth + 1 : rightDepth + 1; 
      }
    };
  }
}

public class Example {
  public static void main(String[] argv) {
    Tree<String> leftBiased = new Node<String>(
      new Node<String>(
        new Node<String>(
          new Leaf<String>("hello"),
          new Empty<String>()
        ),
        new Leaf<String>("world")
      ),
      new Empty<String>()
    );

    assert(leftBiased.accept(TreeUtil.<String>depth()) == 3);
  }
}
TreeVisitor encodes the f-algebra for the Tree data type; the accept method is the catamorphism for Tree. Moreover, given this definition, you can also see that the visitor forms a monad (example in Scala), giving rise to lots of nice compositional properties. Implemented in this fashion, Visitor is actually nothing more (and nothing less) than a multiple dispatch function over the algebraic data type in question. So stop returning void from your visitors, and ramp up their power in the process!

Wednesday, August 17, 2011

The magical Specs/ScalaCheck defaultPrettyParams configuration implicit

When using Specs with ScalaCheck, I have occasionally found myself annoyed that, when a function under test threw an exception, ScalaCheck didn't show me enough of a stack trace to actually figure out what was going on. There is a means to configure this, but to the best of my knowledge it's essentially not documented anywhere. In any case, to get full stack traces in your spec, all that you need is this:
import org.specs._
import org.scalacheck._

class MySpec extends Specification with ScalaCheck {
  override val defaultPrettyParams = Pretty.Params(2)

  "your scalachecky specs here" should {
     //...
  }
}
The magical '2' in Pretty.Params(2) tells Scalacheck to spit out the full stack trace instead of just the error message from the exception.

Sunday, February 27, 2011

Code Retreat Boulder: Conway's Life

I had a ton of fun yesterday at Code Retreat Boulder, even though I ended up as the Scala guy and thus didn't get to play in other languages as much as I would have liked. Anyway, this morning I decided to recreate my favorite of the solutions. Small, simple, pure, and relatively flexible:
import scala.annotation.tailrec

trait Life {
  type Entity
  type Population = Set[Entity]

  def generation(pop: Population) = pop.flatMap(neighborhood).filter {
    entity => willLive(pop.contains(entity), neighbors(entity).count(pop.contains))
  }

  def neighbors(entity: Entity): Seq[Entity]
  def neighborhood(entity: Entity) = neighbors(entity) :+ entity

  def willLive(now: Boolean, n: Int): Boolean
  def render(pop: Population): Unit = println(pop)

  @tailrec final def run(pop: Population): Unit = { 
    render(pop)
    if (!pop.isEmpty) run(generation(pop))
  }
}

class Life2D extends Life {
  type Entity = (Int, Int)

  override def willLive(now: Boolean, n: Int) = ((now && n == 2) || n == 3)

  override def neighbors(entity: Entity) = entity match {
    case (x, y) => for (i <- x-1 to x+1; j <- y-1 to y+1 if !(i == x && j == y)) yield (i, j)
  }
}

object Blinker extends Life2D {
  def main(argv: Array[String]) = run(Set((1, 0), (1, 1), (1, 2)))
}

Thursday, July 29, 2010

A monad for failure-tolerant computations.

I was working on a problem today where some of the computations could fail, but degrade gracefully while providing information about how exactly they failed so that clients could choose whether or not to use the result. This is what I came up with:
/**
 * A monad used to entrain computations on a value that might fail.
 * This is distinct from Either in that it allows computations to continue
 * in the presence of errors, with the possibility of a degraded result.
 * The user of the result can then decide its suitability for use based upon
 * any errors that might be returned. Errors must form a semigroup.
 */
sealed trait Attempt[E, +A] {
  def value: A
  def map[B](f: A => B): Attempt[E, B]
  def flatMap[B](f: A => Attempt[E, B]): Attempt[E, B]

  def error: Option[E]
  def either: Either[E, A]
}

case class Success[E, +A](value: A) extends Attempt[E, A] {
  def map[B](f: A => B): Attempt[E, B] = Success(f(value))
  def flatMap[B](f: A => Attempt[E, B]): Attempt[E, B] = f(value)
  def error = None
  def either = Right(value)
}

case class Failure[E, +A](err: E, value: A)(implicit append: (E, E) => E) extends Attempt[E, A] {
  def map[B](f: A => B): Attempt[E, B] = Failure(err, f(value))
  def flatMap[B](f: A => Attempt[E, B]): Attempt[E, B] = f(value) match {
    case Failure(e, b) => Failure(append(e, err), b)
    case Success(b)   => Failure(err, b)
  }

  def error = Some(err)
  def either = Left(err)
}
Pretty trivial, but maybe occasionally useful. Here's how it looks in my code:
  def mergeComponents(f: String => JSONReport): Attempt[List[String], JSONReport] = {
    components.map(f).map(_.mergeComponents(f)).foldLeft[Attempt[List[String], JSONReport]](Success(this)) {
      (result, comp) => for {
        cur <- result
        jr  <- comp
        properties     <- mergeProperties(cur, jr)
        queries        <- mergeQueries(cur, jr)
        dataAliases    <- mergeAliases(cur, jr, queries.values)
        dataTransforms <- mergeTransforms(cur, jr, dataAliases.values)
        dataMailers    <- mergeMailers(cur, jr, dataTransforms.values)
      } yield {
        JSONReport(
          cur.reportId, cur.active, cur.version, properties,
          queries, dataAliases, dataTransforms, dataMailers,
          cur.dataRange, Nil
        )
      }
    }

Here each of the various mergeX methods return an Attempt[List[String], X] where X is something needed to build the ultimate report. Here I'm just aggregating lists of strings describing the errors, but of course any type for which (E, E) => E is defined would work.

Attempt. For all those times where you want a monad that says, "Hey, I maybe couldn't get exactly what you asked for. Maybe it's little broken, maybe it won't work right, but this the best I could do. You decide."

EDIT:

After a bit of thinking, I realized that this monad is really more general than being simply related to success or failure - it simply models a function that may or may not produce some additional metadata about its result. Then a lightbulb went off, and quick google search confirmed... yup, I just reinvented the writer monad. It's not *exactly* like Writer, because it just requires a semigroup for E instead of a monoid, and the presence of a "log" is optional, so maybe it's better suited than Writer for a few instances.

The one really nice thing about rediscovering well-known concepts is that doing the derivation for yourself, in the context of some real problem, gives you a much more complete understanding of where the thing you reintevented is applicable!

Inspired by michid's example in the comments below, here's a simpler example that demonstrates some utility his doesn't quite capture.

implicit val append = (l1: List[String], l2: List[String]) => l1 ::: l2

def succ(s: String) = Success[List[String], String]("result: success")
val fail: String => Attempt[List[String], String] = { 
  var count = 0 
  (s: String) => {
    count += 1
    Failure(List("failure " + count), s + " then failure")
  }
}

val r = for {
  x <- succ("here we go!")
  y <- fail(x)
  z <- fail(y)
} yield z

println(r)
This results in:
Failure(List(failure 2, failure 1),result: success then failure then failure)

Thursday, July 15, 2010

An unfortunate situation.

I've been working a bit with Hadoop lately. As many will know, there are essentially two versions of Hadoop that coexist within the Hadoop API; a deprecated legacy API and a newer one that is not nearly so feature-complete. It's a source of confusion to say the least.

Earlier today, I asked in the #hadoop channel on freenode, why were the breaking changes not simply made on a separate branch, the major version number incremented, and so forth such that everyone could move forward? The answers that I got back illuminated a troublesome trend that I think is worth considering for those projects (like my beloved Scala) that are moving towards professional support models of funding.

During the discussion, I commented that there is an unfortunate reality that the largest companies, who may provide the greatest support any given open-source project are also those who will likely be most conservative with respect to change. So as soon as open-source projects grow large enough that companies selling commercial support for those projects survive, there is an immediate disincentive for the support companies to make breaking changes. And stagnation ensues.

The response was startling. People who had been involved with numerous prominent businesses and projects immediately replied with stories of their own projects suffering such a fate, sometimes to the point that the project died entirely.

I think that this "success effect" can be traced to a single, simple cause, with a single, simple solution. Companies (and the individuals that make them up) need to stop thinking of mission-critical software as something that is ever complete. Marketing is never complete; sales is never done with their job, purchasing and HR are never done, nor is accounting. Software is no different - as engineers, we do not (and should not purport to) produce solutions to the problems that companies have; instead, we automate and support the operation of businesses so that businesses can do more, serve more customers, and in the end create more wealth. A project is finished when nobody is using it; never before then.

Friday, February 05, 2010

Trick of the day

James Iry just mentioned something I'd never noticed before in #scala:
scala> def f(i: Int, x: Int) = i match { case `x` => println("x!"); case _ => println("nope") }
f: (i: Int,x: Int)Unit

scala> f(1, 1)
x!

scala> f(1, 2)
nope
That's one of those things that I'd idly wondered how to do, but hadn't had a use case so hadn't really gone looking. Good to know it's possible and simple... and more or less makes sense. More or less. Actually, I guess I'm kind of glad that the following doesn't compile:
scala> def f(i: Int) = i match { case `yield` => println(`yield`) }                           
<console>:4: error: not found: value yield
       def f(i: Int) = i match { case `yield` => println(`yield`) }
I can live with the restriction as not being able to use a reserved word for a variable on the lhs of a pattern match, though it does make for a weird inconsistency:
scala> val (`yield`, x) = (1, 1)
<console>:4: error: not found: value yield
       val (`yield`, x) = (1, 1)
            ^

scala> val `yield` = 1
yield: Int = 1
Chalk up one more to the box of "pain points caused by using pattern matching for parallel assignment."

Wednesday, February 03, 2010

On Extraction

Scala case classes are great; they give you a tremendous leg up when it comes to implementing pattern matching. However, there's a drawback: the unapply method of case class companion objects take arguments of the type of the case class, and return Option[TupleN[...]] where N is the number of variables to the case class constructor. What's more, it's not possible to overload or override the unapply method to deconstruct other types of by creating your own companion object on the side, as shown in this REPL session:
scala> case class  Foo(val i: Int, val s: String)
defined class Foo

scala> object Foo {
     |   def unapply(s: String): Option[Foo] = Some(new Foo(s.substring(0, 2).toInt, s.substring(2)))
     | }
defined module Foo

scala> "12abcd" match {
     |   case Foo(i, s) => println(s, i)
     | }
<console>:10: error: wrong number of arguments for object Foo
         case Foo(i, s) => println(s, i)
              ^
<console>:10: error: wrong number of arguments for method println: (Any)Unit
         case Foo(i, s) => println(s, i)
Sadly, it doesn't even seem to be possible to move this extractor to another class and get the desired result:
scala> object SFoo {                                                                                 
     |   def unapply(s: String): Option[Foo] = Some(new Foo(s.substring(0, 2).toInt, s.substring(2)))
     | }
defined module SFoo

scala> "12abcd" match {                 
     |   case SFoo(i, s) => println(s, i)
     | }
<console>:9: error: wrong number of arguments for object SFoo
         case SFoo(i, s) => println(s, i)
              ^
<console>:9: error: wrong number of arguments for method println: (Any)Unit
         case SFoo(i, s) => println(s, i)

Fortunately, there is a way around this issue: simply define your own class that extends ProductN! Then, you can define as many extractors as you want in different pseudo-companion objects and use them idiomatically:
scala> class Foo(val i: Int, val s: String) extends Product2[Int, String] {
     |   val (_1, _2) = (i, s)
     | }
defined class Foo

scala> object Foo {
     |   def unapply(f: Foo): Option[(Int, String)] = Some((f.i, f.s))
     | }
defined module Foo

scala> object SFoo {
     |   def unapply(s: String): Option[Foo] = Some(new Foo(s.substring(0, 2).toInt, s.substring(2)))
     | }
defined module SFoo

scala> "12abcd" match {
     |   case SFoo(i, s) => println(s, i)
     | }
(abcd,12)

scala> new Foo(1, "hi") match {
     |   case Foo(i, s) => println(s, i)
     | }
(hi,1) 

Also, naturally, the trivial extractor (Foo => Foo) can supplant the Foo => Tuple2 extractor above:
scala> object Foo {
     |   def unapply(f: Foo): Option[Foo] = Some(f)
     | }

scala> new Foo(1, "hi") match {                  
     |   case Foo(i, s) => println(s, i)         
     | }                                
(hi,1)

Tuesday, December 08, 2009

A monadic Visitor

This Reader-esque monad emerged from my code this morning:
trait A {
  def accept[T](v: V[T]): T
}

class B(val s: String) extends A {
  var i = 0
  override def accept[T](v: V[T]): T = v.visit(this)
}

class C(val s: String) extends A {
  var i = 0
  override def accept[T](v: V[T]): T = v.visit(this)
}

class D(val s: String) extends A {
  var i = 0
  override def accept[T](v: V[T]): T = v.visit(this)
}

trait V[T] {
  outer =>
  def visit(b: B): T
  def visit(c: C): T
  def visit(d: D): T

  def map[U](f: T => U): V[U] = new V[U] {
    def visit(b: B): U = f(outer.visit(b))
    def visit(c: C): U = f(outer.visit(c))
    def visit(d: D): U = f(outer.visit(d))
  }

  def flatMap[U](f: T => V[U]): V[U]= new V[U] {
    def visit(b: B): U = f(outer.visit(b)).visit(b)
    def visit(c: C): U = f(outer.visit(c)).visit(c)
    def visit(d: D): U = f(outer.visit(d)).visit(d)
  }
}

object V {
  def pure[T](t: T): V[T] = new V[T] {
    def visit(b: B) = t
    def visit(c: C) = t
    def visit(d: D) = t
  }
}
And the test:
object Test {
  def main(argv: Array[String]) {
    class VI(i: Int) extends V[Int] {
      def visit(b: B) = error("not supported")

      def visit(c: C) = {
        println("vi: " + c.s + " = " + c.i)
        c.i = c.i + i
        c.i
      }

      def visit(d: D) = {
        println("vi: " + d.s + " = " + d.i)
        d.i = d.i + i
        d.i
      }
    }

    def f(i: Int): V[Int] = {
      println("f: " + i)
      new VI(i)
    }

    def g(i: Int): V[Int] = {
      println("g: " + i)
      new VI(i)
    }

    val c1: A = new C("c1")
    val c2: A = new C("c2")
    val d1: A = new D("d1")
    val v = new VI(1)
    println(c1.accept(v.flatMap(f).flatMap(g)))
    println(c2.accept(v.flatMap(x => f(x).flatMap(g))))
    println(d1.accept(v.flatMap(f).flatMap(g).flatMap(f).flatMap(g)))
  }
}
As far as I can tell, this satisfies the monad laws. Am I missing anything? If not, this is the first monad I've "found in the wild!"

Wednesday, December 02, 2009

So much from so little

My solution to Tony's catamorphism challenge:
trait MyOption[+A] {
  import MyOption._

  def cata[X](some: A => X, none: => X): X
  def map[B](f: A => B): MyOption[B] = cata(a => some(f(a)), none[B])
  def flatMap[B](f: A => MyOption[B]): MyOption[B] = cata(f, none[B])
  def getOrElse[AA >: A](e: => AA): AA = cata(a => a, e)
  def filter(p: A => Boolean): MyOption[A] = if (cata(p, false)) cata(a => some(a), none[A]) else none[A]
  def foreach(f: A => Unit): Unit = cata(f, ())
  def isDefined: Boolean = cata(_ => true, false)
  def isEmpty: Boolean = cata(_ => false, true)
  def get: A = cata(a => a, error("get on none"))
  def orElse[AA >: A](o: MyOption[AA]): MyOption[AA] = cata(a => some(a), o)
  def toLeft[X](right: => X): Either[A, X] = cata(a => Left(a), Right(right))
  def toRight[X](left: => X): Either[X, A] = cata(a => Right(a), Left(left))
  def toList: List[A] = cata(a => List(a), Nil)
  def iterator: Iterator[A] = toList.elements
}

object MyOption {
  def none[A] = new MyOption[A] {
    def cata[X](s: A => X, n: => X) = n
  }

  def some[A](a: A) = new MyOption[A] {
    def cata[X](s: A => X, n: => X) = s(a)
  }
}

Saturday, November 07, 2009

Composable Bindings in Lift, part II

In my previous post I presented the DataBinding[T] type (T => NodeSeq => NodeSeq) and illustrated how classes inhabiting this type could be used along with an implicit binder function to create modular, reusable rendering logic for objects. In this post, I will discuss how to assemble these components and strategies for handling rendering over inheritance hierarchies.

I left off last time with this DataBinding trait:

trait OrderBinding extends DataBinding[Order] {
  implicit val userBinding: DataBinding[User]
  implicit val addressBinding: DataBinding[Address]
  implicit val lineBinding: DataBinding[OrderLine]
  implicit val txnBinding: DataBinding[Transaction]

  def apply(order: Order): Binding = (xhtml: NodeSeq) => {
    val itemTemplate = chooseTemplate("order", "lineItem", xhtml)
    val txnTemplate = chooseTemplate("order", "transaction", xhtml)

    bind("order", xhtml,
      "user" -> order.user.bind("templates-hidden/user/order-display".path),
      "shipTo" -> order.shipTo.bind("templates-hidden/address/order-display".path),
      "lineItems" -> order.items.flatMap(_.bind(itemTemplate)).toSeq,
      "transactions" -> order.transactions.flatMap(_.bind(txnTemplate)).toSeq
    )
  }
}

This DataBinding[Order] is a trait, not an object like the previous binding instances. In addition, the other DataBinding instances that it delegates to are abstract implicit vals - meaning that I will need to create a concrete implementation of this trait to actually use it. The interesting thing here is that these vals are declared implicit within the scope of the trait, but do not need to be declared implicit when the trait is extended and made concrete. I can then compose what I want my binding to look like at the use site (say an ordinary snippet) like this:

import Bindings._

class MySnippet {
  private implicit object ShowOrderBinding extends OrderBinding { 
    val userBinding = UserBinding
    val addressBinding = MailingAddressBinding
    val lineBinding = OrderLineBinding
    val txnBinding = TransactionBinding //we'll implement this in a minute
  }

  def showOrder(xhtml: NodeSeq): NodeSeq = {
    val order: Order = //get an order from somewhere
    order.bind(xhtml)
  }
}

The ShowOrderBinding selects the bindings I want for the components of the order in a declarative fashion, and since it is an implicit object in scope, it is selected as the DataBinding instance to render the order when order.bind is called. If I decide that I want to use the AdminAddressBinding from the previous post instead, I simply substitute it for MailingAddressBinding at the use site and I'm done!

There's one more thing that needs to be done in the above code - to implement TransactionBinding. This is a little tricky because there are several subclasses of Transaction that may be in the list of transactions for an order, so I'm going to need to recover type information about those instances at runtime in order to be able to choose the correct binding for a given Transaction. Furthermore, there's some information in the base Transaction type that I don't want to have to repeat bindings for in each subtype binding, so I'm going to use function composition to "add together" two different bindings.

There are two different approaches I'm going to show for how one might go about doing this: first, I'll try using a straightforward Scala match statement to figure out the object's type. That looks like this:

trait TransactionBinding extends DataBinding[Transaction] {
  implicit val authBinding: DataBinding[AuthTransaction]
  implicit val captureBinding: DataBinding[CaptureTransaction]
  implicit val shipBinig: DataBinding[ShipTransaction]

  def apply(txn: Transaction): Binding = {
    val base: Binding = bind("txn", _,
      "scheduleDate" -> Text(txn.scheduledAt.toString),
      "completionDate" -> Text(txn.completedAt.getOrElse("(incomplete)").toString),
      "state" -> Text(
        txn.state match {
          case COMPLETE => "complete"
          case _ => "weird"
        }
      )
    )

    base compose (
      txn match {
        case auth: AuthTransaction => auth.binding
        case capture: CaptureTransaction => capture.binding
        case ship: ShipTransaction => ship.binding
        case _ => error("Unrecognized transaction type!")
      }
    )
  }
}

Here, I've created a base binding that defines some simple behavior for how to bind elements that are common to all transactions. I then compose that generated binding with a binding for the appropriate subclass. The determination of the subclass binding is of necessity a bit boilerplate-laden, but that's what happens when one throws away compile-time type information. The alternative, of course, would be to store each type of transaction in a separate collection associated with the order, but that's not the way I've gone for now.

There is, however, a hidden bug that may lurk in the pattern match above, depending upon your environment. Let's supposed for a minute that our model classes are persisted by Hibernate - and that the list of Transactions is annotated such that it is lazily, instead of eagerly, fetched from the database. In this case, the transactions returned may be translucently wrapped in runtime proxies. I say translucently instead of transparently because there is one critical respect in which a proxy does not behave in the same manner as the class being proxied - how it responds to the instanceof check that underlies the Scala pattern match. In this case, we will need to use a Visitor to correctly get the type for which we want to delegate binding. To do this, we'll modify our transaction classes as so:

trait TransactionFunc[T] {
  def apply(auth: AuthTransaction): T
  def apply(capture: CaptureTransaction): T
  def apply(ship: ShipTransaction): T
}

abstract class Transaction(val scheduledAt: DateTime, val completedAt: Option[DateTime], state: State) {
  def accept[T](f: TransactionFunc[T]): T
}

case class AuthTransaction(cc: CreditCard, amount: BigDecimal, scheduledAt: DateTime, completedAt: Option[DateTime], state: State) extends Transaction(scheduledAt, completedAt) {
  def accept[T](f: TransactionFunc[T]): T = f.apply(this)
}

case class CaptureTransaction(auth: AuthTransaction, ...) extends Transaction(...)  {
  def accept[T](f: TransactionFunc[T]): T = f.apply(this)
}

case class ShipTransaction(items: List[OrderLine], ...) extends Transaction(...)  {
  def accept[T](f: TransactionFunc[T]): T = f.apply(this)
}

Then, the implementation of our transaction binding becomes this:

trait TransactionBinding extends DataBinding[Transaction] {
  implicit val authBinding: DataBinding[AuthTransaction]
  implicit val captureBinding: DataBinding[CaptureTransaction]
  implicit val shipBinig: DataBinding[ShipTransaction]

  def apply(txn: Transaction): Binding = {
    val base: Binding = //same as before...

    base compose txn.accept(
      new TransactionFunc[Binding] {
        def apply(auth: AuthTransaction) = auth.binding
        def apply(capture: CaptureTransaction) = capture.binding
        def apply(ship: ShipTransaction) = ship.binding
      }
    )
  }
}

Since the correct overloading of TransactionFunc.apply will be statically determined, there is no possibility of a proxy getting in the way of our binding.

So, now we have a set of tools for binding composition. Here are some additional ideas of where this might be able to go, given the appropriate changes to Lift.

In my example of the showOrder snippet above, the implementation is trivial... so trivial, in fact, that it almost doesn't need to be there. If the notion of a binding was fully integrated into Lift, we could make Loc somewhat more powerful. Loc is one of the primary classes in Lift used to generate the SiteMap, which declares what paths are accessible within a Lift application. A path in a Loc typically corresponds to the path to an XML template that will specify the snippets and markup for a request. An important thing to note though is that Loc is not simply Loc - it is in fact Loc[T], where by default T is NullLocParams, a meaningless object essentially equivalent to Unit.

When one looks at making T something other than Unit, Loc starts to become significantly more interesting - first, it declares a rewrite method that can be overriden to populate the Loc with an instance of T as part of its processing, and secondly, Loc can specify explicitly the template to be used to render the Loc, instead of using the one that corresponds to the path. Wait... we have an instance, and we have a template... what if we could specify a DataBinding[T] as well? In this case, we might not need for there to be a snippet at all!

Tuesday, September 29, 2009

Composable bindings in Lift

I've been using Lift as the framework behind a small internal web application at my company for the past year or so. The experience has been generally positive, despite a few quibbles I have with how state is typically maintained and passed around within an application.

Lift takes a "view first" approach to building web applications, as opposed to the commonly favored MVC pattern. In practice, "view first" boils down to essentially this: when you create an XHTML template for a page, that template will contain one or more XML tags where the name of the tag refers to a method to be called with the body of the tag as an argument. Such a method will return the XHTML to be substituted in place of the original tag and its contents. In Lift parlance such a method is called a "snippet," and has the type NodeSeq => NodeSeq. In order to more fully understand the rest of this article, I recommend you read more about snippets here if you're not already familiar with them.

The interesting thing about this signature is that functions with this type can easily be composed. What's more, the method most frequently used to implement snippets, BindHelpers.bind, can be partially applied to its arguments to give a closure with the same signature.

When I'm designing a website, most of the time rendering a page (or even a snippet within a page) isn't about displaying the data from a single object - instead, I often want to render a whole object graph. Ideally, I want to be able to choose a rendering for each type of object I'm concerned with, then compose these elements to produce the final page. Templates for objects should be reusable and modular, and for any given page I should be able to pick between multiple renderings of any given object, in some places displaying a concise summary and elsewhere a detailed exposition of all the available data.

Furthermore, while I believe that "the meaning should belong with the bytes" I also think that the rendering should be as decoupled from the bytes as possible to allow for maximum flexibility. Hence, I don't want my model classes to have any sort of "render" method - the display is something that should be layered on above the semantics provided by the names and types of the members of the object.

With these goals and composition in mind, I started with the following tiny bit of code:


import net.liftweb.http.TemplateFinder.findAnyTemplate
import net.liftweb.util.{Box,Full,Empty,Failure,Log}
import scala.xml._

object Bindings {
type Binding = NodeSeq => NodeSeq

type DataBinding[T] = T => NodeSeq => NodeSeq

//adds a bind() function to an object if an implicit DataBinding is available for that object
implicit def binder[T](t: T)(implicit binding: DataBinding[T]): Binder = Binder(binding(t))
implicit def binder(binding: Binding): Binder = Binder(binding)

//decorator for a binding function that allows it to be called as bind() rather than apply()
//and also provides facilities for binding to a specific template
case class Binder(val binding: Binding) {
def bind(xhtml: NodeSeq): NodeSeq = binding.apply(xhtml)
def bind(templatePath: List[String]): NodeSeq = {
findAnyTemplate(templatePath) map binding match {
case Full(xhtml) => xhtml
case Failure(msg, ex, _) => Text(ex.map(_.getMessage).openOr(msg))
case Empty => Text("Unable to find template with path " + templatePath.mkString("/", "/", ""))
}
}
}

object NullBinding extends Binding {
override def apply(xhtml : NodeSeq) : NodeSeq = NodeSeq.Empty
}

object StringBinding extends DataBinding[String] {
override def apply(msg: String) = (xhtml: NodeSeq) => Text(msg)
}

implicit def path2list(s: String) = new {
def path: List[String] = s.split("/").toList
}
}


The principle idea is that in my application, I will create one or more DataBinding instances for each of my model types, and that then by simply declaring an implicit val containing the instance I want in the narrowest scope possible, I can call .bind on any object for which an implicit DataBinding is available, or .binding if I want to compose multiple such binding functions and close over the model instance. Furthermore, this gives me good ways to compose both over composition and inheritance hierarchies, as I'll show below.

Consider that I have the following simplified, store-oriented model objects. In Lift, these would probably be implemented using Mapper or JPA - in the app that I've been working on, these are Java classes persisted by Hibernate - certainly not things that I want to change in order to support rendering by Scala.


case class User(fname: String, lname: String, email: String, joinedOn: DateTime)
case class Address(
recipient: String,
addr1: String,
addr2: String,
city: String,
state: String,
country: String,
postalCode: String
)
case class Product(name: String, basePrice: BigDecimal, shippingCost: Option[BigDecimal])
case class OrderLine(product: Product, quantity: Int)
class Order(
val user: User,
val placedOn: DateTime,
val shipTo: Address,
var items: List[OrderLine],
var transactions: List[Transaction]
)

sealed trait State
object PENDING extends State
object IN_PROGRESS extends State
object COMPLETE extends State
object FAILED extends State
object CANCELED extends State

abstract class Transaction(val scheduledAt: DateTime, val completedAt: Option[DateTime], state: State)
case class AuthTransaction(
cc: CreditCard,
amount: BigDecimal,
scheduledAt: DateTime,
completedAt: Option[DateTime],
state: State
) extends Transaction(scheduledAt, completedAt)
case class CaptureTransaction(auth: AuthTransaction, ...) extends Transaction(...)
case class ShipTransaction(items: List[OrderLine], ...) extends Transaction(...)


With this set of models, I have a few different ways that they will be rendered - the display to the users will not be the same as the display to site administrators, I may (or may not) want to reuse the same rendering code for a line item in an order as in a shipping transaction, and so forth. Let's take a look at how this ends up.

First, I want to create some DataBinding objects for my classes that are built just on primitive types. Let's start with User.


import net.liftweb.util.Helpers._ //this is where BindHelpers.bind comes from
import scala.xml._
import Bindings._

object UserBinding extends DataBinding[User] {
def apply(user: User): Binding = bind("user", _,
"name" -> Text(user.fname+" "+user.lname),
"email" -> Text(user.email)
)
}

object MailingAddressBinding extends DataBinding[Address] {
def apply(addr: Address): Binding = bind("address", _,
"line1" -> Text(addr.recipient),
"line2" -> Text(addr.addr1),
"line3" -> Text(addr.addr2),
"line4" -> Text(addr.city+", "+addr.state+" "+addr.postalCode+" "+addr.country)
)
}

object AdminAddressBinding extends DataBinding[Address] {
def apply(addr: Address): Binding = bind("address", _,
"recipient" -> Text(addr.recipient),
// imagine more literal bindings like the previous one here
)
}

object OrderLineBinding extends DataBinding[OrderLine] {
//more of the same
}


These implementations take advantage of partial application and the Scala type inferencer to turn the call to bind(...) into a closure with the type Binding, which is NodeSeq => NodeSeq. At this point, we're not taking advantage of the fact that these functions can compose, so we'll do so now a we build bindings for our more complex objects.


//same imports as before

trait OrderBinding extends DataBinding[Order] {
implicit val userBinding: DataBinding[User]
implicit val addressBinding: DataBinding[Address]
implicit val lineBinding: DataBinding[OrderLine]
implicit val txnBinding: DataBinding[Transaction]

def apply(order: Order): Binding = (xhtml: NodeSeq) => {
val itemTemplate = chooseTemplate("order", "lineItem", xhtml)
val txnTemplate = chooseTemplate("order", "transaction", xhtml)

bind("order", xhtml,
"user" -> order.user.bind("templates-hidden/user/order-display".path),
"shipTo" -> order.shipTo.bind("templates-hidden/address/order-display".path),
"lineItems" -> order.items.flatMap(_.bind(itemTemplate)).toSeq,
"transactions" -> order.transactions.flatMap(_.bind(txnTemplate)).toSeq
)
}
}


There's a lot more going on in this one. First off, we define our DataBinding as a trait rather than a concrete object, in order to take advantage of abstract implicit val declarations. When we go to instantiate this trait, we will provide concrete implementations of the various bindings that we want applied to the objects from which an Order instance is composed. The compiler will then be able to apply the "binder" conversion to get a Binder instance which closes over the instance with the appropriate DataBinding so that we can simply call instance.bind and pass either a path to a template, or a chunk of XHTML extracted from the input with chooseTemplate to get our final value.

To be continued...

Friday, April 24, 2009

Paulp speaks great truth

"After you have spent a certain amount of time in the compiler, you will
come to feel angry resentment at every comment you encounter, because it
might be trying to mislead you! I have modified my editor not to show me
any comments in scala sources so they cannot tempt me with their siren
songs of reasons and explanations. AH THE LIMITLESS SERENITY"

Seems to me that this technique is much more widely applicable than just to scala internals...

Tuesday, April 21, 2009

Deconstruct(or)ing Scala

The question of whether to use a constructor or a factory method for object construction is not new; we've had this discussion for years in the Java community. Scala's approach to object construction has a few features that will undoubtedly reignite this debate.

On the one hand, ordinary object construction is significantly more powerful than in Java - first, the all the ordinary Java boilerplate of assigning constructor parameters to member variables is abolished. What in Java would look like this:

public class Foo {
final int myInt;
final String myString;

public Foo(int myInt, String myString) {
this.myInt = myInt;
this.myString = myString;
}
}

becomes the concise and sensible

class Foo(val myInt: Int, val myString: String)

More significantly, Scala's trait mechanism allows for extension of a class definition at the use site; for example, if I have a trait that adds rendering logic to a class, I can mix it in only when I need it.

trait FooRenderer extends Foo {
def render: String = "I'm a Foo! My int is "+myInt+" and my string is "+myString
}

val f1 = new Foo(1, "hi") // normal Foo
val f2 = new Foo(2, "well, hello!") with FooRenderer // renderable Foo

So Scala's constructors are really powerful and you really want to use them, right? But wait...

It turns out that there are some subtle issues that arise if you start adding more logic to constructors in Scala. The logic of a Scala constructor goes directly in the body of the class, and here's the tricky bit: this is also where other member variables that aren't just blindly assigned as constructor parameters are declared and assigned. In Java, any intermediate variables that you used within a constructor were unavoiably local; in Scala they can easily (and will, if care is not taken) become a permanent part of the object.

Let's look at something a little more complex. Consider a class that, as part of its construction, finds the most common element in a list and assigns both that element and the number of occurrences to member variables.

class Common[T](l: Iterable[T]) {
val (value, count) = l.foldLeft(Map.empty[T,Int]) {(m, v) =>
m + (v -> (m.getOrElse(v, 0) + 1))
}.reduceLeft {(a, b) =>
if (a._2 > b._2) a else b
}
}

Now, there may be ways to implement this that avoid constructing and decomposing a tuple, but this is the most straightforward and efficient implementation I could come up with. A peek at the generated bytecode reveals something interesting, however:

93: putfield #83; //Field x$1:Lscala/Tuple2;
96: aload_0
97: aload_0
98: getfield #83; //Field x$1:Lscala/Tuple2;
101: invokevirtual #73; //Method scala/Tuple2._1:()Ljava/lang/Object;
104: putfield #85; //Field value:Ljava/lang/Object;
107: aload_0
108: aload_0
109: getfield #83; //Field x$1:Lscala/Tuple2;
112: invokevirtual #76; //Method scala/Tuple2._2:()Ljava/lang/Object;
115: invokestatic #91; //Method scala/runtime/BoxesRunTime.unboxToInt:(Ljava/lang/Object;)I
118: putfield #93; //Field count:I

What's up with line 93? I didn't want that spurious Tuple2 to hang around as a member field - I just needed it as an intermediate value in the construction of the object!

As it turns out, this problem is not restricted to tuples; if you use intermediate variables in the construction of your objects, they will become permanent residents. This may not be a real problem in most circumstances, but it feels messy. Now, the standard thing to do in this situation would to be to create a factory method on the companion object:

object Common {
def apply[T](l: Iterable[T]): Common[T] = {
val (value, count) = l.foldLeft(Map.empty[T,Int]) {(m, v) =>
m + (v -> (m.getOrElse(v, 0) + 1))
}.reduceLeft {(a, b) =>
if (a._2 > b._2) a else b
}

new Common(value, count)
}
}

class Common[T](val value: T, val count: Int)

Now there's no intermediate variable stored in the bytecode for Common, but we have a new problem: we can no longer construct the object from an iterable while mixing in an additional trait at the instantiation site!

Scala supports the use of auxiliary constructors, with the caveat (similar to that present in Java) that the first statement in an auxiliary constructor must be either a call to the primary constructor, or another auxiliary constructor. Because of this constraint, we can't simply use the contents of the factory method above in an auxiliary constructor. We can, however, evaluate a method within the chained call, and that gives us a workable, if somewhat boilerplate-laden solution.

class Common[T](val value: T, val count: Int) {

private def this(t: Tuple2[T, Int]) = this(t._1, t._2)

def this(l: Iterable[T]) = this(
l.foldLeft(Map.empty[T,Int]) {(m, v) =>
m + (v -> (m.getOrElse(v, 0) + 1))
}.reduceLeft {(a, b) =>
if (a._2 > b._2) a else b
}
)
}

By threading the decomposition through a private constructor that takes a tuple, we can now avoid the spurious intermediate values getting incorporated into the class, and still enjoy the benefits of instantiation-site mix-ins. What's more, there has been a bunch of talk on the Scala mailing list about a future unification of tuples with method (and hopefully constructor) parameters - in which event the extra private constructor could disappear entirely!

Wednesday, April 15, 2009

A Scala Koan

I just came across this post from James Iry on the Scala list.

"It's not part of the language so you write your own and decide if it's too confusing to be useful."


class LazyVar[T](init : => T) {
private var thunk : (() => T) = {() => {thunk = {() => init}; thunk()}}

def ! = synchronized { thunk() }
def :=(newVal : => T) = synchronized { thunk = {() => {thunk = {() => newVal}; thunk()}}}

def morph(f : T => T) = synchronized { val oldThunk = thunk ; thunk = { () => { thunk = {() => f(oldThunk())}; thunk()}}}
}

object LazyVar {
def apply[T](init : => T) = new LazyVar({init})
}

scala> val x = LazyVar(1)
x: LazyVar[Int] = LazyVar@5354a

scala> x := {println("assigning 2"); 2}

scala> x := {println("assigning 3"); 3}

scala> x!
assigning 3
res16: Int = 3


It's a Scala koan!

Wednesday, December 05, 2007

In looking around the web a bit, I've had a hard time finding any capistrano plugins that handle deployment of SVN tags and branches well. So, I wrote a little one of my own.



Capistrano.configuration(:must_exist).load do

set :svn_trunk_dir, 'trunk'
set :svn_tag_dir, 'tags'
set :svn_branch_dir, 'branches'

before "deploy" do
if rails_env == 'production'
abort "ERROR: Production deploys must be from a tag or branch." unless repository =~ /#{svn_tag_dir}|#{svn_branch_dir}/
end
end

desc <<-DESC
Deploy from a tag instead of from the trunk. This will prompt if no TAG value
is specified on the command line. No sanity checks for whether the tag actually
exists are performed.
DESC
task :deploy_tag do
set :repository, get_svn_url(svn_tag_dir, 'TAG')
run_deploy
end

desc "Create the specified tag and deploy from that tag."
task :deploy_new_tag do
trunk = "#{repository[/(.*?)(?:\/trunk)?$/, 1]}/#{svn_trunk_dir}"
set :repository, get_svn_url(svn_tag_dir, 'TAG')
puts "Creating new tagged release of trunk as #{ENV['TAG']}."
system "svn cp #{trunk} #{repository}"
run_deploy
end

desc "Deploy from a branch instead of a tag; same caviats as for :deploy_tag."
task :deploy_branch do
set :repository, get_svn_url(svn_branch_dir, 'BRANCH')
run_deploy
end

desc "Create a branch from the specified tag and deploy from that branch."
task :deploy_new_branch do
tag_url = get_svn_url(svn_tag_dir, 'TAG')
branch_url = get_svn_url(svn_branch_dir, 'BRANCH')
puts "Creating new branch #{ENV['BRANCH']} from tag #{ENV['TAG']}."
system "svn cp #{tag_url} #{branch_url}"
set :repository, branch_url
run_deploy
end

# Executes the ordinary deploy task
def run_deploy
puts "Deploying using repository url #{repository}"
execute_task(find_task("deploy"))
end

# Prompts the user for a tag and updates the repository url based upon that
# tag and the svn conventions.
def get_svn_url(tag_dir, env_var = 'TAG')
while ENV[env_var].nil? || ENV[env_var].strip.empty?
ENV[env_var] = Capistrano::CLI.ui.ask("Please specify #{env_var.downcase} name: ")
ENV[env_var].strip!
end

repo_root = repository[/(.*?)(?:\/trunk)?$/, 1]
"#{repo_root}/#{tag_dir}/#{ENV[env_var]}"
end
end


I just stuck all of this in lib/cap/tag_deployer.rb, then added


require 'lib/cap/tag_deployer.rb'


to my deploy.rb file, and away I go!

About Me

My photo
aspiring to elegant simplicity