These are my notes for the book Head first design patterns.

Overview

• Give objects new responsibilities without making any code changes to the underlying class, even at runtime.

Example problem

• Starbuzz's cost() function for beverage + condiments combinations

• Solution:

/**
 * Created by jsun on 11/7/2015 AD.
 */
abstract class Beverage { // abstract component
  def cost:Double
}

class DarkRoast extends Beverage{ // concrete component
  def cost = 2.00
}

// we pass in the decorated object in constructor; decorator object is like a wrapper
// decorator object mirror the types of decorated object (both extends Beverage here)
// a class for condiments only --> not really necessary? 
abstract class CondimentDecorator(beverage: Beverage) extends Beverage // abstract decorator

class Mocha(beverage: Beverage) extends CondimentDecorator(beverage){
  def cost = 0.5 + beverage.cost // "composing" behaviors
}

class Whip(beverage: Beverage) extends CondimentDecorator(beverage){
  def cost = 0.2 + beverage.cost
}

object Main{
  def main(args:Array[String]):Unit = {
    val drink = new Whip(new Mocha(new DarkRoast))
    println(drink.cost) // 2 + 0.5 + 0.2 = 2.7
  }
}

Decorator Pattern

• The Decorator Pattern attached additional responsibilities to an object dynamically. Decorators provide a flexible alternative to subclassing for extending functionality.

• Decorators have the same supertype as decorated.

• Each decorator HAS-A a component, i.e the decorator has an instance variable that holds a reference to a component

• Decorators are typically created by using other patterns like Factory and Builder.

• Downside: often result in a large number of small classes that can be overwhelming; can increase component instantiation complexity (can be mitigated by Factory/Builder pattern)

• Basically "wrappers"

Design Principle

• Classes should be open for extension, but closed for modification. Concentrate on areas that are most likely to change in the future.

Example in real life: java.io package

• all decorators: eg. LineNumberInputStream -> BufferedInputStream -> FileInputStream
• abstract component: InpustStream
• concrete component: FileInputStream, StringBufferInputStream, etc
• abstract decorator: FilterInputStream
• concrete decorator: BufferedInputStream, LineNumberInputStream, etc.

Scala

• Use implicits to add behavior without changing original class code (not runtime though)

object ExtendedModels{
  implicit class ExtendedBeverage(beverage: Beverage){
    def costWithMilk = 0.2 + beverage.cost // Beverage now has a costWithMilk() method, not the best example
  }
}

Overview

observer pattern = publisher + subscriber

Pattern definition:

The observer pattern defines a one-to-many dependency between objects so that when one object changes stats, all of its dependents are notified and updated automatically.

Starting example: weather monitoring application

class WeatherStation {

  def measurementsChanged():Unit = {
    val temp = getTemperature
    val humidity = getHumidity
    val pressure = getPressure

    // we want to make the displays extensible (i.e. decouple them), also maybe subscribe/unsubscribe during runtime
    currentConditionsDisplay.update(temp, humidity, pressure)
    statisticsDisplay.update(temp, humidity, pressure)
    forecastDisplay.update(temp, humidity, pressure)
  }

}

Solution round 1: bad data encapsulation

Note the Double temperature is passed everywhere!

trait DisplayElement {
  def display:Unit
}

trait Observer{
  def update(temp:Double):Unit
}

trait Subject {

  val observers:ListBuffer[Observer] = ListBuffer[Observer]()

  def registerObserver(observer: Observer) = observers += observer

  def removeObserver(observer: Observer) = observers.remove(observers.indexOf(observer))

  def notifyObservers(temp:Double):Unit
}

class WeatherStation extends Subject{

  var temperature:Double = 0.0

  def notifyObservers(temp:Double) = observers.foreach(_.update(temp))

  def setMeasurement(temp:Double) = {
    temperature = temp
    notifyObservers(temp)
  }

}


class CurrentConditionDisplay(subject: Subject) extends Observer with DisplayElement{
  // we're passing the subject in constructor so that we can call its register methods and other methods for pull

  subject.registerObserver(this)

  var temperature:Double = 0.0

  def display = println(s"current condition display, temperature: $temperature")

  def update(temp:Double) = {
    temperature = temp
    display
  }

  // NOTE: if using the pull model, would be something like this: 

  def update = {
    temperature = subject.asInstanceOf[WeatherStation].temperature
    display
  }

}

object Main {
  def main(args:Array[String]):Unit = {
    val station = new WeatherStation
    val display = new CurrentConditionDisplay(station)

    station.setMeasurement(10.0)
  }
}

Solution round 2: better data encapsulation

The java.util.Observer use Object type to pass data around: update(Observable o, Object arg)

All the others the same, but we encapsulate data:

Note: scala's trait proves superior again as you can have concrete implementation but still can compose (as opposed to in Java, it uses a class for Observable, thus limiting its reuse potential)

// DisplayElement and Main stays the same

trait Data

trait Observer{
  def update(subject: Subject, data: Data):Unit // note we pass both subject (so that observers can also use pull model) and data here
}

trait Subject {
  val observers:ListBuffer[Observer] = ListBuffer[Observer]()

  def registerObserver(observer: Observer) = observers += observer

  def removeObserver(observer: Observer) = observers.remove(observers.indexOf(observer))

 def notifyObservers(data: Data):Unit // we pass data to all observers
}

case class WeatherData (temperature:Double) extends Data

class WeatherStation extends Subject{

  var temperature:Double = 0.0

  def notifyObservers(data: Data) = observers.foreach(_.update(this, data)) 

  def setMeasurement(temp:Double) = {
    temperature = temp
    notifyObservers(WeatherData(temp))
  }

}

class CurrentConditionDisplay(subject: Subject) extends Observer with DisplayElement{

  subject.registerObserver(this)

  var temperature:Double = 0.0

  def display = println(s"current condition display, temperature: $temperature")

  def update(subject: Subject, data: Data) = {
    temperature = data.asInstanceOf[WeatherData].temperature // we need to cast to WeatherData
    display
  }

}

Design Principle

Strive for loosely coupled designs between objects that interact.

Others

• push and pull style
• java.util.Observer also has a setChanged() method to control notifications.
• Never depend on a specific notification order.

This is my notes about the book Head first design patterns.

Overview

Strategy pattern: encapsulating behaviors / algorithms.

Starting example

We start with a duck base class and several concrete duck classes:

abstract class Duck {

  def quack:Unit = println("quack")

  def display:Unit

}

class RedDuck extends Duck{
  def display:Unit = println("RedDuck")
}

class RubberDuck extends Duck {
  def display:Unit = println("rubber duck")
}

Now we want to add fly behavior, but only to some of the ducks.

Inheritance is not the right solution, because it'll add fly to all subclasses.

General problem:

Duck behavior keeps changing across subclasses, and it's not appropriate for all subclasses to have those behaviors

Book's solution:

Encapsulate behavior in classes. And assign behavior to the instances of Duck, and able to change dynamically.

trait FlyBehavior{
  def fly:Unit
}

class FlyWithWings extends FlyBehavior{
  def fly = println("fly with wings")
}

class FlyNoWay extends FlyBehavior{
  def fly = () // do nothing
}


abstract class Duck
{

  val flyBehavior:FlyBehavior // use super type here,

  // NOTE: we can also use var for flyBehavior and a setter method to change behavior in runtime

  def performFly:Unit = flyBehavior.fly // delegate to behavior class

  def display:Unit

}


class RedDuck extends Duck{

  val flyBehavior = new FlyWithWings

  def display:Unit = println("RedDuck")
}

class RubberDuck extends Duck {

  val flyBehavior = new FlyNoWay // rubber duck cannot fly

  def display:Unit = println("rubber duck")
}

object Main{
  def main(args:Array[String]): Unit ={
    val redDuck = new RedDuck
    redDuck.performFly
  }
}

Design principle:

• Identify the aspects of your application that vary and separate them from what stays the same. (In this case, behavior is what varies)
• Program to an interface/supertype, not an implementation. (Use interface for behaviors)
• Favor composition over inheritance: "HAS-A" rather than "IS-A"

Pattern definition:

Strategy pattern defines a family of algorithms, encapsulates each one, and makes them interchangeable. Strategy lets the algorithm vary independently from clients that use it.

In our example, behavior is our "algorithm"

Other notes:

• The importance of having a shared vocabulary
• OO basics: abstraction; encapsulation; polymorphism; inheritance

How about using traits?

Unlike interface in Java, traits in scala can have concrete implementations, so we have code reuse.

Compared to book's solution, however, I'm unable to update the behavior in runtime.

abstract class Duck {

  def quack:Unit = println("quack")

  def display:Unit

}

trait FlyableDuck{
  def fly = println("flying!")
}

class RedDuck extends Duck with FlyableDuck{
  def display:Unit = println("RedDuck")
}

class RubberDuck extends Duck {
  def display:Unit = println("rubber duck")
}

There are some days I wish Python is strongly typed. And today is one of them.

Background

I work on an algorithm trading desk. Our algorithm trading engine is in Python. For a few days my team has been working on a new feature which can assign a "weight" to each trading strategy, so for example if a strategy has a weight of 2, then each trading order it sends will automatically has its size doubled up.

Last night the development work was done. I run the tests, merged back to development branch, and deployed to our simulation environment.

This morning when I came into office, I was greeted by an error: basically it was shouting at me that the volume field in an order should be a number, not a string.

This has never happened before.

What happened

My first instinct is I could just add a simple int() wrapper around the volume field and back to my breakfast. Luckily I didn't and instead sat down to find the bug.

It turns out as we're reading the weights from a config file, but without properly converting it into an integer, so the weight stays as a string and everytime it's applied to order volume, Python won't complain about type mismatch, instead something like below happens:

1 * '1' = '1'

And an order with size '1', not number 1 is sent.

Why didn't my tests fail?

We have both unit tests and integration tests (where a dummy strategy will run throught the whole market data --> trading signal --> trading execution process).

In this case I didn't update the integration tests to include the weight. And in our code the default weight is 1 (NOTE: not '1'), so the tests happily passed.

Remediation

Just add an int() coersion when reading the weight from the config file.

I also included the weight field in my tests.

The real danger

It only occurred to me in the afternoon that if this has not been properly fixed, real disaster could happen:

Imagine a strategy with weight = 1 emits a trading signal with size = 3:

3 * '1' = '111'
int('111') = 111  #ahhhh fat finger

Now I don't even want to think about what 5 * '1' would turn out.

Takeaways

• NEVER blindly coerce types.
• Always update tests with new features.
• It's important to trace down to the root cause of a bug.

In the startup I work we use selenium for automated web testing. Previously the testing flow involves manually open the selenium plugin in firefox, set the environment-specific variables and click the "Play entire test suite" button. Recently I migrated the task to a Debian server which frees us from manual work and enables us to do much more, eg. test scheduling, integrating to the deployment process, etc.

Below is my notes on achieving this.

Running selenium headless

Selenium normally requires a display. On a Linux server, I use xvfb to simulate the display. The command is:

xvfb-run --server-num=10 <python commands here>

eg:

xvfb-run --server-num=10 python -m unittest discover --pattern=*.py

I have also found out that the method driver.find_element_by_link_text often fails with the message "cannot scroll into view". I have tried the fix with maximize_window and window.scrollTo(x, y) but with no luck. In the end I have to replace some of the clicks with driver.get(url) calls.

POST request in logged in session

Selenium does not support POST request naturally (if it does please let me know!), this is a bit of headache when some jquery POST calls need to be tested within a logged in session.

Luckily we use cookies to verify logged in sessions and thus can have the super awesome requests library to the rescue.

The tricky part(I am not sure if it is a universal case) is that when I login through POST request to our authenticate service, the returned cookie cannot be found in the request.cookies, rather it is part of the r.request.headers['Cookie'].

So for example to test the like button in a post (something similar to the facebook like button):

# step1 login
r = request.post(authen_url, data = {'username': username, 'password': password})

# get cookie from headers 
# note r.request.headers['Cookie'] is a comma separated string
# eg: name1=val1; name2=val2
cookie_str = r.request.headers['Cookie']
cookies = {}
for c in cookie_str.split(';'):
    cname = c.split('=')[0]
    cval = c.split('=')[1]
    cookies[cname] = cval

# step2 test other functions with the obtained cookie
# eg. a "like" POST function here
u = 'domain.com/service/likefeed'
d = {'feedid':feedid}
r = requests.post(u, data = d, cookies = cookies)

# step3 use selenium to test frontend effect, eg. a thumbs up symbol
# selenium webdriver functions here

Supporting different environments

We have 3 sets of development environments (demo --> pilot --> production). To switch among these 3 seamlessly, I created a config.ini file writing down the environment-specific variables (eg. base URL, test account login, test post contents, etc.) for each environment, and pass this config (using the excellent configobj module) when initializing each test suite:

So I have a utility function like below:

#utils.py

from configobj import ConfigObj

def read_env(env):
    c = ConfigObj('config.ini')
    return env, c[env + '_baseurl'], c[env + '_account'], c[env + '_passwd']

Then in each test suite script I add a customized __init__ call that will initialize properties to respective values according to the passed environment, these values can then be used in later testing functions:

For example:

# test_suite_1.py

from utils import read_env

class TestSuite1(unittest.TestCase):
    def __init__(self, env):
        super(TestSuite1, self).__init__('test_case_1')
        self.env, self.base_url, self.account, self.passwd = read_env(en)

Then I have a master file controlling the complete test run:

# run.py

from test_suite_1 import TestSuite1

if __name__ == '__main__':
    env = sys.argv[1]
    suite = unittest.TestSuite()

    suite.addTest(TestSuite1(env)) # initialize with specific env

    unittest.TextTestRunner().run(suite)

To make things easier I also add a bash alias in .bash_aliases:

alias mftest = 'xvfb-run --server-num=10 python /path/to/tests/run.py'

In this way the complete test process can be kicked off by simply running mftest <env> where <env> can be either demo or pilot or prod, and the program will pick up the right config.

Since it took me a surprisingly long time to solve this simple problem -- it's not in Nikola's official docs(where they only have a section on galleries) and my Google search was unfruitful, I thought I'd write it down.

The simple trick is based on the fact that Nikola will automatically copy all contents under files/ folder to output/ folder when you run nikola build.

So here are the steps:

1. Create a subdirectory in files/. Say regex_pic/ (where I put my images for the previous post on parsing regex) -- this is more about better practice, since I don't like to clutter my output/ folder.

2. Put your images into that folder.

3. In your post, use regular markdown syntax to reference those images. For example: ![simple image1](/regex_pic/simple1.jpg) will insert regex_pic/simple1.jpg.

4. nikola build. Done.