Wednesday, May 30, 2012

Working with models in patio

Loading Models in Patio

One question that I often get is how to work with models in patio. Models in patio are different in that they are not synchronous in the loading of them. The reason is because we load the schema when you define the model object.

The reason we did this when designing patio was that it allows for a clean seperation of database DDL. You may ask what difference does it make, well…

This can come in handy when you have multiple pieces of software reading from a single database.

Imagine you have a ruby implementation of your model and a javascript implementation, for whatever reason (maybe your migrating from ruby to javascript and you cannot do it all at once). Well if this is the case then with patio you can just add your models and not have to worry about schema design as patio will dynamically load it at run time.

This allows the javascript developers to focus on gettings things done rather than setting up their models to match the ruby schemas field for field!

Getting started

When I work with patio I typically follow the same basic directory structure

- lib
 - models
   - Model1.js
   - Model2.js
   - Model3.js
 - index.js

Then in index.js I have a method like the following

(function(){
    "use strict";
    var path = require("path"),
        comb = require("comb"),
        patio = require("patio");
    var CONNECT_URI = "mysql://test:testpass@localhost:3306/db?minConnections=1&maxConnections=5";
    patio.connect(CONNECT_URI);

    //add patio as an export so I dont have any 
    //module issues where I have the wrong instance of patio.
    exports.patio = patio;

    exports.load = (function(){
        //create a shared promise so we only load the models once, and a loaded var to store if
        //we have called load before
        var loadPromise = new comb.Promise(), loaded = false; 
        return function(cb){       
            if(!loaded){
                //we havent loaded yet so load the models directory at once then call the loadPromise
                patio.import(path.resolve(__dirname, "./models")).then(loadPromise);
                loaded = true;
            }
            if(cb){
                 //classic accepts a callback like a normal node callback i.e. function(err, result){};
                 loadPromise.classic(cb);
            }
            //return the promise incase you want to use the promise api
            return loadPromise;
        };
    })();

})();

I usually have my model in its own module so I can use it accross services we’ll call our model package my-model

So in other modules I add it as a dependency and use it like so

(function(){
   "use strict";
   var models = require("my-model"),
    patio = models.patio;

    models.load(function(err){
       if(err){
          console.error(err.stack);
          process.exit();
       }else{
          var Model1 = patio.getModel("model1"),
              Model2 = patio.getModel("model2");
         //do some thing with them
       }
    });

})();

Inter-model Dependencies

Sometimes you have models that are dependent among eachother to get around this the model has a patio property or you can just require patio. Then in the constructor of the model you can set up your dependencies.

So if Model2 needed Model1 for a query you can do the following


(function(){
    "use strict";
    var patio = require("patio"), 
    Model1;

    patio.addModel("model2", {

        instance : {

            constructor : function(){
                if(!Model1){
                    //now you can use it
                    Model1 = patio.getModel("model1");
                }

            }

        },

        static : {

            someQueryMethod : function(){
                //using the patio property
                var Model1 = this.patio.getModel("model1");
                //do something with it              
            }

        }

    });
});

Wrapup

Hopefully this boiler plate can help you get started with patio and the model loading pattern.

I have set up a repo at https://github.com/doug-martin/patio-boilerplate

Wednesday, May 23, 2012

Getting started with gofigure

gofigure

gofigure is a configuration tool for node to help in the gathering and monitoring of configuration files in node.

Installation

npm install gofigure

or

git clone git@github.com:Pollenware/gofigure.git

Getting Started

Loading configurations

gofigure currently handles the loading of JSON files for configurations.

To Get an instance of a configuration object use the gofigure method. The gofigure method takes an object that accepts the following options

  • locations : an array of directories that contain your configurations.
  • files : an array of files that contain your configurations.
  • monitor : set to true to monitor changes to configuration files.
  • ignoreMissing : By default gofigure will ignore missing directories. Set this to false to prevent the ignoring of missing configuration directories.
  • environment : By default will look for process.env.NODE_ENV if this is not set then gofigure will read all properties. If you wish to explicity set the environment then set this property.

var gofigure = require("gofigure");

//Loader for directory of configurations
var dirLoader = gofigure({
  locations : [__dirname + "/configs"]
});


//Loader for files of configurations
var dirLoader = gofigure({
  files : [process.env.HOME + "/configs/config1.json", __dirname + "/config1.json"]
});

You can load configurations asynchronously

loader.load(function(err, config){
    var PORT = config.port, HOST = config.host;
});

or synchronously

var gofigure = require("gofigure");

var loader = gofigure({locations : [__dirname + "/configs"]});
var config = loader.loadSync();

Directories of configurations

To load directories that contain configuration files in the options object provide locations property that is an array of directories than contain your configurations.


var gofigure = require("gofigure");

var loader = gofigure({locations : [__dirname + "/configs"]});
loader.load(function(err, config){
    var PORT = config.port, HOST = config.host;
});

The order of the locations matter as it defines a precedence for files. For example suppose you have a directory of default configuration files, and on production you want to override those configuration with environment specific configurations with out changing your module or source controlled files.

var gofigure = require("gofigure");

var loader = gofigure({locations : ["/prod/configs", __dirname + "/configs"]});
loader.load(function(err, config){
    var PORT = config.port, HOST = config.host;
});

Here any production configuration files found in /prod/configs will override the properties in __dirname + "/configs".

Another use case might be in development where you have default properties and instead of altering the source controlled files the developer can override them by putting them in their home directory.

var gofigure = require("gofigure");
var HOME = process.env.HOME;

var loader = gofigure({locations : [ HOME + "/yourApp/configs", __dirname + "/configs"]});
loader.load(function(err, config){
    var PORT = config.port, HOST = config.host;
});

Files

You may also load specific files rather than entire directories.

var gofigure = require("gofigure");

var loader = gofigure({files : ["/prod/configs/config1.json", __dirname + "/config.json"]});
loader.load(function(err, config){
    var PORT = config.port, HOST = config.host;
});

Again order matters /prod/configs/config1.json will override __dirname + "/config.json"

Monitoring

gofigure supports the monitoring of changes to configuration files.

All files

To enable monitoring you can specify monitor to true in the options.

var gofigure = require("gofigure");

var loader = gofigure({monitor : true, files : ["/prod/configs/config1.json", __dirname + "/config.json"]});
var config = loader.loadSync();

loading.on("my.cool.property", function(newValue){
  //...do something
});

Individual Files

To monitor certain files you can use the files property and with object that have a monitor : true KV pair.

var gofigure = require("gofigure");

var loader = gofigure({files : [
  { 
    file : "/prod/configs/config1.json", 
    monitor : true

  }, 
  __dirname + "/config.json"
]});
var config = loader.loadSync();

loading.on("my.cool.property", function(newValue){
  //...do something
});

Just config1.json will be monitored for changes.

Property topic syntax

To listen to all properties

loading.on(function(config){
  //...do something
});

loading.on(function(nameOfPropertyChanged, config){
  //...do something
});

loading.on(function(nameOfPropertyChanged, value, config){
  //...do something
});

To listen to specific properties

loading.on("my.cool.property", function(newValue){
  //...do something
});

loading.on("my.cool.property", function(newValue, config){
  //...do something
});

loading.on("my.cool.property", function(nameOfPropertyChanged, value, config){
  //...do something
});

Wild cards


//listen to any property changed on the my.cool object
loading.on("my.cool.*", function(propName, newValue){
  //...do something
});


//listen to the change of a property named 'property' on any object
//that is a member of my
loading.on("my.*.property", function(propName, newValue){
  //...do something
});

//listen to the change of a property named 'property' that is
//a member of a property called cool
loading.on("*.cool.property", function(propName, newValue){
  //...do something
});

//listen to the change of property or otherProperty on the my.cool object.
loading.on("my.cool.{property|otherProperty}", function(propName, newValue){
  //...do something
});

//listen to the change of property or otherProperty on the my cool or 
//notCool object.
loading.on("my.{cool|notCool}.{property|otherProperty}", function(propName, newValue){
  //...do something
});

Callback Arguments

The property change callback will pass in the following values depending on the arity of the callback.

If 1 argument is expected then just the callback invoked with the new value is a.


loading.on("my.cool.property", function(newValue){
  //...do something
});

If two arguments are expected then it is invoked with the property name and the new value.


loading.on("my.cool.property", function(propName, newValue){
  //...do something
});

Other wise the callback is invoked with the propertyName, newValue and the configuration object.


loading.on("my.cool.property", function(propName, newValue, configObject){
  //...do something
});

Environments

gofigure also supports environments, by default it will look for NODE_ENV and if it is set then it will use it.

The following is an example configuration file


{
    "development": {
        "logging":{
            "patio":{
                    "level":"DEBUG",
                    "appenders":[
                        {
                            "type":"RollingFileAppender",
                            "file":"/var/log/myApp/patio.log"
                        },                              
                        {
                            "type":"ConsoleAppender"
                        }
                    ]
                }
        },
        "app" : {
          "host" : "localhost",
          "port" : "8088"
        },
        "MYSQL_DB" : "mysql://test:testpass@localhost:3306/dev_db",
        "MONGO_DB" : "mongodb://test:testpass@localhost:27017/dev_db"
    },
    "production": {
        "logging":{
            "patio":{
                    "level":"ERROR",
                    "appenders":[
                        {
                            "type":"RollingFileAppender",
                            "file":"/var/log/myApp/patio.log"
                        }
                    ]
                }
        },
        "app" : {
          "host" : "prod.mydomain.com",
          "port" : "80"
        },
        "MYSQL_DB" : "mysql://test:testpass@prod.mydomain.com:3306/prod_db",
        "MONGO_DB" : "mongodb://test:testpass@prod.mydomain.com:27017/prd_db"        
    },
    "test": {
        "logging":{
            "patio":{
                    "level":"INFO",
                    "appenders":[
                        {
                            "type":"RollingFileAppender",
                            "file":"/var/log/myApp/patio.log"
                        }                                                       
                    ]
                }
        },
        "app" : {
          "host" : "test.mydomain.com",
          "port" : "80"
        },
        "MYSQL_DB" : "mysql://test:testpass@test.mydomain.com:3306/test_db",
        "MONGO_DB" : "mongodb://test:testpass@test.mydomain.com:27017/test_db"        
    }
}

To load just the development properties set the environment to development.


var gofigure = require("gofigure"),
    patio = require("patio"),
    mongoose = require("mongoose"),
    comb = require("comb"),
    DB, HOST, PORT;


var loader = gofigure({
  files : [__dirname + "/config-env.json"],
  environment : "development"
})
  .on("MYSQL_DB", function(uri){
    patio.connect(uri);
  })
  .on("MONGO_DB", function(uri){
    mongoose.connect(uri);
  })
  .on("logging", function(logging){
    new comb.logging.PropertyConfigurator().configure(logging);
    patio.configureLogging(logging);
  })
  .on("app", function(app){
    //...
  })
  .load(function(){
    //do something
  })

Wrap up

I hope this is useful is getting you started with gofigure and the potential uses of the library.

Please submit issues to github issues.

As always pull requests are welcome!

Tuesday, May 22, 2012

Getting started with nools

What is nools?

Nools is a rete based rules engine written entirely in javascript, with a target platform of node.js.
Nools was written for Pollenware to allow them to status offers in their market clearing events in a more dynamic way.

Getting Started

To install nools run
npm install nools
Or
git clone git@github.com:Pollenware/nools.git
To run the tests
make test
To run the benchmarks
make benchmarks

Terms

First lets start with a few terms that are used in nools.
  • rule - A collection of constraints that must be satisfied for in order for a action to occur.
  • action - The code that will execute when a all of a rule's constraints have been satisfied.
  • fact - An object/item inserted into a session that the rule's constraints match against.
  • flow - This is container for rules that will be executed, you can think of it as a rule book.
  • session - This is an "instance" of a flow used to match facts. Sessions are obtained from a flow.

Writing rules

Nools includes a parser to parse nools files into a flow.
Lets look at an example
//Define our object classes, you can
//also declare these outside of the nools
//file by passing them into the compile method
define Fibonacci {
    value:-1,
    sequence:null
}
define Result {
    value : -1
}

rule Recurse {
    priority:1,
    when {
        //you can use not or or methods in here
        not(f : Fibonacci f.sequence == 1);
        //f1 is how you can reference the fact else where
        f1 : Fibonacci f1.sequence != 1;
    }
    then {
        assert(new Fibonacci({sequence : f1.sequence - 1}));
    }
}

rule Bootstrap {
   when {
       f : Fibonacci f.value == -1 && (f.sequence == 1 || f.sequence == 2);
   }
   then{
       modify(f, function(){
           this.value = 1;
       });
   }
}

rule Calculate {
    when {
        f1 : Fibonacci f1.value != -1 {sequence : s1};
        //here we define constraints along with a hash so you can reference sequence
        //as s2 else where
        f2 : Fibonacci f2.value != -1 && f2.sequence == s1 + 1 {sequence:s2};
        f3 : Fibonacci f3.value == -1 && f3.sequence == s2 + 1;
        r : Result
    }
    then {
        modify(f3, function(){
            this.value = r.result = f1.value + f2.value;
        });
        retract(f1);
    }
}

Defining Objects

In the above flow we define two classes Fibonacci and Result Both of these classes can be referenced by name throughout the file you can also access them by through the flow by calling getDefined.
In the nools language any valid javascript hash can be put inside of the define block, however if you define a constructor property then that will override the default constructor
For example we could change Fibonacci to to accept a sequence and value directly
define Fibonacci{
   constructor : function(sequence, value){
       this.sequence = sequence || -1;
       this.value = value || -1;
   }
}
You could then use the new constructor in your rules
var fib = new Fibonacci(10, 1);

Rule Definition

The rules Recurse, Bootstrap and Calculate use the nool rule syntax
rule <name> {
   when {<constraints>}
   then {<action>}
}
Lets look at the Recurse rule a little closer.
//Define a rule recurse
rule Recurse {
    //Give it a priority. The priority is 1 which is above all
    //other rules as the default priority/salience is 0
    priority:1,
    //Our LHS when block
    when {
        //you can use not or or methods in here
        //give the fact an alias of f and we checking that there is
        //NOT a Fibonacci object with a sequence of 1
        not(f : Fibonacci f.sequence == 1);

        //look for a Fibonacci object who's sequence does not equal 1
        f1 : Fibonacci f1.sequence != 1;
    }
    then {
        //assert a new fact into the engine
        assert(new Fibonacci({sequence : f1.sequence - 1}));
    }
}
Note In the when block each constraint must end with a ;

But my action is async

The actions also support async actions through a next(err?) call.
rule Recurse {
    priority:1,
    when {
        not(f : Fibonacci f.sequence == 1);
        f1 : Fibonacci f1.sequence != 1;
    }
    then {
        process.nextTick(function(){
           assert(new Fibonacci({sequence : f1.sequence - 1}));
           next();
        });
    }
}
Note if next is called with an argument then it will be assumed to be an error and the session execution will stop and an error will be passed into the callback.

Defining Helper Functions

Nools also allows the definition of helper functions, the function can be any valid javascript.
function subtract(a, b){
   return a, b;
}

function add(a, b){
   return a + b;
}

function sum(){
   var arr = Array.prototype.slice.call(arguments);
   var sum = 0;
   arr.forEach(function(a){sum += 0});
   return sum;
}
You can reference the methods in your actions or within other defined functions.

Working With Flows

The flow object is the container for all rules engines, a flow is created once and you can get multiple sessions from the flow.
To get a flow for the above fibonacci rules
//compile the file into a flow
var flow = nools.compile(__dirname + "/fibonacci.nools");

//Get defined classes
var Fibonacci = flow.getDefined("Fibonacci"), 
    Result = flow.getDefined("Result");

Sessions

The flow object is simply a conatainer for rules. To run the engine you need to obtain a session.
var r1 = new Result(),
    //Get a new session inserting a new Fibonacci object and a result to gather the
    //final number
    session1 = flow.getSession(new Fibonacci({sequence:10}), r1),
    s1 = +(new Date());
//call match to execute the engine
session1.match(function(err){
   if(err){
       console.error(err);
   }else{
      console.log("%d [%dms]", r1.result, +(new Date()) - s1);
      //cleanup the session
      session1.dispose();
   }
});

Session Events

Then session object also emits different events
  • fire - emitted when a new rule is fired. The event is emitted with the name of the rule fired and associated facts.
  • assert - emitted when a new fact is asserted
  • retract - emitted when a fact is retracted(removed) from the session
  • modify - emitted when a fact is modified

Wrap Up

I hope this was a good introduction into nools for more information visit :