Wiki

System Analyser

kean — Thu, 18 Oct 2007 08:33:33 Z

Introduction

Welcome to the System Analyser for Common Lisp home page. Here you can find different types of documentation and the source code of the applicaiton.

I wrote the System Analyser to provide an automated means of detecting whether a system had developed unwanted characteristics within itself and also with it’s coupling to other systems. These characteristics tend to become more prominent as the scale of the system increases and it’s better to detect and address them before they get out of hand. The kind of characteristics you want to look for include:

unbalanced structure of the module relationships within a system
bloated dependencies between systems
circular dependencies (although modern system definition facilities should pick this up)
bloated modules/systems whose components have poor cohesion
insufficient decoupling of a module’s interface and implementation

For a more detailed introduction to System Analyser I recommend following the links below and reading the documentation, starting with “What Is It?”.

Documentation

Downloads

Version	Source Code	Release Notes	API Reference
0.2	package public key	n/a	package public key

License

System Analyser is released under a BSD style license. See the LICENSE file in package for more details.

Roadmap

At the moment I don’t have an issue/defect tracking system setup. I still haven’t decided on a suitable implementation. If anyone has some recommendations please let me know.

Here is the current list of enhancements under consideration for upcoming releases:

TODO: an abstraction for grouping analysis checks into a suite, similar to how unit tests are grouped into test suites
TODO: a service for drawing the system definition into a visual graph using Graphviz
TODO: write a developer's guide and/or tutorial
TODO: support mk-defsystem system definitions

Other more mundane house-cleaning tasks that need to be done for future releases:

TODO: create a chained-condition under core/condition
TODO: move item-not-found-error and already-exists-error to core/condition
TODO: move the ASDF utililty functions in service/sys-graph to util/sysdef

Biography

kean — Thu, 18 Oct 2007 07:34:14 Z

Bio

My name is Kean Lau. I’m a 25 year old software and release engineer living in Sydney, Australia. I’ve been working in the industry for 6 years and I imagine I’ll be in it for a long while yet. Back in 1st year of uni I used to chuckle at the old lecturers who were really eccentric and cynical. It’s not so funny now that I realise I’m following their same paths!

I’m a geek at heart, so my hobbies include gadgets, sci-fi shows & books and anime. Like most guys I’m always on the look out for a tasty meal, though I’m conscious of staying healthy and fit. My other interests include prog music, snooker & poker. The sports I play and watch the most are tennis, cricket and football (an I’m not refering to rugby league, ruby union or aussie rules).

If anyone is interested, you can find my CV here

System Analyser - Getting Started

kean — Thu, 18 Oct 2007 03:45:01 Z

Requirements

System Analyser does not use any implemention dependent features. If your Common Lisp implementation is compliant with the ANSI Common Lisp standard it should work with System Analyser. Having said that so far I’ve only tried it on Steel Bank Common Lisp (SBCL). If it doesn’t work with your implementation let me know.

The software pre-requesites for System Analyser that need to be installed manually are the following:

ASDF-install – see the Cliki page for installation instructions
ASDF – see the Cliki page for installation instructions

System Analyser depends on 2 other systems, FiveAM for unit testing and Albert for documentation generation. If for some reason the current versions of these systems available through ASDF-install won’t compile in your environment, you can try install it through your OS’ distributed package management facility. There are Debian packages available for FiveAM (cl-fiveam) and Albert (albert), which can be installed through apt-get or aptitude. I haven’t checked whether they are available on other distros or BSD variants.

To generate HTML documentation from the code you will need to install either jade or openjade on your system.

Installation

To get System Analyser up and running, enter these instructions in the REPL of your CL implementation:


1. CL-USER> (asdf-install:install 'sys-analyser) - download System Analyser and it's dependent systems
2. CL-USER> (asdf:oos 'asdf:compile-op 'sys-analyser) - compile System Analyser
3. CL-USER> (asdf:oos 'asdf:test-op 'sys-analyser) - run the unit tests
4. CL-USER> (asdf:oos 'asdf:doc-op 'sys-analyser) - generate Docbook documentation

The installation location of System Analyser will depend on your CL implementation.
For example under SBCL the location will be $HOME/.sbcl/site/sys-analyser-<ver_num>.

To generate HTML API documentation you need to jump out to a shell session:


1. some-shell) cd <sys-analyser_install_dir>/doc/api
2. some-shell) openjade -E 99999 -d ./albert.dsl -t sgml <albert_install_dir>/xml.dcl ./book.xml

To view the documentation point your web browser to <sys-analyser_install_dir>/doc/api/index.html.

Jade will generate the HTML files inside <sys-analyser_install_dir>/doc/api. Unfortunately Albert doesn’t generate API HTML documents that match the quality of Javadoc or Doxygen. It looks like no formatting or markup is available for the documentation strings in the code.

Trying it out

To get an idea of what System Analyser can do have a look at the examples under <sys-analyser_install_dir>/eg/business. You can run the examples through the REPL:

CL-USER> (sys-analyser.business:suite-example1)

which will create a file called example-deprecated-report.txt in your current directory.

Uninstallation

Uninstallation is done through ASDF-install

CL-USER> (asdf-install:unnstall 'sys-analyser)

System Analyser - Architectural Overview

kean — Thu, 18 Oct 2007 02:36:35 Z

Application Architecture

System Analyser’s application is divided into 6 main modules:

Business Layer – This layer contains all of the business logic to analyse system definitions for unwanted characteristics. Please note that System Analyser doesn’t provide a Business Layer built in; it provides a framework to write your own Business Layer to suit your requirements.

System Analysis Language (SAL) – This module contains the definition of the domain specific language SAL.

Service Layer – This module encapsulates all of the operations performed in the conceptual domain model. Domain Service is intended to be common layer used in multiple applications involved in the same domain.

Domain Datatype – This module contains datatype representations of the entities in the domain model. For the System Analyser, only 2 entities exist, the system graph and the analysis report.

Core – A collection of fundamental libraries that are considered so essential that every software system is going to need them

Util – A range of fundamental libraries that are more specialised, where each library is likely to be used in a minority of software systems.

The modules interact with each other in the following fashion:

The business logic in Business Layer is expressed in the various components of the System Analysis Language module

Operations requested in Business Layer is carried out by Services Layer

Most of the data manipulation in Business Layer is done on the datatypes from Domain Datatypes

Service Layer is implemented using Core and Utility

System Analysis Language is implemtned using Core

Hobbies

kean — Thu, 18 Oct 2007 01:46:57 Z

Currently in transition. Need to move the Hobby pages from my old Wiki to this one.

Blog

kean — Thu, 18 Oct 2007 01:45:30 Z

Follow the Bliki link in the navigation bar at the top of the page.

HomePage

kean — Thu, 18 Oct 2007 01:36:51 Z

Welcome

Hi, welcome to my home page. It’s basically a dumping ground for my hobbies and thoughts. The website you’re looking at now is actually a snapshot from my Wiki. It’s currently in a transition stage as I move content from my old Wiki to this one. I can’t run the Wiki on my website as my ISP won’t support the required infrastructure on their webserver.

Some of the things you’ll find here include:

source code and information on the open source projects I’ve worked on
a blog for my ramblings and rants
tips and nuggets of information on various topics that be useful for other people

To navigate around the website, I recommend using the Menu on the left frame.

System Analyser - What Is It

kean — Wed, 17 Oct 2007 11:22:44 Z

Overview

System Analyser is a simple application framework for analysing the properties of a system definition. The main feature of the framework is a declarative domain specific language called System Analysis Language (I can’t be bothered thinking of a more catchy name), which is supported by some services for analysis reports and system definition graphs.

Currently System Analyser will work with ASDF system definitions, but Mk-Defsystem support is planned for future releases.

To start this introduction will go through the basics of the System Analysis Language (SAL) and then move on to some of the domain specific services provided.

System Analysis Language

At the heart of SAL is a group of primitives that provide functionality very similar to set/list comprehension. If you are not familiar with set/list comprehension have a look here and here .

Set/List comprehension is an expression that returns a set of values that meet certain criteria. For the analysis of system definitions, set/list comprehension can be used to pick out modules from the system definition that have unwanted characteristics. More importantly set/list comprehension is declarative, ie it expresses what should be in the set, not how the set should be generated, and this makes it a powerful domain specific abstraction for system definition analysis.

Like set/list comprehension SAL provides the 3 primitives for building sets. These are:

quantifiers
generators
filters

The differences between the set/list comprehension provided in SAL and the more traditional versions are:

The set built is a set of system module IDs
The set generators draw the system modules IDs from a system definition graph, rather than some pre-defined range

The best way of describing the capabilities of SAL is to give a few examples.

Example 1

Let’s start with the simplest example, where we want to use System Analyser to find all modules that depend on another module that we have declared to be deprecated.


1    (for-some ((module (<-target-of sys-graph sys-name)))
2              (and (member module *deprecated-list* :test #'equalp)
3                   (not (eq '() (get-all-required-by-links-from-module sys-graph module)))))

On the left hand side of the example, I have provided line numbers to help explain the code. I’ll describe what’s happening line by line and provide an explanation of how all the parts work together at the end.

Line 1
`(for-some ...)`	The outer most expression is a quantifier; in this case `for-some` is an existential quantifier. Each quantifier takes 2 arguments, a list of set generators and a filter
`(module ...)`	This is a declared variable that will be bound to an element from the generated set. The variable can be referenced inside the filter
`(<-target-of ...)`	`<-target-of` is a set generator returns the IDs of all the system modules that are a required to be built before the some specified module can be built. A set generator can take as many arguments as it requires. In this case it only requires 2
`sys-graph`	This set generator argument is the system definition contained in a graph data structure
`sys-name`	This set generator argument is the ID of the module whose build targets are being returned in the set. In this case we are passing in the ID of a top-level module, ie a system

Line 2
`(and ...)`	This is the set comprehension filter, which is a predicate asserting what properties a system module must have to qualify for inclusion in the generated set. In this case the filter is a compound predicate combined through the `and` function
`(member module deprecated-list :test #'equalp)`	This 1st predicate asserts that the module ID must match one of the IDs in some list (`deprecated-list`) representing the deprecated modules

Line 3
`(not (eq '() (get-all-required-by-links-from-module sys-graph module)))`	This 2nd predicate asserts that we’re only interested in deprecated modules that are required by some other module

So what is this set comprehension expression trying to do? It is building a set of module IDs that are build targets of sys-name and have the properties of being deprecated and being required by other modules.

Now that we have the list of modules, what can with do with it? SAL provides other primitives to help report the analysis findings. Let’s explore this by continuing with the example used above:


1    (when-matches
2       (set (for-some ((module (<-target-of sys-graph sys-name)))
3                      (and (member module *deprecated-list* :test #'equalp)
4                           (not (eq '() (get-all-required-by-links-from-module sys-graph module))))))
5       (add-deprecated-chapters! sys-graph report set)))

Line 1
`(when-matches ...)`	This is a conditional expression that takes 2 arguments – an expression that returns a list and an expression that is evaluated if the first argument returns a non-empty list

Line 2
`(set ...)`	This is a declared variable that will be bound to the list returned by the set comprehension

The rest of line 2 and lines 3 & 4 were explained above

Line 5
`(add-deprecated-chapters! sys-graph report set)`	This is a call to a function that takes the list of culprits (bound to the variable `set` that was declared in line 2), generates a deprecated report chapter for each of them and adds the chapters to the analysis report `report`. It will only be evaluated if the set generator returned a non-empty set

Overview
We now have a means to tie the module IDs generated by the set comprehension with a mechanism that can describe why they require attention in an analysis report. This example shows the core of SAL.

Example 2

The first example is rather simplistic and doesn’t highlight the true benefits of expressing system analysis in SAL. A more complex example will highlight this. In this example we want to find all modules that has one or more critical path dependencies whose weight is greater than some threshold.


1   (when-matches
2       (set (for-some* ((node (<-target-of sys-graph sys-name))
3                        (leaf (<-dependent-leaf-of sys-graph node)))
4                       (> (critical-path-weight node leaf) *weight-threshold*)))
5       (add-deep-dependency-chapters! sys-graph report set))

Again on the left hand side of the example, I have provided line numbers to help explain the code. I’ll describe what’s happening line by line and explain how all of the parts work together at the end.

Line 1
`(when-matches ...)`	The same `when-matches` conditional from example1

Line 2
`(set ...)`	The declared variable to bound to the return value from the set comprehension
`(for-some* ...)`	This is an existential quantifier that nests each set generator passed as an argument. `for-some` and `for-some` have the same relationship as `let` has with `let`.
`(node (<-target-of ...))`	This is the same set generator used in the 1st example

Line 3
`(leaf ...)`	This is a declared variable that will be bound to each module produced by the set generator to its right
`(<-dependent-leaf-of ...)`	This is a set generator produces modules that are dependencies of the argument module, but are also leaves, ie modules that don’t have any dependencies themselves
`sys-graph`	This set generator argument is the system definition contained in a graph data structure
`node`	This set generator argument is the same variable that was declared in Line 2! This is possible because the quantifier is `for-some*` which nests the set generators.

Line 4
`(> (critical-path-weight node leaf) weight-threshold)`	This predicate asserts whether the critical path weight between a node and one of it’s leaf depenedencies is greater than some threshold value. Notice that the arguments `node` and `leaf` are the variables declared in set generators of lines 2 & 3

Line 5
`(add-deep-dependency-chapters! sys-graph report set)`	A call to another function that takes `set` and records them in the analysis report, but it records a differnt type of chapter (a deep dependency chapter) to the `add-deprecated-chapters!` function seen in the 1st example.

Overview
The key to understanding the semantics of example 2 is the nested existential quantifier for-some*. Due to the nesting <-dependent-leaf-of produces a set of leaf dependency modules for each module produced by <-target-of. As a result the predicate is checked against each combination the 2 set generators can produce.

For example if <-target-of generates module A & B and according to <-dependent-leaf-of module A’s leaf dependencies are modules W & X and for module B it’s leaf dependencies are modules Y & Z, the predicate will be checked against the following combination of modules:

node = A, leaf = W
node = A, leaf = X
node = B, leaf = Y
node = B, leaf = Z

If each of these combinations yeilds a true value from the predicate, the set generated is a list of module tuples:

((A W) (A X) (B Y) (B Z))

Other Features

The examples shown so far use an existential quantifier. SAL also provides the universal quantifiers for-all and for-all*. In set comprehension a universal quantifier asserts that all elements in the set pass the filter. In SAL for-all and for-all* also assert that every module in the generated set also passes the filter. If 1 or more modules returned by the set generators do not meet the filter then the universal quantifier will return an empty set, otherwise it will return the whole set.

Domain Services

At the moment the only 2 domain services provided

write-report! – writes a formatted analysis report to a file
read-asdf-sysdef – reads an ASDF system definition and structures the information into a system graph

There are many other services that could be added in future, such as:

drawing the system definition as a visual graph
highlighting modules in the visual graph that are culprits in the analysis report
persisting the system graph & analysis report to disk, which requires serialisation and deserialisation services
persisting the system graph & analysis report to a Berkeley database or a relational database

KhlScheme

kean — Mon, 15 Oct 2007 12:16:44 Z

Introduction

After I had learnt MIPS assembly, I started to become interested in finding out how high level languages translated into assembly. This is of course meant understanding how compilers work.

I guess I should have started off writing a C compiler first, but for some reason wanted to jump straight to a compiler for a language in the Lisp family. Writing a Common Lisp compiler is out of the question. The ANSI CL specification is just too huge to attempt on my own. The Scheme RSR5 specification on the other hand is more managable, albeit still quite a challenge. So I settled on attempting to write a Scheme compiler.

Goals

Output Language

I wanted the compiler’s output language to be an intermediate language rather than a quasi high level language like C because it would carry across too many high level semantics that would complicate the design and implementation of the compiler. I managed to hear about the Low Level Virtual Machine (LLVM) project, which is well documented and provides excellent compiler tools and so I chose the LLVM instruction set as the output language.

Definition Language

There are some open source Scheme compilers written in C, but they’re too large and complex to learn from. All of the simpler Scheme compilers I’ve found in compiler textbooks and on the internet are written in Scheme. I’ve never been fond of source-to-source compilers. They seem impractical to me due to their catch 22 nature.

So I decided to go down the route of writing my compiler in some other language that is portable and lower level. LLVM provides a C++ API so to leverage this as much as possible the definition language is C++.

Features

The first goal is to implement the primtive operator’s described in John McCarthy’s original Lisp paper. From there I’ll develop piece by piece the rest of the RSR5 specification.

News

I managed to develop the scanner (using the re2c library), tokeniser and the abstract syntax tree parser (using the lemon parser library), but decided to halt because C++ and/or any available libraries could not provide the necessary abstractions to cleanly write the subsequent compiling stages (construct parser, operational semantics analysis & denotational semantic analysis).

To solve this I have decided to first write a bootstrap compiler that reads a low level meta language. The abstractions that I’m currently lacking can be written in the meta language and compiled by the bootstrap compiler to create the real Scheme compiler.

Projects

kean — Mon, 15 Oct 2007 06:43:06 Z

Introduction

Here are the open source projects that I have started or planning to start sometime in the future. I only work on them during my spare time, so progress is slow. Some of them are ambitious enough that I expect to be working on it for at least 5 years.

Why have I started these projects? A combination of curiosity, the desire to tackle a challenge and genuine interest in the subject. I guess I’ll always keep looking for something new to learn. I never considered my graduation from uni as a signal to stop learning. At first I studied from various textbooks and did the exercises in them, but I reached a point where I wanted to take on something more ambitious and bring my study up to the next level. So these projects are as much as about studying as they are about developing and maintaining a product or service.

Wiki

System Analyser

Introduction

Documentation

Downloads

License

Roadmap

Biography

Bio

System Analyser - Getting Started

Requirements

Installation

Trying it out

Uninstallation

System Analyser - Architectural Overview

Application Architecture

Hobbies

Blog

Menu

HomePage

Welcome

System Analyser - What Is It

Overview

System Analysis Language

Example 1

Example 2

Other Features

Domain Services

KhlScheme

Introduction

Goals

Output Language

Definition Language

Features

News

Projects

Introduction

Project Directory