I've joined MSF - the Microsoft Solutions Framework.  MSF is about providing process "guidance". Software development processes are about who does what and in what order - role X writes a spec or a story, role Y writes the software, role Z tests it, etc.  There are heaps of variations: different companies have widely differing sets of roles and responsibilities. In some outfits the workflow is simple: there's a list of tasks and people pick them off and do them; in some it's complex, involving multiple stages of work and signoffs. I put the quotes round process "guidance" because it used to mean a load of narrative text describing the responsibilities of each role - so if you're a tester, you can look up what to do in each circumstance.  But these days - particularly with the software factories movement - we're more interested in tooling up the guidance: "guidance" means everything in the user's environment (particularly Visual Studio) that supports the locally-adopted process.  Work-tracking tools, tools for doing particular tasks like UI design, languages specific to the company's business, constraints that help you remember to run your tests before you check in, queries that enable you to find how the project is progressing - everything that helps the team work together effectively; and everything that helps them work in the way that's best tuned to their business. Visual Studio already supports a lot of this customized process-oriented stuff. GAT provides a way to extend VS so that a developer working on a particular type of job - designing UI or databases or some specific aspect of your project - can invoke tools specific to that type of work.  The DSL Tools provide ways to generate projectfuls of code from notations specific to your business.   On a more process-oriented level, VS Team System lets you track progress of work items and set up a workflow that's suited to your project's way of working.  Customizing and tuning the process to your company's business is a very important aspect. So a good chunk of my job over the coming months is going to be improving the tools for doing the customization. We're shifting the emphasis from the narrative style of guidance (though not losing it entirely) to the tooled-up style, and making it easy for people to make the tools suit them.  

The Models Conference is in Genova, Italy this year (October 1-6). If you have developed some interesting stuff in DSL Tools, this is the place to share it and get useful feedback. If you're wondering about the best ways of using models to drive software, this is the place to hear about and discuss others' experience and ideas. It's where you'll meet the most innovative practitioners and tool makers in the field. If you have interesting questions you want to discuss, run a workshop.  If you've got some interesting bits running on DSL Tools, show them off at a Poster Session - you'll get lots of good ideas and make interesting contacts. Or if you've advice to give, run a Tutorial. Models grew out of the old UML conference, and is the place to learn about model-driven software and development. Members of the DSL Tools team will be there; and have chaired and contributed to the conference on many occasions in the past.

The DSL Tools are extensible - as well as writing a specification of your own language, you can add your own code to extend the language and its tools in various ways. One of the best ways to learn the extensions is by reading the DSL Samples. They include detailed writeups on how to customize the tools, with sections on validation, deployment, templates, customized shapes, menu commands, and other goodies.

The Models/UML conference has posted its call for papers http://www.modelsconference.org/. Steve, Stuart and I take it in turns on the programme committee for this. (Stuart was Programme Chair last year.) "The MoDELS series of conferences are devoted to the topic of model-driven engineering, covering both languages and systems used to create complex systems. These conferences are both an expansion and a re-direction of previous Unified Modeling Language (UML) conferences, and replaced that series of conference since last year."

We've just published an updated set of samples for the DSL Tools. As well as updates to the previous examples, which show how to customize your DSL, there's now a complete small end-to-end showing how a DSL can be used to generate the code of a project, and work as part of a larger solution. The scenario in the end-to-end is in a team where they often have to design Windows wizards. The layout of the wizard pages is constant, with Next and Back buttons and so on; but the sequence of pages can change. So the team decide to create a Wizard Sequence Definition language. From the language, most of the code for the wizards is generated. The developer assigned to author the language actually has two jobs: as well as creating the wizard sequence DSL itself, she has to create the generators that turn it into code. She begins by designing a typical wizard, and then gradually turns it into a template which reads the page sequence from a DSL instance. There are multiple code files in the wizard, so there is a complete project full code all generated from one DSL file. When she is confident she has the language and the generated wizard project working, she packages it into a standard installer (MSI), with the help of the DSL Setup project. After her colleagues run the installer, they have a new project template available in theVisual Studio "new project" menu, which creates their own copy of the wizard project. They can now add a wizard to any application, and quickly generate the wizard code from a drawing of the page sequence. The sample also makes good use of the new validation framework. Not all wizard sequences are valid - for example, you shouldn't have pages that can't be reached from the start page. The language author only has to create the code for the constraints themselves, and the validation framework applies them at the right times and manages the error list.

We use partial classes a lot in the DSL Tools. We need them to help compose code that's generated from different places, and generated with handwritten code. If you're not familiar with it, a partial class definition is just a set of methods, properties etc that gets combined with another partial definition at link time, to form a complete class. The parts can be in different source files, which makes it easier to manage them separately: different generators can output to different files. So in the DSL Tools Use Case template, there are two generated files Designer.dsldm.cs and Shapes.cs that each contain aspects of every shape you have defined in your language. You can add a separate file with your own definitions. There is a particularly useful pattern for providing a hook for optional manually defined features.  Each generated class - let's take Container.cs in the Use Case template - has a framework superclass in which default values are defined as overridable properties. For example, HasBackgroundGradient is defined in the framework superclass OutlinedShape. This enables you to override HasBackGroundGradient if you wish in a partial Container class, but to leave it to default to the OutlinedShape value otherwise. Generated: public partial class ContainerShape : Microsoft.VisualStudio.Modeling.Utilities.OutlinedShape {... Framework:  public class OutlinedShape { protected virtual bool HasBackgroundGradient { get { return true; } } ... Optional manual override: public class OutlinedShape { protected override bool HasBackgroundGradient { get { return false; } } ... It isn't as sophisticated or flexible as delegation or aspect weaving, but does the trick. Unfortunately, it only works for program code! If we're generating tables, configuration files, or reports, we need a way of merging them together. One way is to get the generator to read the file and only replace particular bits, corresponding to one view. Another, cleaner, way is to have each source generating to a separate file, and then weave them together in a final step.  

I do get irritated by over-officious tools. "You can't use that name, you've used it already over there." Well, I know, but I'm going to change that in a moment, and it'll be alright in the end, really. In the DSL Tools Customization Samples & Guide, there is a section showing you how to prevent the user from making undesirable connections, such as loops in inheritance graphs. As the user moves the mouse to complete the last link in a loop, a tooltip pops up saying "that connection would create a loop" and the arrow changes to a not-allowed sign. This is quite effective, but of course you might prefer to let your user make the loop, and then cut it somewhere else later. Also, the computation to find the loop - or whatever other restriction - has to be fast, to respond to the user's mouse movements in real time. In some cases, that might prohibit this method. The solution is to use the new Validation framework, which is coming in the next version of DSL Tools. It allows you to specify constraints on any class (or superclass, or relation, or the whole model). These are applied to all instances of the type when the model is opened or saved, or on the user's explicit command. Error messages appear in the VS task list, and clicking on an error selects the offending objects. You write just the C# code for the constraint itself, and the framework applies it as required. This means you have a choice of when to apply a constraint - at the moment it is about to be contravened, or at validation time.

In the DSL Tools  Customization Samples & Guide there's some examples that use MDF rules to spot a change and react to it. (1) In the class diagram sample, attributes are displayed as a string like "aName : SomeType", and in the properties grid, the name and type are in separate properties. All three forms are accessible to code, for example in any generator template. A rule fires when any of the properties changes, and propagates the change to the others. (2) In the activity diagram sample, rules are used to snap the Sync Bar shape always to be a constant width or height, no matter how the user pulls the corners. (3) There is a (rather primitive!) container shape in the activities and use-case samples, vaguely reminiscent of a swimlane and a system boundary respectively. The shape manages the trick of always remaining behind other shapes; and when it moves, so do any shapes within its boundaries. In all these cases, you'll notice that if you undo a change, the whole thing undoes in one go: you don't, for example, have to first undo the readjustment to the shape, and then another undo for the original change. This is because the rule framework is part of the MDF transaction system. (MDF is the framework underlying the DSL Tools and other graphical designers in Visual Studio.) All the properties of an object - size and color of a shape, the value properties you've specified in your language, etc - are accessed through a framework that knows about the rules, and schedules firings as appropriate. Changes are always made within a transaction, and when the transaction is committed, the rules fire. All the firings are counted as being part of the transaction, which is the unit of undo/redo. As you can see from the samples, rules are attached to specific types (such as SyncBarShape), using a class attribute. Each rule is a class inheriting from one of a few abstract parents. The most useful of these are ChangeRule, which monitors property changes, AddRule, RemovingRule (about to delete) and RemovedRule (after deletion). The main thing to be careful of is infinite reiterations: if a rule makes a change, it the appropriate rule will be scheduled on the firing list. However, ChangeRules only fire if there is an actual change of value - a set operation by itself doesn't fire. So in a rule that re-adjusts the same property that caused the firing, you just have to make sure things always converge rapidly on a stable value that gets adjusted to itself. (If you're fascinated by scrolling through that long list you get after typing "override" in one of your language classes, you'll notice some methods called "On...Changed". You can override these to get an effect like the rules, but it doesn't work so well. The On... method is called after the end of the transaction, so you have to open another transaction to make your adjustments. Then of course, the user gets a horrible two-stage undo (which might sometimes be what you want, I suppose). And the biggest pitfall is that you mustn't make changes if the original change is part of an undo or redo - so if you decide you really must use this method, bracket the code with "if (!Store.InUndoRedoOrRollback)..."  But it's really not recommended, and may not work in future versions.)

Interesting post and comments on Harry Pierson's blog: http://devhawk.net/2005/10/05/Code+Is+Model.aspx  

One of the features of the DSL Tools is extensibility. With the language definition files, you can define a wide range of thing-and-link style graphical notations. The current version of the Tools comes with a set of templates for creating class, use case and activities diagrams; but the idea is that you adapt the language definitions to suit your own purposes - for example to add new sorts of classes, or cut out some sorts of association. That kind of change can be made in the language definition files. But by plugging in your own custom code, you can extend the features of your language to obtain even more interesting effects. For example, in the Activity Diagrams template in the latest version of the Tools, there's a shape called Synchronisation Bar. It comes onto the diagram as a long thin black rectangle; but of course, the user can pull the corners so that it doesn't look like a Sync Bar any more. But with a bit of custom code, you can constrain the height to be constant - so that the user can only make it longer or shorter. (Actually, a neat trick is to snap it either to a thin horizontal bar or a thin vertical one, depending on how the user tries to reshape it - so they can reorient it and change the length, but it's always a bar, one way or the other.) Another useful extension is connectivity restrictions. With the Class Diagrams template, you can make a language with classes, associations and inheritance etc. Again, your first step would probably be to adapt the language definition to do what your team needs; but pretty soon, you'll notice that it's perfectly possible to draw diagrams in which Class A inherits from Class B inherits from Class A - round in a loop. If your application needs to exclude this, you can write code that stops the user making a loop -- popping up a warning as soon as the mouse hovers over the target of the last link. There are also a few features you can obtain with custom code in the current version of the Tools, that will probably be controlled through the language definition files in the Release version -- for example, whether connectors are straight or rectangular, background color is graduated, and other basic stuff.  Other features require fairly complex code at present, but will be simplified in the future - for example, if you want to add your own menu commands to the designer. We've recently released a Customization Samples & How-To Guide package that explains these techniques, and include code for all these features. Follow the link, download, and enjoy!

JAOO  in Aarhus, Denmark, is getting big - almost 900 participants, I heard. There was a Domain Specific Languages track on Wednesday - DSLs is really gathering steam!  The day kicked off with an excellent plenary on Software Factories and DSLs by Microsoft architects Arvindra Sehmi and Beat Schwegler. Markus Voelter gave a very interesting talk on how to design DSLs, and Juha-Pekka Tolvanen - who has been doing it for a decade - about how to put them to use. MetaCase (usually represented by Juha-Pekka or Steve Kelly) always emphasise how you can build whole systems from DSLs, without any (or much) extra 'glue' code. I guess this is because in general they're working with mature product lines - often in embedded systems like phones. There, the generic framework is sufficiently evolved with sufficiently well-defined bounds.  In the MS work, we tend to talk more about cases where the DSL is just covering some part of it.

There's now a DSL Tools FAQ, which I've pulled together from the multifariouse Q&A on the DSL Tools Forum over the past few months. Thanks to all the questioners and answerers.

Interesting progress - there's an independent company offering courses on DSLs, including DSL Tools as an exemplar. http://www.clipcode.biz/workshops/Clipcode_DSL_Workshop_Data_Sheet.pdf

DSL + Agile workshop  I spent most of Monday running a DSL workshop at XP2005, with Steven Kelly of MetaCase. There were 22 participants including ourselves, and people said it was an enjoyable and useful day.   Results posted at http://www.dsmforum.org/Events/ADDSL05/results.html  Main interesting points: ·         Quite a lot of interest in using DSLs (both graphical and textual) to script and describe tests, especially in a model-based way – where the diagram is a state model, for example. ·         How to develop a DSL: three inputs: o        Variability in the developed software §         Which bits vary between delivered instances or across the life of the system? – These are the aspects that need to be at least separated out from the more constant stuff; and are good candidates for generating or generalizing and driving from a DSL. o        Ontology/vocabulary of the domain (as seen in user stories) §         Make a model of the whole domain. Separate out the variable bits as derived from the software variability: this gives you the concepts you want to express in your language. o        Existing user notations §         Look for drawings ·         The issue of “do DSLs help scale up agile” was more or less accepted without question. Of course they do, provided you manage carefully the evolution of language and framework – don’t do too much up front, stick to agile principles. Other stuff at the conference:  Jutta Eckstein, a well-respected process consultant, gave a talk on “Agile: Coming of Age”. Apparently the German armed forces (among others) now stipulate many principles and practices of agile methods for software contracts - in particular 2-way transparency and feedback; so it seems we’re getting past the early adopters stage.   The essence of agile methods is aiming to meet the real needs of the customer by looking for early feedback, and planning for change. Also planning to tune your process -- Jutta said "If it doesn't include Retrospectives , it's not agile". Acceptance Testing. Session on tools for  managing acceptance tests, including FIT, which is about integrating requirements with acceptance testing. It’s a bit like if we associated a test script with every feature in our features DB, and then had the testing apparatus automatically record the results there.  Part of the emphasis of FIT, and the other acceptance test approaches on show, was to make the tests and their results more accessible to the customer, by working from and to readable documents.  

Software development is teamwork. The most important techniques you learn for project success -- or get wrong otherwise -- are about working in teams. The methods that have made the biggest improvements in development success rates are not tools or technical stuff, they're about how people work together -- most notably, the agile methods. Tools are useful to support good ways of working, once you've sorted out what they are. So it's unfortunate that most computing degrees, in most parts of the world, don't teach these teamwork aspects. If you're lucky, you get some stuff about agility, but actually running a project -- with all the practical stuff that entails -- not much. Sheffield University in the UK is different. There, Professor Mike Holcombe has been running a course for some years in which the undergraduates get to work on real agile projects in a real company for real customers.  He uses group assessment methods, as applied in other disciplines where teamwork is a necessary part of the curriculum. His students are in great demand by industry, and “more Sheffield CS students than from any other UK University made it to the finals of the 2004 Microsoft Imagine Cup”.   Why don't we see more courses like this? A big factor is that academics haven't often worked in a commercial environment. Lecturers are chosen for the number of papers they've written, so it's difficult (though not by any means unknown) for anyone to move from an industrial into a decent academic job. This is unfortunate in an engineering discipline -- one that owes its existence to the industrial activity.  Even though very many academics have close ties with industry -- well, it isn't quite like actually being on the job. So Mike's methods haven't spread through Universities like the wildfire they should. Which is sad.