Reflecting on my first tour of duty in New York

So as I lay here in a hotel in Queens, New York I have had a chance to reflect a little on the experience I have gained over the last few months as I worked for the client out on Long Island.  There were many high points and a few low points, mistakes made and successes gained.  All in all, I really couldn’t have asked for things to have gone better.  I wanted to take some time and share with you some of the the things I learned and/or experienced.

1) The virtues of planning
As a programmer we often hear the stories about staying out late or working long hours to finish a project.  I can recall many people in college pulling all nighters to finish projects and the constant stream of rumors from the West Coast that programmers out there often sleep at the office.  I have never been a fan of such a thing and usually feel that when you find yourself in such a situation it is most often because you did not plan accordingly.

With this project, I made every attempt to plan. I planned what I was going to do, I planned what I was going to concentrate and think about for the future. I thought about what I would train my colleagues on for a given day. And in the end it really paid off, I worked very little hours and accomplished a great deal.  The application that was built was built in accordance with known practices.  In fact, a colleague was sent to a Microsoft technology seminar where practices for developing an application was discussed. Upon his return he shared with us what we learned, and essentially described the application’s architecture as I had intended it.  This was all thanks to proper planning as well as the invaluable lessons I learned from working with a host of people at RCM.

2) Communication is vital and can be redundant
I have always had a hard time remembering things, so I write stuff down.  But when I write stuff down I become too detailed and I miss stuff being said as I am writing stuff down.  Thus I am a person who likes to try to understand small detailed chunks and maintain a high level conceptual understanding of what I am doing.  I am also not someone who can understand everything via reading, I am very kinetic learner.  Thus sometimes I had to ask questions repeatedly to make sure I understood things, especially when it came to the sometimes complex business rules of my client.

To help with this I created a great working relationship with all of the members of the team by being myself.  By the end, I felt like one of the members in the company, not a consultant and I was able to speak candidly around them, always maintaining who was in charge.  There were times I had to take charge of a situation and explain what can and cannot be done on the fly.  I remember once that I had to explain that adding values to a table was great but unless the application knew what the value mean’t you could not add functionality this way, at least with the architecture that was put into place.

I owe this great level of communication to the high satisfaction the client has with what was produced and what they learned from this experience.  At the start of the project a couple people knew some .NET others had only ever done AS400 programming. By the end today, they were able to speak about various abstract concepts that they had come across in their own research or things that I had spoke about during training. And they were able to understand why I had made the decisions that I had and how to help each other continue with the design concepts already in place.

3) Dont accept tools and frameworks blindly
If you read and follow this blog you know that we decided to use the Coolite UI framework for this project.  Coolite is an set of controls built around the ExtJS framework that is designed to help developers quickly create a rich and visually appealing websites quickly.  With the lack of a designer on the team this framework was very appealing and was chosen as the backbone for the application; this was all before I arrived on the scene.

Initially, people were ready to go with MVC and Linq2Sql so we started down that path. But I quickly noticed that this application was going to be complicated and require an advanced architecture to support the requirements.  I have no doubt that such an architecture is possible in MVC and I would really loved to have done this application in MVC.  However, given that I have far more experience with Webforms and Coolite worked better with the event model offered by Webforms I convinced the client switch to Webforms.  While I also tried to convince them to move away from Linq2Sql in favor of the the Kinetic Framework, which we have used at EIS for numerous large web projects.

In the end, because the application was designed so modularly it really could support a change in ORM without a major change, but the reason that was given to me was since the client wanted to use SQL stored procedures for the operations Linq2Sql seemed the best choice to them, so I lost that argument. However, by the end I spoke with the client again about this decision, and he agreed that given the chance he would have revised his decisions and decided to use Linq in place of the stored procedures.

All in all the experience was exceptionally positive for both myself and the client. The application that was created was visually appealing and rich in functionality, and all through the QA sessions very few bugs were found and no critical bugs.  Of course, as the application moves to the official QA team I expect that problems will be found but with the excellent communication regarding project requirements I feel that these problems will not be serious.  In addition, I was able to impress upon those I was training the value of not just .NET and its languages, but also the value and ability to think abstractly enough to work in a object oriented environment.  I look forward to my return in two weeks to assist the client with a second .NET project while the approval for phase two of the current project is decided.

Experiments with ASP .NET MVC Model Binding

I am really becoming a fan of the ASP .NET MVC framework, it has a lot of flexibility and lends itself well to design patterns that I currently use.  I also like the emerging MVVm pattern that has really become widely used among RIA apps.  It functions well with MVC and SilverLight because it gives us the flexibility to use ORMs like Entity Framework and Linq2Sql without having to battle with the lost context problem.

I decided this weekend to sit down with MVC and play around with a feature that I really dig: Model Binding.  For the uninitiated, model binding in MVC offers developers a way to specify an entity as the argument to a function and based on data sent over via the request create the object for you.  The out of the box binder works great and will cover the vast majority of cases.  I decided to take this one step further and I wanted to be able to totally bind up my proxy classes (essentially the ViewModels) without having throw away properties that were just used for this purpose and without creating an inheritance hierarchy to allow subclass proxy objects to be used for the binding, offering essentially the same thing as option 1.  I wanted this to all happen inside the model binder, so it would be totally transparent.

My result was pretty good, but not perfect.  It relies on a dependency and therefore is not 100% transparent and standalone.  This is because of what we are doing, we need to abstract the set of the property we need, this could be done via an enforced rule that constructors for proxy entities must follow or have you entities inherit from a common interfaces to guarantee that you have such a setter.  I choose to go with the later and defined IEntity as such:

public interface IEntity
{
     int ID { get; set; }
     string Name { get; set; }
}

For our example, the Name setter will not be used, but it is there because it really makes sense that an Entity should provide a name for its object.

So very briefly, the view model pattern relies on the creation of entity classes which are translated from generated entities.  This ensures that extraneous information is not passed to the client which can happen frequently when you use generated code in an all-or-nothing scenario.  Furthermore, you can reuse these objects within each other and thus gain higher amounts of code reuse which usually translates to greater maintainability.  In this example we are going to be focusing on my SeriesEntity class, defined as such:

public class SeriesEntity : IEntity
{
     // simple data references
     public int SeriesId { get; set; }
     public string Name { get; set; }
     public bool IsActive { get; set; }

     // complex data references
     public IList<SeasonEntity> Seasons { get; set; }
     public IList<GenreEntity> Genres { get; set; }
     public StudioEntity Studio { get; set; }
     
     #region IEntity Members
     public int ID
     {
          get
          {
               return SeriesId;
          }
          set
          {
               SeriesId = value;
          }
     }
     #endregion
}

The use of IList here is crucial to the solution I am going to propose, though you could make this any collection type, with the exception of an array.

One of the really neat things about MVC is that in addition to allowing you the ability to override the default binder, you can also specify a binder of a specific type, all you need to is define the binders dictionary in Global.asax.

protected void Application_Start()
{
     RegisterRoutes(RouteTable.Routes);
     RegisterModelBinders(ModelBinders.Binders);
}

private void RegisterModelBinders(
     ModelBinderDictionary binderDictionary)
{
     binderDictionary.DefaultBinder =
          new Binders.AnimeModelBinder();
     binderDictionary.Add(
          typeof(SeriesEntity), new SeriesModelBinder());
}

I am showing this as an example of what you can do with the binding at a type level.  We will be using the the DefaultModelBinder base class, so lets start with that class, firstly it must inherit from DefaultModelBinder.  This class provides a variety of virtual methods that you can override, the first one we will look at is BindProperty, this is my implementation:

protected override void BindProperty(

     ControllerContext controllerContext,

     ModelBindingContext bindingContext,

     PropertyDescriptor propertyDescriptor)

{

     // need name of the property we are trying to bind

     var propertyName = propertyDescriptor.Name;

     // find the property on the object we are binding

     var property = bindingContext.Model.GetType()

          .GetProperty(propertyName);

     if (property != null)

          property.SetValue(bindingContext.Model,

          GetObjectValueFromProperty(property,

               bindingContext.ValueProvider[propertyName]),

               null);

}

The BindProperty method is called each time a new property is discovered that could potentially be binded to.  You can control this action by specifying Exclude when defining the parameter list for your action, example:

public ActionResult Create([Bind(Exclude = "SeriesId,Seasons")]SeriesEntity series)

Notice the use of Exclude here, which ensures that the binder will not get SeriesId and Seasons to bind.  While our binder can handle it not being there, the lessens the load on the binder.

So lets understand the Binder, first we make a call into GetObjectValueFromProperty function which essentially determines if we need to do anything special with the values, if we dont we make a call to Convert.ChangeType:

return Convert.ChangeType(result.AttemptedValue, property.PropertyType);

I love this method, as it calls the underlying parsing logic to convert the value from (in this case) a string to whatever I desire.  It will fail, of course, if the value is not as expected, but this would happen with the normal binder as well.  So at this point, we can handle simple one-to-one arguments for value types now.  Lets add a means to work with complex objects.

At this point we need to introduce a dependency within our code, that is we need a way to construct our proxy entities and since these are generally custom based on the application, this is where IEntity comes into play.  Thus we create a method to generate an instance of a type that, using IEntity, we can set a value to:

private object GetInstanceWithId(int entityId, Type type)
{
     var constructor = type.GetConstructor(new Type[] { });
     var value = constructor.Invoke(null);

     ((IEntity) value).ID = entityId;
     return value;
}

Looking at this code there is one flaw and one major area of improvement.  The first one is this casting is unsafe, we have no idea if we can convert the type passed to IEntity and we are hardcoding IEntity rather then letting it being provided.  This is not that big of a deal as this is not code most developers working with this binder would ever see, but it is a hard dependency.  The best way to check this is before we call the function, we validate the the property type CAN be cast to IEntity, example:

private bool CanConvertToType(Type type, Type conversionType)
{
     var interfaces = type.GetInterfaces();
     return interfaces.Length > 0 &&
          interfaces.Select(t => t.Name)
               .Contains(conversionType.Name);
}

Simply pass the type of the property and the a type reference to the type we want to check the cast for and this method will tell you, though it only works for interfaces.

The final pieces to this is binding a collection of values.  One of the things we can do with HTML is send a comma delimited list of values from the view by naming controls with the same name.  This will form the basis for our collection binding.  The reason we use IList for our entities is so we can perform the binding and as I came to find, creating generic arrays is something .NET does not allow.  So the first step is to determine if the property we are looking at IS in fact a collection, we can reuse our CanConvertToType function and pass typeof(ICollection<>) as the conversionType parameter.

The next step is determining the actual type being held by the collection, so we can determine if we can make objects that can be stored in this collection.  We can make a simple call to get this information:

Type elementType = property.PropertyType.GetGenericArguments()[0];

Because this is a generic, there will always be at least one argument, and because its an IList we know that the first argument is the contained type.  Once we confirm that this type can be used with IEntity we can break apart incoming value and create the IList reference we are going to return.  These two lines handle these tasks:

var valueArray = result.AttemptedValue.Split(',');
var valueList = (IList)Activator.CreateInstance(
     (typeof(List<>).MakeGenericType(elementType)));

This code is fairly self explanation, but to reiterate, create a string array from the values coming from the form submit, and use the Activator class to create an instance of List<> that we then cast to an IList.

Finally, we iterate over the string array, and call our GetInstanceWithId function to generate an instance of the underlying type with the ID property set via the abstraction provided by IEntity.

So there you have it, with some very clean reflection we can build an entire proxy entity via the custom model binder, this sets us up for validation and then actions within the controller method.  This also centralizes how we set the values for our objects helping with maintainability.  Finally, if the case arises that a totally different binder is needed for a certain type, MVC does allow type level model binding configuration which allows MVC to use a certain binder for a certain type.  I wonder if you can specify a interface type thus you can declare an model binder for a set of entities.

I hope to add validation to this example for the next entry.

The MVVM pattern and some developer reflection

It is not uncommon in the world of web programming to see a new acronym floating around every other week.  I saw MVVM for the first time a few months ago, but I put off really learning what it was, I was focused on other things and since all the conversation pertained to its use with SilverLight and I was not focused on SilverLight at the time I put it on the list of things to look at.

Then I attended the ASP .NET MVC FireStarter in New York city where they talked about the ViewModel pattern being used for MVC applications.  I immediately saw the benefits of this pattern with respect to Microsoft Entity Framework as it took the hassle out of the lost/unloaded context problem I had to circumvent whenever using the framework.  I was able to fully create a set of objects that translated the generated entities into classes that could be efficiently used by the View, in addition, I found added benefits from having these classes inherit from each other.  And since every OO programmer is continually striving for a way to reuse as much as possible without creating too much complexity, this was a win in my opinion.

What I did not realize at the time was, this is the essence of the MVVM pattern that has been gaining popularity.  It happened recently when I was thinking about the application I am developing for the client out here in New York.  I realized that in fact I had architected the application to follow this pattern without even realizing it.  As is the case whenever you realize such a thing, you start wanting to refactor bits that you did not design according to the pattern, without taking too much time in doing so.  I have a number of cases that I am correcting where we are using anonymous objects where we could be using these “proxy” objects as I call them, but they really are ViewModels, I am even starting to create facades for other objects that are composites of other properties within the object.

What I did was I had Linq2Sql generate a bunch of classes based on the tables we are using, each of these classes is proceeded by either a t (table) or a vw (view).  The ActiveRecord pattern is used to make these calls into the model, thereby creating a clean separation that allows data access calls to be refactored without changing calling code.  At the start, we were returning the data from these calls to the View via anonymous objects. (reasons for using this architecture where mostly due to using the Coolite ExtJs ASP .NET UI Framework).  However, as we began to develop the controllers which handled the brunt of the more involved functionality, I began creating classes for use by these controllers.  It became almost instantly apparent that these classes could also be used in the Read calls.  And now, I am rewriting old code to use objects themselves.  The value of this was recently demonstrated when the client asked for a formatting change that required a minimal to change, but the effect was application wide.

The one thing I would like to do differently next time, is make the static methods use to call into the model hang off the view model objects instead, this way there is no visible usage of the generated model classes, this is how my current personal application is setup, though it uses a service based pattern for data access, this is the recommended approach when using a context driven ORM as is the case with Linq2Sql or EF.  I have handled the cases in which I want to share my context across many functions (for transacted scenarios) by creating internal functions that can accept an existing context as a parameter, though it is only visible within the assembly it is defined in, this keeps calls in the view from being able to see these functions, thereby hiding the use of a context reference entirely from the view.

The idea of taking the static methods off the generated views and onto the view models themselves is that, like the service pattern, the translation is shared and not implemented all over the place.  This means that if you want to change a property that needs to come back, you do it in one spot.  For custom queries, you can still define the custom translation in a custom view model.  But the point is that you can better control over your translation which helps with maintenance and code cleanliness.  One of the ways I am doing this now is creating constructors that build the ViewModel objects for me.  But if I can find the time to perform this refactor, it is something I will do.  The other concern is that I am training other developers to develop this application, so I need to be careful changing patterns on them.

Experiments with MVC 2

This weekend I once again sat down to work with my existing MVC application and see what I could find and how I could better improve the pattern and my overall understanding of the framework.  I found a few gotchas and came up with some tips to help improve development.

• Unit Testing
• I was disappointed to find that as of right now, due to breaking changes, updated unit test project templates will be needed to use your favorite unit testing framework with MVC 2.  This includes my favorite MbUnit v3.  This does not mean that you cannot use the framework, you will just have to do the setup manually.
• Validation
• One of the things that I am most excited for moving forward is the new way to define validations on the server side.  Using the DataAnnotations library, located curiously in the System.ComponentModel parent namespace, you can easily define many of the most common validation routines, from Required to Range.  This validation ties tightly in with the MVC framework
• One of the downsides to this, however, is that it does not generate client side validation via JavaScript.  Thus requests must always be validated on the server, which can be expensive for a busy site.
• In addition, you cannot validate complex types (objects, arrays, etc) via this method.  I tried with an array that was assigned via ModelBinding, and the validation routine is not even called.  However, based on what I read regarding .NET 4, this functionality will be available via the CustomValidator attribute.  I really must comment on how it is strange that you can define your own custom validation attribute, but if it is attached to a complex type, you get nothing.

That all aside, I am really impressed with how well this ViewModel pattern works with a deferred execution system like Entity Framework.  Since my EF entities never cross into the web layer, only their “dumb” proxy counterparts, the separation is really very clean and easy to maintain.  At this point the main problem is the distinct Contexts that are open each session, but I plan to address that next week with Ninject.

The MVC View Model pattern

Microsoft ASP .NET MVC Framework is Microsoft’s implementation of the popular MVC pattern which was made popular for web applications chiefly by Ruby on Rails.  The purpose of the framework is to give developers a choice over using ASP .NET webforms which, until the advent of the MVC framework, had been the only way to develop a web application with ASP .NET.

As with any new framework, especially the web, having a pattern for doing this is essential.  Without a pattern developing an application that is clean and a codebase that properly leverages reusability of code in very difficult to do.  One of the popular patterns for MVC is the ViewModel pattern.  In this entry I will talk about developing an application that users this pattern and also explain why we make the decisions we do in developing the application so that we follow this pattern:

In the project I developed I created a web project to house my MVC project which has the following folder structure:

As you can see I have mutated the default project and removed the Models folder and replaced it with the ViewModels folder.  In this folder are the classes that I will be passing to my strongly typed views.

I have created two other projects for this solution that are both class libraries: AM.DataAccess and AM.Models.

This pattern follows some of the advice that I got from @sbohlen at the ASP .NET MVC Firestarter I attended on 10/3.  His advice was that it is ok to use an ORM like the ADO .NET Entity Framework to generate your entities but you should not pass them to the View.  The reason for this makes sense because the generated entities come with a lot of “extra” functionality.  This extra functionality is great in data access scenarios, but really do not make sense in the view.

This is why the ADO .NET Entities are in their own separate project.  The Service classes in the DataAccess project are responsible for the translation of these generated entities to the entities that will be used by the view models.

Below is an example of this translation:

This is simply a query that talks to the database via the EntityContext and takes the results and makes a List of these SeriesEntity objects.  This list is then passed to my Controller.  Once the controller receives it it then creates the necessary ViewModel class to be passed to the View, this is an example of such a piece of code:

The advantages of this model is the isolation of the generated entities with the entities designed for the web.  This can reduce the amount of data sent across the wire and decreasing load size.  In addition, there entities are more readily serialized into JSON results by MVC. The key thing to keep in mind is the separation of the Web from the Model layer, the two should never cross.

However, a disadvantage with using the Entity Framework in this way is the management of the context object (EntityContext in the example code).  The way I have this setup is a lazy loaded property in a base class.  The base class also has a destructor which disposes of the EntityContext. It doesn’t permit the sharing of the context by the request.  This means you have as many EntityContext references as the number of Service instances that you have.  Generally the best way to handle this problem is to use something like StrucutreMap or Ninject for Dependency Injection, however, that conversation is outside the scope of this post.

The main goal of this pattern is to provide a very clean and lightweight means to get data from a database and pass it to the view while maintaining type safety and leveraging code reuse.  For example, the ViewModel classes can inherit from each other, like such.  First this is the base class for all the ViewModel classes that are in use within this application:

So this base class defines a couple properties that will be in use on just about every page.  (Side note: The validation in use here is called FluentValidation).  Now lets look at the next level, the SeriesViewModel class:

So this defines all the information we need for the Details view.  Now we can use this ViewModel and inherit from it to provide the view with additional information for say the Edit/Create view.

(Note: CheckboxItem is a custom class that I created) Notice what we have available to us now in the View.  We have two properties that can tie into existing MVC HtmlHelper methods to provide us with select boxes or a set of Checkboxes.

In conclusion, the ViewModel pattern is a very popular pattern among MVC developers because it reduces clutter in the controller and views, which is what the essence of MVC is: reduction of code and simplification of responsibilities.  Furthermore, using ViewModel allows you to share and reuse code across multiple view and can truly make your View agnostic from the controller. This was one of the main weaknesses of the MVP pattern used by ASP .NET, the controller (code behind) was very tightly tied to the ASPX view.  

My Trip to the New York ASP .NET MVC Firestarter

Being in New York has its many perks, one is the awesome big city an hour train ride away. This city and its surrounding area offer one of the biggest markets for technology in the world and it is no surprise that Microsoft has a tremendous presence here.  I was informed of this event by my colleague here in New York, Anis.  Upon seeing it I immediately registered, not just for its content, but also its location: At the Grand Hyatt in New York City next to Grand Central station.  Any excuse I can find to go into the city is a valid one in my book.

I want to thank Peter Laudati, Stephen Bohlen, and Sara Chipps for taking the time to put on this wonderful event, while a lot of the information for me was review, it was still very helpful to get other people’s perspective on the technology as well as meet local members of the New York technology community.

I really enjoyed this session because it got me into thinking mode and finally got a design pattern to click in my head for MVC, I have to thank Stephen for this as he pointed out something that I was doing which I see now is a bit of an overstep in terms of the responsibility of the control.  I assumed that I should use EF or L2SQL entity classes as the models that are passed to my strongly typed views.  Stephen pointed out that why should I pass these objects, which contain so much extra information, to the model. Make simple versions, he said, and passed those.

At first I didn’t agree with this point as I thought it would lead to class clutter.  But as I started to think about it I realized this is already what I am doing on the project out here in New York, where we use Linq2Sql unfortunately.  When you look at the application the L2SQL entity NEVER leave the Model layer except as an anonymous object or as a simplified entity object (realizing this made me recognize some areas that should be refactored starting Monday).  By using this approach with MVC we can open the ability to TRUELY leverage the idea of the Universal Model.  Where a single View can be used by n controllers to display data.

Think about a maintenance screen for satellite data.  Generally you just have a Name field and then your normal Create or Update logic.  You could design a Web level model class with an interface that has the desired methods on it and simply work with that in the View passing it off as necessary without ever know what you are actually working with.  In turn this would use just one view, and save you the headache of many satellite maintenance screens.

That aside, the majority of what was discussed was review, though the conversations were interesting and I was able to get to know a few of the local technologists.  Thinking I may look into visiting the New York .NET User group.