Refactoring a Mobile Client Logon Process

Recently we encountered a problem with the client application I had been developing.  The application features a force logoff, whereby when the device is connected to an external power source the application will log the current user off.  This was designated a requirement by the client to prevent confusion amongst factory workers.  However, the flaw was that the force logoff caused dispose problems with the main form.  I wanted to detail the process of solving the problem.

When I initially designed the system I designated the Login form act as a “host” for the application, that is used the ShowDialog method to effectively place the main form over the login form.  While this seemed to work, after some time the QA team began complaining about a crash when the application was removed from a charging cradle.  We examined this problem over the period of a day and determined that the form was seeing part of it garbage collected.  You see, this form is rather large so my attempt was to keep a reference to it and simply hide/show the main form as needed per the login process.

Our first attempt was to completely decouple this process: thus a new instance would be created each time within a using statement.  We knew this would hamper performance as the application transitioned; but we would rather not have the application crash.  On the whole, this approach seemed to work and so QA was made happy….for a day.  It started when one of the QA members had the application crash with a similar error, but this time coming from a different place.  Furthermore, the change seemed to permit the existence of ”zombie” forms under certain situations.  Data became corrupted due to double execution of certain operations.  It was clear we had to return to the drawing board.

A two hour brainstorm session yielded some ideas, but nothing we felt would actually work.  That night at my hotel I was watching Futurama and playing with idea on a notepad when suddenly it hit me.  The application uses a strategy pattern to carry out workflows which map to business operations.  My thought was, what if I consider the entire application to be one giant workflow.  This would allow me to lose the multi-form interactions that were giving us grief and make the application simpler.  A firm belief of mine is that a refactor should  be more about removing code then about adding code.

The next morning I presented the idea to the client and my team.  At this stage I am on my way out with this client as I return to Michigan, so there is a need for me, as the lead, to begin writing concept documentation.  However, it was decided that I was the only one who  had the vision and could carry this change out quickly, thus I took the task.

At its root the problem is very easy.  1) Transform the login form into a login user control 2) Split the existing MainForm apart so that a user control is used for this process that fits into the “workflow” 3) update type loading so that we can operate on a higher abstraction then those simply for control involved in workflows.

This provides an example of why it is important to decouple components of responsibilities in an application and why the use of a consistent abstraction is recommended for application.  UserControls in this system all inherent from one of two classes: WorkflowUserControlBase and UserControlBase.  As  you might guess from the naming, WorkflowUserControlBase inherits from UserControlBase.  This is so when working with the components within the plumbing I dont need to worry about what control I am working with, I can operate on its abstraction; however these were all coded for WorkflowUserControlBase because only workflow control were involved in this process.  This is why the definition of workflow within the application had to be changed.

The workflows operate via .NET events which allow for highly decoupled pieces.  It made this substantially easier as my changes never went beyond the entry point tier, a good sign of an orthagonal design.  The only concession I had to make was while it would be ideal to fully implement the concept of a ProcessWorkflow which contains subworkflows upon subworkflows managed through the Strategy pattern, the time given to me simply did not allow for it, thus I improvised.

I coupled (loosely) the LoginUserControl and MainMenuUserControl in a symbiotic relationship.  The login will take the user to the main menu on a successful login and the main menu will load the login user control if a logout is requested.  I was even able to move our splash screen from the login form into the main form.  Remember the main menu acted (and still does) as a launch pad for the rest of the application.  In a properly developed application Point C should be launched from Point B. But Point C should NOT care about Point B was reached.  This is the very essence of decoupling and orthagonality.

The end result is a much cleaner, straightforward, and simpler application; which is what a refactor should always yield.  With a single form managing the entire application, the chances of the bug rearing  its head again are nearly non-existent (never say never, especially with computers).  There are still some pending issues with the application which I will not be able to get to ahead of my departure back to Michigan on Friday, however, I can rest easy knowing that I went from knowing nothing about developing with Compact Framework to developing a solid stable application that will help to improve the clients productivity; and lets not forget the lessons that I have learned that will me with future development projects.

Hierarchical Data in SilverLight

I had an idea recently to start a new side project that would help me learn Silverlight as well as provide a valuable service to RCM, the company that I am currently employed with.  My goal was to use to provide a clean interface to allow managers to get at certain bits of information needed to make decisions.

One of the requirements was the ability to display employees by office, thus I decided I would see how the scenario of binding Hierarchial data would be done using XAML; also I wanted to start building my experience with MVVm.

Note: As of SilverLight 4 the TreeView is now a standard component

Here is the basic XAML for the Treeview on my XAML page:

Side Note: Great resource for understanding many of the components in SilverLight 4:
Silverlight Toolkit Samples

Notice that we can easily create a static structure using TreeViewItem tags:

Of course, honestly, this isnt very useful and you can figure this much out about the control just by looking.  Lets talk about data binding, which is one of the forte’s for XAML.

The first thing to understand is that you will not be using the typical ItemTemplate for this because it does not handle hierarchial data, instead we use the HierarchicalDataTemplate:

Now, my first thought was the HierarchialDataTemplate would take, as its ItemsSource, something which inherits IDictionary, not true, it actually requires a ObservableCollection<T>.  Before we get to addressing that lets flush out the data we are going to use for this exercise: a collection of sports teams from various leagues.

Finally, I developed the following method to do the grouping in anticipation that this would be the construct that would be used to display the data:

However, as I mentioned before this needed to be converted into a listing that could be used, but I couldnt just use GetAllTeams, it had to have a particular structure. A common property, for example, to act as the display. Thus I created the HierarchyGrouping class to act as a proxy for groupings. The structure is general enough:

This is the necessary structure for the HierarchialDataTemplate to operate properly. Because of this conversion the MVVm pattern makes excellent sense as a means to facilitate the conversion, here is the TestPageViewModel class:

What we are doing here is defining a common property with a private setter so that the ViewModel class sets it.  Ideally, we would like this to be an ObservableCollection, but in this case we are not going to be changing this list.

With our list set, it is time to  update our XAML to bind the source:

I have added three things to this code that are of note:

• ItemsSource on sdk:TreeView – this indicates the property off the ViewModel that will server as the source of items for the control as a whole.  Note that this corresponds to our GroupListing property that we defined on the TestPageViewModel – this is not a coincidence.
• ItemsSource on the HierarchicalDataTemplate – This is where things start to get a bit curious, notice that I am specifying the property on the HierarchyGrouping which contains the sub-elements relating to the parent.  This is actually very flexible, because the possibility of n levels is easily supported.
• TextBlock – this really confused me at first. This is how  you want to display each item in the list, because every thing is considered to be of the same; hence the same type being used to hold the values.  Here I am simply binding to our Display property, which is ubiquitously available.

Now, I can already guess the next question on your mind, “but I want to style the parent nodes differently then the leaf nodes!”  The answer to this question is using a Nested HierarchicalDataTemplate.  You can get information for that here: http://www.silverlight.net/content/samples/sl4/toolkitcontrolsamples/run/default.html

Now lets finish up, we just need to add a bit of code and witness how MVVm ensure we write less code and leverage the power of the XAML interpreter.  This small bit of code will take care of all the binding to be done:

   1: var viewModel = new TestPageViewModel();

   2: viewModel.SetGroupByLeague(Team.GetTeamsGroupedByLeague());

   3:  

   4: treeView.DataContext = viewModel;

This is the essence of MVVm; as every control in XAML has a DataContext property we can leverage this fact and set the ViewModel as the DataContext of the page and all of our binding is handled by the XAML. Combine this with Observable properties and the ease of writing interactive code skyrockets.  So if your application is working you should see something like this:

Understanding Web References with Compact Framework

One of the core requirements of the client application for the client I am working with is it must communication with a web service. To do this, we created an interface that was implemented in two different ways: a local mocked data source and a remote data source which housed our web service calls.  The mock was little more then a way to mock out local scenarios locally as this application could not be properly debugged when it ran with the remote data source.

This application is built on the Windows Mobile 6 framework and its original concept was conceived by a consultant the client brought in. We decided to utilize WCF for the web service, this would come at a price, however, compact framework only supports basicHttpBinding; which is the slowest and least efficient of all the bindings available in WCF.

To create the proxy classes, we used a tool called NetCfSvcUtil which is proved by the Power Toys for .NET Compact Framework 3.5, this is also what our consultant, with many years of experience in mobile development said we had to use after doing his research.  Thus we did it; it was a very manual process requiring a command line tool to generate the classes and then a copy/paste into a file that at its peak was 14k+ lines of code.  The consultant was removed from the team after his performance proved less then satisfactory.

I took over the project and had to first had to create a solid architecture to prevent the application from crashing randomly and without warning.  Once this was complete we evolved and flushed the application out into what it is now.  A special thanks to Microsoft for their help and guidance in getting the concept into my head of why the problem was occurring.

A second problem cropped up with respect to performance.  Despite what we did we could not get calls to take less then 4s.  After some timing research we determine that our server side methods were running in under .5s in the extreme high case (yes our machines on the back are insane).  Thus we dove into why this might be.  Spent 5 days in and out of the proxy code and scouring the Internet; I will say that after developing this application it is not surprising why Microsoft is getting killed by RIM, Apple, and Google in the mobile device market, really hoping 7 proves to be better.  But there are so few solid examples online of using WinMo 6.

Anyway, thankfully, our client has Premier support with Microsoft, so we decided to throw this issue to them.  Instantly we were told that our previous consultant had made a gross error; I also fault myself for not also doing my own investigation.  Microsoft had us use Web References and instantly we saw a 50% speed improvement.  Apparently doing this, despite the fact that you are effectively doing ASMX over WCF, is that it allows the proxy code to take certain shortcuts that allow it to operate faster.  Still cant use anything besides basicHttpBinding, but Ill take whatever I can get.

So bottom line: do NOT use the Net CF power toys, they arent supported by Microsoft and does not generate efficient code.  When speed is an necessity, use Web References (though 7 does support Service References).

NYC ALT .NET Meeting – NoRM and MongoDB

I recently had the opportunity to attend the ALT .NET meeting in New York City and catch the presentation on NoRM and MongoDB by John Zablocki.  As someone who was educated and has worked with only relational databases throughout his career it was kind of a mind bending experience. The concept of literally storing data ad-hoc within the “database”, without any real relational model to speak of.

The reasons for this shift in thinking vary, but the main reason is the increased need to support horizontal scaling. Traditionally, we have known vertical scaling, whereby a bigger machine is brought in as needs increase. In today’s computer world where we are starting to see limits on processor speed, this is simply not feasible to large high volume sites such as Amazon, Facebook, Twitter, and Google.  Horizontal scaling is based on the principle that you can get better performance by increasing the number of machines hosting the service, even if the individual machines themselves are standard consumer boxes.  However, as we begin to leverage this distributed model, the limitations of RDBMS become apparent, hence the need for something different.

Database persons will often talk about ACID with respect to RDBMS as a means for ensuring data consistency. ACID often refers to the transactions within the database: Atomicity, Consistency, Isolation, Durability.  But when we start moving into distributed systems this becomes much harder to utilize without compromising performance.  In 2000, Eric Brewer introduced the CAP theorem.  CAP stands for Consistency, Availability, and Partition Tolerance.  It postulates that a database system can ONLY ever provide two of these principles.  Most RDBMS system are CP (Consistency, Partition tolerance) systems. MongoDb (and other NoRM systems) is an example of AP systems, or highly available with “eventual” consistency.  It means that the MongoDb system does not care about immediate consistency, but that the system will eventually be consistent.  This is a rather hard concept to get your head around and I am not here to sell you on the idea; I leave that to members of the NoRM community.

MongoDB is a Document Store type NoRM system, whose databases are comprised of collections (“tables”) which are actually sets of schema-less JSON objects, yes this database stores things in JavaScript. The objects have not set format, they just exist. There are automatically given a unique key (primary key) as a means of identification and can contain references to other “documents”.  MongoDB is an example of a Document Store NoRM database.

I am not going to go into a whole in-depth discussion on MongoDB or its technical facts.  But it is definitely something worth checking out.  I was quite impressed by the concepts being pitched by the NoRM community and how it could  be used.  The even better news is that drivers do exist to allow .NET integration and use of the LINQ syntax, though be careful as many NoRM systems are built on the Linux platform and can only be used in such an environment.  MongoDB, however, is one of the ones that is supported on Windows.  I admit, it is a bit strange using JavaScript as the primary means for storage and querying of data, but I am still fascinated and looking forward to checking it out.

Official link to MongoDB project: http://www.mongodb.org/

What I learned in New York

I think its finally to a point where I can talk about this with some level of safety.  For the past year, I have been working in New York on a top secret project for a large company. Initially, this started as a 3 month gig, but quickly turned into something much bigger as the scope of the project became clear and I was placed in the position of Lead Developer.

The application is a typical n-tier application with both a web and web service component.  What made the application so difficult was the insane number of rules the system had to enforce.  The company is a conglomerate, very similar in fact to RCM, and with that comes the differing ways things are done within each of the divisions or, in this case, brands.  This meant that a great deal of research had to go into the system to make sure it upheld the rules for all division as well as forcing them to adhere to standardization, one of the long term goals for the company.  Having to deal with this not only taught me patience but also brought an iron truth to the forefront: these processes could change at a moments notice and if we didnt design the application to change we would be in a world of hurt.

One of my greatest interests is in software development practices and study of patterns. I am often amazed at how many developers I meet, many with greater experience then I, who seem to know nothing about design and patterns. These patterns saved my skin so many times, from keeping this structured to making things more reusable and extensible.  Some of the notable patterns that we used were State, Strategy, and Template Method patterns.

Perhaps the first biggest example of how these patterns helped was in their main central process.  The process itself is central to the entire company.  The process was initially designed to have four separate implementations, one of which was relatively simple and dealt only with the items themselves, the others had to contend with ensuring that an inventory was maintained as well.  What I ended up creating was a two-tiered abstraction which housed components central to each level.  After discussions with the client we came up with a set of guidelines that these process would follow; and you can guess what happened next.  It was at this time the design showed its merit by making it very easy for us to make the change, though in all honesty, I was guilty of coupling to an assumption I was told was “set in stone”.  The design showed its merit yet again when another portion of the project, a backend process, needed to utilize it in a different way, rather then taking time to create the module my colleague was able to do it in just under one day.

That was perhaps the biggest lesson for me: patience.  I was told before I took the gig that I would be expected to train AS400 programmers how to use .NET.  I initially had three students, today, a year later, I am proud to say that two of them have become excellent programmers and are fully supporting the web piece.  However, the biggest challenge ended up being the mobile portion of the project.

You see, I had never done any sort of serious mobile programming before, and what little I had done was on Android.  However, as the mobile piece showed signs of stalling, my client took a gamble and showed faith in me and asked me to take it over and see it through.  Upon inspection of the code base it was a clear case of “you gotta rewrite this”.  Basically it was totally abusing system resources, which you might get away with on a Windows app, but certainly not a mobile app.  This was causing the application to randomly crash without any sort of error. This led to the company developing a second application to watchdog the app and restart it when it died.

The problem with the application was: structure, or lack thereof.  I will never understand the reason why developers do not understand that a house must have a solid foundation to stand; the same is true with software.  I think more often then not the answer is lack of knowledge; I have seen so many 9-5 programmers I am just sickened.  I view being a developer as a lifestyle more then a job.  I have to keep myself sharp or I will become a dinosaur.  I know programmers who work 8hrs then go home and dont touch a computer until the next day, and yes, it does show.  Thus began my crusade to rewrite the application.

What I learned here was that having successfully implemented the web application I had credit, and was able to give assurances to my client that I would keep this under control and it would not run away.  After two months, I am proud to say that we have completed our first pass and that all features have been implemented.  I basically ended up writing a full on mobile framework which allows for workflows, this is what guides their processes.  In the end, this ended up separating things so nicely, we were even able to introduce sub workflows, all of which was based on the State machine design.

So after one year in New York what is the biggest and most important thing that I have learned: I can do it. It has given me the self confidence to know that I have the ability to be a great consultant and that I am able to take on that leadership role and guide a project technically.  The thing I have had to be most careful about is: over confidence.  I would be lying if I said I didnt consider moving to New York, but if you wait long enough reality sets in and being away from Michigan for so long has really made me appreciate where I come from.  Much as I like the big city, right now I am still a small town guy, though Grand Rapids is hardly a small town.

The reality is, I have learned more then I can say, and I am sure not everything was good.  I know I made mistakes and in the future I must work to avoid those mistakes and learn from them.  I must continue to grow.  And I must keep the people who keep my feet on the ground as close to me as possible.

Persisting oAuth using DotNetOpenAuth

To continue my series in talking about oAuth I am now moving to one of the most critical aspects of any oAuth application: persistence. There is no value in oAuth unless you can automatically authenticate the user without asking them to allow you access to their profile.  This is the core feature of Twitter and Facebook apps on mobile devices, since you dont want to allow the application every time.  I will admit that getting this to work with DotNetOpenAuth was surprisingly very difficult and in the end required what I consider a bit of a hack.  But lets walk through the basics of this approach.

The oAuth Dance
The first thing to understand is the oAuth dance whereby we gain the users trust. This is the process of authorization and must be done to confirm the user is ok with us accessing the service to which they belong.  In our example, we are using foursquare. Once this authorization takes place you will receive an access token and access secret from the service. Storing these allows you to make authenticated requests now and in the future without asking for authorization. This article does not deal with the expiration of these tokens.

In DotNetOpenAuth we use a WebConsumer to carry out the requests over the oAuth service line. This requires service endpoints and a ConsumerSecret and ConsumerKey which is provided by the service so as to indentify the application. You can then use these to get the authenticated token as noted above.

Our application
Disclaimer: This application is a proof of concept and does not follow proper coding standards, do not use this code in a production setting.

So our application uses a local SQL Express database to store a simple username and password with the related foursquare data stored in a separate table linked by a foreign key. An Entity Framework layer sits on top of this and allows us to extract these values based on the login and make authorized requests to the foursquare service without the user needing to do anything.

The chunk of code which allows this to happen is here:

Line 11 is most critical.  You see, DotNotOpenAuth creates this concept of a TokenManager which contains the token/secret combinations for the user to access authorized data.  This is a very ugly implementation and is not something you would use in production.

To clarify, there are two sets of credentials involved: the set identifying the application (ConsumerKey, ConsumerSecret) and the set proving authorization to the service (AccessKey(token) and AccessSecret).

For the application, if the user does not exist in the database they are given a message stating the login is invalid. If they have a user account but have not provided foursquare credentials we redirect them to foursquare for authorization.  This part was especially hairy.  I use a second page to handle the callback, though the page is never visible just does a redirect.  The idea is we want to save the access information so we can use it when they log in again.

Remember, our application is a web application so multiple users will hit the same interface.  This differs from a device where the device itself is associated with the person.  We can, of course, use cookies to circumvent the login, but that is not a matter discussed by this article

With the foursquare information saved we redirect back to our main application and present the user with the interface for access the various services provided by foursquare.  The following code chunk is an example of calling the users checkin history and displaying it in a Repeater:

Ahh the beauty of Linq to XML and how deliciously evil it makes parsing XML.  Granted foursquare does also support JSON which I intend to look at later.

To conclude, oAuth is quite interesting in the way it handles authentication. Still in need of some fine tuning before I can take this to the level needed to support the application I am planning to write.  For one thing I would like to encapsulate a lot of the piping for parsing data and the plumbing for the connection and storage of credentials.

Download Source Code

Working oAuth and .NET

I recently had an idea for a mashup that would use foursquare.  I look at this as an opportunity to understand how I might leverage oAuth in the future as a solution for a Single SignOn solution.

To start with I knew that I would need a library to handle the oAuth communication. A Google searched turned up the oAuth Community Code site, which maintains a listing of popular libraries for various platforms. Among those for .NET are DotNetAuth and oAuth for .NET.  I decided to go with oAuth for .NET first.  It seemed like a solid library but for one minor drawback: it uses DI.  Dependency Injection is not a bad thing, but its not something that that I should need know to use a library.

So I decided to check out DotNetAuth which I came to found out is the same library used by Stack Overflow.  After a fair amount of testing I got a working example which walks through my first milestone: getting the token for authorization.  To start with, however, you will need to get a Key and Secret to provide to foursquare to prove you have authorization to use the service. You can get that information here.

So the first step is understanding how to get the token that proves the app is authorized to access the account.

This uses the DotNetAuth WebConsumer class to setup for our call into the foursquare oAuth service.  The service addressing is defined within Provider definition is shown below.

Looking at this code you can see what we are doing, basically pointing at where to get the oAuth token from the foursquare service.  The information is here for foursquare, though it is pretty self explanatory.

The call to Send on the Channel property will cause a redirect to the foursquare auth page where the user can enter their crerdentials and allow the app to access their data. One of the things that I found curious was that my token kept changing whenever I ran my test.  This makes me wonder if I will be able to store the token in a cookie and thus refrain from authorizing every time, which seems counterintuitive.

Thus after the allow the application will redirect back to the page that the request originated unless a callback Uri is defined.  The best way to understand how to use the library is to use this example from the foursquare HowTo: http://tinyurl.com/2b9x66a.

At this point I am very confident in my understanding of the oAuth workflow and my next step will be developing an understanding of how I can store the token and then use it repeatedly without reauthorizing and also to actually pull data from my foursquare account.

Thanks to the DotNetAuth team, the Twitter interaction sample was also very helpful to understanding the library and how it can be used.