Trackable Entities: N-Tier Support for Entity Framework

tracking-logo

Writing N-Tier apps can get complicated fast.  Consider the assortment of n-tier technologies now consigned to the ash heap of history: WCF RIA Services, Self-Tracking Entities, and good old typed DataSets.  These have all suffered from lack of interoperability and tight coupling with technologies that were later deprecated.  DataSets were coupled to the .NET Framework, RIA Services was tightly integrated with Silverlight, and Self-Tracking Entities bit the dust with EF5 when CodeFirst was released and the team didn’t want to upgrade a solution that was somewhat flawed to begin with.

The problem all these approaches attempted to solve is how to commit a batch of entity updates in a single transaction with a single round-trip to the server.  For example, if I have an order with associated details, and I alter the order details by modifying one of the details, removing one, and inserting a new detail, I would like to send the order to a service operation where all the updates can take place atomically.  One approach would be to separate out each set of changes and add them as parameters.  For example: UpdateOrder(Order order, OrderDetail[] addedDetails, OrderDetail[] modifiedDetails, OrderDetail[] deletedDetails).  But that would require me to massage the order by removing all the details and creating different collections of details based on the types of changes, and I would have to cache deleted entities so they could be passed in separately.  That involves a fair amount of tedious work, and the service API becomes rather clunky. It would be preferable for my service to expose an UpdateOrder operation that simply accepts an Order with details that have been added, modified or deleted.

public Order UpdateOrder(Order order)
{
    // Add, update and delete order details ...
}

The trick is to pass entity state along, but to do so in a way that is technology and platform agnostic and that does not add much overhead to the operation.  There is a technology that achieves this goal: OData, implemented by WCF Data Services and OData for ASP.NET Web API.  But the intent of OData and Data Services is to expose an Data Model (EF or an alternative source) as a REST-based service and is geared toward rendering data as a syndication feed.  While this is certainly a plausible choice, it might be overkill for scenarios where you simple want to expose a set of fine-grained operations.

A simpler and more straightforward approach would be to attach a little bit of metadata to each entity to indicate its state.  Then read that state on the server-side, performing all the updates in a single transaction.  That is the purpose of my new library and Visual Studio 2012 extension: Trackable Entities.

new-trackable-project

The easiest way to get it is to install it from within Visual Studio: Tools, Extensions and Updates, Online Visual Studio Gallery, then search for “Trackable”.  But you can get more goodness by visiting the Trackable Entities CodePlex site where you can download samples and source code. (Note that a prerequisite for the VS extension is the Entity Framework Power Tools.)

trackable-vs-gallery

Core functionality is contained in a set of TrackableEntities NuGet packages, which provide both client-side change-tracking and a server-side DbContext extension which can walk an object graph and inform the DbContext of each entity’s state so it can be persisted in a transaction when SaveChanges is called.  The client Nuget package has a ChangeTrackingCollection<T>, which extends ObservableCollection<T> by monitoring entity changes and marking them as Added, Modified or Deleted.  There is a GetChanges that returns a cloned object graph containing only changed entities.  The client-side package is implemented as a Portable Class Library which supports .NET 4.5, Silverlight 4-5, Windows Phone 7-5 or greater, and Windows Store applications.

Entity state is tracked by means of the ITrackable interface:

public interface ITrackable
{
    TrackingState TrackingState { get; set; }
    ICollection<string> ModifiedProperties { get; set; }
}

TrackingState is simply an enum, and ModifiedProperties contains a list of properties that have been updated on an entity, so that only changed properties can be persisted.

public enum TrackingState
{
    Unchanged,
    Added,
    Modified,
    Deleted
}

While the NuGet packages provide change-tracking and persistence capability, the sweet spot for productively using these libraries lies in the Trackable Entities Visual Studio 2012 extension, deployed as a VSIX file that installs a number of Visual Studio project templates.  First there is a Client Entities template, which creates a portable library and includes a T4 template for reverse engineering client entities using Entity Framework Power Tools.  Keep in mind that on the client you will remove all EF-specific classes and packages after running the tool, and there will be no coupling to EF on the client, or any other persistence technology for that matter.  (I have submitted a feature request to the EF team for allowing the EF Power Tools to generate entities separately from EF-specific classes, so they can be placed in an assembly that does not reference EF.)  Each generated entity implements both ITrackable and INotifyPropertyChanged interfaces to facilitate change-tracking, and child collections are typed as ChangeTrackingCollection<T>, so that they are properly change-tracked. For example, here is the Order entity reverse engineered from the Northwind database.

[JsonObject(IsReference = true)]
[DataContract(IsReference = true, Namespace = "http://schemas.datacontract.org/2004/07/TrackableEntities.Models")]
public partial class Order : ModelBase<Order>
{
    [DataMember]
    public int OrderId
    { 
        get { return _OrderId; }
        set
        {
            if (value == _OrderId) return;
            _OrderId = value;
            NotifyPropertyChanged(m => m.OrderId);
        }
    }
    private int _OrderId;

    [DataMember]
    public string CustomerId
    { 
        get { return _CustomerId; }
        set
        {
            if (value == _CustomerId) return;
            _CustomerId = value;
            NotifyPropertyChanged(m => m.CustomerId);
        }
    }
    private string _CustomerId;

    [DataMember]
    public Customer Customer
    {
        get { return _Customer; }
        set
        {
            if (value == _Customer) return;
            _Customer = value;
            NotifyPropertyChanged(m => m.Customer);
        }
    }
    private Customer _Customer;

    [DataMember]
    public ChangeTrackingCollection<OrderDetail> OrderDetails
    {
        get { return _OrderDetails; }
        set
        {
            if (Equals(value, _OrderDetails)) return;
            _OrderDetails = value;
            NotifyPropertyChanged(m => m.OrderDetails);
        }
    }
    private ChangeTrackingCollection<OrderDetail> _OrderDetails;
}

ITrackable and INotifyPropertyChanged interfaces are implemented in ModelBase<T>, which has a property NotifyPropertyChanged method that accepts lambda expressions.  You might also notice the use of [DataContract] and [JsonObject] attributes, which support serialization of cyclical references with an IsReference property when it is set to True.

On the server side you can use the Service Entities project template to create a .NET 4.5 class library. It also has a set of T4 templates for use with the EF Power Tools that reverse engineer CodeFirst classes and entities that implement ITrackable.  It comes with an ApplyChanges extension method to DbContext which walks an entity object graph, reads the tracking state and sets the entity state.  It can be tricky to apply state changes in the right order as you recursively traverse an object graph, but the extension method takes care of that for you so you don’t have to worry about it.  All you do is call ApplyChanges just before SaveChanges, and you’re good to go.

var db = new NorthwindContext();
db.ApplyChanges(order);
db.SaveChanges();

The Client and Service Entities project templates give you an amazing amount of functionality, but there’s more!  The Trackable Entities extension also installs a Trackable Web Api multi-project template, which includes both the Client and Service Entities projects, as well as an ASP.NET Web API project with a T4 template that customizes code generation when you add a controller.  The call to ApplyChanges is inserted in the right place, and there are other improvements to the default template.  For example, the Put method returns the updated entity so that concurrency checks will take place, Post loads related entities, and Delete loads child entities.

add-controller

The Web API template also includes a console client project that references the Client Entities project and uses HttpClient to invoke service operation on change-tracked entities.  It includes a ReadMe file with step-by-step instructions to get you started.  Here are some helper methods produced for the console client, which can be ported to any kind of .NET client (for example, WPF, Phone, or Windows Store app).

// TODO: Replace 'Entities', 'Entity', 'EntityId, 'entity' with class name (for ex, Order)

private static Entity GetEntity(HttpClient client, int entityId)
{
    string request = "api/Entities/" + entityId;
    var response = client.GetAsync(request).Result;
    response.EnsureSuccessStatusCode();
    var result = response.Content.ReadAsAsync<Entity>().Result;
    return result;
}

private static Entity CreateEntity(HttpClient client, Entity entity)
{
    string request = "api/Entities";
    var response = client.PostAsJsonAsync(request, entity).Result;
    response.EnsureSuccessStatusCode();
    var result = response.Content.ReadAsAsync<Entity>().Result;
    return result;
}

private static Entity UpdateEntity(HttpClient client, Entity entity)
{
    string request = "api/Entities";
    var response = client.PutAsJsonAsync(request, entity).Result;
    response.EnsureSuccessStatusCode();
    var result = response.Content.ReadAsAsync<Entity>().Result;
    return result;
}

private static void DeleteEntity(HttpClient client, Entity entity)
{
    string request = "api/Entities/" + entity.EntityId;
    var response = client.DeleteAsync(request);
    response.Result.EnsureSuccessStatusCode();
}

I began thinking about trackable entities way back in 2008, when I published an MSDN Magazine article exploring the topic, and then again in 2010 when I updated the code and incorporated T4 templates which read WCF service metadata.  These ideas are carried forward with this project, but it is not tied to WCF and allows for any service or persistence framework (it could, for example, be extended for NHibernate and other object-relational mappers).  The main design goal is to provide a robust client-side change tracker and code-generation of persistence-ignorant classes (POCO) that carry state information with minimal overhead and are fully interoperable.  Service entities are generated independently so they can be mapped to a data context, and they can implement ITrackable without referencing the client library.

The code is fairly stable and there is good code coverage with unit and integration tests, but I’ve released the NuGet packages and VS extension as beta (you need to include Prerelease packages to find it) so that you can start playing with it, and I can respond to feature requests before going RTM.  (Note that the EF Power Tools are also in Beta but will be folded into EF6 when it is released.)  So please let me know what you think and I’ll do my best to address any issues that come up.  Enjoy!

About Tony Sneed

Married with three children.
This entry was posted in Technical and tagged , , , . Bookmark the permalink.

18 Responses to Trackable Entities: N-Tier Support for Entity Framework

  1. nicolaspyck says:

    Hi Anthony,

    first off, great job on the plugin. It is very similar to what we have manually implemented at work over the past 3 weeks.

    I am however unable to install the package using the NuGet package manager.
    The following error occurs:
    Could not install package ‘TrackableEntities.Common 1.0.0-beta1’. You are trying to install this package into a project that targets ‘.NETFramework,Version=v4.5’, but the package does not contain any assembly references that are compatible with that framework. For more information, contact the package author.

    I suspect this is an issue with the package specification.

    Best regards,
    Nicolas

    • nicolaspyck says:

      Update:
      Seems the problem was my NuGet package manager being out of date.
      It is working fine now, time to get playing with this badboy 🙂

      • Tony Sneed says:

        Great to hear you got it installed. I will have to increase the min version requirement for NuGet in the VSIX installer.

        I also plan to do a Getting Started guide, but the Web API project template does almost everything, and the Sample readme is a good guide.

  2. Tony,

    Great work on this plugin. I’d like your opinion on an architectural approach to something I’m being asked to create for my current assignment which I think this could help with, but we want to use L2E and WCF and I’m wondering if the approach I’m looking at makes sense. When you get a moment, drop me a line at my email address and I’ll send you a more complete question assuming you’re willing and available to provide some direction.

    Thanks
    Larry
    larry.whipple@gmail.com

  3. John Radley says:

    Hello Tony,
    I was googling a few days ago researching DTOs, and came across your 2010 video, which I enjoyed immensely, and surprised myself by understanding it!! Since it was already 2 years old you can imaging I had questions on significance to the current day – and to my luck have been answered greatly by this, your blog post here.
    But for me the reverse engineering of the database is not ideal, (and I can see why you’ve made a feature request), but it has shown me interesting features about EFPowerTools that I did not know.
    For my own interest I have made a version of your TrackableEnties that reads from the EDMX file, and thus I feel it is easier to use when designing a new model and then making changes to the client. I’ve put my first attempt on my Github repository, at https://github.com/jradxl, hope you find it interesting and are not offended by my changes.
    Many thanks for your videos and I am pleased to make acquaintance with the DevelopMentor site, which has more free training material that I am looking forward to viewing.
    Thanks
    John

    • Tony Sneed says:

      Excellent! I will be sure to have a look at your project. I hope the EF team invests in improving the Code First reverse engineering tool, to allow selection of specific tables.

    • Marius C. says:

      Hello John,
      I had a look on your project and i have a question which anyway it’s is not related necessary with the approach used to generate POCO classes (Database First or Reverse Engineering Code First). If I’m using WCF Service Application to expose the server entities i don’t see any way on the client side to make the wire up between the proxy classes and the client entities than writing my own code to make the conversion. Any advises?
      Thx in advance
      Regards,
      Marius

      • Tony Sneed says:

        Place your POCO’s in a separate class library, then reference from the client it BEFORE adding the advice reference. The POCO classes will be used and not regenerated as part of the service reference

  4. gherold says:

    I looks like you have reinvented STE. No ? what is the difference ?

    • Tony Sneed says:

      Support for STE’s has been dropped entirely from EF. I recently created a replacement called Trackable Entities which can be used both with WCF and ASP.NET Web API. Check out my blog post on Trackable Entities, which I implemented as a VS 2012 extension.

    • Tony Sneed says:

      First, my implementation is a lot more efficient. The old STE’s were bloated and tightly coupled, similar to DataSets and diff grams, which made them less interoperable. My trackable entities have just one numeric property (an enum) to indicate entity state, which plays nicely with non-Microsoft services.

      The other difference is that I implemented my solution as NuGet packages and Visual Studio extensions which plug neatly into the ASP.NET Web API template that ships with Visual Studio. All the correct service-side code is generated for batch inserts, updates and deletes within a single transaction. In short, it’s cutting edge. 🙂

  5. Bart says:

    Interesting article. But part of the reason why STE was dropped was because people were finding out that the STE is more or less equivalent to the .NET DataSet. STEs just made your objects more bloated. So why recreate the STE in a different form again?

    • Tony Sneed says:

      I’m familiar with that assessment, which was coined by a colleague of mine at DevelopMentor. And I agree that STE’s were bloated, and too tightly coupled, just like DataSets. But that does not mean the idea of trackable entities is bad or that they have to be bloated. It just means that the EF team at Microsoft did not design them correctly.

      If you take a close look at my implementation, you’ll see that the entities only carry one extra property, an enum indicating state (plus changed properties in the case of an update).

      The other thing the EF team did wrong was to use code gen to replicate the change tracker in each project that used STE’s. My solution puts that in an assembly deployed via NuGet. Mine also leverages a customized template for ASP.NET Web API, which serializes entities as JSON. This is far more lightweight than the old XML format used by STE’s.

      Trackable Entities do solve a real-world problem of batch updates in a single round trip in a single transaction. And they do it much more efficiently than the old STE’s, which mostly deserve the bad wrap they got. 🙂

  6. john says:

    Quick question, how would this interact with a model post back? Would I need to make sure every field of the model is mapped, and wouldn’t the model binding cause every parameter to be marked as changed?

    • Tony Sneed says:

      It all works fine. 🙂 The client-side change-tracker is a collection that simply updates the TrackingState property of the entity whenever items are added or removed, and when individual properties are updated. The change-tracker uses the same mechanism as data binding: INotifyPropertyChanged. It subscribes to the PropertyChanged event, which is fired from each property setter on the model. Properties are not individually marked as changed, per se; the entire entity is marked as Modified when any property has been changed. However, when it comes to updates, there is a ModifiedProperties property, which keeps a list of names of properties that have been updated, so that the database will only update properties that have changed, versus all the properties. (Data binding has no effect on the change-state of the entity.) If you want to revert changes, that’s done by implementing IEditableDataObject, which my Simple MVVM Toolkit does out of the box. 🙂 Cheers!

  7. Amour Rashid says:

    Tony this is very wonderful.
    I came across this blog after a lot of googling for a solution of this kind.
    I have a question and some suggestions . First my question is that I want to pass a bundle of changes for example my entityContext.GetChanges() into the web API instead of passing a specific object such as Customer. For example in breeze the save method receives JObject as a bundle that can contain changes from different entities. How can we achieve this with your implementation.

    Now my suggestion is that , with technologies such as asp.net web api and on the other side your implementation of trackable entities you can achieve something like the BreezeJS has done with JavaScript.

    Can you please visit the breezejs website http://www.breezejs.com and finish the game with something like that in Microsoft.Net that will be portable accross platforms.
    I think my friend you can bring us where we want to go. Something that microsoft should have done with Entity Framework.

    Thanks for all.

    • Tony Sneed says:

      @Amour: Glad you like it! Let me see if I can address your question. Yes, it is true that ChangeTrackingCollection.GetChanges will only return items of T, because the change tracker is generic and strongly typed. However, there’s nothing to stop you from calling GetChanges on more than one ChangeTrackingCollection and simply adding them to a List (Of Object) for passing to a Web API operation for persistence. On the service side you would just need to tease out each type using the OfType LINQ operator in order to call ApplyChanges on each one.

      I took a look at breezejs, and found a nice write-up on it. In terms of portability, it turns out the Client project generated by the Trackable Entities VS extension *is* implemented as a Portable Class Library and therefore can be used on a variety of client UI platforms, including WPF, Silverlight, Windows Store and Windows Phone – as well as iOS and Android (using Xamarin). What is cool about my solution is that the client-side is absolutely ignorant of Entity Framework (or any ORM stack or persistence concerns for that matter). So it’s already portable. 🙂

      On a related note, I just released a new version of my Simple MVVM Toolkit with support for Portable Class Libraries that can share a set of models and view models across all the native client UI platforms out there. In fact, one of my next goals is to put together a sample of using Simple MVVM Toolkit together with Trackable Entities and Web API. Should be fun!

      Cheers,
      Tony

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s