Updating a Core Data database to a new data model version is pretty hard, even if you just added a new attribute to an entity. Sometimes you want to just add an attribute without worrying about migration code.

Sometimes you want to add attributes at runtime like Andy Finnell trying to save unknown NNTP headers.

Here is how I did it.

Update: Read Part 2 for a better way of doing this.

I added a persistent binary attribute, extendedData, to my entity description. This attribute stores a binary property list. The top level object is a dictionary with attribute names as keys. The values are property lists, so I get to store arrays and dictionaries as a bonus.

In my NSManagedObject subclass I represent the new attributes with a two-layer scheme. An NSDictionary contains the attributes as they are in the persistent store, and an NSMutableDictionary contains the unsaved changes.

@interface ExtendedManagedObject : NSManagedObject {
    NSDictionary *extendedSnapshot;
    NSMutableDictionary *extendedChanges;
}
@end

The snapshot is computed on demand:

- (NSDictionary*)extendedSnapshot
{
    if (extendedSnapshot!=nil)
        return extendedSnapshot;
    id data = [[self committedValuesForKeys:[NSArray arrayWithObject:EXTDATAKEY]] 
                objectForKey:EXTDATAKEY];
    if (data!=nil && data!=[NSNull null]) {
        NSString *error=nil;
        extendedSnapshot = [[NSPropertyListSerialization propertyListFromData:data
                            mutabilityOption:NSPropertyListImmutable 
                            format:nil errorDescription:&error] retain];
        if (error!=nil) {
            NSLog(@"Error loading extended data: %@", error);
            [error release];
        }
    }
    else {
        extendedSnapshot = [[NSDictionary alloc] init];
    }
    return extendedSnapshot;
}

Note that I am using committedValuesForKeys rather than valueForKey. That is important.

The extendedChanges dictionary contains the unsaved changes with the exception that [NSNull null] indicates that the key has been set to nil. (Nil indicates that the key has not been changed).

- (id)valueForExtendedKey:(NSString*)key
{
    if (extendedChanges!=nil) {
        id value = [extendedChanges objectForKey:key];
        if (value!=nil)
            return value==[NSNull null] ? nil : [[value retain] autorelease];
    }
    return [[[[self extendedSnapshot] objectForKey:key] retain] autorelease];
}

The extended attributes completely bypass Core Data's change management, so we have to notify the undo manager manually when changing them:

- (void)setValue:(id)value forExtendedKey:(NSString*)key
{
    id oldValue = [self valueForExtendedKey:key];
    [[[[self managedObjectContext] undoManager] prepareWithInvocationTarget:self] 
            setValue:oldValue forExtendedKey:key];
    if (value==nil)
        value = [NSNull null];
    [self willChangeValueForKey:key];
    if (extendedChanges==nil)
        extendedChanges = [[NSMutableDictionary alloc] init];
    [extendedChanges setObject:value forKey:key];
    [self didChangeValueForKey:key];
}

The KVO calls have the additional effect of marking the object as dirty, even though the keys are unknown to Core Data.

When saving, we combine the two dictionaries and set the binary attribute:

- (void)willSave
{
    if (extendedChanges!=nil) {
        // merge changes into snapshot
        NSMutableDictionary *dict = [[self extendedSnapshot] mutableCopy];
        NSEnumerator *e = [extendedChanges keyEnumerator];
        NSString *key;
        while (key=[e nextObject]) {
            id value = [extendedChanges objectForKey:key];
            if (value==[NSNull null])
                [dict removeObjectForKey:key];
            else
                [dict setObject:value forKey:key];
        }

        // archive as binary
        NSData *data = nil;
        if ([dict count]>0) {
            NSString *error=nil;
            data = [NSPropertyListSerialization dataFromPropertyList:dict 
                    format:NSPropertyListBinaryFormat_v1_0 errorDescription:&error];
            if (error!=nil) {
                NSLog(@"Error saving extended data: %@", error);
                [error release];
            }
        }
        [dict release];
        [self setPrimitiveValue:data forKey:EXTDATAKEY];
    }
    [super willSave];
}

Note that we are not changing any instance variables above. We have to wait until the save is committed.

- (void)didSave
{
    [super didSave];
    // save success, changes now in snapshot
    if (extendedChanges!=nil) {
        [extendedChanges release];
        extendedChanges = nil;
    }
    // snapshot should be reloaded
    if (extendedSnapshot!=nil) {
        [extendedSnapshot release];
        extendedSnapshot = nil;
    }
}

Re-faulting is similar, lose the instance variables.

- (void)didTurnIntoFault
{
    [super didTurnIntoFault];
    if (extendedChanges!=nil) {
        [extendedChanges release];
        extendedChanges = nil;
    }
    if (extendedSnapshot!=nil) {
        [extendedSnapshot release];
        extendedSnapshot = nil;
    }
}

The code up to this point pretty much works, but it is overly complicated. What is with the two-layer storage? The answer is [refreshObject:mergeChanges:YES] is not working yet. [refreshObject:mergeChanges:NO] works just fine -- the local changes are discarded and a new snapshot loaded. However, on [refreshObject:mergeChanges:YES] we want to merge the local changes with any changes made to the persistent store by a different context. We don't want to overwrite the entire dictionary with our copy if we only changed one key, and somebody else changed another key. We want to merge the keys individually, not as a block.

That is where the two-layer storage comes in: If, during [refreshObject:mergeChanges:YES], we keep extendedChanges, but reload extendedSnapshot, the merge works per-attribute as we want.

Here is my less-than-elegant way of doing that:

- (void)willRefreshAndMergeChanges;
{
    [self trimExtendedChanges];
    [[self managedObjectContext] setRefreshToken:extendedChanges];
}

- (void)didRefreshAndMergeChanges;
{
    extendedChanges = [[[self managedObjectContext] refreshToken] retain];
}

Hey, is that some kind of secret API? No, unfortunately you have to call these methods from your own NSManagedContext subclass:

- (void)refreshObject:(NSManagedObject *)object mergeChanges:(BOOL)flag
{
    [self setRefreshToken:nil];
    if (flag && ![object isFault]) {
        [object willRefreshAndMergeChanges];
        [super refreshObject:object mergeChanges:flag];
        [object didRefreshAndMergeChanges];
    }
    else {
        [super refreshObject:object mergeChanges:flag];
    }
    [self setRefreshToken:nil];
}

The refreshToken is just an id instance variable with standard get/set methods. The managed object uses it to keep state as it passes through oblivion during the refresh.

After I wrote this code I did my transient property research, and I believe it would be a better solution to turn extendedChanges into a transient attribute. That way you can avoid the NSManagedObjectContext subclass, and you get automatic undo support. You would have to copy the dictionary on each attribute change, though.

To automatically add extended attributes when they are used, use the unknown key fallbacks:

- (id)valueForUndefinedKey:(NSString*)key
{
    return [self valueForExtendedKey:key];
}

- (void)setValue:(id)value forUndefinedKey:(NSString*)key
{
    [self setValue:value forExtendedKey:key];
}

This can be error prone, of course, so you may want some checks.

What is in a change?

There is one final issue: the mystery trimExtendedChanges method called above. The method looks like this:

- (void)trimExtendedChanges
{
    NSDictionary *snapshot = [self extendedSnapshot];
    NSEnumerator *e = [[extendedChanges allKeys] objectEnumerator];
    NSString *key;
    while (key=[e nextObject]) {
        id oldValue = [snapshot objectForKey:key];
        id newValue = [extendedChanges objectForKey:key];
        if ((oldValue==nil && newValue==[NSNull null]) 
                || (oldValue!=nil && [newValue isEqual:oldValue])) {
            [extendedChanges removeObjectForKey:key];
        }
    }
}

It removes entries from extendedChanges that are no-ops as determined by [isEqual:]. This mimics how Core Data determines changed properties when merging, so our extended attributes behave like normal attributes in every respect. The only difference is that you cannot use them in fetch requests.

I am not a big fan of this method.

Take a look at the following code:

[obj setValue:@"blue" forKey:@"color"];
[moc refreshObject:obj mergeChanges:YES];
NSString *color = [obj valueForKey:@"color"];

What would you expect color to be? Well, I just changed the color to "blue". Then I refreshed the object, merging my changes. Then I read the color back. Obviously color should be "blue", and usually it will be. However, if accidentally color was "blue" before I set the attribute, and somebody else has changed it to "red" in the persistent store, I get their change! This seems very inconsistent to me. When I set an attribute, it should stay changed until I save, no matter what value I give it. After I save, refreshObject:mergeChanges:YES can change it, but before? Core Data determines if an attribute is changed by comparing values, not by flagging attributes that I changed. (And it knows perfectly well which values I changed, c.f. [NSManagedObject changedValues]).

Of course you could argue that "changing" an attribute from "blue" to "blue" is not a change at all, and you would be right. However, I think that making the logic independent of data values makes for fewer surprises.

Here is an example of the surprises I am talking about:

// here color is "red"
[obj setValue:@"blue" forKey:@"color"];
// here "blue"
[moc refreshObject:obj mergeChanges:YES];
// still "blue"
// somebody saves "green" from a different context
[moc refreshObject:obj mergeChanges:YES];
// still "blue"
// somebody saves "blue" from a different context
[moc refreshObject:obj mergeChanges:YES];
// still "blue"
// somebody saves "red" from a different context
[moc refreshObject:obj mergeChanges:YES];
// now we're suddenly "red"??

Somebody else "hijacked" my change by saving the same value as my change, then changing it again. From my viewpoint, I changed "red" to "blue", then refreshed a few times, merging my changes. Suddenly my change disappeared. Surprise!

You can get either behaviour with the extended attributes. Use trimExtendedChanges for the Core Data way, leave it out for the sane way. Your choice.

If you implement extendedChanges as a transient attribute, you don't need the extra call backs, and you have no place to call trimExtendedChanges. Another clue that comparing values is not the natural way.