Copy an entity in Google App Engine datastore in Python without knowing property names at 'compile' time

Solution 1:

Here you go:

def clone_entity(e, **extra_args):
  """Clones an entity, adding or overriding constructor attributes.

  The cloned entity will have exactly the same property values as the original
  entity, except where overridden. By default it will have no parent entity or
  key name, unless supplied.

  Args:
    e: The entity to clone
    extra_args: Keyword arguments to override from the cloned entity and pass
      to the constructor.
  Returns:
    A cloned, possibly modified, copy of entity e.
  """
  klass = e.__class__
  props = dict((k, v.__get__(e, klass)) for k, v in klass.properties().iteritems())
  props.update(extra_args)
  return klass(**props)

Example usage:

b = clone_entity(a)
c = clone_entity(a, key_name='foo')
d = clone_entity(a, parent=a.key().parent())

EDIT: Changes if using NDB

Combining Gus' comment below with a fix for properties that specify a different datastore name, the following code works for NDB:

def clone_entity(e, **extra_args):
  klass = e.__class__
  props = dict((v._code_name, v.__get__(e, klass)) for v in klass._properties.itervalues() if type(v) is not ndb.ComputedProperty)
  props.update(extra_args)
  return klass(**props)

Example usage (note key_name becomes id in NDB):

b = clone_entity(a, id='new_id_here')

Side note: see the use of _code_name to get the Python-friendly property name. Without this, a property like name = ndb.StringProperty('n') would cause the model constructor to raise an AttributeError: type object 'foo' has no attribute 'n'.

Solution 2:

If you're using the NDB you can simply copy with: new_entity.populate(**old_entity.to_dict())

Solution 3:

This is just an extension to Nick Johnson's excellent code to address the problems highlighted by Amir in the comments:

1. The db.Key value of the ReferenceProperty is no longer retrieved via an unnecessary roundtrip to the datastore.
2. You can now specify whether you want to skip DateTime properties with the auto_now and/or auto_now_add flag.

Here's the updated code:

def clone_entity(e, skip_auto_now=False, skip_auto_now_add=False, **extra_args):
  """Clones an entity, adding or overriding constructor attributes.

  The cloned entity will have exactly the same property values as the original
  entity, except where overridden. By default it will have no parent entity or
  key name, unless supplied.

  Args:
    e: The entity to clone
    skip_auto_now: If True then all DateTimeProperty propertes will be skipped which have the 'auto_now' flag set to True
    skip_auto_now_add: If True then all DateTimeProperty propertes will be skipped which have the 'auto_now_add' flag set to True
    extra_args: Keyword arguments to override from the cloned entity and pass
      to the constructor.
  Returns:
    A cloned, possibly modified, copy of entity e.
  """

  klass = e.__class__
  props = {}
  for k, v in klass.properties().iteritems():
    if not (type(v) == db.DateTimeProperty and ((skip_auto_now and getattr(v, 'auto_now')) or (skip_auto_now_add and getattr(v, 'auto_now_add')))):
      if type(v) == db.ReferenceProperty:
        value = getattr(klass, k).get_value_for_datastore(e)
      else:
        value = v.__get__(e, klass)
      props[k] = value
  props.update(extra_args)
  return klass(**props)

The first if expression is not very elegant so I appreciate if you can share a better way to write it.

Solution 4:

I'm neither Python nor AppEngine guru, but couldn't one dynamically get/set the properties?

props = {}
for p in Thing.properties():
    props[p] = getattr(old_thing, p)
new_thing = Thing(**props).put()

Solution 5:

A variation inspired in Nick's answer which handles the case in which your entity has a (repeated) StructuredProperty, where the StructuredProperty itself has ComputedProperties. It can probably be written more tersely with dict comprehension somehow, but here is the longer version that worked for me:

def removeComputedProps(klass,oldDicc):
  dicc = {}
  for key,propertType in klass._properties.iteritems():
      if type(propertType) is ndb.StructuredProperty:
          purged = []
          for item in oldDicc[key]:
              purged.append(removeComputedProps(propertType._modelclass,item))
          dicc[key]=purged
      else:
          if type(propertType) is not ndb.ComputedProperty:
              dicc[key] = oldDicc[key]
  return dicc

def cloneEntity(entity):
  oldDicc = entity.to_dict() 
  klass = entity.__class__
  dicc = removeComputedProps(klass,oldDicc)
  return klass(**dicc)