Google App Engine Datastore - OneToMany update operation fails due to bug in JPA datanucleus

NOTICE  Google Cloud Platform documentation states that JPA and JDO should not be used with Datastore (see: JPA overview, JDO overview) Developers should consider using low level Datastore API (for simple cases) or use Objectify library which is specifically created for Datastore.

This issue has been reported to GAE issue tracker: https://code.google.com/p/googleappengine/issues/detail?id=11995 Code to reproduce this issue is available HERE

When this issue occurs

GAE SDK Version: 1.9.17 and higher (probably lower versions too - not tested)
DataNucleus JPA: 3.1.3, 3.1.4
DataNucleus-Appengine: 2.1.1, 2.1.2

When you use JPA to persist your domain objects and you try to update OneToMany relation by removing old objects and adding new objects.
Ex. "User" class has "items" field - classic one to many relation as shown below:

User has many Items (OneToMany relation represented by "User.items" field)

Why it occurs?

This issue is probably related to issues with updating collections resolved in datanucleus 3.2.1: http://www.datanucleus.org/news/access_platform_3_2_1.html.

When you fetch an object and its OneToMany items and then you remove all items to add new items, JPA entity manger tracks these actions.
DataNucleus implementation unfortunatelly schedules these operations in wrong order - instead of remove all and add, it executes it as add and then remove all.

#	Expected order	Execution order in DataNucleus
1	remove all items	add new items
2	add new items	remove all items
RESULT	new items only	empty list

NOTICE: GoogleAppEngine is compatible with Datanucleus up to v. 3.1.x (https://code.google.com/p/datanucleus-appengine/wiki/Compatibility). Higher versions are not supported (and there is no plan).

How to reproduce this issue

To reproduce this issue, find existing object, then remove all OneToMany items it has, and finally add new items. Then commit transaction.
You will find that your object will end up with empty "items" list when you will fetch it again.

        EntityManager em = emf.createEntityManager();
        EntityTransaction tx = em.getTransaction();
        tx.begin();
        try {
            User user = em.find(User.class, userId);
        
            Item item1 = new Item("Tomatoe", 20.0d);
            Item item2 = new Item("Carrot", 15.0d);
        
            user.getItems().clear();
            //em.flush(); // WORKAROUND: Flush "clear" 
                          // operation to be executed in proper order
            user.addItem(item1);
            user.addItem(item2);
        
            tx.commit();
        } finally {
            if( tx.isActive() ) {
                tx.rollback();
            }
            em.close();
        }

 

How to workaround this issue

To workaround this problem, you should flush "clear" operation to be executed instantly after it's requested on the object.
So, adding em.flush(); just after user.getItems().clear(); fixes this issue.