We were recently working on large scale migration batches (on ~550.000 entities) using NDB on App Engine. As we are limited to 2048MB of ram, memory consumption is a big subject when doing this kind of things.
When trying to lower it, I noticed something that I found an unexpected (in my opinion) behaviour of fetch_page.
I would have expected the memory consumption to be close to a plateau, as the new entities will weight approx. the same as the previous ones. I found that it instead grows linearly until it ends, even without processing the results at all.

Environment details

1. Specify the API: Datastore using NDB 1.11.1
2. OS type and version: Ubuntu 20.04.3 LTS x86_64
3. Python version and virtual environment information: 3.8.10
4. google-cloud- version: 1.11.1

Code example

import os

import psutil  # https://pypi.org/project/psutil/
from google.cloud import ndb

from app.models import Enterprise

print(ndb.__version__)


def memory_usage():
    pid = os.getpid()
    process = psutil.Process(pid)
    return process.memory_info().rss / 10 ** 6


with ndb.Client().context():
    more = True
    cursor = None
    standard_usage = memory_usage()
    while more:
        # 'Enterprise' is rather a big entity in our datastore, which makes the memory consumption increase more visible
        result, cursor, more = Enterprise.query().fetch_page(200, start_cursor=cursor, use_cache=False, use_global_cache=False)
        print(memory_usage() - standard_usage)

Output

I truncated the output, as it loops ~400 times. Using the best App Engine servers (B8) we would not be able to finish that script.

37.437439999999995
51.515392000000006
60.62079999999999
74.268672
83.07507199999999
92.27878399999999
101.040128
114.597888
129.179648
138.15398399999998
136.94976000000003
156.188672
164.958208
[...]
3663.07328
3672.14592
3681.140736
3690.008576
3698.93376
3708.043264
3710.746624