11 Things To Understand Before Migrating Your Python 2 GAE App To Python 3

28 Jan 2019

If you're reading this it might be because you have at least one web app running on the old GAE Standard Environment that supports only Python 2 (and of course Java, PHP and Go).

Meanwhile, the new generation of the GAE Standard Environment has arrived and it enables us to build web apps using Python 3. But switching from the old GAE environment to the new one is not just about changing a few lines of code in a Python 2 web app to make it compatible with Python 3.

Instead, the new GAE Standard Environment brought some significant changes that you should be aware of if you are planning to migrate your Python web app from the old to the new Standard Environment. Let's take a look at them.

1. requirements.txt

Let's start with good news. You won't have to store all your third-party libraries in a special libs folder anymore!

Instead, you'll define them in the requirements.txt file, just like in any regular Python project. The new GAE Standard environment will automatically detect this file and install the libraries.

It was about time, right?

2. Datastore vs. Firestore

Firestore is basically a new generation Datastore. So, should you migrate your data from Datastore to Firestore?

Actually, Google will do this for you in the background. The only difference is that if you used Datastore before, you would still need to use Firestore in a "Datastore mode" (or maybe better put: in the Datastore way).

You can read more about it here: Automatic Upgrade to Cloud Firestore.

So the good news is that you won't have to do anything to migrate data from Datastore to Firestore.

But that doesn't mean you won't have to do anything when migrating your Python 2 GAE app code to Python 3. The main reason is that the ndb library is not (yet?) supported on Python 3.

Update (11 Apr 2019): According to a talk at the Cloud Next 2019 conference, Google is actively working on a Python 3 ndb library.

Instead, you have to use a new library called google-cloud-datastore. This library helps you communicate with your Datastore (or with Firestore in the Datastore Mode). But the new library will require you to write the query code a bit differently than you were used to. See some examples here and here.

There are some gotchas to consider though. One is that you'll have to put data in the Datastore as dictionaries (JSON) and you'll also receive data back as dictionaries (not objects, as you did with the ndb library). So don't forget to add the dict_to_obj() method to your classes. ;)

index.yaml for creating composite indexes still works as before.

3. Running different Python instances

More good news. You can run your new Python 3 instance next to the Python 2 instance in the same Cloud project.

This means you don't have to create a new Google Cloud project, but instead you can run Python 3 instance(s) in the existing project.

For example, you would be able to create a Python 3 instance with a special version name (like new or beta). This instance, of course, would have access to the same database as your Python 2 instance does. And you would be able to give your users a preview of the new/beta web app on the separate appspot.com subdomain (e.g.: https://new-dot-projectname.appspot.com).

Pro-tip: just make sure to use the --no-promote flag when you deploy your Python 3 instance:

gcloud app deploy --version new --no-promote

This means the new version will not become the default version.

4. webapp2

I personally really liked the webapp2 Python microframework, mainly because of its simplicity.

But it's time to say goodbye.

Why? Because Google isn't really doing anything to actively maintain it. They don't even consider it as their official product, they "only host it" on their GitHub account and "can review any pull request" made by the community volunteers.

Even though webapp2 3.0.0b1 (beta) does support Python 3, I would advise against using it. Instead, you should rewrite your web app into Flask.

You'll have to rewrite large parts of your web app anyway because many of the old Google Python 2 libraries (such as the ndb) won't work on the Python 3 environment. So why not change the whole framework along.

5. Cron jobs

This is the area where you won't have to change anything (for now).

According to this documentation you can use cron.yaml in a Python 3 GAE app in the same way as you have used it in the Python 2 environment.

But be aware that Google created a new service called Cloud Scheduler (still in beta) which is set to replace App Engine Cron Jobs.

It is not yet clear if you'll be able to use Cloud Scheduler via the cron.yaml file. But even if you won't, you'll be able to create scheduling jobs either visually at the GCP Console or using the gcloud command line tool. So no worries with this one.

6. Task Queue API (background tasks)

Here's another GAE Standard library that cannot be used in the Python 3 environment: App Engine task queues.

Instead, you'll have to use Cloud Tasks via the google-cloud-tasks Python 3 library. Beware: the service is still in beta.

7. Memcache

Memcache is not (yet) available for the Python 3 GAE Standard Environment. But according to Google, they are "actively working on making a Memcache solution available to the Python 3 runtime."

If you'd want to migrate your GAE app from Python 2 to Python 3 today, you'd have to use Redis instead (see Cloud Memorystore).

Update (11 Apr 2019): There were no mentions of memcache for the Python 3 runtime at Cloud Next 2019 conference. Cloud Memorystore was mentioned as a replacement when migrating from py2 to py3, but the Memorystore documentation does not specify how to connect to it from your Python 3 GAE web app.

8. Mail API

GAE allowed you to send emails via the Mail API. This worked for smaller web apps, but for anything serious, you had to turn to a professional third-party solution such as Sendgrid or Mailgun.

So, in case you did use the Mail API - guess what? Not supported on Python 3. Time to say goodbye. :)

9. Users API

This was one of the nicest features of the old GAE Standard Environment. Enabling Google login with just a few lines of code via the Users API.

The good old days are over and the Users API is not available on Python 3. Instead, you'll have to use Firebase Authentication or, even better, the Requests OAuthlib.

10. dev_appserver.py

You can run your new Python 3 web app with the "old" dev_appserver.py on localhost. Which is kind of funny, because gcloud and dev_appserver.py still require Python 2 to run. So you still need both Python versions installed.

Also be aware that the Datastore emulator is moving out of dev_appserver.py and is already available as an extra feature in gcloud (although still in beta for now). See this link for more info.

11. No plans for deprecating the old Standard Environment

Finally, just to make you a little bit less stressed about everything: Google has no official plans to deprecate the old Standard Environment.

Even though Python 2 has the "end of life" in 2020, you will still be able to use it on Google App Engine. According to some Google insider, too much of the Google's infrastructure still relies on Python 2 so Google will take care of patching Python 2 if needed.

To conclude, some of the services a normal project would need on GAE are not available yet (Memcache) and others are still in beta (Scheduler, Tasks, etc.). So there's no hurry with the migration. But you should start planning it.


Let me know in the comments what else one should be aware of when migrating, and I will update the article if needed. Thanks!

Useful links:


Please share this article via your social media channels:

Comments (5)

Your comment
Enter your real email address, because you'll have to verify your comment (anti-spam).

"ndb" is currently being rewritten to support Python 3 at https://github.com/googleapis/python-ndb. There are going to be some breaking changes as a result, but it should assist in migrating forward.

- Chris on 15 Mar 2019

Hey Bruce! The "ndb" is NOT a database, but instead a library that helps you access the Datastore database. You can access the same Datastore database in py2 and py3 GAE standard environments (I already tested it and it works). The only difference is that you can't use the ndb library in the py3 environment. You will have to completely rewrite your web app in order to migrate from py2 to py3 environment. Unless Google makes the ndb library to work on py3 too. The good news is that there's no rush. But I suggest you do start testing the py3 environment with some test web apps.

- Matt on 24 Feb 2019

I have an existing (Python 2.7) standard GAE with 60,000 users' data in an ndb database. From what you say, I'm not clear on whether my ndb database is the same as what you call Datastore, so it's also not clear to me what I will have to do to access old user data in a new Python 3 environment, nor how to code the Python server appropriately.

- Bruce Sherwood on 12 Feb 2019

@David Freeman: with the new google-cloud-datastore library you enter data into the Datastore in a JSON format. So you can create custom models in a normal Pythonic way and add the default behavior there. Or don't even use model classes at all.

- Matt on 29 Jan 2019

Great article, thanks. Any thoughts on subclasses? I have many, many instances of myid = fsintegerproperty(verbose_name='My Id Number') where fsintegerproperty is an extension of ndb.IntergerProperty Thanks David

- David Freeman on 29 Jan 2019

Newsletter


Don't miss any new post! Sign up for our newsletter (we promise not to spam)!