28 Jan 2019
If you're reading this it might be because you have at least one web app running on the old GAE Standard Environment that supports only Python 2 (and of course Java, PHP and Go).
Meanwhile, the new generation of the GAE Standard Environment has arrived and it enables us to build web apps using Python 3. But switching from the old GAE environment to the new one is not just about changing a few lines of code in a Python 2 web app to make it compatible with Python 3.
Instead, the new GAE Standard Environment brought some significant changes that you should be aware of if you are planning to migrate your Python web app from the old to the new Standard Environment. Let's take a look at them.
Let's start with good news. You won't have to store all your third-party libraries in a special
libs folder anymore!
Instead, you'll define them in the
requirements.txt file, just like in any regular Python project. The new GAE Standard environment will automatically detect this file and install the libraries.
It was about time, right?
Firestore is basically a new generation Datastore. So, should you migrate your data from Datastore to Firestore?
Actually, Google will do this for you in the background. The only difference is that if you used Datastore before, you would still need to use Firestore in a "Datastore mode" (or maybe better put: in the Datastore way).
You can read more about it here: Automatic Upgrade to Cloud Firestore.
So the good news is that you won't have to do anything to migrate data from Datastore to Firestore.
But that doesn't mean you won't have to do anything when migrating your Python 2 GAE app code to Python 3. The main reason is that the
ndb library is not (yet?) supported on Python 3.
Update (11 Apr 2019): Google is actively working on a Python 3 ndb library: https://github.com/googleapis/python-ndb.
Instead, you have to use a new library called
google-cloud-datastore. This library helps you communicate with your Datastore (or with Firestore in the Datastore Mode). But the new library will require you to write the query code a bit differently than you were used to. See some examples here and here.
There are some gotchas to consider though. One is that you'll have to put data in the Datastore as dictionaries (JSON) and you'll also receive data back as dictionaries (not objects, as you did with the
ndb library). So don't forget to add the
dict_to_obj() method to your classes. ;)
index.yamlfor creating composite indexes still works as before.
More good news. You can run your new Python 3 instance next to the Python 2 instance in the same Cloud project.
This means you don't have to create a new Google Cloud project, but instead you can run Python 3 instance(s) in the existing project.
For example, you would be able to create a Python 3 instance with a special version name (like
beta). This instance, of course, would have access to the same database as your Python 2 instance does. And you would be able to give your users a preview of the new/beta web app on the separate
appspot.com subdomain (e.g.:
Pro-tip: just make sure to use the
--no-promote flag when you deploy your Python 3 instance:
gcloud app deploy --version new --no-promote
This means the
new version will not become the default version.
I personally really liked the webapp2 Python microframework, mainly because of its simplicity.
But it's time to say goodbye.
Why? Because Google isn't really doing anything to actively maintain it. They don't even consider it as their official product, they "only host it" on their GitHub account and "can review any pull request" made by the community volunteers.
Even though webapp2 3.0.0b1 (beta) does support Python 3, I would advise against using it. Instead, you should rewrite your web app into Flask.
You'll have to rewrite large parts of your web app anyway because many of the old Google Python 2 libraries (such as the
ndb) won't work on the Python 3 environment. So why not change the whole framework along.
This is the area where you won't have to change anything (for now).
According to this documentation you can use
cron.yaml in a Python 3 GAE app in the same way as you have used it in the Python 2 environment.
But be aware that Google created a new service called Cloud Scheduler (still in beta) which is set to replace App Engine Cron Jobs.
It is not yet clear if you'll be able to use Cloud Scheduler via the
cron.yaml file. But even if you won't, you'll be able to create scheduling jobs either visually at the GCP Console or using the
gcloud command line tool. So no worries with this one.
Here's another GAE Standard library that cannot be used in the Python 3 environment: App Engine task queues.
Instead, you'll have to use Cloud Tasks via the
google-cloud-tasks Python 3 library.
Memcache will not be available for the Python 3 GAE Standard Environment (2nd Gen).
Instead, you'd have to use Redis (see Cloud Memorystore).
How to connect your GAE app with Cloud Memorystore? VPC Connectors.
Using Memorystore will be more complex than using memcache on the old GAE, unfortunately. It will also be more expensive, because you'll have to run a Redis instance.
The cheapest Cloud Memorystore Redis instance costs $0.049/hour, which means you'll be charged at least $36/month for each GAE app that uses the Memorystore (because you can't share Memorystore across many projects).
GAE allowed you to send emails via the Mail API. This worked for smaller web apps, but for anything serious, you had to turn to a professional third-party solution such as Sendgrid or Mailgun.
So, in case you did use the Mail API - guess what? Not supported on Python 3. Time to say goodbye. :)
This was one of the nicest features of the old GAE Standard Environment. Enabling Google login with just a few lines of code via the Users API.
You can run your new Python 3 web app with the "old"
dev_appserver.py on localhost. Which is kind of funny, because gcloud and
dev_appserver.py still require Python 2 to run. So you still need both Python versions installed.
Also be aware that the Datastore emulator is moving out of
dev_appserver.py and is already available as an extra feature in gcloud (although still in beta for now). See this link for more info.
Finally, just to make you a little bit less stressed about everything: Google has no official plans to deprecate the old Standard Environment.
Even though Python 2 has the "end of life" in 2020, you will still be able to use it on Google App Engine. According to some Google insider, too much of the Google's infrastructure still relies on Python 2 so Google will take care of patching Python 2 if needed.
To conclude, some of the services a normal project would need on GAE are not available yet (Memcache) and others are still in beta (Scheduler, Tasks, etc.). So there's no hurry with the migration. But you should start planning it.
Let me know in the comments what else one should be aware of when migrating, and I will update the article if needed. Thanks!
(Last update: 4 Oct 2019)
Very helpful for all of us with py 2 GAE aps. Will be great if it is updated from time to time. Thanks!
- acegl on 01 Oct 2019
Do we have any alternatives for ndb tasklets in 2nd gen App Engine?
- Biren Prajapati on 05 Jul 2019
Cloud Tasks is GA since April. This is a good article, worth maintaining and updating
- Michael on 28 Jun 2019
"ndb" is currently being rewritten to support Python 3 at https://github.com/googleapis/python-ndb. There are going to be some breaking changes as a result, but it should assist in migrating forward.
- Chris on 15 Mar 2019
Hey Bruce! The "ndb" is NOT a database, but instead a library that helps you access the Datastore database. You can access the same Datastore database in py2 and py3 GAE standard environments (I already tested it and it works). The only difference is that you can't use the ndb library in the py3 environment. You will have to completely rewrite your web app in order to migrate from py2 to py3 environment. Unless Google makes the ndb library to work on py3 too. The good news is that there's no rush. But I suggest you do start testing the py3 environment with some test web apps.
- Matt on 24 Feb 2019
I have an existing (Python 2.7) standard GAE with 60,000 users' data in an ndb database. From what you say, I'm not clear on whether my ndb database is the same as what you call Datastore, so it's also not clear to me what I will have to do to access old user data in a new Python 3 environment, nor how to code the Python server appropriately.
- Bruce Sherwood on 12 Feb 2019
@David Freeman: with the new google-cloud-datastore library you enter data into the Datastore in a JSON format. So you can create custom models in a normal Pythonic way and add the default behavior there. Or don't even use model classes at all.
- Matt on 29 Jan 2019
Great article, thanks. Any thoughts on subclasses? I have many, many instances of myid = fsintegerproperty(verbose_name='My Id Number') where fsintegerproperty is an extension of ndb.IntergerProperty Thanks David
- David Freeman on 29 Jan 2019