在Google应用程序引擎中部署应用程序时出现内存错误

Question

我试图在谷歌应用程序引擎中部署一个FlaskAPI，但遇到了内存错误

运行gcloud app deploy时，我通过终端收到以下消息：

Updating service [default]...failed.
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build f857cba3-1bf5-4384-acec-f1ee3ba97c8b status: FAILURE.
Build error details: {"error":{"errorType":"BuildError","canonicalCode":"INVALID_ARGUMENT","errorId":"26CADD66","errorMessage":"14 May 2020 13:48:07 INFO     Arguments: ['--parser_script=/usr/local/bin/ftl.par', '--src=', '--entrypoint=', '--name=us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0', '--directory=/workspace', '--destination=/srv', '--cache-repository=us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00']\n14 May 2020 13:48:07 INFO     Unparsed arguments: ['--name=us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0', '--directory=/workspace', '--destination=/srv', '--cache-repository=us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00']\n14 May 2020 13:48:07 INFO     Executing ['/usr/local/bin/ftl.par', '--name=us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0', '--directory=/workspace', '--destination=/srv', '--cache-repository=us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d', '--cache', '--builder-output-path=\"\"', '--additional-directory=/.googleconfig', '--python-cmd=/opt/python3.7/bin/python3.7', '--pip-cmd=/env/bin/python3.7 -m pip', '--venv-cmd=/opt/python3.7/bin/python3.7 -m venv /env', '-v=DEBUG', '--base=us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00']\nINFO     FTL version python-v0.17.0\nINFO     Beginning FTL build for python\nINFO     FTL arg passed: virtualenv_dir /env\nINFO     FTL arg passed: ttl 168\nINFO     FTL arg passed: python_cmd /opt/python3.7/bin/python3.7\nINFO     FTL arg passed: cache True\nINFO     FTL arg passed: virtualenv_cmd virtualenv\nINFO     FTL arg passed: entrypoint None\nINFO     FTL arg passed: exposed_ports None\nINFO     FTL arg passed: pip_cmd /env/bin/python3.7 -m pip\nINFO     FTL arg passed: tar_base_image_path None\nINFO     FTL arg passed: export_cache_stats False\nINFO     FTL arg passed: builder_output_path \"\"\nINFO     FTL arg passed: destination_path /srv\nINFO     FTL arg passed: sh_c_prefix False\nINFO     FTL arg passed: base us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00\nINFO     FTL arg passed: cache_key_version v0.17.0\nINFO     FTL arg passed: cache_salt \nINFO     FTL arg passed: cache_repository us.gcr.io/aerospider/app-engine-tmp/build-cache/ttl-7d\nINFO     FTL arg passed: venv_cmd /opt/python3.7/bin/python3.7 -m venv /env\nINFO     FTL arg passed: name us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0\nINFO     FTL arg passed: global_cache False\nINFO     FTL arg passed: upload True\nINFO     FTL arg passed: fail_on_error True\nINFO     FTL arg passed: output_path None\nINFO     FTL arg passed: directory /workspace\nINFO     FTL arg passed: additional_directory /.googleconfig\nINFO     FTL arg passed: verbosity DEBUG\nINFO     starting: full build\nINFO     starting: builder initialization\nINFO     Loading Docker credentials for repository 'us.gcr.io/gae-runtimes/python37:python37_20200405_3_7_7_RC00'\nINFO     Loading Docker credentials for repository 'us.gcr.io/aerospider/app-engine-tmp/app/ttl-2h:7e15a1a4-8ddf-4fdb-b668-f8d48552bfa0'\nINFO     builder initialization took 0 seconds\nINFO     starting: build process for FTL image\nINFO     starting: checking_cached_interpreter_layer\nINFO     starting: check python version\nINFO     `python version` full cmd:\n/opt/python3.7/bin/python3.7 --version\nINFO     `python version` stderr:\n\nINFO     check python version took 0 seconds\nDEBUG   .

在检查GCP上的构建详细信息时，我得到以下信息：

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/bin/ftl.par/__main__.py", line 65, in <module>
  File "/usr/local/bin/ftl.par/__main__.py", line 54, in main
  File "/usr/local/bin/ftl.par/__main__/ftl/python/builder.py", line 123, in Build
  File "/usr/local/bin/ftl.par/__main__/ftl/common/layer_builder.py", line 42, in BuildLayer
  File "/usr/local/bin/ftl.par/__main__/ftl/common/ftl_util.py", line 116, in zip_dir_to_layer_sha
MemoryError

我的发言如下：

scikit-learn==0.20.3
pandas==1.0.3
xgboost==1.0.2
fuzzywuzzy==0.18.0
python-Levenshtein==0.12.0
gensim==3.8.1
nltk==3.3
Flask==1.1.2
flask-cors==3.0.7
requests==2.23.0

我的发言如下：

runtime: python37
instance_class: F4_1G
resources:
  cpu: 3
  memory_gb: 16
  disk_size_gb: 30

我尝试了很多与CPU和内存相关的配置，但都没有成功

我想问题是我试图加载一个3.5gb的word2vec模型：

model = gensim.models.KeyedVectors.load_word2vec_format('./GoogleNews-vectors-negative300.bin', binary=True)

Answer 1

在应用程序引擎中app.yaml可以使用instance_class和resources之间的一个标记，因为这两个标记都定义了将分配给应用程序引擎实例的机器类型

在App Engine Standar中，使用的标记是前面提到的“instance_class”here所有具有instance class的iInstance都带有1GB的磁盘，因此在尝试上载大于1GB的文件时可能会出现错误

如果您想更多地指定将要使用的机器类型，如内存、磁盘大小等，则需要在[App Engine Flex]中部署，您可以在其中指定resources标记

要在flex上部署app.yaml需要指定环境是flex，以及在本例中实例的自定义配置

runtime: python
env: flex

runtime_config:
  python_version: 3

resources:
  cpu: 4
  memory_gb: 16
  disk_size_gb: 30

我将CPU从3修改为4，因为该值应为1或偶数

认为应用程序引擎Flex比标准更昂贵，需要更多的时间来部署。

有时，这可能会发送Error Response: [4] Timed out waiting for the app infrastructure to become healthy.错误，预计可能会抛出该错误，如果您遇到此问题，请稍后再试，如果再次发生，请尝试在其他区域部署该应用