Runicorn Flask App Term Signals

jamiecattocodePRO

a year ago

I’m trying to deploy a Flask app using Runicorn Web Server and a Docker image.

However I am repeatedly getting term signals which stop the app from running. Here are some deployment logs showing this happening:
[2024-02-26 19:02:01 +0000] [7] [INFO] Starting gunicorn 21.2.0
Feb 26 16:02:01
thassistant [2024-02-26 19:02:01 +0000] [7] [INFO] Listening at: http://0.0.0.0:8080 (7) Feb 26 16:02:01 thassistant
[2024-02-26 19:02:01 +0000] [7] [INFO] Using worker: gthread
Feb 26 16:02:01
thassistant [2024-02-26 19:02:01 +0000] [8] [INFO] Booting worker with pid: 8 Feb 26 16:02:01 thassistant
[2024-02-26 19:02:01 +0000] [9] [INFO] Booting worker with pid: 9
Feb 26 16:02:30
thassistant [2024-02-26 19:02:30 +0000] [7] [INFO] Handling signal: term Feb 26 16:02:30 thassistant
[2024-02-26 19:02:30 +0000] [8] [INFO] Worker exiting (pid: 8)
Feb 26 16:02:30
thassistant OpenAI version is compatible. Feb 26 16:02:30 thassistant
[2024-02-26 19:02:30 +0000] [9] [INFO] Worker exiting (pid: 9)
Feb 26 16:02:30
thassistant OpenAI version is compatible. Feb 26 16:02:31 thassistant
[2024-02-26 19:02:31 +0000] [7] [INFO] Shutting down: Master

I cannot find any reason why this is happening – my plan allows me enough memory and storage for the Docker image to run correctly.

Solved

11 Replies

Hey! I'm going to un-mark this thread as private so the community can chip in on project-related help, as this is not a Railway issue.

[INFO] Handling signal: term

Looking at the log here, it looks like something is shutting down the process in your app? This might be an incorrect start cmd where it runs once and exits. Couple of questions to help narrow this down:

  • How are you starting gunicorn?

  • Do you have a timeout set?

  • Do you have the PORT set to the one Railway provides (seems you're listening on 8080)?


jamiecattocodePRO

a year ago

  1. The Dockerfile entry point is:
    ENTRYPOINT ["gunicorn", "--config", "gunicorn_config.py", "main:app"]

  2. There's no timeout set in my gunicorn config. The gunicorn_config.py file:
    import os

Use Railway-provided PORT environment variable, with a fallback to 8080 if not set

port = os.getenv('PORT', '8080')

workers = int(os.getenv('GUNICORNPROCESSES', '2')) threads = int(os.getenv('GUNICORNTHREADS', '4'))
bind = f"0.0.0.0:{port}"

forwardedallowips = '*'
secureschemeheaders = {'X-Forwarded-Proto': 'https'}

  1. I have been using the PORT variable in my code as you can see above e.g. in the gunicorn config. Also in the app's entry point as you see here:
    if name == 'main':
    app.run(host='0.0.0.0', port=os.getenv("PORT", default=8080))

However, I did not include the port option in the gunicorn start command as mentioned in the docs you linked. I will try this right now and report back.


jamiecattocodePRO

a year ago

Actually I just realised I misread the docs and this port option is not for gunicorn, it's for uvicorn - so yes, I have the PORT set to the one provided by Railway.

I think this is why the health checks are passing - so it's strange that the term signal is sent shortly after the app starts.

I'm also receiving print statements which are executed inside my main.py such as this one:
print("OpenAI version is compatible.")


Make sure the main:app you're passing into Gunicorn is indeed an instance of the app, i.e. it should be exactly the instance you get from calling Flask(__name__).


jamiecattocodePRO

a year ago

That is indeed the case - snippet from main.py:
app = Flask(__name__) CORS(app) @app.route("/health", methods=["GET"]) def health(): return "Health check successful", 200


Could you share your full Dockerfile and [main.py](main.py)?


jamiecattocodePRO

a year ago

Dockerfile:

FROM python:3.12-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    libffi-dev \
    python3-dev \
    gfortran \
    liblapack-dev \
    libblas-dev

RUN apt-get clean && rm -rf /var/lib/apt/lists/*

RUN pip install --upgrade pip && pip install poetry==1.6.1

RUN mkdir /app
COPY . /app
WORKDIR /app
EXPOSE 8080
RUN poetry config virtualenvs.create false
RUN poetry install
ENTRYPOINT ["gunicorn", "--config", "gunicorn_config.py", "main:app"]

main.py:

import os
import json
import traceback
from time import sleep
import dotenv
from openai import OpenAI
import openai
from flask import Flask, request, jsonify, abort
from flask_cors import CORS
import pandas as pd
from packaging import version
from functions import *
from objects import Hamper, ProductSuggester, HamperManager, HamperBuildingTeam, Assistant


dotenv.load_dotenv()

# Check for OpenAI API key
assert os.environ.get('OPENAI_API_KEY'), "OPENAI_API_KEY not found in .env file."
OPENAI_API_KEY = os.environ.get('OPENAI_API_KEY')

# Check OpenAI version compatibility
required_version = version.parse("1.1.1")
current_version = version.parse(openai.__version__)
OPENAI_API_KEY = os.environ['OPENAI_API_KEY']
ASSISTANT_ID = os.environ['ASSISTANT_ID']
if current_version < required_version:
    raise ValueError(
        f"Error: OpenAI version {openai.__version__} is less than the required version 1.1.1"
    )
else:
    print("OpenAI version is compatible.")

# Initialize OpenAI client
client = OpenAI(api_key=OPENAI_API_KEY)


# Create Flask app
app = Flask(__name__)
CORS(app)

@app.errorhandler(400)
def bad_request(e):
    error_description = e.description
    return jsonify(error_description), 400

@app.errorhandler(500)
def server_error(e):
    error_description = e.description
    return jsonify(error_description), 500

@app.route("/health", methods=["GET"])
def health():
    return "Health check successful", 200

@app.route("/rank", methods=["POST"])
def rank():
    # redacted

@app.route("/oop", methods=["POST"])
def oop():
    # redacted


@app.route("/test", methods=["POST"])
def test():
    # redacted

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=os.getenv("PORT", default=8080))

a year ago

your deploy logs speak of gthread, where is that being set?

have you overwrote the ENTRYPOINT via the service settings start command field?


jamiecattocodePRO

a year ago

@brody I've not put anything into the start command field in service settings.
Gthread is set in my gunicorn_config.py referenced in the entry point command - snippet below:

port = os.getenv('PORT', '8080')

workers = int(os.getenv('GUNICORN_PROCESSES', '2'))
threads = int(os.getenv('GUNICORN_THREADS', '4'))
bind = f"0.0.0.0:{port}"

jamiecattocodePRO

a year ago

Updates:

  • Created a hello world Flask app with the same gunicorn config file and Dockerfile. Although it's working locally, having the same problem when uploading to Railway (term signals shortly after deployment and passed health checks).

  • Commented out the threads in the gunicorn config. Still having the same problem.

  • Changed the ENTRYPOINT to: ENTRYPOINT gunicorn -w 2 -b :${PORT:-8080} main:app (bypassing the gunicorn config file) and this was SUCCESSFUL (on the hello world app). Made this change on my real app and the term signal problem still persisted in the exact same way


Could you try changing ENTRYPOINT to CMD? Also, any reason you're using a Dockerfile instead of letting nixpacks build it for you?


Status changed to Solved railway[bot] over 1 year ago