- This blog is under progress.
- How to handle threading correctly in your django app.
- How to take care of not-thread safe objects like boto sessions and resources.
- How to handle exception handling in app middleware and what are the potential problems that could happen if it is not handled properly (e.g,. the issue when removing session db in the last response stage.)
- The problem when running uwsgi in preforking mode only along with app threading. (The random behavior of http-serving suspension issue!) and how can we get around it? (https://github.com/unbit/uwsgi/issues/1925)
Until now we conclude:
– The existence of master process along with preforking mode could cause the suspension issue
– Disabling the master process with preforking solves the issue but we lose the workers management (e.g, in case of harakiri.) Moreover, we need to enable stats with APM.
– To solve the issue: we enabled master and emperror but with lazy mode (i.e., load after fork). However, this causes end_point resolver issue.
– Causes of endpoint_resolve issue:
— First off, our implementation of the boto session is not true and we should create a session and resource per process/thread. See this: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html#multithreading-multiprocessing
— In all cases, the fork() always copies the parent process with copy-on-write startegy. Now, the strange thing is that, in the preforking mode, we load and copy each process, so point to same session var, but no issues appeared!!! When we enable the lazy, still each process after copy points to same session var, but untill written, since we load it in each process, now it should have its own boto session. Given that, the issue of end_point resolver happens. Now according to initial investigation, the issue could be a bug or the issue might be related to EC2 profile itself where we cannot have more than endpoint resolver per instance (i.e., per profile).
— Another theory could be is that, becasue we create the boto session as a constant, it could somehow be shared across the processes, which lead to smth like race condition.
— Another strong theory is that:
— Before, when running uwsgi with master and preforking mode, the backgroud thread runs within the master process not in the forked workers, so the the request thread and that background thread cannot corolate with each other. In the preforking mode with master = false, we have one worker having the background thread and recieves requests, which is smth that makes the threads interleave. When running master with lazy mode (load after fork), the possibility of interleaving increases because now each process has background thread and request thread. (edited)