Skip to content

NORTH dockerspawner is not creating jupyterhub container

Hey Nomad,

after chatting and debugging a bit with @afekete i am still stuck. This happened after updating to to 1.2.0-dev. On non updated versions it still works. When i launch a jupyter container i get:

nomad_oasis_north | 2023-05-12T11:09:03.963605963Z [I 2023-05-12 11:09:03.963 JupyterHub app:2849] JupyterHub is now running at http://:9000/nomad-oasis/north
nomad_oasis_north | 2023-05-12T11:09:43.635811535Z [W 2023-05-12 11:09:43.635 JupyterHub log:189] 404 GET /nomad-oasis/north/hub/api/users/michaelgoette/servers/jupyter/progress (nomad-service@::ffff:192.168.48.5) 14.35ms
nomad_oasis_north | 2023-05-12T11:09:43.647857907Z [W 2023-05-12 11:09:43.647 JupyterHub log:189] 404 GET /nomad-oasis/north/hub/api/users/michaelgoette/servers/nionswift/progress (nomad-service@::ffff:192.168.48.5) 3.80ms
nomad_oasis_north | 2023-05-12T11:09:43.659415848Z [W 2023-05-12 11:09:43.659 JupyterHub log:189] 404 GET /nomad-oasis/north/hub/api/users/michaelgoette/servers/mpes/progress (nomad-service@::ffff:192.168.48.5) 3.64ms
nomad_oasis_north | 2023-05-12T11:09:43.667369183Z [W 2023-05-12 11:09:43.667 JupyterHub log:189] 404 GET /nomad-oasis/north/hub/api/users/michaelgoette/servers/nexustools/progress (nomad-service@::ffff:192.168.48.5) 3.52ms
nomad_oasis_north | 2023-05-12T11:09:43.675099187Z [W 2023-05-12 11:09:43.674 JupyterHub log:189] 404 GET /nomad-oasis/north/hub/api/users/michaelgoette/servers/ellips/progress (nomad-service@::ffff:192.168.48.5) 3.41ms
nomad_oasis_north | 2023-05-12T11:09:43.684064034Z [W 2023-05-12 11:09:43.683 JupyterHub log:189] 404 GET /nomad-oasis/north/hub/api/users/michaelgoette/servers/xps/progress (nomad-service@::ffff:192.168.48.5) 3.56ms
nomad_oasis_north | 2023-05-12T11:09:43.702304658Z [W 2023-05-12 11:09:43.702 JupyterHub log:189] 404 GET /nomad-oasis/north/hub/api/users/michaelgoette/servers/webtop/progress (nomad-service@::ffff:192.168.48.5) 4.05ms
nomad_oasis_north | 2023-05-12T11:09:43.713002220Z [W 2023-05-12 11:09:43.712 JupyterHub log:189] 404 GET /nomad-oasis/north/hub/api/users/michaelgoette/servers/apmtools/progress (nomad-service@::ffff:192.168.48.5) 3.73ms
nomad_oasis_north | 2023-05-12T11:09:43.720782870Z [W 2023-05-12 11:09:43.720 JupyterHub log:189] 404 GET /nomad-oasis/north/hub/api/users/michaelgoette/servers/abtem/progress (nomad-service@::ffff:192.168.48.5) 3.41ms
nomad_oasis_north | 2023-05-12T11:09:43.728316580Z [W 2023-05-12 11:09:43.728 JupyterHub log:189] 404 GET /nomad-oasis/north/hub/api/users/michaelgoette/servers/fiji/progress (nomad-service@::ffff:192.168.48.5) 3.48ms
nomad_oasis_north | 2023-05-12T11:09:43.735499522Z [W 2023-05-12 11:09:43.735 JupyterHub log:189] 404 GET /nomad-oasis/north/hub/api/users/michaelgoette/servers/frwr/progress (nomad-service@::ffff:192.168.48.5) 3.35ms
nomad_oasis_north | 2023-05-12T11:09:45.159724733Z [W 2023-05-12 11:09:45.159 JupyterHub log:189] 404 GET /nomad-oasis/north/hub/api/users/michaelgoette (nomad-service@::ffff:192.168.48.5) 4.18ms
nomad_oasis_north | 2023-05-12T11:09:45.185369149Z [I 2023-05-12 11:09:45.185 JupyterHub log:189] 201 POST /nomad-oasis/north/hub/api/users/michaelgoette (nomad-service@::ffff:192.168.48.5) 21.59ms
nomad_oasis_north | 2023-05-12T11:09:45.205949273Z [W 2023-05-12 11:09:45.205 JupyterHub log:189] 404 GET /nomad-oasis/north/hub/api/users/michaelgoette/servers/jupyter/progress (nomad-service@::ffff:192.168.48.5) 4.73ms
nomad_oasis_north | 2023-05-12T11:09:45.237338388Z [I 2023-05-12 11:09:45.237 JupyterHub provider:574] Creating oauth client jupyterhub-user-michaelgoette-jupyter
nomad_oasis_north | 2023-05-12T11:09:45.266328521Z [I 2023-05-12 11:09:45.266 JupyterHub dockerspawner:988] Container 'noamd_oasis_north-michaelgoette--jupyter' is gone
nomad_oasis_north | 2023-05-12T11:09:45.318152266Z [I 2023-05-12 11:09:45.317 JupyterHub dockerspawner:1272] Created container noamd_oasis_north-michaelgoette--jupyter (id: a6a5581) from image jupyter/datascience-notebook
nomad_oasis_north | 2023-05-12T11:09:45.318254123Z [I 2023-05-12 11:09:45.318 JupyterHub dockerspawner:1296] Starting container noamd_oasis_north-michaelgoette--jupyter (id: a6a5581)
nomad_oasis_north | 2023-05-12T11:09:48.659880087Z [E 2023-05-12 11:09:48.659 JupyterHub user:718] Unhandled error starting michaelgoette's server: 404 Client Error for http+docker://localhost/v1.42/containers/a6a5581ae13d502f10495fab4ec28e7a8442ac51d1d139bc50ccc6b0488e6c8b/json: Not Found ("No such container: a6a5581ae13d502f10495fab4ec28e7a8442ac51d1d139bc50ccc6b0488e6c8b")
nomad_oasis_north | 2023-05-12T11:09:48.662142913Z [I 2023-05-12 11:09:48.662 JupyterHub dockerspawner:988] Container 'noamd_oasis_north-michaelgoette--jupyter' is gone
nomad_oasis_north | 2023-05-12T11:09:48.662310625Z [W 2023-05-12 11:09:48.662 JupyterHub dockerspawner:963] Container not found: noamd_oasis_north-michaelgoette--jupyter
nomad_oasis_north | 2023-05-12T11:09:48.689539041Z [E 2023-05-12 11:09:48.686 JupyterHub web:1798] Uncaught exception POST /nomad-oasis/north/hub/api/users/michaelgoette/servers/jupyter (::ffff:192.168.48.5)
nomad_oasis_north | 2023-05-12T11:09:48.689555779Z     HTTPServerRequest(protocol='http', host='north:9000', method='POST', uri='/nomad-oasis/north/hub/api/users/michaelgoette/servers/jupyter', version='HTTP/1.1', remote_ip='::ffff:192.168.48.5')
nomad_oasis_north | 2023-05-12T11:09:48.689559756Z     Traceback (most recent call last):
nomad_oasis_north | 2023-05-12T11:09:48.689562664Z       File "/usr/local/lib/python3.9/site-packages/tornado/web.py", line 1713, in _execute
nomad_oasis_north | 2023-05-12T11:09:48.689565894Z         result = await result
nomad_oasis_north | 2023-05-12T11:09:48.689568570Z       File "/usr/local/lib/python3.9/site-packages/jupyterhub/apihandlers/users.py", line 472, in post
nomad_oasis_north | 2023-05-12T11:09:48.689571470Z         await self.spawn_single_user(user, server_name, options=options)
nomad_oasis_north | 2023-05-12T11:09:48.689574212Z       File "/usr/local/lib/python3.9/site-packages/jupyterhub/handlers/base.py", line 996, in spawn_single_user
nomad_oasis_north | 2023-05-12T11:09:48.689585138Z         await gen.with_timeout(
nomad_oasis_north | 2023-05-12T11:09:48.689587846Z       File "/usr/local/lib/python3.9/site-packages/jupyterhub/handlers/base.py", line 907, in finish_user_spawn
nomad_oasis_north | 2023-05-12T11:09:48.689590713Z         await spawn_future
nomad_oasis_north | 2023-05-12T11:09:48.689593336Z       File "/usr/local/lib/python3.9/site-packages/jupyterhub/user.py", line 736, in spawn
nomad_oasis_north | 2023-05-12T11:09:48.689596130Z         raise e
nomad_oasis_north | 2023-05-12T11:09:48.689598629Z       File "/usr/local/lib/python3.9/site-packages/jupyterhub/user.py", line 635, in spawn
nomad_oasis_north | 2023-05-12T11:09:48.689601394Z         url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
nomad_oasis_north | 2023-05-12T11:09:48.689604134Z       File "/usr/local/lib/python3.9/site-packages/dockerspawner/dockerspawner.py", line 1309, in start
nomad_oasis_north | 2023-05-12T11:09:48.689606856Z         ip, port = await self.get_ip_and_port()
nomad_oasis_north | 2023-05-12T11:09:48.689609557Z       File "/usr/local/lib/python3.9/site-packages/dockerspawner/dockerspawner.py", line 1347, in get_ip_and_port
nomad_oasis_north | 2023-05-12T11:09:48.689612393Z         resp = await self.docker("inspect_container", self.container_id)
nomad_oasis_north | 2023-05-12T11:09:48.689615132Z       File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
nomad_oasis_north | 2023-05-12T11:09:48.689617738Z         result = self.fn(*self.args, **self.kwargs)
nomad_oasis_north | 2023-05-12T11:09:48.689620287Z       File "/app/nomad/jupyterhub_config.py", line 113, in _docker
nomad_oasis_north | 2023-05-12T11:09:48.689632813Z         result = super()._docker(method, *args, **kwargs)
nomad_oasis_north | 2023-05-12T11:09:48.689635346Z       File "/usr/local/lib/python3.9/site-packages/dockerspawner/dockerspawner.py", line 948, in _docker
nomad_oasis_north | 2023-05-12T11:09:48.689637851Z         return m(*args, **kwargs)
nomad_oasis_north | 2023-05-12T11:09:48.689640842Z       File "/usr/local/lib/python3.9/site-packages/docker/utils/decorators.py", line 19, in wrapped
nomad_oasis_north | 2023-05-12T11:09:48.689643364Z         return f(self, resource_id, *args, **kwargs)
nomad_oasis_north | 2023-05-12T11:09:48.689645727Z       File "/usr/local/lib/python3.9/site-packages/docker/api/container.py", line 783, in inspect_container
nomad_oasis_north | 2023-05-12T11:09:48.689648182Z         return self._result(
nomad_oasis_north | 2023-05-12T11:09:48.689657390Z       File "/usr/local/lib/python3.9/site-packages/docker/api/client.py", line 274, in _result
nomad_oasis_north | 2023-05-12T11:09:48.689662094Z         self._raise_for_status(response)
nomad_oasis_north | 2023-05-12T11:09:48.689664450Z       File "/usr/local/lib/python3.9/site-packages/docker/api/client.py", line 270, in _raise_for_status
nomad_oasis_north | 2023-05-12T11:09:48.689666959Z         raise create_api_error_from_http_exception(e) from e
nomad_oasis_north | 2023-05-12T11:09:48.689669253Z       File "/usr/local/lib/python3.9/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
nomad_oasis_north | 2023-05-12T11:09:48.689671699Z         raise cls(e, response=response, explanation=explanation) from e
nomad_oasis_north | 2023-05-12T11:09:48.689674146Z     docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.42/containers/a6a5581ae13d502f10495fab4ec28e7a8442ac51d1d139bc50ccc6b0488e6c8b/json: Not Found ("No such container: a6a5581ae13d502f10495fab4ec28e7a8442ac51d1d139bc50ccc6b0488e6c8b")
nomad_oasis_north | 2023-05-12T11:09:48.689676808Z     
nomad_oasis_north | 2023-05-12T11:09:48.690153960Z [E 2023-05-12 11:09:48.690 JupyterHub log:181] {
nomad_oasis_north | 2023-05-12T11:09:48.690164320Z       "X-Forwarded-Host": "north:9000",
nomad_oasis_north | 2023-05-12T11:09:48.690167103Z       "X-Forwarded-Proto": "http",
nomad_oasis_north | 2023-05-12T11:09:48.690169558Z       "X-Forwarded-Port": "9000",
nomad_oasis_north | 2023-05-12T11:09:48.690171903Z       "X-Forwarded-For": "::ffff:192.168.48.5",
nomad_oasis_north | 2023-05-12T11:09:48.690174339Z       "Content-Type": "application/json",
nomad_oasis_north | 2023-05-12T11:09:48.690176683Z       "Content-Length": "4348",
nomad_oasis_north | 2023-05-12T11:09:48.690178986Z       "Authorization": "Bearer [secret]",
nomad_oasis_north | 2023-05-12T11:09:48.690181257Z       "Connection": "close",
nomad_oasis_north | 2023-05-12T11:09:48.690183625Z       "Accept": "*/*",
nomad_oasis_north | 2023-05-12T11:09:48.690185869Z       "Accept-Encoding": "gzip, deflate",
nomad_oasis_north | 2023-05-12T11:09:48.690188180Z       "User-Agent": "python-requests/2.28.2",
nomad_oasis_north | 2023-05-12T11:09:48.690190518Z       "Host": "north:9000"
nomad_oasis_north | 2023-05-12T11:09:48.690192955Z     }

It really breaks in the beginning, i also checked on my machine the image is not even created in \var\lib\docker. I checked that \var\run\docker.socket is exposed and that the 1000 user maps to my docker user id.