Rewrite/refactor of our API with FastAPI
Motivation
Flask + REST Plus is bad
- outdated and not really maintained
- cumbersome, restrictive REST Plus specific request/response models
- argparser (Flask) and models (REST Plus) are unrelated
- no manipulation of generated OpenAPI spec
- optimade Python tools are not based on Flask REST Plus
Fast API seems to solve a lot of problems
- models based on type annotations (pydantic)
- object oriented models
- query parameter parsing from function parameter annotations
- based on more modern uvicorn; prod==dev server
our API is messy
- lots of inconsistencies between repo/archive/raw/... related to queries, pagination, etc.
- lots of doubled CnC code pieces
- lots of functionality/parameter crammed into GET requests, e.g.
/repo/
- grown tests
Operations
Before we look at API operations (Fast API and Open API call the basic API building blocks operations), we should look at our datamodel it's core entities and principle views on data.
NOMAD entities
- (info)
- users
- uploads
- entries
- datasets
- materials
- (metainfo)
views
- metadata (repo)
- archive
- raw
- encyclopedia, currently a mix of metadata and archive
- processing/upload (should be just part of metadata)
In contrast to the old API, we structure the API based on entities. The metadata view is the default, and there is access to the other views via subpaths raw
, archive
.
models
Many operations use the same input and output structures over and over again:
- Query
- Pagination, includes response size, order, aggregation
- (Metadata|Archive)Required, a list or structure of things to include in return
- Statistics, additional histogram information on metadata results
- FileFiler, for file downloads
- Models for our entities: User, UserMetadata, Dataset, Entry, Upload
operations
http | path | input | output |
---|---|---|---|
GET | /entries |
SimpleQuery, Pagination, MetadataRequired | Metadata* |
POST | /entries/edit |
Query, UserMetadata, EditActions | Edit or EditActions |
POST | /entries/query |
Query, Statistics, Pagination, MetadataRequired, | Metadata*, Stat. |
GET | /entries/<id> |
MetadataRequired | Metadata |
GET | /entries/raw |
SimpleQuery, FileFilter | .zip+manifest |
POST | /entries/raw |
Query, FileFilter | .zip+manifest |
GET | /entries/<id>/raw |
FileFilter | .zip+manifest |
GET | /entries/<id>/raw/<path> |
file | |
GET | /entries/archive |
SimpleQuery, Pagination | Archive* |
POST | /entries/archive |
Query, ArchiveRequired, Pagination | Archive* |
GET | /entries/<id>/archive |
ArchiveRequired | Archive* |
GET | /entries/<id>/archive/<path> |
Archive* | |
GET | /uploads |
SimpleQuery, Pagination, MetadataRequired | (Upload, Metadata*)* |
POST | /uploads/query |
Query, Statistics, Pagination, MetadataRequired | (Upload, Metadata*)* |
PUT | /uploads |
file | Upload |
GET | /uploads/<id> |
Pagination, MetadataRequired | Upload, Metadata* |
POST | /uploads/<id> |
Upload+Actions | Upload |
GET | /uploads/<id>/query |
Query, Statistics, Pagination, MetadataRequired | Upload, Metadata* |
DELETE | /uploads/<id> |
Upload | |
GET | /uploads/<id>/raw |
FileFilter | .zip+manifest |
GET | /uploads/<id>/<glob> |
FileFilter | .zip+manifest |
GET | /uploads/<id>/<path> |
file | |
GET | /datasets |
SimpleQuery, Pagination, MetadataRequired | (Dataset, Metadata*)* |
POST | /datasets/query |
Query, Statistics, Pagination, MetadataRequired | (Dataset, Metadata*)* |
PUT | /datasets |
Dataset | Dataset |
GET | /datasets/<id> |
SimpleQuery, Pagination, MetadataRequired | Dataset, Metadata* |
GET | /datasets/<id>/query |
Query, Statistics, Pagination, MetadataRequired | Dataset, Metadata* |
POST | /datasets/<id> |
Dataset+Actions | Dataset |
DELETE | /datasets/<id> |
Dataset | |
GET | /material |
SimpleQuery, Pagination, MetadataRequired | (Material, Metadata*)* |
POST | /material/query |
Query, Statistics, Pagination, MetadataRequired | (Material, Metadata*)* |
GET | /material/<id> |
SimpleQuery, Pagination, MetadataRequired | Material, Metadata* |
GET | /material/<id>/query |
Query, Statistics, Pagination, MetadataRequired | Material, Metadata* |
GET | /materials/suggestions |
? | Suggestion* |
GET | /groups |
SimpleQuery, Pagination, MetadataRequired | (Group, Metadata*)* |
POST | /groups/query |
Query, Statistics, Pagination, MetadataRequired | (Group, Metadata*)* |
GET | /groups/<id> |
SimpleQuery, Pagination, MetadataRequired | Group, Metadata* |
GET | /groups/<id>/query |
Query, Statistics, Pagination, MetadataRequired | Group, Metadata* |
GET | /group/<id>/query |
Query, Statistics, Pagination, MetadataRequired | Group, Metadata* |
GET | /users |
Pagination | User* |
GET | /users/me |
User, Auth | |
GET | /users/<id> |
User | |
GET | /info |
Info | |
GET | /metainfo |
MetainfoQuery | Archive* |
GET | /metainfo/<path> |
Archive or json-value |
Tasks
-
learn fast api and build skeleton API with a few model and static example response -
think of a documentation scheme: swagger, reference, tutorials -
restructure the tests based on the skeleton (test first) -
move the implementation step by step (search, upload, edit, archive, raw, metainfo, ...) -
implement everything
Ideally we would already have end-to-end GUI tests that use the API for "real"
Edited by Markus Scheidgen