Commit 367b9f83 authored by Philipp Schubert's avatar Philipp Schubert
Browse files

minor changes

parent 47d45dbb
Pipeline #116267 passed with stage
in 1 minute and 56 seconds
......@@ -11,4 +11,5 @@ omit =
syconn/analysis/syconn_knossos_viewer.py
syconn/analysis/syconn_graphwalker.py
syconn/analysis/server.py
syconn/version.py
*/__init__.py
......@@ -25,7 +25,7 @@ We also present more general information about SyConn on our [Website](https://s
Documentation
-------------
The documentation including API docs can be found as [readthedocs build](https://syconn.readthedocs.io/en/latest/) or partially as markdown [here](docs/doc.md).
The documentation including installation procedure, requirements and API docs can be found as [readthedocs build](https://syconn.readthedocs.io/en/latest/) or partially as markdown [here](docs/doc.md).
The Team
--------
......
......@@ -68,8 +68,8 @@ m2r2_disable_inline_math = False
master_doc = 'index'
# General information about the project.
project = u'SyConn'
copyright = u'2018, SyConn team'
project = u'SyConn2'
copyright = u'2021, SyConn team'
author = u'Philipp Schubert, Jörgen Kornfeld'
# The version info for the project you're documenting, acts as replacement for
......@@ -77,9 +77,9 @@ author = u'Philipp Schubert, Jörgen Kornfeld'
# built documents.
#
# The short X.Y version.
version = u'0.3'
version = u'2.0'
# The full version, including alpha/beta/rc tags.
release = u'0.3a'
release = u'2.0'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
......
......@@ -38,15 +38,18 @@ If you encounter
with open3d, you need to upgrade your system or downgrade open3d to `<=0.9`.
## Example run
Place the example and model data (provided upon request) in `SyConn/examples/`,
Place the example data and models (provided upon request) in `~/SyConnData/`,
cd to `SyConn/examples/` and run
python start.py [--working_dir=..]
The example script analyzes the EM data based on KnossosDatasets (see `knossos_utils`) of the cell segmentation,
The example script analyzes the EM data together with the cell segmentation,
probability maps of sub-cellular structures (mitochondria, vesicle clouds and synaptic junctions) and synapse type
(inhibitory, excitatory). For adding further cell organelles to this pipeline see [here](cellorganelle_integration.md).
The data format for raw image and segmentation data is based on ``KnossosDataset``
(see [knossos_utils](https://github.com/knossos-project/knossos_utils)).
On a machine with 20 CPUs (Intel Xeon @ 2.60GHz) and 2 GPUs (NVidia Quadro RTX 5000) SyConn
finished the following analysis steps for an example cube of shape \[1100 1100 600] (1.452e-06 mm^3; 0.726 GVx) after 00h:31min:46s.
......@@ -73,11 +76,15 @@ finished the following analysis steps for an example cube of shape \[1100 1100 6
\[11/11] Matrix export 0d:0h:0min:7s 0.4%
## Example scripts and API usage
An introduction on how to use the example scripts can be found [here](examples.md)
and API code examples [here](api.md).
## Flowchart of SyConn
<img src="https://docs.google.com/drawings/d/e/2PACX-1vSY7p2boPxb9OICxNhSrHQlvuHTBRbSMeIOgQ4_NV6pflxc0FKJvPBtskYMAgJsX_OP-6CNmb08tLC5/pub?w=2880&amp;h=1200">
## Package structure and data classes
The basic data structures and initialization procedures are explained in the following sections:
......@@ -90,7 +97,7 @@ handled by the `SegmentationDatasets`. For a more detailed description see [here
(such as lustre, Google Cloud Filestore or AWS Elastic File System).
* Agglomerated supervoxels (SVs) are implemented as SuperSegmentationObjects ([SSO](super_segmentation_objects.md)). The collection
of super-SVs are usually defined in a region adjacency graph (RAG) which is used to initialize the SuperSegmentationDataset
of super-SVs are usually defined in a region supervoxel graph which is used to initialize the SuperSegmentationDataset
([SSD](super_segmentation_datasets.md)).
* [Skeletons](skeletons.md) of (super-) supervoxels, usually computed from variants of the TEASAR algorithm (https://ieeexplore.ieee.org/document/883951)
......@@ -98,15 +105,10 @@ handled by the `SegmentationDatasets`. For a more detailed description see [here
* [Mesh](meshes.md) generation and representation of supervoxels
* Multi-view representation of neurpn reconstructions for [glia](glia_removal.md) and
* Multi-view representation of neuron reconstructions for [glia](glia_removal.md) and
[neuron](neuron_analysis.md) analysis (published in [Nature Communications](https://www.nature.com/articles/s41467-019-10836-3))
## Flowchart of SyConn
<img src="https://docs.google.com/drawings/d/e/2PACX-1vSY7p2boPxb9OICxNhSrHQlvuHTBRbSMeIOgQ4_NV6pflxc0FKJvPBtskYMAgJsX_OP-6CNmb08tLC5/pub?w=2880&amp;h=1200">
## Analysis steps
After initialization of the SDs (cell and sub-cellular structures, step 1 in the example run) and the SSD
containing the agglomerated cell SVs (step 3), several analysis steps can be applied:
......
......@@ -31,7 +31,7 @@ parser.add_argument('--sp', type=int, default=15000, help='Number of sample poin
parser.add_argument('--scale_norm', type=int, default=5000, help='Scale factor for normalization')
parser.add_argument('--co', action='store_true', help='Disable CUDA')
parser.add_argument('--seed', default=0, help='Random seed', type=int)
parser.add_argument('--ctx', default=15000, help='Context size in nm', type=float)
parser.add_argument('--ctx', default=20000, help='Context size in nm', type=float)
parser.add_argument(
'-j', '--jit', metavar='MODE', default='disabled', # TODO: does not work
choices=['disabled', 'train', 'onsave'],
......@@ -92,7 +92,7 @@ if cellshape_only:
if name is None:
name = f'semseg_pts_nb{npoints}_ctx{ctx}_{gt_type}_nclass' \
f'{num_classes[gt_type]}_lcp_GN_noKernelSep_AdamW_CE_large'
f'{num_classes[gt_type]}_lcp_GN_noKernelSep_AdamW_CE_large_v2'
if not normalize_pts:
name += '_NonormPts'
if cellshape_only:
......@@ -124,18 +124,18 @@ conv = dict(layer='ConvPoint', kernel_separation=False, normalize_pts=normalize_
act = nn.ReLU
# architecture = None
architecture = [dict(ic=-1, oc=1, ks=48, nn=32, np=-1),
dict(ic=1, oc=1, ks=32, nn=32, np=2048),
dict(ic=1, oc=1, ks=48, nn=32, np=2048),
dict(ic=1, oc=1, ks=32, nn=16, np=1024),
dict(ic=1, oc=2, ks=32, nn=16, np=256),
dict(ic=2, oc=2, ks=16, nn=16, np=128),
dict(ic=2, oc=2, ks=16, nn=8, np=64),
dict(ic=2, oc=2, ks=16, nn=4, np=32),
dict(ic=2, oc=2, ks=32, nn=16, np=128),
dict(ic=2, oc=2, ks=16, nn=16, np=64),
dict(ic=2, oc=2, ks=16, nn=16, np=32),
dict(ic=2, oc=2, ks=16, nn=4, np='d'),
dict(ic=4, oc=2, ks=16, nn=8, np='d'),
dict(ic=4, oc=1, ks=16, nn=16, np='d'),
dict(ic=3, oc=1, ks=16, nn=16, np='d'),
dict(ic=2, oc=1, ks=16, nn=16, np='d'),
dict(ic=2, oc=1, ks=16, nn=16, np='d')]
dict(ic=4, oc=2, ks=16, nn=4, np='d'),
dict(ic=4, oc=1, ks=32, nn=4, np='d'),
dict(ic=3, oc=1, ks=32, nn=8, np='d'),
dict(ic=2, oc=1, ks=32, nn=8, np='d'),
dict(ic=2, oc=1, ks=48, nn=8, np='d')]
model = ConvAdaptSeg(input_channels, num_classes[gt_type], get_conv(conv), get_search(search), kernel_num=64,
architecture=architecture, activation=act, norm='gn')
......@@ -181,10 +181,10 @@ else: # no additional validation data
train_ds = CloudDataSemseg(npoints=npoints, transform=train_transform, use_subcell=use_subcell,
batch_size=batch_size, ctx_size=ctx, mask_borders_with_id=ignore_l,
source_dir=train_dir, remap_dict=remap_dicts[gt_type])
valid_ds = None
# valid_ds = CloudDataSemseg(npoints=npoints, transform=valid_transform, train=False, use_subcell=use_subcell,
# batch_size=batch_size, ctx_size=ctx, mask_borders_with_id=ignore_l,
# source_dir=valid_dir, remap_dict=remap_dicts[gt_type])
valid_ds = None
# PREPARE AND START TRAINING #
# set up optimization
......
......@@ -1822,7 +1822,7 @@ def infere_cell_morphology_ssd(ssv_params, mpath: Optional[str] = None, pred_key
postproc_kwargs=postproc_kwargs, postproc_func=pts_postproc_embedding,
show_progress=False, mpath=mpath, **loader_kwargs, **default_kwargs)
if not np.all(list(out_dc.values())) or len(out_dc) != len(ssv_params):
raise ValueError('Invalid output during glia prediction.')
raise ValueError('Invalid output during cell morphology extraction.')
def predict_celltype_ssd(ssd_kwargs, mpath: Optional[str] = None, ssv_ids: Optional[Iterable[int]] = None,
......
[pytest]
addopts = -v --cache-clear --tb=native --cov --cov-config=.coveragerc
python_files = test_*.py
python_files = tests/test_*.py
norecursedirs = .git build
\ No newline at end of file
......@@ -25,19 +25,21 @@ def example_run():
process = subprocess.Popen(
["python", startpy_fname, f"--working_dir={working_dir}"],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
stderr=subprocess.PIPE,
universal_newlines=True,
)
out, _ = process.communicate()
out, err = process.communicate()
if os.environ.get('syconn_wd') is not None:
del os.environ['syconn_wd']
shutil.rmtree(example_wd, ignore_errors=True)
return process.returncode
return process.returncode if process.returncode == 0 else err
def test_example_run():
assert example_run() == 0
ret = example_run()
if ret != 0:
raise RuntimeError(ret)
if __name__ == '__main__':
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment