Commit c7c708bf authored by Philipp Schubert's avatar Philipp Schubert
Browse files

updated references in docs, minor refinements

parent 3fa063a3
......@@ -6,22 +6,33 @@ Refactored version of SyConn for automated synaptic connectivity inference based
Current features:
- introduction of supervoxel and agglomerated supervoxel classes
- added support for (sub-) cellular compartment (spines, axon/dendrite/soma) and cell type classification with skeleton- [\[1\]](https://www.nature.com/articles/nmeth.4206) and multiview-based [\[2\]](https://www.biorxiv.org/content/early/2018/07/06/364034) approaches
- added support for (sub-) cellular compartment (spines, axon/dendrite/soma) and cell type classification with skeleton- [\[1\]](https://www.nature.com/articles/nmeth.4206) and multiview-based [\[2\]](https://www.nature.com/articles/s41467-019-10836-3) approaches
- cell organelle prediction, extraction and mesh generation
- glia identification and supervoxel graph splitting [\[2\]](https://www.biorxiv.org/content/early/2018/07/06/364034)
- glia identification and separation [\[2\]](https://www.nature.com/articles/s41467-019-10836-3)
- generation of connectivity matrix
If you use parts of this code base in your academic projects, please cite the corresponding publication.
Documentation
--------------
To get started, please have a look at our [documentation](https://structuralneurobiologylab.github.io/SyConn/documentation/), but be aware that it is currently outdated and applies only to SyConn v1. We also present more general information about SyConn on our [Website](https://structuralneurobiologylab.github.io/SyConn/).
The documentation for the refactored version is still work-in-progress and can be found [here](docs/doc.md). Alternatively see the latest [readthedocs build](https://syconn.readthedocs.io/en/latest/).
SyConn v1 Publication
---------------------
The first version of SyConn (see branch [dorkenwald2017nm](https://github.com/StructuralNeurobiologyLab/SyConn/tree/dorkenwald2017nm)) was published in [Nature Methods](http://www.nature.com/nmeth/journal/vaop/ncurrent/full/nmeth.4206.html) on February 27th 2017. If you use parts of this code base in your academic projects, please cite the corresponding publication. <br />
```
# The Team
The Synaptic connectivity inference toolkit is developed at the Max-Planck-Institute of Neurobiology in Martinsried by
Philipp Schubert, Maria Kawula, Carl Constantin v. Wedemeyer, Atul Mohite, Gaurav Kumar and Joergen Kornfeld.
# Acknowledgements
We thank deepmind for providing egl extension code to handle multi-gpu rendering on the same machine, which is under the Apache License 2.0. The original code snippet used in our
project can be found [here](https://github.com/deepmind/dm_control/blob/30069ac11b60ee71acbd9159547d0bc334d63281/dm_control/_render/pyopengl/egl_ext.py).
# References
\[1\] [Automated synaptic connectivity inference for volume electron microscopy][https://www.nature.com/articles/nmeth.4206]
```
@ARTICLE{SyConn2017,
title = "Automated synaptic connectivity inference for volume electron
microscopy",
......@@ -41,13 +52,24 @@ The first version of SyConn (see branch [dorkenwald2017nm](https://github.com/St
}
```
# The Team
The Synaptic connectivity inference toolkit is developed at Max-Planck-Institute of Neurobiology, Munich.
Authors: Philipp Schubert, Sven Dorkenwald, Rangoli Saxena, Joergen Kornfeld
# Acknowledgements
We thank deepmind for providing egl extension code to handle multi-gpu rendering on
the same machine, which is under the Apache License 2.0. The original code snippet used in our
project can be found [here](https://github.com/deepmind/dm_control/blob/30069ac11b60ee71acbd9159547d0bc334d63281/dm_control/_render/pyopengl/egl_ext.py).
\ No newline at end of file
\[2\] [Learning cellular morphology with neural networks][https://doi.org/10.1038/s41467-019-10836-3]
```
@Article{Schubert2019,
author={Schubert, Philipp J.
and Dorkenwald, Sven
and Januszewski, Michal
and Jain, Viren
and Kornfeld, Joergen},
title={Learning cellular morphology with neural networks},
journal={Nature Communications},
year={2019},
volume={10},
number={1},
pages={2736},
abstract={Reconstruction and annotation of volume electron microscopy data sets of brain tissue is challenging but can reveal invaluable information about neuronal circuits. Significant progress has recently been made in automated neuron reconstruction as well as automated detection of synapses. However, methods for automating the morphological analysis of nanometer-resolution reconstructions are less established, despite the diversity of possible applications. Here, we introduce cellular morphology neural networks (CMNs), based on multi-view projections sampled from automatically reconstructed cellular fragments of arbitrary size and shape. Using unsupervised training, we infer morphology embeddings (Neuron2vec) of neuron reconstructions and train CMNs to identify glia cells in a supervised classification paradigm, which are then used to resolve neuron reconstruction errors. Finally, we demonstrate that CMNs can be used to identify subcellular compartments and the cell types of neuron reconstructions.},
issn={2041-1723},
doi={10.1038/s41467-019-10836-3},
url={https://doi.org/10.1038/s41467-019-10836-3}
}
```
\ No newline at end of file
......@@ -140,7 +140,7 @@ organized in [SegmentationDatasets](segmentation_datasets.md).
* [Mesh](meshes.md) generation and representation of SOs
* Multi-view representation of SSOs (see docs for [glia](glia_removal.md) and [neuron](neuron_analysis.md) analysis; [preprint](https://www.biorxiv.org/content/early/2018/07/06/364034) on biorXiv)
* Multi-view representation of SSOs (see docs for [glia](glia_removal.md) and [neuron](neuron_analysis.md) analysis and [article](https://www.nature.com/articles/s41467-019-10836-3) in Nature Communications)
## Flowchart of SyConn
......
......@@ -33,13 +33,13 @@ working_dir = so_kwargs['working_dir']
global_params.wd = working_dir
kwargs = args[2]
n_parallel_jobs = global_params.NCORES_PER_NODE # -> uses more threads than available (increase
n_parallel_jobs = global_params.NCORES_PER_NODE // global_params.NGPUS_PER_NODE # -> uses more threads than available (increase
if 'add_cellobjects' in kwargs and kwargs['add_cellobjects']:
n_parallel_jobs = global_params.NCORES_PER_NODE // global_params.NGPUS_PER_NODE
# usage of GPU)
multi_params = ch
# this creates a list of lists of SV IDs
multi_params = chunkify(multi_params, n_parallel_jobs * 2)
multi_params = chunkify(multi_params, n_parallel_jobs)
# list of SSV IDs and SSD parameters need to be given to a single QSUB job
multi_params = [(ixs, so_kwargs, kwargs) for ixs in multi_params]
# first SV should always be unique
......
......@@ -588,8 +588,9 @@ def run_glia_prediction(e3=False):
else:
missing_contained_in_rag.append(el)
if len(missing_contained_in_rag) != 0:
msg = "Not all SVs were predicted! Missing:\n" \
"{}".format(missing_contained_in_rag)
msg = "Not all SVs were predicted! {}/{} missing:\n" \
"{}".format(len(missing_contained_in_rag), len(all_sv_ids_in_rag),
missing_contained_in_rag[:100])
log.error(msg)
raise ValueError(msg)
else:
......@@ -633,15 +634,14 @@ def run_glia_splitting():
"".format(global_params.config.working_dir + "/glia/"))
def _run_huge_ssv_render_worker(q):
def _run_huge_ssv_render_worker(q, q_out):
while True:
if q.empty():
inp = q.get()
if inp == -1:
break
kk, g, version = q.get()
kk, g, version = inp
# Create SSV object
sv_ixs = np.sort(list(g.nodes()))
# log.info("Processing SSV [{}] with {} SVs on whole cluster.".format(
# kk + 1, len(sv_ixs)))
sso = SuperSegmentationObject(sv_ixs[0], working_dir=global_params.config.working_dir,
version=version, create=False, sv_ids=sv_ixs)
# nodes of sso._rag need to be SV
......@@ -653,6 +653,7 @@ def _run_huge_ssv_render_worker(q):
sso.render_views(add_cellobjects=False, cellobjects_only=False,
skip_indexviews=True, woglia=False, overwrite=True,
qsub_co_jobs=global_params.NGPU_TOTAL)
q_out.put(0)
def run_glia_rendering(max_n_jobs=None):
......@@ -693,7 +694,8 @@ def run_glia_rendering(max_n_jobs=None):
bbs = sds.load_cached_data('bounding_box') * sds.scaling
for ii in range(len(sds.ids)):
sv_size_dict[sds.ids[ii]] = bbs[ii]
ccsize_dict = create_ccsize_dict(cc_gs, sv_size_dict, is_connected_components=True)
ccsize_dict = create_ccsize_dict(cc_gs, sv_size_dict,
is_connected_components=True)
multi_params = cc_gs
big_ssv = []
......@@ -713,14 +715,21 @@ def run_glia_rendering(max_n_jobs=None):
log.info("Processing {} huge SSVs in {} threads on the entire cluster"
".".format(len(big_ssv), n_threads))
q_in = Queue()
q_out = Queue()
for kk, g in enumerate(big_ssv):
q_in.put((kk, g, version))
ps = [Process(target=_run_huge_ssv_render_worker, args=(q_in, )) for _ in range(n_threads)]
for _ in range(n_threads):
q_in.put(-1)
ps = [Process(target=_run_huge_ssv_render_worker, args=(q_in, q_out)) for _ in range(n_threads)]
for p in ps:
p.start()
time.sleep(0.5)
q_in.close()
q_in.join_thread()
for p in ps:
p.join()
if q_out.qsize() != len(big_ssv):
raise ValueError('Not all `_run_huge_ssv_render_worker` jobs completed successfully.')
# render small SSV without overhead and single cpus on whole cluster
multi_params = small_ssv
np.random.shuffle(multi_params)
......@@ -744,8 +753,9 @@ def run_glia_rendering(max_n_jobs=None):
else:
missing_contained_in_rag.append(el)
if len(missing_contained_in_rag) != 0:
msg = "Not all SVs were rendered completely! Missing:\n" \
"{}".format(missing_contained_in_rag)
msg = "Not all SVs were rendered completely! {}/{} missing:\n" \
"{}".format(len(missing_contained_in_rag), len(all_sv_ids_in_rag),
missing_contained_in_rag[:100])
log.error(msg)
raise ValueError(msg)
else:
......
......@@ -67,7 +67,8 @@ def QSUB_script(params, name, queue=None, pe=None, n_cores=1, priority=0,
sge_additional_flags=None, iteration=1, max_iterations=3,
params_orig_id=None, python_path=None, disable_mem_flag=False,
disable_batchjob=False, send_notification=False, use_dill=False,
remove_jobfolder=False, log=None, show_progress=True):
remove_jobfolder=False, log=None, show_progress=True,
allow_resubm_all_fail=False):
# TODO: Switch to JobArrays!
"""
QSUB handler - takes parameter list like normal multiprocessing job and
......@@ -129,6 +130,9 @@ def QSUB_script(params, name, queue=None, pe=None, n_cores=1, priority=0,
use_dill : bool
show_progress : bool
Currently only applies for `batchjob_fallback`
allow_resubm_all_fail : bool
Will resubmit failed jobs even if all failed. Useful for e.g. unexpected memory
requirements.
Returns
-------
......@@ -362,7 +366,7 @@ def QSUB_script(params, name, queue=None, pe=None, n_cores=1, priority=0,
out_files = glob.glob(path_to_out + "*.pkl")
# only stop if first iteration and script was not resumed (params_orig_id is None)
if len(out_files) == 0 and iteration == 1 and params_orig_id is None:
if len(out_files) == 0 and iteration == 1 and params_orig_id is None and not allow_resubm_all_fail:
msg = 'All submitted jobs have failed. Re-submission will not be initiated.' \
' Please check your submitted code.'
log_batchjob.error(msg)
......@@ -389,7 +393,7 @@ def QSUB_script(params, name, queue=None, pe=None, n_cores=1, priority=0,
# set number cores per job higher which will at the same time increase
# the available amount of memory per job, ONLY VALID IF '--mem' was not specified explicitly!
n_cores += 2 # increase number of cores per job by at least 1
n_cores += 2 # increase number of cores per job by at least 2
# TODO: activate again
# n_cores = np.max([np.min([global_params.NCORES_PER_NODE, float(n_max_co_processes) //
# len(missed_params)]), n_cores])
......@@ -616,7 +620,8 @@ def fallback_exec(cmd_exec):
if log_mp.level == 10:
reported = False
if 'error' in out.decode().lower() or \
'error' in err.decode().lower():
'error' in err.decode().lower() or 'killed' in\
out.decode().lower() or 'killed' in err.decode.lower():
log_mp.error(out.decode())
log_mp.error(err.decode())
reported = True
......
......@@ -251,7 +251,7 @@ class SuperSegmentationDataset(object):
def load_cached_data(self, name):
if os.path.exists(self.path + name + "s.npy"):
return np.load(self.path + name + "s.npy")
return np.load(self.path + name + "s.npy", allow_pickle=True)
def sv_id_to_ssv_id(self, sv_id):
return self.id_changer[sv_id]
......
......@@ -1354,7 +1354,7 @@ class SuperSegmentationObject(object):
qu.QSUB_script(
params, "render_views_partial", suffix="_SSV{}".format(self.id),
n_cores=global_params.NCORES_PER_NODE // global_params.NGPUS_PER_NODE,
n_max_co_processes=qsub_co_jobs, remove_jobfolder=True,
n_max_co_processes=qsub_co_jobs, remove_jobfolder=True, allow_resubm_all_fail=True,
resume_job=resume_job, additional_flags="--gres=gpu:1")
else:
# render raw data
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment