Discussion of the main Technology choices, basically always several options were evaluated, here we normally do not present a full evaluation, but just the choice with a a small discussion. If contention arises this might be expanded. Hints, infos on technical aspects is also given.

Language

scala

Scala is the main language for the base layer, it is compatible with the JVM, widely used in the machine learning, flink, spark communities (see also emma), and more concise than java. See link for more information.

python

is more widespread and known by scientists, and is used in some parsers

Interfaces

REST

REpresentational State Transfer, a software architectural style, sometimes also called a programming paradigm, the WWW's functioning is based on.

Nice introductory video explaining the concept and usage: video

Building

sbt was chosen as it is well integrated with scala and offers incremental builds

Testing

• specs2 should be used for fixed point testing (verification of input/output).
• Currently version 2.x is used, as using 3.x with spray needs forking, with akka http or if spray will support 3.x this might change (simple tests should work without changes).
• it seems that specs2 can easily increase compilation times, so we might want to switch to scalatest in the future
• specs2 tests of bla/Class.scala are expected to be in tests/bla/ClassSpec.scala
• scalacheckshould be used for generation (random) testing, and verificatin of invariants
• scalacheck tests of bla/Class.scala are expected to be in tests/bla/ClassTests.scala
• every class should have at least one kind of test, ideally both

Debugging/Monitoring

a good discussion of JVM montoring and debugging capabilities is in modern java part 2 (the other two ports are not so relevant, and I partially disagree, but part 2 is nice).

Logging

scala-logging (i.e. SLF4J) with log4j2 as backend

Docker

We use Docker heavily

• We use sbt-docker to generate automatically docker images when compiling (sbt-native-packager would be another option).
• We have a docker issues wiki page with problem we had.

Interesting articles:

• Docker traps
• Monitoring Docker
• Joyent approach to safe Docker (triton) and also their manta object storage is worth being followed closly
• Evaluation of storage drivers for docker:
  • nice presentation and description of the various storage options
  • performance test of various options

Kubernetes

For containers orchestration in the cluster we want to use Kubernetes

Evaluation of queuing systems

A good evaluation of various queuing systems can be found here. To handle data flow to containers we use a queuing system.

AI links (could become interesting)

• Facebook opensource AI hardware
• torch an opensource neural network software
• opensource NLP projects, concentrated on unstructured/natural language processing, used in Watson
• tensorflow google deep neural networks project
• Open AI

Notebooks

Notebooks offer a way to combine all different pieces in a coherent user visible interface that a user can interact with. Thus they represent an excellent glue for WP4, but also potentially for all other work packages.

We evaluated a bit the various possibilities like jupyter and repated, for now we did choose beaker because although relatively new it has a nice interface, and critically, the ability to use several languages in the same notebook. See our beaker wiki page for the current status how to start and tips and tricks.