"* Add a simple workflow to the central database via the command line\n",
"* Run that workflow\n",
"* Monitor your job status with the FireWorks database\n",
"* Get a flavor of the Python API\n",
"\n",
"\n",
"## Start FireWorks\n",
"\n",
"A MongoDB database (containing the FireWorks database) is running in your docker.\n",
"\n",
"Reset/Initialize the FireWorks database (the LaunchPad) using the command line:\n",
"\n",
"```lpad reset```\n",
"\n",
"Note: All FireWorks commands come with built-in help. For example, type lpad -h or lpad reset -h. There often exist many different options for each command.\n",
"\n",
"Note2: Resetting the FireWorks removes all your workflows and jobs from your database. During this tutorial, you may use this \"often\" but when you are in production and actually using FireWorks, you will most likely almost never use this reset.\n",
"\n",
"\n",
"\n",
"## Add a Workflow\n",
"\n",
"There are many ways to add Workflows to the database, including a Python API. Let’s start with an extremely simple example that can be added via the command line:\n",
"This added a two-job linear workflow. The first jobs prints hello to the command line, and the second job prints goodbye. We gave names (optional) to each step as “hello” and “goodbye”. We named the workflow overall (optional) as “test_workflow”.\n",
"\n",
"Let’s look at our test workflow:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "opponent-strip",
"metadata": {},
"outputs": [],
"source": [
"!lpad get_wflows -n test_workflow -d more"
]
},
{
"cell_type": "markdown",
"id": "deadly-gravity",
"metadata": {},
"source": [
"We get back basic information on our workflows. The second step “goodbye” is waiting for the first one to complete; it is not ready to run because it depends on the first job.\n",
"\n",
"\n",
"## Run all Workflows\n",
"\n",
"You can run jobs one at a time (“singleshot”) or all at once (“rapidfire”). Let’s run all jobs:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "destroyed-flooring",
"metadata": {},
"outputs": [],
"source": [
"!rlaunch rapidfire"
]
},
{
"cell_type": "markdown",
"id": "pressing-spiritual",
"metadata": {},
"source": [
"Clearly, both steps of our workflow ran in the correct order.\n",
"\n",
"Let’s again look at our workflows:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "presidential-macintosh",
"metadata": {},
"outputs": [],
"source": [
"!lpad get_wflows -n test_workflow -d more"
]
},
{
"cell_type": "markdown",
"id": "designing-bruce",
"metadata": {},
"source": [
"FireWorks automatically created launcher_ directories for each step in the Workflow and ran them. We see that both steps are complete. Note that there exist options to choose where to run jobs, as well as to tear down empty directories after running jobs.\n",
"\n",
"\n",
"## Look at the web GUI\n",
"\n",
"If you have a web browser, you can launch the web GUI to see your workflows using ```lpad webgui```. In this tutorial, the web GUI is directly integrated in the jupyter:\n",
"\n",
"\n",
"## Python code\n",
"\n",
"The following Python code achieves the same behavior:"
* Add a simple workflow to the central database via the command line
* Run that workflow
* Monitor your job status with the FireWorks database
* Get a flavor of the Python API
## Start FireWorks
A MongoDB database (containing the FireWorks database) is running in your docker.
Reset/Initialize the FireWorks database (the LaunchPad) using the command line:
```lpad reset```
Note: All FireWorks commands come with built-in help. For example, type lpad -h or lpad reset -h. There often exist many different options for each command.
Note2: Resetting the FireWorks removes all your workflows and jobs from your database. During this tutorial, you may use this "often" but when you are in production and actually using FireWorks, you will most likely almost never use this reset.
## Add a Workflow
There are many ways to add Workflows to the database, including a Python API. Let’s start with an extremely simple example that can be added via the command line:
This added a two-job linear workflow. The first jobs prints hello to the command line, and the second job prints goodbye. We gave names (optional) to each step as “hello” and “goodbye”. We named the workflow overall (optional) as “test_workflow”.
Let’s look at our test workflow:
%% Cell type:code id:opponent-strip tags:
``` python
!lpad get_wflows -n test_workflow -d more
```
%% Cell type:markdown id:deadly-gravity tags:
We get back basic information on our workflows. The second step “goodbye” is waiting for the first one to complete; it is not ready to run because it depends on the first job.
## Run all Workflows
You can run jobs one at a time (“singleshot”) or all at once (“rapidfire”). Let’s run all jobs:
%% Cell type:code id:destroyed-flooring tags:
``` python
!rlaunch rapidfire
```
%% Cell type:markdown id:pressing-spiritual tags:
Clearly, both steps of our workflow ran in the correct order.
Let’s again look at our workflows:
%% Cell type:code id:presidential-macintosh tags:
``` python
!lpad get_wflows -n test_workflow -d more
```
%% Cell type:markdown id:designing-bruce tags:
FireWorks automatically created launcher_ directories for each step in the Workflow and ran them. We see that both steps are complete. Note that there exist options to choose where to run jobs, as well as to tear down empty directories after running jobs.
## Look at the web GUI
If you have a web browser, you can launch the web GUI to see your workflows using ```lpad webgui```. In this tutorial, the web GUI is directly integrated in the jupyter:
## Python code
The following Python code achieves the same behavior:
%% Cell type:code id:black-avatar tags:
``` python
from fireworks import Firework, Workflow, LaunchPad, ScriptTask
from fireworks.core.rocket_launcher import rapidfire
"* Add a simple workflow to the central database via a file\n",
"\n",
"* Run that workflow in a few modes\n",
"\n",
"* Get a flavor of the Python API\n",
"\n",
"The purpose of this tutorial is to get you set up as quickly as possible; it isn’t intended to demonstrate the features of FireWorks or explain things in great detail. This tutorial can be safely completed from the command line, and requires no programming.\n",
"\n",
"First, reset again the FireWorks database with ```lpad reset```.\n",
"\n"
]
},
{
"cell_type": "markdown",
"id": "conceptual-subdivision",
"metadata": {},
"source": [
"## Add a Firework to the LaunchPad\n",
"\n",
"A Firework contains a list of computing tasks (Firetasks) to be performed. For this tutorial, we will use a Firework that consists of only a single step. We’ll tackle more complex workflows in other tutorials. Our workflow consisting of one Firework and one Firetask thus looks like this:\n",
"\n",
"\n",
"\n",
"Let's add the firework to the database of jobs:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "shared-printing",
"metadata": {},
"outputs": [],
"source": [
"!lpad add fw_test.yaml"
]
},
{
"cell_type": "markdown",
"id": "conceptual-smart",
"metadata": {},
"source": [
"This command added a simple workflow to the database which was serialized into a file called fw_test.yaml. This workflow is just a single step that print some text to a file. Look inside fw_test.yaml with a text editor to see how that workflow was defined:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "elect-omega",
"metadata": {},
"outputs": [],
"source": [
"!cat fw_test.yaml"
]
},
{
"cell_type": "markdown",
"id": "short-liberia",
"metadata": {},
"source": [
"You should have received confirmation that the Firework got added. You can query the database for this Firework as follows:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "packed-sound",
"metadata": {},
"outputs": [],
"source": [
"!lpad get_fws -i 1 -d all"
]
},
{
"cell_type": "markdown",
"id": "great-divide",
"metadata": {},
"source": [
"This prints, in JSON format, all details of the Firework with fw_id = 1 (the first Firework entered into the database):\n",
"\n",
"Some of the Firework is straightforward, but a few sections deserve further explanation:\n",
"\n",
"The spec of the Firework contains all the information about what job to run and the parameters needed to run it.\n",
"\n",
"Within the spec, the _tasks section tells you what jobs will run. The ScriptTask is a particular type of task that runs commands through the shell. Other sections of the spec can be also be defined, but for now we’ll stick to just _tasks. Later on, we’ll describe how to run multiple _tasks or customized _tasks.\n",
"\n",
"This Firework runs the script echo \"howdy, your job launched successfully!\" >> howdy.txt\", which prints text to a file named howdy.txt.\n",
"\n",
"The state of READY means the Firework is ready to be run.\n",
"\n",
"The name is an optional field that we can set to help query for FireWorks later on. In this case, we did not specify one so a default name was used.\n",
"You have now stored a Firework in the LaunchPad, and it’s ready to run!"
]
},
{
"cell_type": "markdown",
"id": "textile-chinese",
"metadata": {},
"source": [
"## Launch jobs\n",
"\n",
"We can launch jobs using ```rlaunch``` (\"Rocket\" launch) command:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "committed-cardiff",
"metadata": {},
"outputs": [],
"source": [
"!rlaunch singleshot"
]
},
{
"cell_type": "markdown",
"id": "united-banner",
"metadata": {},
"source": [
"This command fetches an available Firework from the FireWorks database and runs it.\n",
"\n",
"Verify that the desired task ran:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "hungry-correlation",
"metadata": {},
"outputs": [],
"source": [
"!cat howdy.txt"
]
},
{
"cell_type": "markdown",
"id": "increasing-absence",
"metadata": {},
"source": [
"You should see the text: \"howdy, your job launched successfully!\"\n",
"\n",
"In addition to howdy.txt, you should also see a file called FW.json. This contains a JSON representation of the Firework that the Rocket ran and can be useful later for tracking down a launch or debugging.\n",
"\n",
"Check the status of your Firework:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "interstate-glossary",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!lpad get_fws -i 1 -d all"
]
},
{
"cell_type": "markdown",
"id": "excellent-moldova",
"metadata": {},
"source": [
"You will now see lots of information about your Rocket launch, such as the time and directory of the launch. A lot of it is probably unclear, but you should notice that the state of the Firework is now COMPLETED.\n",
"\n",
"Try launching another rocket:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "every-setting",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!rlaunch singleshot"
]
},
{
"cell_type": "markdown",
"id": "distinguished-marble",
"metadata": {},
"source": [
"The message \"No FireWorks are ready to run and match query!\" indicates that it tried to fetch a Firework from the database, but none could be found. Indeed, we had previously run the only Firework that was in the database.\n",
"\n",
"## Launch many Rockets (rapidfire mode)\n",
"\n",
"If you just want to run many jobs on the central server itself, the simplest way is to run the Rocket Launcher in “rapidfire mode”. Let’s try this feature:\n",
"\n",
"Let’s add a Fireworks 3 times:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "limited-stewart",
"metadata": {},
"outputs": [],
"source": [
"!lpad add fw_test.yaml"
]
},
{
"cell_type": "markdown",
"id": "breeding-delaware",
"metadata": {},
"source": [
"Confirm that the three Fireworks got added to the database, in addition to the one from before (4 total):"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "exceptional-bradley",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!lpad get_fws -d less"
]
},
{
"cell_type": "markdown",
"id": "thousand-guitar",
"metadata": {},
"source": [
"We could also just get information for jobs that are ready to run (our 3 new FireWorks):"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "packed-worse",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!lpad get_fws -s READY -d less"
]
},
{
"cell_type": "markdown",
"id": "small-murder",
"metadata": {},
"source": [
"Let’s launch jobs in “rapidfire” mode, which will keep repeating until we run out of Fireworks to run:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "double-contrary",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!rlaunch rapidfire"
]
},
{
"cell_type": "markdown",
"id": "stretch-knife",
"metadata": {},
"source": [
"You should see three directories starting with the tag launcher_. Inside each of these directories, you’ll find the results of one of your FireWorks (a file named howdy.txt):"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "documentary-acrobat",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"!cat launch*/howdy.txt"
]
},
{
"cell_type": "markdown",
"id": "received-immune",
"metadata": {},
"source": [
"## Running FireWorks automatically\n",
"\n",
"We can set our Launcher to continuously look for new FireWorks to run. Let’s try this feature.\n",
"\n",
"Start the Launcher in a terminal so that it looks for new FireWorks every 10 seconds:\n",
