Merge pull request #53 from pyiron/cleanup

Final changes: All types of potentials are now included in the validation notebooks! Hope this works!

day_1/ex_02_creating_structure_databases.ipynb

@@ -2,7 +2,7 @@
  "cells": [
   {
    "cell_type": "markdown",
-   "id": "greek-medicare",
+   "id": "appointed-stylus",
    "metadata": {},
    "source": [
     "# [**Workflows for atomistic simulations**](http://potentials.rub.de/) "
@@ -10,7 +10,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "champion-twins",
+   "id": "straight-bicycle",
    "metadata": {},
    "source": [
     "## **Day 1 - Atomistic simulations with [pyiron](https://pyiron.org)**\n",
@@ -20,17 +20,27 @@
     "\n",
     "Before the excercise, you should:\n",
     "\n",
-    "* Finish exercises 1 and 2\n",
+    "* Finish exercise 1\n",
     "\n",
     "The aim of this exercise is to make you familiar with:\n",
     "\n",
     "* Creating structure databases and working with them for potential fitting (day 2)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "durable-leone",
+   "metadata": {},
+   "source": [
+    "## **Importing necessary modules and creating a project**\n",
+    "\n",
+    "This is done the same way as shown in the first exercise"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": 1,
-   "id": "seeing-application",
+   "id": "novel-wisconsin",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -42,68 +52,91 @@
   {
    "cell_type": "code",
    "execution_count": 2,
-   "id": "general-sensitivity",
+   "id": "sitting-religious",
    "metadata": {},
    "outputs": [],
    "source": [
-    "from pyiron import Project\n",
-    "import pyiron_contrib"
+    "from pyiron import Project"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 3,
-   "id": "corresponding-asset",
+   "id": "technical-newport",
    "metadata": {},
    "outputs": [],
    "source": [
     "pr = Project(\"creating_datasets\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "based-kentucky",
+   "metadata": {},
+   "source": [
+    "## Creating a structure \"container\" from the data\n",
+    "\n",
+    "We now go over the jobs generated in the first notebook to store structures, energies, and forces into a structure container which will later be used for potential fitting\n",
+    "\n",
+    "**Note**: Usually these datasets are created using highly accurate DFT calculations. But for practical reasons, we only demonstrate how to do this using data from LAMMPS calculations (the workflow remain the same)"
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": 4,
-   "id": "adequate-treasury",
+   "execution_count": 9,
+   "id": "attached-germany",
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Access the project created in exercise 1 \n",
     "pr_fs = pr[\"../first_steps\"]"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
-   "id": "usual-characterization",
+   "execution_count": 16,
+   "id": "filled-natural",
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Create a TrainingContainer job (to store structures and databases)\n",
     "container = pr.create.job.TrainingContainer('dataset_example')"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "roman-possible",
+   "id": "married-storm",
    "metadata": {},
    "source": [
-    "## **Add structures from the E-V curves**"
+    "## **Add structures from the E-V curves**\n",
+    "\n",
+    "For starters, we append structures from the energy volume curves we calculated earlier"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
-   "id": "above-sunrise",
+   "execution_count": 17,
+   "id": "mediterranean-upset",
    "metadata": {},
    "outputs": [],
    "source": [
-    "\n",
+    "# Iterate over the jobs in this sub-project and append the final structure, potential energy, and forces\n",
     "for job in pr_fs[\"E_V_curve\"].iter_jobs(status=\"finished\"):\n",
     "    container.include_job(job, iteration_step=-1)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "acquired-missile",
+   "metadata": {},
+   "source": [
+    "We can obtain this data as a `pandas` table"
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": 7,
-   "id": "revolutionary-response",
+   "execution_count": 18,
+   "id": "greatest-person",
    "metadata": {},
    "outputs": [
     {
@@ -762,7 +795,7 @@
        "46              1.0  "
       ]
      },
-     "execution_count": 7,
+     "execution_count": 18,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -773,33 +806,36 @@
   },
   {
    "cell_type": "markdown",
-   "id": "still-western",
+   "id": "imperial-belarus",
    "metadata": {},
    "source": [
     "## **Add structures from the MD**\n",
     "\n",
-    "Structures sampled every 10 snapshots (we may need to minimize the structures from the MD)"
+    "We also add some structures obtained from the MD simulations"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
-   "id": "accomplished-heart",
+   "execution_count": 20,
+   "id": "healthy-structure",
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Reloading the MD job\n",
     "job_md = pr_fs[\"lammps_job\"]"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
-   "id": "finite-improvement",
+   "execution_count": 21,
+   "id": "outside-bhutan",
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Iterate over the MD-trajectory to append structures\n",
+    "\n",
     "traj_length = len(job_md[\"output/generic/positions\"])\n",
-    "stride = 10\n",
+    "stride = 10 # append structures every 10 steps\n",
     "\n",
     "for i in range(0, traj_length, stride):\n",
     "    container.include_job(job_md, iteration_step=i)"
@@ -807,27 +843,30 @@
   },
   {
    "cell_type": "markdown",
-   "id": "informational-helen",
+   "id": "minus-blink",
    "metadata": {},
    "source": [
-    "## **Add some wierd structures (vacancies, surfaces, etc)**"
+    "## **Add some defect structures (vacancies, surfaces, etc)**\n",
+    "\n",
+    "It's necessary to also include some defect structures, and surfaces to the training dataset"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
-   "id": "fabulous-upset",
+   "execution_count": 22,
+   "id": "loaded-sheriff",
    "metadata": {},
    "outputs": [
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "2021-03-06 08:30:54,890 - pyiron_log - WARNING - The job lammps_job_vac is being loaded instead of running. To re-run use the argument 'delete_existing_job=True in create_job'\n"
+      "2021-03-08 11:55:40,043 - pyiron_log - WARNING - The job lammps_job_vac is being loaded instead of running. To re-run use the argument 'delete_existing_job=True in create_job'\n"
      ]
     }
    ],
    "source": [
+    "# Setup a MD calculation for a structure with a vacancy\n",
     "job_lammps = pr.create.job.Lammps(\"lammps_job_vac\")\n",
     "job_lammps.structure = pr.create_ase_bulk('Cu', cubic=True, a=3.61).repeat([3, 3, 3])\n",
     "del job_lammps.structure[0]\n",
@@ -838,19 +877,20 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 11,
-   "id": "smart-evanescence",
+   "execution_count": 28,
+   "id": "billion-shade",
    "metadata": {},
    "outputs": [
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "2021-03-06 08:30:56,197 - pyiron_log - WARNING - The job lammps_job_surf is being loaded instead of running. To re-run use the argument 'delete_existing_job=True in create_job'\n"
+      "2021-03-08 11:56:36,874 - pyiron_log - WARNING - The job lammps_job_surf is being loaded instead of running. To re-run use the argument 'delete_existing_job=True in create_job'\n"
      ]
     }
    ],
    "source": [
+    "# Setup a MD calculation for a surface structure\n",
     "job_lammps = pr.create.job.Lammps(\"lammps_job_surf\")\n",
     "job_lammps.structure = pr.create_surface(\"Cu\", surface_type=\"fcc111\", size=(4, 4, 8), vacuum=12, orthogonal=True)\n",
     "job_lammps.potential = '2012--Mendelev-M-I--Cu--LAMMPS--ipr1'\n",
@@ -860,8 +900,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 12,
-   "id": "future-louis",
+   "execution_count": 29,
+   "id": "constitutional-throw",
    "metadata": {},
    "outputs": [
     {
@@ -870,7 +910,7 @@
        "{'groups': ['tc'], 'nodes': ['lammps_job_vac', 'lammps_job_surf', 'dataset_example']}"
       ]
      },
-     "execution_count": 12,
+     "execution_count": 29,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -879,45 +919,53 @@
     "pr"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "alpine-cooking",
+   "metadata": {},
+   "source": [
+    "We now add these structures to the dataset"
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": 13,
-   "id": "former-marriage",
+   "execution_count": 33,
+   "id": "italian-cleanup",
    "metadata": {},
    "outputs": [],
    "source": [
-    "for job_md in pr.iter_jobs(convert_to_object=True, status=\"finished\"):\n",
-    "    # only check Lammps jobs\n",
+    "for job_name in [\"lammps_job_vac\", \"lammps_job_surf\"]:\n",
+    "    job_md = pr.load(job_name)\n",
     "    pos = job_md[\"output/generic/positions\"]\n",
-    "    if pos is not None:\n",
-    "        traj_length = len(pos)\n",
-    "        stride = 10\n",
-    "        for i in range(0, traj_length, stride):\n",
-    "            container.include_job(job_md, iteration_step=i)"
+    "    traj_length = len(pos)\n",
+    "    stride = 10\n",
+    "    for i in range(0, traj_length, stride):\n",
+    "        container.include_job(job_md, iteration_step=i)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 14,
-   "id": "genuine-booth",
+   "execution_count": 34,
+   "id": "exotic-asset",
    "metadata": {},
    "outputs": [
     {
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "2021-03-06 08:31:06,560 - pyiron_log - WARNING - The job dataset_example is being loaded instead of running. To re-run use the argument 'delete_existing_job=True in create_job'\n"
+      "2021-03-08 11:59:00,874 - pyiron_log - WARNING - The job dataset_example is being loaded instead of running. To re-run use the argument 'delete_existing_job=True in create_job'\n"
      ]
     }
    ],
    "source": [
+    "# We run the job sto store this dataset in the pyiron database\n",
     "container.run()"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 15,
-   "id": "warming-intake",
+   "execution_count": 32,
+   "id": "british-kelly",
    "metadata": {},
    "outputs": [
     {
@@ -1039,7 +1087,7 @@
        "2  pyiron@cmdell17#1  TrainingContainer        0.4     None     None  "
       ]
      },
-     "execution_count": 15,
+     "execution_count": 32,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1048,10 +1096,20 @@
     "pr.job_table()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "ideal-volunteer",
+   "metadata": {},
+   "source": [
+    "## **Reloading the dataset**\n",
+    "\n",
+    "This dataset can ow be reloaded anywhere to use in the potential fitting procedures"
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": 16,
-   "id": "physical-liverpool",
+   "execution_count": 35,
+   "id": "numeric-museum",
    "metadata": {},
    "outputs": [
     {
@@ -1619,7 +1677,7 @@
        "39            128.0  "
       ]
      },
-     "execution_count": 16,
+     "execution_count": 35,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -1629,10 +1687,18 @@
     "dataset.to_pandas()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "level-dimension",
+   "metadata": {},
+   "source": [
+    "We can now inspect the data in this dataset quite easily"
+   ]
+  },
   {
    "cell_type": "code",
-   "execution_count": 17,
-   "id": "female-thumb",
+   "execution_count": 36,
+   "id": "visible-execution",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -1641,18 +1707,26 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 18,
-   "id": "standard-whale",
+   "execution_count": 37,
+   "id": "becoming-integral",
    "metadata": {},
    "outputs": [],
    "source": [
     "structures, energies, forces, num_atoms  = dataset.to_list()"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "novel-usage",
+   "metadata": {},
+   "source": [
+    "The datasets used in the potential fitting procedure for day 2 (obtained from accurate DFT calculations) will be accessed in the same way"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "worldwide-freight",
+   "id": "forced-scotland",
    "metadata": {},
    "outputs": [],
    "source": []