Blog

Scaling On the Cheap with Dask, Gigantum, and DigitalOcean

Posted by Dav Clark on May 14, 2020 7:13:44 AM

Below, we’ll sketch out a smart approach for using lots of CPU cores without breaking the bank: using your laptop when feasible along with a DIY approach to working on bigger cloud resources as needed. We’ll use Gigantum to automate Git and Docker, along with most details of our cloud environment. With the following approach, you can be up and running Dask on 32 CPU cores on DigitalOcean in about 10 minutes - look at those tasks fly in parallel!

Read More

Topics: Reproducibility, Data Science, Open Science

Rebooting reproducibility: From re-execution to replication

Posted by Tyler Whitehouse, Dav Clark and Emmy Tsang on Jul 12, 2019 12:26:00 PM

6000x4000-5379227-mountain-rock-road-line-yellow-line-hill-red-rock-cliff-nature-national-park-state-park-nevada-open-road-adventure-travel-road-trip-creative-commons-images

Computational reproducibility should be trivial but it is not. Though code and data are increasingly shared, the community has realised that many other factors affect reproducibility, a typical example of which is the difficulty in reconstructing a work’s original library dependencies and software versions. The required level of detail documenting such aspects scales with the complexity of the problem, making the creation of user-friendly solutions very challenging.

Read More

Topics: Reproducibility, Data Science

Making Reproducibility Reproducible

gigantum blog post 12

Reproducibility doesn’t have to be magic, anymore. This image is provided by Abstruse Goose under the Creative Commons License

TL;DR - We believe the following

  • Approaches to the transmission of scientific knowledge are currently broken, mainly due to the criticality of software in modern research.
  • Calling re-execution of static results “reproducibility” isn’t enough. Reproducibility should be functionally equivalent to collaboration.
  • Academic emphasis on best practices is ineffective and should switch to a product based approach that minimizes effort rather than maximizes it.
  • By focusing on the needs of the end user, people can actually improve how scientific knowledge is communicated and shared.
Read More

Topics: Science, Reproducibility, Data Science, Containers, Jupyter

Gigantum – a simple way to create and share reproducible data science and research

Today, we present Gigantum, an open source platform for creating and collaborating on computational and analytic work, complete with:

  • Automated, high-resolution versioning of code, data and environment for reproducibility and rollback
  • Work and version history illustrated in a browsable activity feed
  • Streamlined environment management with customization via Docker snippets
  • One-click transfer between laptop and cloud for easy sharing
  • Seamless integration with development environments such as JupyterLab
Read More

Topics: Data Science, Software