Databricks natively stores it’s notebook files by default as DBC files, a closed, binary format. A .dbc file has a nice benefit of being self-contained. One dbc file can consist of an entire folder of notebooks and supporting files. But other than that, dbc files are frankly obnoxious.
- I can’t view them outside of the Databricks notebook/workspace UI
- The native nbviewer in github doesn’t recognize them (nbviewer is what allows github to view ipynb files (Jupyter notebook files))
- they are binary (that makes git diffs impossible)
- ipynb files are json and I posit that they actually aren’t human-readable, but I can at least do a modicum of diffing/merging if I needed to.
The git integration in the Databricks UI is passable, but lacking. One example, each notebook must be saved as a separate commit even though any given feature/bug may span multiple notebooks.
So, what’s the solution?
I’ve never seen this published before, but I was poking around the Databricks CLI and noticed it can actually do all of this for you. Here’s a couple of sample scripts that demonstrate some methods of doing notebook lifecycle using the CLI:
pip3 install --upgrade databricks-cli databricks --version # let's use a personal access token # databricks|user settings|access tokens # https://eastus2.azuredatabricks.net databricks configure --token databricks workspace -h databricks workspace ls /Usersfirstname.lastname@example.org databricks workspace ls /Usersemail@example.com/OpenHack # default format is SOURCE, also the only(???) format for export_dir databricks workspace export_dir /Usersfirstname.lastname@example.org/OpenHack . databricks workspace export --format JUPYTER /Usersemail@example.com/OpenHack/01_useful_functions .
All we really need to do is download our notebooks with a little sh script and then call
git commit, etc, as needed.
Are you convinced your data or cloud project will be a success?
Most companies aren’t. I have lots of experience with these projects. I speak at conferences, host hackathon events, and am a prolific open source contributor. I love helping companies with Data problems. If that sounds like someone you can trust, contact me.
Thanks for reading. If you found this interesting please subscribe to my blog.
- Marketing Trends for 2021
- E-Commerce is Dead, Long Live A-Commerce
- PolarisCon Conference Speaker
- Forrester 2020 Enterprise Architecture Award Winner
- MTC MLOps Workshops and Hackathons
Dave Wentzel CONTENT
azure data lake data science