Whale supports custom metadata scrapers through plain python scripts, as long as these satisfy two criteria:
A list of all tables should be generated in
Table "stubs" should be stored as markdown files in
To register a build script, simply run
wh init and follow the
build-script workflow, manually add details (see Connection configuration).
This can be particularly useful for organizations where metadata is already collected and stored in an intermediary repository (e.g. Airbnb or Lyft), or if you want to get metadata from a source we do not support. For an example, see the
databuilder/build_script.py file in the whale github repo.