Skip to content

hdx.utilities.retriever

module hdx.utilities.retriever

Classes

  • Retrieve — Retrieve class which takes in a Download object and can either download, download and save or use previously downloaded and saved data. It also allows the use of a static fallback when downloading fails.

class Retrieve(downloader: Download, fallback_dir: Path | str, saved_dir: Path | str, temp_dir: Path | str, save: bool = False, use_saved: bool = False, prefix: str = '', delete: bool = True, log_level: int = logging.INFO)

Bases : BaseDownload

Retrieve class which takes in a Download object and can either download, download and save or use previously downloaded and saved data. It also allows the use of a static fallback when downloading fails.

Parameters

  • downloader : Download — Download object

  • fallback_dir : Path | str — Directory containing static fallback data

  • saved_dir : Path | str — Directory to save or load downloaded data

  • temp_dir : Path | str — Temporary directory for when data is not needed after downloading

  • save : bool — Whether to save downloaded data. Defaults to False.

  • use_saved : bool — Whether to use saved data. Defaults to False.

  • prefix : str — Prefix to add to filenames. Defaults to "".

  • delete : bool — Whether to delete saved_dir if save is True. Defaults to True.

  • log_level : int — Level at which to log messages. Defaults to logging.INFO.

Methods

  • check_flags — Check flags. Also delete saved_dir if save and delete are True.

  • get_url_logstr — Url string that will be logged. It is limited to 100 characters if necessary.

  • clone — Clone a given retriever but use the given downloader.

  • get_filename — Get filename from url and given parameters.

  • set_bearer_token — Set bearer token in downloader

  • download_file — Retrieve file.

  • download_text — Download text.

  • download_yaml — Retrieve YAML.

  • download_json — Retrieve JSON.

  • get_tabular_rows — Returns header of tabular file(s) pointed to by url and an iterator where each row is returned as a list or dictionary depending on the dict_rows argument.

  • generate_retrievers — Generate retrievers. Retrievers are generated from downloaders so Download.generate_downloaders() needs to have been called first. Each retriever can either download, download and save or use previously downloaded and saved data. It also allows the use of a static fallback when downloading fails.

  • get_retriever — Get a generated retriever given a name. If name is not supplied, the default one will be returned.

staticmethod Retrieve.check_flags(saved_dir: Path | str, save: bool, use_saved: bool, delete: bool) → None

Check flags. Also delete saved_dir if save and delete are True.

Parameters

  • saved_dir : Path | str — Directory to save or load downloaded data

  • save : bool — Whether to save downloaded data

  • use_saved : bool — Whether to use saved data

  • delete : bool — Whether to delete saved_dir if save is True

Returns

  • None — None

Raises

  • ValueError

staticmethod Retrieve.get_url_logstr(url: Path | str) → str

Url string that will be logged. It is limited to 100 characters if necessary.

Parameters

  • url : Path | str — URL to download

Returns

  • str — Url string to use in logs

method Retrieve.clone(downloader: Download) → Retrieve

Clone a given retriever but use the given downloader.

Parameters

  • downloader : Download — Downloader to use

Returns

method Retrieve.get_filename(url: Path | str, filename: str | None = None, possible_extensions: tuple[str, ...] = tuple(), **kwargs: Any) → tuple[str, Any]

Get filename from url and given parameters.

Parameters

  • url : Path | str — Url from which to get filename

  • filename : str | None — Filename to use. Defaults to None (infer from url).

  • possible_extensions : tuple[str, ...] — Possible extensions to look for in url

  • **kwargs : Any — See below

  • format : str — Given extension to look for in url

  • file_type : str — Given extension to look for in url

Returns

  • tuple[str, Any] — Tuple of (filename, kwargs)

method Retrieve.set_bearer_token(bearer_token: str) → None

Set bearer token in downloader

Parameters

  • bearer_token : str — Bearer token

Returns

  • None — None

method Retrieve.download_file(url: Path | str, filename: str | None = None, logstr: str | None = None, fallback: bool = False, resume: bool = False, retries: int = 0, log_level: int = None, **kwargs: Any) → Path

Retrieve file.

Parameters

  • url : Path | str — URL to download

  • filename : str | None — Filename of saved file. Defaults to getting from url.

  • logstr : str | None — Text to use in log string to describe download. Defaults to filename.

  • fallback : bool — Whether to use static fallback if download fails. Defaults to False.

  • resume : bool — Whether to resume a partial download using Range requests where the server supports it. Defaults to False.

  • retries : int — Number of times to retry a mid-stream failure. Only effective when resume=True. Defaults to 0.

  • log_level : int — Level at which to log messages. Overrides level from constructor.

  • **kwargs : Any — Parameters to pass to download_file call

Returns

  • Path — Path to downloaded file

method Retrieve.download_text(url: Path | str, filename: str | None = None, logstr: str | None = None, fallback: bool = False, log_level: int = None, **kwargs: Any) → str

Download text.

Parameters

  • url : Path | str — URL to download

  • filename : str | None — Filename of saved file. Defaults to getting from url.

  • logstr : str | None — Text to use in log string to describe download. Defaults to filename.

  • fallback : bool — Whether to use static fallback if download fails. Defaults to False.

  • log_level : int — Level at which to log messages. Overrides level from constructor.

  • **kwargs : Any — Parameters to pass to download_text call

Returns

  • str — The text from the file

method Retrieve.download_yaml(url: Path | str, filename: str | None = None, logstr: str | None = None, fallback: bool = False, log_level: int = None, **kwargs: Any) → Any

Retrieve YAML.

Parameters

  • url : Path | str — URL to download

  • filename : str | None — Filename of saved file. Defaults to getting from url.

  • logstr : str | None — Text to use in log string to describe download. Defaults to filename.

  • fallback : bool — Whether to use static fallback if download fails. Defaults to False.

  • log_level : int — Level at which to log messages. Overrides level from constructor.

  • **kwargs : Any — Parameters to pass to download_yaml call

Returns

  • Any — The data from the YAML file

method Retrieve.download_json(url: Path | str, filename: str | None = None, logstr: str | None = None, fallback: bool = False, log_level: int = None, **kwargs: Any) → Any

Retrieve JSON.

Parameters

  • url : Path | str — URL to download

  • filename : str | None — Filename of saved file. Defaults to getting from url.

  • logstr : str | None — Text to use in log string to describe download. Defaults to filename.

  • fallback : bool — Whether to use static fallback if download fails. Defaults to False.

  • log_level : int — Level at which to log messages. Overrides level from constructor.

  • **kwargs : Any — Parameters to pass to download_json call

Returns

  • Any — The data from the JSON file

method Retrieve.get_tabular_rows(url: Path | str | Sequence[str], headers: int | Sequence[int] | Sequence[str] = 1, dict_form: bool = False, filename: str | None = None, logstr: str | None = None, fallback: bool = False, **kwargs: Any) → tuple[list[str], Iterator[list | dict]]

Returns header of tabular file(s) pointed to by url and an iterator where each row is returned as a list or dictionary depending on the dict_rows argument.

The headers argument is either a row number or list of row numbers (in case of multi-line headers) to be considered as headers (rows start counting at 1), or the actual headers defined as a list of strings. It defaults to 1. The dict_form arguments specifies if each row should be returned as a dictionary or a list, defaulting to a list.

Parameters

  • url : Path | str | Sequence[str] — A single or list of URLs or paths to read from

  • headers : int | Sequence[int] | Sequence[str] — Number of row(s) containing headers or list of headers. Defaults to 1.

  • dict_form : bool — Return dict or list for each row. Defaults to False (list)

  • filename : str | None — Filename of saved file. Defaults to getting from url.

  • logstr : str | None — Text to use in log string to describe download. Defaults to filename.

  • fallback : bool — Whether to use static fallback if download fails. Defaults to False.

  • **kwargs : Any — Parameters to pass to download_file and get_tabular_rows calls

Returns

  • tuple[list[str], Iterator[list | dict]] — Tuple (headers, iterator where each row is a list or dictionary)

classmethod Retrieve.generate_retrievers(fallback_dir: Path | str, saved_dir: Path | str, temp_dir: Path | str, save: bool = False, use_saved: bool = False, ignore: Sequence[str] = tuple(), delete: bool = True, **kwargs: Any) → None

Generate retrievers. Retrievers are generated from downloaders so Download.generate_downloaders() needs to have been called first. Each retriever can either download, download and save or use previously downloaded and saved data. It also allows the use of a static fallback when downloading fails.

Parameters

  • fallback_dir : Path | str — Directory containing static fallback data

  • saved_dir : Path | str — Directory to save or load downloaded data

  • temp_dir : Path | str — Temporary directory for when data is not needed after downloading

  • save : bool — Whether to save downloaded data. Defaults to False.

  • use_saved : bool — Whether to use saved data. Defaults to False.

  • ignore : Sequence[str] — Don't generate retrievers for these downloaders

  • delete : bool — Whether to delete saved_dir if save is True. Defaults to True.

  • **kwargs : Any — Any other arguments to pass.

Returns

  • None — None

classmethod Retrieve.get_retriever(name: str | None = None) → Retrieve

Get a generated retriever given a name. If name is not supplied, the default one will be returned.

Parameters

  • name : str | None — Name of retriever. Defaults to None (get default).

Returns