Skip to content

hdx.utilities.path

module hdx.utilities.path

Directory Path Utilities.

Classes

Functions

  • script_dir — Get current script's directory.

  • script_dir_plus_file — Get current script's directory and then append a filename.

  • get_temp_dir — Get a temporary directory. Looks for environment variable TEMP_DIR and falls back on os.gettempdir if a root temporary directory is not supplied. If a folder is supplied, creates that folder within the temporary directory. Optionally deletes and recreates it if it already exists.

  • temp_dir — Get a temporary directory optionally with folder appended (and created if it doesn't exist)

  • read_or_create_batch — Get batch or create it if it doesn't exist.

  • temp_dir_batch — Get a temporary directory and batch id. Yields a dictionary with key folder which is the temporary directory optionally with folder appended (and created if it doesn't exist). In key batch is a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls.

  • get_wheretostart — Evaluate WHERETOSTART.

  • progress_storing_folder — Store progress in folder in key folder of info dictionary parameter. Yields 2 dictionaries. The first is the info dictionary. It contains in key folder the folder being used to store progress and in key progress the current position in the iterator. If store_batch is True, that dictionary will also contain the key batch containing a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls. The second dictionary is the next dictionary in the iterator.

  • wheretostart_tempdir_batch — Get a temporary directory and batch id. Deletes any existing folder if WHERETOSTART environment variable is set to RESET. Yields a dictionary with key folder which is the temporary directory optionally with folder appended (and created if it doesn't exist). In key batch is a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls.

  • progress_storing_tempdir — Store progress in temporary directory. The folder persists until the final iteration allowing which iteration to start at and the batch code to be persisted between runs. Yields 2 dictionaries. The first contains key folder which is the temporary directory optionally with folder appended (and created if it doesn't exist). In key progress is held the current position in the iterator. It also contains the key batch containing a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls. The second dictionary is the next dictionary in the iterator. The WHERETOSTART environment variable can be set to RESET to force the deletion and recreation of the temporary directory or to a key value pair in the form key=value eg. iso3=PAK indicating where to start.

  • multiple_progress_storing_tempdir — Store progress in temporary directory. The folder persists until the final iteration of the last iterator allowing which iteration to start at and the batch code to be persisted between runs. Yields 2 dictionaries. The first contains key folder which is the temporary directory optionally with folder appended (and created if it doesn't exist). In key progress is held the current position in the iterator. It also contains the key batch containing a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls. The second dictionary is the next dictionary in the iterator. The WHERETOSTART environment variable can be set to RESET to force the deletion and recreation of the temporary directory or to a key value pair in the form key=value eg. iso3=PAK indicating where to start.

class NotFoundError()

Bases : Exception

script_dir(pyobject: Any, follow_symlinks: bool = True) → Path

Get current script's directory.

Parameters

  • pyobject : Any — Any Python object in the script

  • follow_symlinks : bool — Follow symlinks or not. Defaults to True.

Returns

  • Path — Current script's directory

script_dir_plus_file(filename: str, pyobject: Any, follow_symlinks: bool = True) → Path

Get current script's directory and then append a filename.

Parameters

  • filename : str — Filename to append to directory path

  • pyobject : Any — Any Python object in the script

  • return_path — Whether to return a Path object. Defaults to returning str.

  • follow_symlinks : bool — Follow symlinks or not. Defaults to True.

Returns

  • Path — Current script's directory with filename appended

get_temp_dir(folder: Path | str | None = None, delete_if_exists: bool = False, tempdir: Path | str | None = None) → Path

Get a temporary directory. Looks for environment variable TEMP_DIR and falls back on os.gettempdir if a root temporary directory is not supplied. If a folder is supplied, creates that folder within the temporary directory. Optionally deletes and recreates it if it already exists.

Parameters

  • folder : Path | str | None — Folder to create in temporary folder. Defaults to None.

  • delete_if_exists : bool — Whether to delete the folder if it exists. Defaults to False.

  • tempdir : Path | str | None — Folder to use as temporary directory. Defaults to None (TEMP_DIR or os.gettempdir).

Returns

  • Path — A temporary directory

temp_dir(folder: Path | str | None = None, delete_if_exists: bool = False, delete_on_success: bool = True, delete_on_failure: bool = True, tempdir: Path | str | None = None) → Iterator[Path]

Get a temporary directory optionally with folder appended (and created if it doesn't exist)

Parameters

  • folder : Path | str | None — Folder to create in temporary folder. Defaults to None.

  • delete_if_exists : bool — Whether to delete the folder if it exists. Defaults to False.

  • delete_on_success : bool — Whether to delete folder (if folder supplied) on exiting with statement successfully. Defaults to True.

  • delete_on_failure : bool — Whether to delete folder (if folder supplied) on exiting with statement unsuccessfully. Defaults to True.

  • tempdir : Path | str | None — Folder to use as temporary directory. Defaults to None (TEMP_DIR or os.gettempdir).

Returns

  • Iterator[Path] — A temporary directory

read_or_create_batch(folder: Path, batch: str | None = None) → str

Get batch or create it if it doesn't exist.

Parameters

  • folder : Path — Folder in which to look for or create batch file.

  • batch : str | None — Batch to use if there isn't one in a file already.

Returns

  • str — Batch

temp_dir_batch(folder: Path | str | None = None, delete_if_exists: bool = False, delete_on_success: bool = True, delete_on_failure: bool = True, batch: str | None = None, tempdir: Path | str | None = None) → Iterator[dict]

Get a temporary directory and batch id. Yields a dictionary with key folder which is the temporary directory optionally with folder appended (and created if it doesn't exist). In key batch is a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls.

Parameters

  • folder : Path | str | None — Folder to create in temporary folder. Defaults to None.

  • delete_if_exists : bool — Whether to delete the folder if it exists. Defaults to False.

  • delete_on_success : bool — Whether to delete folder (if folder supplied) on exiting with statement successfully. Defaults to True.

  • delete_on_failure : bool — Whether to delete folder (if folder supplied) on exiting with statement unsuccessfully. Defaults to True.

  • batch : str | None — Batch to use if there isn't one in a file already.

  • tempdir : Path | str | None — Folder to use as temporary directory. Defaults to None (TEMP_DIR or os.gettempdir).

Returns

  • Iterator[dict] — Dictionary containing temporary directory in key folder and batch id in key batch

get_wheretostart(text: str, message: str, key: str) → str | None

Evaluate WHERETOSTART.

Parameters

  • text : str — String to process

  • message : str — Text for logging

  • key : str — Key to comapre with

Returns

  • str | None — A string or None

progress_storing_folder(info: dict, iterator: Iterable[dict], key: str, wheretostart: str | None = None) → Iterator[tuple[dict, dict]]

Store progress in folder in key folder of info dictionary parameter. Yields 2 dictionaries. The first is the info dictionary. It contains in key folder the folder being used to store progress and in key progress the current position in the iterator. If store_batch is True, that dictionary will also contain the key batch containing a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls. The second dictionary is the next dictionary in the iterator.

Parameters

  • info : dict — Dictionary containing folder and anything else to be yielded

  • iterator : Iterable[dict] — Iterate over this object persisting progress

  • key : str — Key to examine from dictionary from iterator

  • wheretostart : str | None — Where in iterator to start

Returns

  • Iterator[tuple[dict, dict]] — A tuple of the form (info dictionary, next object in iterator)

Raises

wheretostart_tempdir_batch(folder: Path | str, batch: str | None = None, tempdir: Path | str | None = None) → Iterator[dict]

Get a temporary directory and batch id. Deletes any existing folder if WHERETOSTART environment variable is set to RESET. Yields a dictionary with key folder which is the temporary directory optionally with folder appended (and created if it doesn't exist). In key batch is a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls.

Parameters

  • folder : Path | str — Folder to create in temporary folder

  • batch : str | None — Batch to use if there isn't one in a file already.

  • tempdir : Path | str | None — Folder to use as temporary directory. Defaults to None (TEMP_DIR or os.gettempdir).

Returns

  • Iterator[dict] — Dictionary containing temporary directory in key folder and batch id in key batch

progress_storing_tempdir(folder: Path | str, iterator: Iterable[dict], key: str, batch: str | None = None, tempdir: Path | str | None = None) → Iterator[tuple[dict, dict]]

Store progress in temporary directory. The folder persists until the final iteration allowing which iteration to start at and the batch code to be persisted between runs. Yields 2 dictionaries. The first contains key folder which is the temporary directory optionally with folder appended (and created if it doesn't exist). In key progress is held the current position in the iterator. It also contains the key batch containing a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls. The second dictionary is the next dictionary in the iterator. The WHERETOSTART environment variable can be set to RESET to force the deletion and recreation of the temporary directory or to a key value pair in the form key=value eg. iso3=PAK indicating where to start.

Parameters

  • folder : Path | str — Folder to create in temporary folder

  • iterator : Iterable[dict] — Iterate over the iterator persisting progress

  • key : str — Key to examine from dictionary from iterator

  • batch : str | None — Batch to use if there isn't one in a file already.

  • tempdir : Path | str | None — Folder to use as temporary directory. Defaults to None (TEMP_DIR or os.gettempdir).

Returns

  • Iterator[tuple[dict, dict]] — A tuple of the form (info dictionary, next object in iterator)

multiple_progress_storing_tempdir(folder: Path | str, iterators: Sequence[Iterable[dict]], keys: Sequence[str], batch: str | None = None) → Iterator[tuple[int, dict, dict]]

Store progress in temporary directory. The folder persists until the final iteration of the last iterator allowing which iteration to start at and the batch code to be persisted between runs. Yields 2 dictionaries. The first contains key folder which is the temporary directory optionally with folder appended (and created if it doesn't exist). In key progress is held the current position in the iterator. It also contains the key batch containing a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls. The second dictionary is the next dictionary in the iterator. The WHERETOSTART environment variable can be set to RESET to force the deletion and recreation of the temporary directory or to a key value pair in the form key=value eg. iso3=PAK indicating where to start.

Parameters

  • folder : Path | str — Folder to create in temporary folder

  • iterators : Sequence[Iterable[dict]] — Iterate over each iterator in the list consecutively persisting progress

  • keys : Sequence[str] — Key to examine from dictionary from each iterator in the above list

  • batch : str | None — Batch to use if there isn't one in a file already.

Returns

  • Iterator[tuple[int, dict, dict]] — A tuple of the form (iterator index, info dictionary, next object in iterator)