hdx.utilities.path

Directory Path Utilities.

script_dir

def script_dir(pyobject: Any, follow_symlinks: bool = True) -> str

[view_source]

Get current script's directory.

Arguments:

pyobject Any - Any Python object in the script
follow_symlinks bool - Follow symlinks or not. Defaults to True.

Returns:

str - Current script's directory

script_dir_plus_file

def script_dir_plus_file(filename: str,
                         pyobject: Any,
                         follow_symlinks: bool = True) -> str

[view_source]

Get current script's directory and then append a filename.

Arguments:

filename str - Filename to append to directory path
pyobject Any - Any Python object in the script
follow_symlinks bool - Follow symlinks or not. Defaults to True.

Returns:

str - Current script's directory and with filename appended

get_temp_dir

def get_temp_dir(folder: Optional[str] = None,
                 delete_if_exists: bool = False,
                 tempdir: Optional[str] = None) -> str

[view_source]

Get a temporary directory. Looks for environment variable TEMP_DIR and falls back on os.gettempdir if a root temporary directory is not supplied. If a folder is supplied, creates that folder within the temporary directory. Optionally deletes and recreates it if it already exists.

Arguments:

folder Optional[str] - Folder to create in temporary folder. Defaults to None.
delete_if_exists bool - Whether to delete the folder if it exists. Defaults to False.
tempdir Optional[str] - Folder to use as temporary directory. Defaults to None (TEMP_DIR or os.gettempdir).

Returns:

str - A temporary directory

temp_dir

@contextlib.contextmanager
def temp_dir(folder: Optional[str] = None,
             delete_if_exists: bool = False,
             delete_on_success: bool = True,
             delete_on_failure: bool = True,
             tempdir: Optional[str] = None) -> str

[view_source]

Get a temporary directory optionally with folder appended (and created if it doesn't exist)

Arguments:

folder Optional[str] - Folder to create in temporary folder. Defaults to None.
delete_if_exists bool - Whether to delete the folder if it exists. Defaults to False.
delete_on_success bool - Whether to delete folder (if folder supplied) on exiting with statement successfully. Defaults to True.
delete_on_failure bool - Whether to delete folder (if folder supplied) on exiting with statement unsuccessfully. Defaults to True.
tempdir Optional[str] - Folder to use as temporary directory. Defaults to None (TEMP_DIR or os.gettempdir).

Returns:

str - A temporary directory

read_or_create_batch

def read_or_create_batch(folder: str, batch: Optional[str] = None) -> str

[view_source]

Get batch or create it if it doesn't exist.

Arguments:

folder str - Folder in which to look for or create batch file.
batch Optional[str] - Batch to use if there isn't one in a file already.

Returns:

str - Batch

temp_dir_batch

@contextlib.contextmanager
def temp_dir_batch(folder: Optional[str] = None,
                   delete_if_exists: bool = False,
                   delete_on_success: bool = True,
                   delete_on_failure: bool = True,
                   batch: Optional[str] = None,
                   tempdir: Optional[str] = None) -> Dict

[view_source]

Get a temporary directory and batch id. Yields a dictionary with key folder which is the temporary directory optionally with folder appended (and created if it doesn't exist). In key batch is a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls.

Arguments:

folder Optional[str] - Folder to create in temporary folder. Defaults to None.
delete_if_exists bool - Whether to delete the folder if it exists. Defaults to False.
delete_on_success bool - Whether to delete folder (if folder supplied) on exiting with statement successfully. Defaults to True.
delete_on_failure bool - Whether to delete folder (if folder supplied) on exiting with statement unsuccessfully. Defaults to True.
batch Optional[str] - Batch to use if there isn't one in a file already.
tempdir Optional[str] - Folder to use as temporary directory. Defaults to None (TEMP_DIR or os.gettempdir).

Returns:

Dict - Dictionary containing temporary directory in key folder and batch id in key batch

get_wheretostart

def get_wheretostart(text: str, message: str, key: str) -> Optional[str]

[view_source]

Evaluate WHERETOSTART.

Arguments:

text str - String to process
message str - Text for logging
key str - Key to comapre with

Returns:

Optional[str] - A string or None

progress_storing_folder

def progress_storing_folder(
        info: Dict,
        iterator: Iterable[Dict],
        key: str,
        wheretostart: Optional[str] = None) -> Tuple[Dict, Dict]

[view_source]

Store progress in folder in key folder of info dictionary parameter. Yields 2 dictionaries. The first is the info dictionary. It contains in key folder the folder being used to store progress and in key progress the current position in the iterator. If store_batch is True, that dictionary will also contain the key batch containing a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls. The second dictionary is the next dictionary in the iterator.

Arguments:

info Dict - Dictionary containing folder and anything else to be yielded
iterator Iterable[Dict] - Iterate over this object persisting progress
key str - Key to examine from dictionary from iterator
wheretostart Optional[str] - Where in iterator to start

Returns:

Tuple[Dict,Dict] - A tuple of the form (info dictionary, next object in iterator)

wheretostart_tempdir_batch

@contextlib.contextmanager
def wheretostart_tempdir_batch(folder: str,
                               batch: Optional[str] = None,
                               tempdir: Optional[str] = None) -> Dict

[view_source]

Get a temporary directory and batch id. Deletes any existing folder if WHERETOSTART environment variable is set to RESET. Yields a dictionary with key folder which is the temporary directory optionally with folder appended (and created if it doesn't exist). In key batch is a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls.

Arguments:

folder str - Folder to create in temporary folder
batch Optional[str] - Batch to use if there isn't one in a file already.
tempdir Optional[str] - Folder to use as temporary directory. Defaults to None (TEMP_DIR or os.gettempdir).

Returns:

Dict - Dictionary containing temporary directory in key folder and batch id in key batch

progress_storing_tempdir

def progress_storing_tempdir(
        folder: str,
        iterator: Iterable[Dict],
        key: str,
        batch: Optional[str] = None,
        tempdir: Optional[str] = None) -> Tuple[Dict, Dict]

[view_source]

Store progress in temporary directory. The folder persists until the final iteration allowing which iteration to start at and the batch code to be persisted between runs. Yields 2 dictionaries. The first contains key folder which is the temporary directory optionally with folder appended (and created if it doesn't exist). In key progress is held the current position in the iterator. It also contains the key batch containing a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls. The second dictionary is the next dictionary in the iterator. The WHERETOSTART environment variable can be set to RESET to force the deletion and recreation of the temporary directory or to a key value pair in the form key=value eg. iso3=PAK indicating where to start.

Arguments:

folder str - Folder to create in temporary folder
iterator Iterable[Dict] - Iterate over the iterator persisting progress
key str - Key to examine from dictionary from iterator
batch Optional[str] - Batch to use if there isn't one in a file already.
tempdir Optional[str] - Folder to use as temporary directory. Defaults to None (TEMP_DIR or os.gettempdir).

Returns:

Tuple[Dict,Dict] - A tuple of the form (info dictionary, next object in iterator)

multiple_progress_storing_tempdir

def multiple_progress_storing_tempdir(
        folder: str,
        iterators: ListTuple[Iterable[Dict]],
        keys: ListTuple[str],
        batch: Optional[str] = None) -> Tuple[Dict, Dict]

[view_source]

Store progress in temporary directory. The folder persists until the final iteration of the last iterator allowing which iteration to start at and the batch code to be persisted between runs. Yields 2 dictionaries. The first contains key folder which is the temporary directory optionally with folder appended (and created if it doesn't exist). In key progress is held the current position in the iterator. It also contains the key batch containing a batch code to be passed as the batch parameter in create_in_hdx or update_in_hdx calls. The second dictionary is the next dictionary in the iterator. The WHERETOSTART environment variable can be set to RESET to force the deletion and recreation of the temporary directory or to a key value pair in the form key=value eg. iso3=PAK indicating where to start.

Arguments:

folder str - Folder to create in temporary folder
iterators ListTuple[Iterable[Dict] - Iterate over each iterator in the list consecutively persisting progress
keys ListTuple[str] - Key to examine from dictionary from each iterator in the above list
batch Optional[str] - Batch to use if there isn't one in a file already.

Returns:

Tuple[int, Dict,Dict]: A tuple of the form (iterator index, info dictionary, next object in iterator)

get_filename_extension_from_url

def get_filename_extension_from_url(
        url: str,
        second_last: bool = False,
        use_query: bool = False) -> Tuple[str, str]

[view_source]

Get separately filename and extension from url.

Arguments:

url str - URL to download
second_last bool - Get second last segment of url as well. Defaults to False.
use_query bool - Include query parameters as well. Defaults to False.

Returns:

Tuple[str,str] - Tuple of (filename, extension)

get_filename_from_url

def get_filename_from_url(url: str,
                          second_last: bool = False,
                          use_query: bool = False) -> str

[view_source]

Get filename including extension from url.

Arguments:

url str - URL
second_last bool - Get second last segment of url as well. Defaults to False.
use_query bool - Include query parameters as well. Defaults to False.

Returns:

str - filename

Keys	Action
`?`	Open this help
`n`	Next page
`p`	Previous page
`s`	Search