hdx.utilities.text

module hdx.utilities.text

Text processing utilities.

Functions

• normalise — Mormalise text for example to support name matching. Accented characters are replaced with non-accented if possible. Any punctuation and whitespace is replaced with a space except for ' which is replaced with blank. Multiple spaces are replaced with a single space. Uppercase is replaced with lowercase. Spaces at start and end are removed. All non-ASCII characters are removed.

• remove_end_characters — Remove any characters at end of string that are in characters_to_remove.

• remove_from_end — Remove list of items from end of string, stripping any whitespace.

• remove_string — Remove string from another string and delete any preceding end characters - by default punctuation (eg. comma) and any whitespace following the punctuation

• get_words_in_sentence — Returns list of words in a sentence.

• number_format — Format float-castable input as string.

• get_fraction_str — Given float-castable numerator and optional float-castable denominator, format as string, returning '' for invalid numerator or 0 denominator.

• only_allowed_in_str — Returns True if test string contains only allowed characters, False if not.

• get_numeric_if_possible — Return val if it is not a string, otherwise see if it can be cast to float or int, taking into account commas and periods.

• smart_split — Splits text into paragraphs based on sentence length and structure.

normalise(text: str) → str

Mormalise text for example to support name matching. Accented characters are replaced with non-accented if possible. Any punctuation and whitespace is replaced with a space except for ' which is replaced with blank. Multiple spaces are replaced with a single space. Uppercase is replaced with lowercase. Spaces at start and end are removed. All non-ASCII characters are removed.

Parameters

• text : str — Text to normalise

Returns

• str — Normalised text

remove_end_characters(string: str, characters_to_remove: str = punctuation) → str

Remove any characters at end of string that are in characters_to_remove.

Parameters

• string : str — Input string

• characters_to_remove : str — Characters to remove. Defaults to punctuation.

Returns

• str — String with any characters at end of string that are in characters_to_remove removed

remove_from_end(string: str, things_to_remove: list[str], logging_text: str | None = None, whole_words: bool = True) → str

Remove list of items from end of string, stripping any whitespace.

Parameters

• string : str — Input string

• things_to_remove : list[str] — Things to remove from the end of string

• logging_text : str | None — Text to log. Defaults to None.

• whole_words : bool — Remove parts of or whole words. Defaults to True (whole words only).

Returns

• str — String with text removed

remove_string(string: str, toremove: str, end_characters_to_remove: str = punctuation) → str

Remove string from another string and delete any preceding end characters - by default punctuation (eg. comma) and any whitespace following the punctuation

Parameters

• string : str — String to process

• toremove : str — String to remove

• end_characters_to_remove : str — Characters to remove. Defaults to punctuation.

Returns

• str — String with other string removed

get_words_in_sentence(sentence: str) → list[str]

Returns list of words in a sentence.

Parameters

• sentence : str — Sentence

Returns

• list[str] — List of words in sentence

number_format(val: Any, format: str = '%.4f', trailing_zeros: bool = True) → str

Format float-castable input as string.

Parameters

• val : Any — Number to format

• format : str — Format to use. Defaults to %.4f.

• trailing_zeros : bool — Leave trailing zeros. Defaults to True.

Returns

• str — Formatted number as string

get_fraction_str(numerator: Any, denominator: Any | None = None, format: str = '%.4f', trailing_zeros: bool = True) → str

Given float-castable numerator and optional float-castable denominator, format as string, returning '' for invalid numerator or 0 denominator.

Parameters

• numerator : Any — Numerator

• denominator : Any | None — Denominator. Defaults to None.

• format : str — Format to use. Defaults to %.4f.

• trailing_zeros : bool — Leave trailing zeros. Defaults to True.

Returns

• str — Formatted number as string

only_allowed_in_str(test_str: str, allowed_chars: set) → bool

Returns True if test string contains only allowed characters, False if not.

Parameters

• test_str : str — Test string

• allowed_chars : set — Set of allowed characters

Returns

• bool — True if test string contains only allowed characters, False if not

get_numeric_if_possible(value: Any) → Any

Return val if it is not a string, otherwise see if it can be cast to float or int, taking into account commas and periods.

Parameters

• value : Any — Value

Returns

• Any — Value

smart_split(text: str, target_length: int = 400) → str

Splits text into paragraphs based on sentence length and structure.

Parameters

• text : str — The input text.

• target_length : int — Approximate characters per paragraph. Defaults to 400.

Returns

• str — The output text