hdx.utilities.downloader

Downloading utilities for urls.

Classes

Download — Download class with various download operations. Requires either global user agent to be set or appropriate user agent parameter(s) to be completed. If the EXTRA_PARAMS, BASIC_AUTH or BEARER_TOKEN environment variable is supplied, the extra_params* parameters will be ignored. extra_params_dict takes precedence over extra_params_json and extra_params_yaml. extra_params_lookup, if supplied, only applies to extra_params_json and extra_params_yaml.

Bases : BaseDownload

Download class with various download operations. Requires either global user agent to be set or appropriate user agent parameter(s) to be completed. If the EXTRA_PARAMS, BASIC_AUTH or BEARER_TOKEN environment variable is supplied, the extra_params* parameters will be ignored. extra_params_dict takes precedence over extra_params_json and extra_params_yaml. extra_params_lookup, if supplied, only applies to extra_params_json and extra_params_yaml.

Parameters

user_agent : str | None — User agent string. HDXPythonUtilities/X.X.X- is prefixed.
user_agent_config_yaml : Path | str | None — Path to YAML user agent configuration. Ignored if user_agent supplied. Defaults to ~/.useragent.yaml.
user_agent_lookup : str | None — Lookup key for YAML. Ignored if user_agent supplied.
use_env : bool — Whether to read environment variables. Defaults to True.
fail_on_missing_file : bool — Raise an exception if any specified configuration files are missing. Defaults to True.
verify : bool — Whether to verify SSL certificates. Defaults to True.
rate_limit : dict | None — Rate limiting per host eg. {"calls": 1, "period": 0.1}. Defaults to None.
**kwargs : Any — See below
auth : tuple[str, str] — Authorisation information in tuple form (user, pass) OR
basic_auth : str — Authorisation information in basic auth string form (Basic xxxxxxxxxxxxxxxx) OR
basic_auth_file : str — Path to file containing authorisation information in basic auth string form (Basic xxxxxxxxxxxxxxxx)
bearer_token : str — Bearer token string OR
bearer_token_file : str — Path to file containing bearer token string OR
extra_params_dict : dict[str, str] — Extra parameters to put on end of url as a dictionary OR
extra_params_json : str — Path to JSON file containing extra parameters to put on end of url OR
extra_params_yaml : str — Path to YAML file containing extra parameters to put on end of url
extra_params_lookup : str — Lookup key for parameters. If not given assumes parameters are at root of the dict.
headers : dict — Additional headers to add to request.
use_auth : str — If more than one auth found, specify which one to use, rather than failing.
status_forcelist : Sequence[int] — HTTP statuses for which to force retry
allowed_methods : iterable — HTTP methods for which to force retry. Defaults t0 frozenset(['GET']).

Methods

close_response — Close response.
close — Close response and session.
get_full_url — Get full url including any additional parameters added to the session.
normal_setup — Setup download from provided url returning the response.
set_bearer_token — Set bearer token
hash_stream — Stream file from url and hash it using MD5. Must call setup method first.
stream_path — Stream file from url and store in provided path. Must call setup method first.
stream_file — Stream file from url and store in provided folder or temporary folder if no folder supplied. Must call setup method first.
download_file — Download file from url and store in provided folder or temporary folder if no folder supplied.
download — Download url.
get_header — Get a particular response header of download.
get_headers — Get response headers of download.
get_status — Get response status code.
get_text — Get text content of download.
get_yaml — Get YAML content of download.
get_json — Get JSON content of download.
download_text — Download url as text.
download_yaml — Download url as YAML.
download_json — Download url as JSON.
get_frictionless_tableresource — Get Frictionless TableResource.
get_tabular_rows — Returns header of tabular file(s) pointed to by url and an iterator where each row is returned as a list or dictionary depending on the dict_rows argument.
get_tabular_rows_as_list — Returns headers and an iterator where each row is returned as a list.
get_tabular_rows_as_dict — Returns headers and an iterator where each row is returned as a dictionary.
download_tabular_key_value — Download 2 column csv from url and return a dictionary of keys (first column) and values (second column).
download_tabular_rows_as_dicts — Download multicolumn csv from url and return dictionary where keys are first column and values are dictionaries with keys from column headers and values from columns beneath.
download_tabular_cols_as_dicts — Download multicolumn csv from url and return dictionary where keys are header names and values are dictionaries with keys from first column and values from other columns.
get_column_positions — Get mapping of headers to column positions.
generate_downloaders — Generate downloaders. Requires either global user agent to be set or appropriate user agent parameter(s) to be completed. The custom_configs dictionary is a mapping from name to a dictionary of custom configuration parameters that is added to the underlying session's params or headers. It can have keys that correspond to the input arguments of Download's constructor init (or the other arguments of this method).
get_downloader — Get a generated downloader given a name. If name is not supplied, the default one will be returned.

method Download.close_response() → None

Close response.

Returns

None — None

method Download.close() → None

Close response and session.

Returns

None — None

method Download.get_full_url(url: Path | str) → str

Get full url including any additional parameters added to the session.

Parameters

url : Path | str — URL for which to get full url

Returns

str — Full url including any additional parameters

Setup download from provided url returning the response.

Parameters

url : Path | str — URL or path to download
stream : bool — Whether to stream download. Defaults to True.
post : bool — Whether to use POST instead of GET. Defaults to False.
parameters : dict | None — Parameters to pass. Defaults to None.
timeout : float | None — Timeout for connecting to URL. Defaults to None (no timeout).
headers : dict | None — Headers to pass. Defaults to None.
encoding : str | None — Encoding to use for text response. Defaults to None (best guess).
json_string : bool — Whether to post parameters as JSON string. Defaults to False.

Returns

requests.Response — requests.Response object

Raises

DownloadError

method Download.set_bearer_token(bearer_token: str) → None

Set bearer token

Parameters

bearer_token : str — Bearer token

Returns

None — None

method Download.hash_stream(url: Path | str) → str

Stream file from url and hash it using MD5. Must call setup method first.

Parameters

url : Path | str — URL or path to download

Returns

str — MD5 hash of file

Raises

DownloadError

method Download.stream_path(path: Path | str, errormsg: str, append: bool = False) → Path

Stream file from url and store in provided path. Must call setup method first.

Parameters

path : Path | str — Path for downloaded file
errormsg : str — Error message to display if there is a problem
append : bool — Whether to append to an existing file. Defaults to False.

Returns

Path — Path of downloaded file

Raises

DownloadError

Stream file from url and store in provided folder or temporary folder if no folder supplied. Must call setup method first.

Parameters

url : Path | str — URL or path to download
folder : Path | str | None — Folder to download it to. Defaults to None (temporary folder).
filename : str | None — Filename to use for downloaded file. Defaults to None (derive from the url).
path : Path | str | None — Full path to use for downloaded file. Defaults to None (use folder and filename).
overwrite : bool — Whether to overwrite existing file. Defaults to False.
keep : bool — Whether to keep already downloaded file. Defaults to False.

Returns

str — Path of downloaded file

Raises

DownloadError

method Download.download_file(url: Path | str, **kwargs: Any) → Path

Download file from url and store in provided folder or temporary folder if no folder supplied.

Parameters

url : Path | str — URL or path to download
**kwargs : Any — See below
folder : str — Folder to download it to. Defaults to temporary folder.
filename : str — Filename to use for downloaded file. Defaults to deriving from url.
path : str — Full path to use for downloaded file instead of folder and filename.
overwrite : bool — Whether to overwrite existing file. Defaults to False.
keep : bool — Whether to keep already downloaded file. Defaults to False.
resume : bool — Whether to resume a partial download using Range requests where the server supports it. Defaults to False.
retries : int — Number of times to retry a mid-stream failure. Only effective when resume=True so that each retry can continue from the partial file. Defaults to 0.
post : bool — Whether to use POST instead of GET. Defaults to False.
parameters : dict — Parameters to pass. Defaults to None.
timeout : float — Timeout for connecting to URL. Defaults to None (no timeout).
headers : dict — Headers to pass. Defaults to None.
encoding : str — Encoding to use for text response. Defaults to None (best guess).
json_string : bool — Whether to post parameters as JSON string. Defaults to False.

Returns

Path — Path of downloaded file

Raises

DownloadError

method Download.download(url: Path | str, **kwargs: Any) → requests.Response

Download url.

Parameters

url : Path | str — URL or path to download
**kwargs : Any — See below
post : bool — Whether to use POST instead of GET. Defaults to False.
parameters : dict — Parameters to pass. Defaults to None.
timeout : float — Timeout for connecting to URL. Defaults to None (no timeout).
headers : dict — Headers to pass. Defaults to None.
encoding : str — Encoding to use for text response. Defaults to None (best guess).
json_string : bool — Whether to post parameters as JSON string. Defaults to False.

Returns

requests.Response — Response

method Download.get_header(header: str) → Any

Get a particular response header of download.

Parameters

header : str — Header for which to get value

Returns

Any — Response header's value

method Download.get_headers() → Any

Get response headers of download.

Returns

Any — Response headers

method Download.get_status() → int

Get response status code.

Returns

int — Response status code

method Download.get_text() → str

Get text content of download.

Returns

str — Text content of download

method Download.get_yaml() → Any

Get YAML content of download.

Returns

Any — YAML content of download

method Download.get_json() → Any

Get JSON content of download.

Returns

Any — JSON content of download

method Download.download_text(url: Path | str, **kwargs: Any) → str

Download url as text.

Parameters

url : Path | str — URL or path to download
**kwargs : Any — See below
post : bool — Whether to use POST instead of GET. Defaults to False.
parameters : dict — Parameters to pass. Defaults to None.
timeout : float — Timeout for connecting to URL. Defaults to None (no timeout).
headers : dict — Headers to pass. Defaults to None.
encoding : str — Encoding to use for text response. Defaults to None (best guess).

Returns

str — Text content of download

method Download.download_yaml(url: Path | str, **kwargs: Any) → Any

Download url as YAML.

Parameters

url : Path | str — URL or path to download
**kwargs : Any — See below
post : bool — Whether to use POST instead of GET. Defaults to False.
parameters : dict — Parameters to pass. Defaults to None.
timeout : float — Timeout for connecting to URL. Defaults to None (no timeout).
headers : dict — Headers to pass. Defaults to None.
encoding : str — Encoding to use for text response. Defaults to None (best guess).

Returns

Any — YAML content of download

method Download.download_json(url: Path | str, **kwargs: Any) → Any

Download url as JSON.

Parameters

url : Path | str — URL or path to download
**kwargs : Any — See below
post : bool — Whether to use POST instead of GET. Defaults to False.
parameters : dict — Parameters to pass. Defaults to None.
timeout : float — Timeout for connecting to URL. Defaults to None (no timeout).
headers : dict — Headers to pass. Defaults to None.
encoding : str — Encoding to use for text response. Defaults to None (best guess).

Returns

Any — JSON content of download

method Download.get_frictionless_tableresource(url: Path | str, ignore_blank_rows: bool = True, infer_types: bool = False, **kwargs: Any) → TableResource

Get Frictionless TableResource.

Parameters

url : Path | str — URL or path to download ignore_blank_rows: Whether to ignore blank rows. Defaults to True. infer_types: Whether to infer types. Defaults to False (strings). **kwargs: See below has_header (bool): Whether data has a header. Defaults to True. headers (int | Sequence[int] | Sequence[str]): Number of row(s) containing headers or list of headers columns (Sequence[int] | Sequence[str] | None): Columns to pick. Defaults to all. format (str | None): Type of file. Defaults to inferring. file_type (str | None): Type of file. Defaults to inferring. encoding (str | None): Type of encoding. Defaults to inferring. compression (str | None): Type of compression. Defaults to inferring. delimiter (str | None): Delimiter for values in csv rows. Defaults to inferring. skip_initial_space (bool): Ignore whitespace straight after delimiter. Defaults to False. sheet (int | str | None): Sheet in Excel. Defaults to inferring. fill_merged_cells (bool): Whether to fill merged cells. Defaults to True. http_session (Session): Session object to use. Defaults to downloader session. columns (Sequence[int] | Sequence[str] | None): Columns to pick. Defaults to all. default_type (str | None): Default field type if infer_types False. Defaults to string. float_numbers (bool): Use float not Decimal if infer_types True. Defaults to True. null_values (list[Any]): Values that will return None. Defaults to [""]. dialect (Dialect): This can be set to override the above. See Frictionless docs. detector (Detector): This can be set to override the above. See Frictionless docs. layout (Layout): This can be set to override the above. See Frictionless docs. schema (Schema): This can be set to override the above. See Frictionless docs.
Returns — frictionless TableResource object

Raises

DownloadError

Returns header of tabular file(s) pointed to by url and an iterator where each row is returned as a list or dictionary depending on the dict_rows argument.

The headers argument is either a row number or list of row numbers (in case of multi-line headers) to be considered as headers (rows start counting at 1), or the actual headers defined as a list of strings. It defaults to 1. The dict_form argument specifies if each row should be returned as a dictionary or a list, defaulting to a list.

Optionally, headers can be inserted at specific positions. This is achieved using the header_insertions argument. If supplied, it is a list of tuples of the form (position, header) to be inserted. A function is called for each row. If supplied, it takes as arguments: headers (prior to any insertions) and row (which will be in dict or list form depending upon the dict_rows argument) and outputs a modified row or None to ignore the row.

Parameters

url : Path | str | Sequence[str] — A single or list of URLs or paths to read from
headers : int | Sequence[int] | Sequence[str] — Number of row(s) containing headers or list of headers. Defaults to 1.
dict_form : bool — Return dict or list for each row. Defaults to False (list)
include_headers : bool — Whether to include headers in iterator. Defaults to False.
ignore_blank_rows : bool — Whether to ignore blank rows. Defaults to True.
infer_types : bool — Whether to infer types. Defaults to False (strings).
header_insertions : Sequence[tuple[int, str]] | None — List of (position, header) to insert. Defaults to None.
row_function : Callable[[list[str], list | dict], list | dict] | None — Function to call for each row. Defaults to None.
**kwargs : Any — See below
format : str | None — Type of file. Defaults to inferring.
file_type : str | None — Type of file. Defaults to inferring.
xlsx2csv : bool — Whether to convert xlsx files. Defaults to False.
encoding : str | None — Type of encoding. Defaults to inferring.
compression : str | None — Type of compression. Defaults to inferring.
delimiter : str | None — Delimiter for values in csv rows. Defaults to inferring.
skip_initial_space : bool — Ignore whitespace straight after delimiter. Defaults to False.
sheet : int | str | None — Sheet in Excel. Defaults to inferring.
fill_merged_cells : bool — Whether to fill merged cells. Defaults to True.
http_session : Session — Session object to use. Defaults to downloader session.
columns : Sequence[int] | Sequence[str] | None — Columns to pick. Defaults to all.
default_type : str | None — Default field type if infer_types False. Defaults to string.
float_numbers : bool — Use float not Decimal if infer_types True. Defaults to True.
null_values : list[Any] — Values that will return None. Defaults to [""].
dialect : Dialect — This can be set to override the above. See Frictionless docs.
detector : Detector — This can be set to override the above. See Frictionless docs.
layout : Layout — This can be set to override the above. See Frictionless docs.
schema : Schema — This can be set to override the above. See Frictionless docs.

Returns

tuple[list[str], Iterator[list | dict]] — Tuple (headers, iterator where each row is a list or dictionary)

method Download.get_tabular_rows_as_list(url: Path | str | Sequence[str], headers: int | Sequence[int] | Sequence[str] = 1, include_headers: bool = True, ignore_blank_rows: bool = True, infer_types: bool = False, header_insertions: Sequence[tuple[int, str]] | None = None, row_function: Callable[[list[str], list | dict], list | dict] | None = None, **kwargs: Any) → tuple[list[str], Iterator[list]]

Returns headers and an iterator where each row is returned as a list.

The headers argument is either a row number or list of row numbers (in case of multi-line headers) to be considered as headers (rows start counting at 1), or the actual headers defined as a list of strings. It defaults to 1 and cannot be None.

Optionally, headers can be inserted at specific positions. This is achieved using the header_insertions argument. If supplied, it is a list of tuples of the form (position, header) to be inserted. A function is called for each row. If supplied, it takes as arguments: headers (prior to any insertions) and row (which will be in dict or list form depending upon the dict_rows argument) and outputs a modified row or None to ignore the row.

Parameters

url : Path | str | Sequence[str] — A single or list of URLs or paths to read from
headers : int | Sequence[int] | Sequence[str] — Number of row(s) containing headers or list of headers. Defaults to 1.
include_headers : bool — Whether to include headers in iterator. Defaults to True.
ignore_blank_rows : bool — Whether to ignore blank rows. Defaults to True.
infer_types : bool — Whether to infer types. Defaults to False (strings).
header_insertions : Sequence[tuple[int, str]] | None — List of (position, header) to insert. Defaults to None.
row_function : Callable[[list[str], list | dict], list | dict] | None — Function to call for each row. Defaults to None.
**kwargs : Any — See below
format : str | None — Type of file. Defaults to inferring.
file_type : str | None — Type of file. Defaults to inferring.
xlsx2csv : bool — Whether to convert xlsx files. Defaults to False.
encoding : str | None — Type of encoding. Defaults to inferring.
compression : str | None — Type of compression. Defaults to inferring.
delimiter : str | None — Delimiter for values in csv rows. Defaults to inferring.
skip_initial_space : bool — Ignore whitespace straight after delimiter. Defaults to False.
sheet : int | str | None — Sheet in Excel. Defaults to inferring.
fill_merged_cells : bool — Whether to fill merged cells. Defaults to True.
http_session : Session — Session object to use. Defaults to downloader session.
columns : Sequence[int] | Sequence[str] | None — Columns to pick. Defaults to all.
default_type : str | None — Default field type if infer_types False. Defaults to string.
float_numbers : bool — Use float not Decimal if infer_types True. Defaults to True.
null_values : list[Any] — Values that will return None. Defaults to [""].
dialect : Dialect — This can be set to override the above. See Frictionless docs.
detector : Detector — This can be set to override the above. See Frictionless docs.
layout : Layout — This can be set to override the above. See Frictionless docs.
schema : Schema — This can be set to override the above. See Frictionless docs.

Returns

tuple[list[str], Iterator[list]] — Tuple (headers, iterator where each row is a list)

Returns headers and an iterator where each row is returned as a dictionary.

The headers argument is either a row number or list of row numbers (in case of multi-line headers) to be considered as headers (rows start counting at 1), or the actual headers defined as a list of strings. It defaults to 1 and cannot be None.

Optionally, headers can be inserted at specific positions. This is achieved using the header_insertions argument. If supplied, it is a list of tuples of the form (position, header) to be inserted. A function is called for each row. If supplied, it takes as arguments: headers (prior to any insertions) and row (which will be in dict or list form depending upon the dict_rows argument) and outputs a modified row or None to ignore the row.

Parameters

url : Path | str | Sequence[str] — A single or list of URLs or paths to read from
headers : int | Sequence[int] | Sequence[str] — Number of row(s) containing headers or list of headers. Defaults to 1.
ignore_blank_rows : bool — Whether to ignore blank rows. Defaults to True.
infer_types : bool — Whether to infer types. Defaults to False (strings).
header_insertions : Sequence[tuple[int, str]] | None — List of (position, header) to insert. Defaults to None.
row_function : Callable[[list[str], list | dict], list | dict] | None — Function to call for each row. Defaults to None.
**kwargs : Any — See below
format : str | None — Type of file. Defaults to inferring.
file_type : str | None — Type of file. Defaults to inferring.
xlsx2csv : bool — Whether to convert xlsx files. Defaults to False.
encoding : str | None — Type of encoding. Defaults to inferring.
compression : str | None — Type of compression. Defaults to inferring.
delimiter : str | None — Delimiter for values in csv rows. Defaults to inferring.
skip_initial_space : bool — Ignore whitespace straight after delimiter. Defaults to False.
sheet : int | str | None — Sheet in Excel. Defaults to inferring.
fill_merged_cells : bool — Whether to fill merged cells. Defaults to True.
http_session : Session — Session object to use. Defaults to downloader session.
columns : Sequence[int] | Sequence[str] | None — Columns to pick. Defaults to all.
default_type : str | None — Default field type if infer_types False. Defaults to string.
float_numbers : bool — Use float not Decimal if infer_types True. Defaults to True.
null_values : list[Any] — Values that will return None. Defaults to [""].
dialect : Dialect — This can be set to override the above. See Frictionless docs.
detector : Detector — This can be set to override the above. See Frictionless docs.
layout : Layout — This can be set to override the above. See Frictionless docs.
schema : Schema — This can be set to override the above. See Frictionless docs.

Returns

tuple[list[str], Iterator[dict]] — Tuple (headers, iterator where each row is a dictionary)

Download 2 column csv from url and return a dictionary of keys (first column) and values (second column).

The headers argument is either a row number or list of row numbers (in case of multi-line headers) to be considered as headers (rows start counting at 1), or the actual headers defined as a list of strings. It defaults to 1 and cannot be None.

Optionally, headers can be inserted at specific positions. This is achieved using the header_insertions argument. If supplied, it is a list of tuples of the form (position, header) to be inserted. A function is called for each row. If supplied, it takes as arguments: headers (prior to any insertions) and row (which will be in dict or list form depending upon the dict_rows argument) and outputs a modified row or None to ignore the row.

Parameters

url : Path | str | Sequence[str] — A single or list of URLs or paths to read from
headers : int | Sequence[int] | Sequence[str] — Number of row(s) containing headers or list of headers. Defaults to 1.
include_headers : bool — Whether to include headers in iterator. Defaults to True.
ignore_blank_rows : bool — Whether to ignore blank rows. Defaults to True.
infer_types : bool — Whether to infer types. Defaults to False (strings).
header_insertions : Sequence[tuple[int, str]] | None — List of (position, header) to insert. Defaults to None.
row_function : Callable[[list[str], list | dict], list | dict] | None — Function to call for each row. Defaults to None.
**kwargs : Any — See below
format : str | None — Type of file. Defaults to inferring.
file_type : str | None — Type of file. Defaults to inferring.
xlsx2csv : bool — Whether to convert xlsx files. Defaults to False.
encoding : str | None — Type of encoding. Defaults to inferring.
compression : str | None — Type of compression. Defaults to inferring.
delimiter : str | None — Delimiter for values in csv rows. Defaults to inferring.
skip_initial_space : bool — Ignore whitespace straight after delimiter. Defaults to False.
sheet : int | str | None — Sheet in Excel. Defaults to inferring.
fill_merged_cells : bool — Whether to fill merged cells. Defaults to True.
http_session : Session — Session object to use. Defaults to downloader session.
columns : Sequence[int] | Sequence[str] | None — Columns to pick. Defaults to all.
default_type : str | None — Default field type if infer_types False. Defaults to string.
float_numbers : bool — Use float not Decimal if infer_types True. Defaults to True.
null_values : list[Any] — Values that will return None. Defaults to [""].
dialect : Dialect — This can be set to override the above. See Frictionless docs.
detector : Detector — This can be set to override the above. See Frictionless docs.
layout : Layout — This can be set to override the above. See Frictionless docs.
schema : Schema — This can be set to override the above. See Frictionless docs.

Returns

dict — Dictionary keys (first column) and values (second column)

Download multicolumn csv from url and return dictionary where keys are first column and values are dictionaries with keys from column headers and values from columns beneath.

The headers argument is either a row number or list of row numbers (in case of multi-line headers) to be considered as headers (rows start counting at 1), or the actual headers defined as a list of strings. It defaults to 1 and cannot be None.

Optionally, headers can be inserted at specific positions. This is achieved using the header_insertions argument. If supplied, it is a list of tuples of the form (position, header) to be inserted. A function is called for each row. If supplied, it takes as arguments: headers (prior to any insertions) and row (which will be in dict or list form depending upon the dict_rows argument) and outputs a modified row or None to ignore the row.

Parameters

url : Path | str | Sequence[str] — A single or list of URLs or paths to read from
headers : int | Sequence[int] | Sequence[str] — Number of row(s) containing headers or list of headers. Defaults to 1.
keycolumn : int — Number of column to be used for key. Defaults to 1.
ignore_blank_rows : bool — Whether to ignore blank rows. Defaults to True.
infer_types : bool — Whether to infer types. Defaults to False (strings).
header_insertions : Sequence[tuple[int, str]] | None — List of (position, header) to insert. Defaults to None.
row_function : Callable[[list[str], list | dict], list | dict] | None — Function to call for each row. Defaults to None.
**kwargs : Any — See below
format : str | None — Type of file. Defaults to inferring.
file_type : str | None — Type of file. Defaults to inferring.
xlsx2csv : bool — Whether to convert xlsx files. Defaults to False.
encoding : str | None — Type of encoding. Defaults to inferring.
compression : str | None — Type of compression. Defaults to inferring.
delimiter : str | None — Delimiter for values in csv rows. Defaults to inferring.
skip_initial_space : bool — Ignore whitespace straight after delimiter. Defaults to False.
sheet : int | str | None — Sheet in Excel. Defaults to inferring.
fill_merged_cells : bool — Whether to fill merged cells. Defaults to True.
http_session : Session — Session object to use. Defaults to downloader session.
columns : Sequence[int] | Sequence[str] | None — Columns to pick. Defaults to all.
default_type : str | None — Default field type if infer_types False. Defaults to string.
float_numbers : bool — Use float not Decimal if infer_types True. Defaults to True.
null_values : list[Any] — Values that will return None. Defaults to [""].
dialect : Dialect — This can be set to override the above. See Frictionless docs.
detector : Detector — This can be set to override the above. See Frictionless docs.
layout : Layout — This can be set to override the above. See Frictionless docs.
schema : Schema — This can be set to override the above. See Frictionless docs.

Returns

dict[str, dict] — Dictionary where keys are first column and values are dictionaries with keys from column headers and values from columns beneath

Download multicolumn csv from url and return dictionary where keys are header names and values are dictionaries with keys from first column and values from other columns.

The headers argument is either a row number or list of row numbers (in case of multi-line headers) to be considered as headers (rows start counting at 1), or the actual headers defined as a list of strings. It defaults to 1 and cannot be None.

Optionally, headers can be inserted at specific positions. This is achieved using the header_insertions argument. If supplied, it is a list of tuples of the form (position, header) to be inserted. A function is called for each row. If supplied, it takes as arguments: headers (prior to any insertions) and row (which will be in dict or list form depending upon the dict_rows argument) and outputs a modified row or None to ignore the row.

Parameters

url : Path | str | Sequence[str] — A single or list of URLs or paths to read from
headers : int | Sequence[int] | Sequence[str] — Number of row(s) containing headers or list of headers. Defaults to 1.
keycolumn : int — Number of column to be used for key. Defaults to 1.
ignore_blank_rows : bool — Whether to ignore blank rows. Defaults to True.
infer_types : bool — Whether to infer types. Defaults to False (strings).
header_insertions : Sequence[tuple[int, str]] | None — List of (position, header) to insert. Defaults to None.
row_function : Callable[[list[str], list | dict], list | dict] | None — Function to call for each row. Defaults to None.
**kwargs : Any — See below
format : str | None — Type of file. Defaults to inferring.
file_type : str | None — Type of file. Defaults to inferring.
xlsx2csv : bool — Whether to convert xlsx files. Defaults to False.
encoding : str | None — Type of encoding. Defaults to inferring.
compression : str | None — Type of compression. Defaults to inferring.
delimiter : str | None — Delimiter for values in csv rows. Defaults to inferring.
skip_initial_space : bool — Ignore whitespace straight after delimiter. Defaults to False.
sheet : int | str | None — Sheet in Excel. Defaults to inferring.
fill_merged_cells : bool — Whether to fill merged cells. Defaults to True.
http_session : Session — Session object to use. Defaults to downloader session.
columns : Sequence[int] | Sequence[str] | None — Columns to pick. Defaults to all.
default_type : str | None — Default field type if infer_types False. Defaults to string.
float_numbers : bool — Use float not Decimal if infer_types True. Defaults to True.
null_values : list[Any] — Values that will return None. Defaults to [""].
dialect : Dialect — This can be set to override the above. See Frictionless docs.
detector : Detector — This can be set to override the above. See Frictionless docs.
layout : Layout — This can be set to override the above. See Frictionless docs.
schema : Schema — This can be set to override the above. See Frictionless docs.

Returns

dict[str, dict] — Dictionary where keys are header names and values are dictionaries with keys from first column and values from other columns

staticmethod Download.get_column_positions(headers: Sequence[str]) → dict[str, int]

Get mapping of headers to column positions.

Parameters

headers : Sequence[str] — List of headers

Returns

dict[str, int] — Dictionary where keys are header names and values are header positions

classmethod Download.generate_downloaders(custom_configs: dict[str, dict], user_agent: str | None = None, user_agent_config_yaml: Path | str | None = None, user_agent_lookup: str | None = None, use_env: bool = True, fail_on_missing_file: bool = True, rate_limit: dict | None = None, **kwargs: Any) → None

Generate downloaders. Requires either global user agent to be set or appropriate user agent parameter(s) to be completed. The custom_configs dictionary is a mapping from name to a dictionary of custom configuration parameters that is added to the underlying session's params or headers. It can have keys that correspond to the input arguments of Download's constructor init (or the other arguments of this method).

Parameters

custom_configs : dict[str, dict] — Optional dictionary of custom configurations.
user_agent : str | None — User agent string. HDXPythonUtilities/X.X.X- is prefixed.
user_agent_config_yaml : Path | str | None — Path to YAML user agent configuration. Ignored if user_agent supplied. Defaults to ~/.useragent.yaml.
user_agent_lookup : str | None — Lookup key for YAML. Ignored if user_agent supplied.
use_env : bool — Whether to read environment variables. Defaults to True.
fail_on_missing_file : bool — Raise an exception if any specified configuration files are missing. Defaults to True.
rate_limit : dict | None — Rate limiting per host eg. {"calls": 1, "period": 0.1}. Defaults to None.
**kwargs : Any — See below
auth : tuple[str, str] — Authorisation information in tuple form (user, pass) OR
basic_auth : str — Authorisation information in basic auth string form (Basic xxxxxxxxxxxxxxxx) OR
basic_auth_file : str — Path to file containing authorisation information in basic auth string form (Basic xxxxxxxxxxxxxxxx)
bearer_token : str — Bearer token string OR
bearer_token_file : str — Path to file containing bearer token string OR
extra_params_dict : dict[str, str] — Extra parameters to put on end of url as a dictionary OR
extra_params_json : str — Path to JSON file containing extra parameters to put on end of url OR
extra_params_yaml : str — Path to YAML file containing extra parameters to put on end of url
extra_params_lookup : str — Lookup key for parameters. If not given assumes parameters are at root of the dict.
headers : dict — Additional headers to add to request.
use_auth : str — If more than one auth found, specify which one to use, rather than failing.
status_forcelist : Sequence[int] — HTTP statuses for which to force retry. Defaults to (429, 500, 502, 503, 504).
allowed_methods : Sequence[str] — HTTP methods for which to force retry. Defaults to ("HEAD", "TRACE", "GET", "PUT", "OPTIONS", "DELETE").

Returns

None — None

classmethod Download.get_downloader(name: str | None = None) → Download

Get a generated downloader given a name. If name is not supplied, the default one will be returned.

Parameters

name : str | None — Name of downloader. Defaults to None (get default).

Returns

Download — Downloader object