Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/python-package.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
python-version: ["3.9", "3.13", "3.14"]
python-version: ["3.10", "3.13", "3.14"]

steps:
- uses: actions/checkout@v6
Expand Down
2 changes: 2 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
**06/23/2026:** **Breaking change (1.2.0):** the minimum supported Python is now **3.10** (`requires-python = ">=3.10"`). 3.9 support was already effectively broken — the `waterdata` module's dependencies (`anyio`, the test stack) require 3.10+, and the `waterdata` test modules already skipped on <3.10. `anyio` is now declared as a direct dependency (it is imported directly by `waterdata`), and the CI/ruff/mypy targets move to 3.10. Also fully removed the deprecated `variable_info` metadata property: the `NWIS_Metadata` override only warned and returned `None` (it relied on the defunct `get_pmcodes`), and the `BaseMetadata` abstract is gone too since nothing implemented it — accessing `.variable_info` now raises `AttributeError`. `site_info` is unaffected.

**06/23/2026:** **Breaking change (1.2.0):** removed the `nadp` module and the deprecated `samples` module ahead of the 1.2.0 release. `nadp` was deprecated on 05/01/2026 — NADP is not a USGS data source, so retrieve NADP data directly from https://nadp.slh.wisc.edu/. The `samples.get_usgs_samples` shim (a deprecated forward to the modern getter) is gone; use `waterdata.get_samples()` instead. `import dataretrieval.nadp` / `import dataretrieval.samples` now raise `ModuleNotFoundError`.

**06/03/2026:** The request-error hierarchy is now unified. Every module (`nwis`, `wqp`, `nldi`, `waterdata`, `nadp`, `streamstats`) raises a subclass of `dataretrieval.DataRetrievalError` on a failed request, so a single `except dataretrieval.DataRetrievalError` spans them all. An HTTP error status surfaces as an `HTTPError` carrying `.status_code` (inspect it to branch on a specific code); the retryable 429/5xx subset is `TransientError` (`RateLimited` / `ServiceUnavailable`, carrying `.retry_after`); and a request too large to satisfy is a `RequestTooLarge` (`URLTooLong` for an over-long single request, `Unchunkable` when the Water Data chunker cannot split a call small enough). Connection-level failures (timeouts, DNS, refused connections) are wrapped as a `NetworkError`, with the underlying `httpx` exception on `__cause__`. Every `DataRetrievalError` also exposes `.status_code` (`None` when there is no HTTP status), `.retry_after`, and `.retryable`, so a single `except dataretrieval.DataRetrievalError as e` clause can branch on the status or retry transient failures without knowing the concrete subclass. **Breaking change:** these exceptions no longer multiply-inherit a built-in — code that caught request failures with `except ValueError` or `except RuntimeError` should switch to `except dataretrieval.DataRetrievalError` (or a specific subclass). A no-data result is **not** an error: the modern getters (`waterdata`, `wqp`, `nldi`) return an empty DataFrame when nothing matches. Only the deprecated `nwis` (waterservices) path still raises `NoSitesError` on no data.
Expand Down
26 changes: 8 additions & 18 deletions dataretrieval/nwis.py
Original file line number Diff line number Diff line change
Expand Up @@ -1033,11 +1033,15 @@ def _read_json(json: dict[str, Any]) -> pd.DataFrame:
# for example, [0, 21, 22] would be the first and last indices
index_list = [0]
index_list.extend(
[i + 1 for i, (a, b) in enumerate(zip(site_list[:-1], site_list[1:])) if a != b]
[
i + 1
for i, (a, b) in enumerate(zip(site_list[:-1], site_list[1:], strict=False))
if a != b
]
)
index_list.append(len(site_list))

for start, end in zip(index_list[:-1], index_list[1:]):
for start, end in zip(index_list[:-1], index_list[1:], strict=False):
# grab a block containing timeseries 0:21,
# which are all from the same site
site_block = json["value"]["timeSeries"][start:end]
Expand Down Expand Up @@ -1133,8 +1137,8 @@ class NWIS_Metadata(BaseMetadata):

Notes
-----
``site_info`` and ``variable_info`` are exposed as properties (documented
below) rather than plain attributes.
``site_info`` is exposed as a property (documented below) rather than a
plain attribute.

"""

Expand Down Expand Up @@ -1196,17 +1200,3 @@ def site_info(self) -> tuple[pd.DataFrame, BaseMetadata] | None:

else:
return None # don't set metadata site_info attribute

@property
def variable_info(self) -> None:
"""
Deprecated. Accessing variable_info via NWIS_Metadata is deprecated.
Returns None.
"""
warnings.warn(
"Accessing variable_info via NWIS_Metadata is deprecated as "
"it relies on the defunct get_pmcodes function.",
DeprecationWarning,
stacklevel=2,
)
return None
2 changes: 1 addition & 1 deletion dataretrieval/ogc/planning.py
Original file line number Diff line number Diff line change
Expand Up @@ -497,7 +497,7 @@ def iter_sub_args(self) -> Iterator[dict[str, Any]]:
chunk_lists = [self.chunks[ax.arg_key] for ax in self.axes]
for combo in itertools.product(*chunk_lists):
sub_args = dict(self.args)
for axis, chunk in zip(self.axes, combo):
for axis, chunk in zip(self.axes, combo, strict=False):
sub_args[axis.arg_key] = axis.render(chunk)
yield sub_args

Expand Down
10 changes: 3 additions & 7 deletions dataretrieval/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -252,19 +252,15 @@ def __init__(self, response: httpx.Response) -> None:
# # disclaimer seems to be only part of importWaterML1
# self.disclaimer = None

# These properties are to be set by `nwis` or `wqp`-specific metadata classes.
# ``site_info`` is set by ``nwis`` / ``wqp``-specific metadata classes; the
# modern ``waterdata`` metadata leaves it unimplemented (use
# ``waterdata.get_monitoring_locations`` to retrieve site descriptions).
@property
def site_info(self) -> Any:
raise NotImplementedError(
"site_info must be implemented by utils.BaseMetadata children"
)

@property
def variable_info(self) -> Any:
raise NotImplementedError(
"variable_info must be implemented by utils.BaseMetadata children"
)

def __repr__(self) -> str:
return f"{type(self).__name__}(url={self.url})"

Expand Down
3 changes: 2 additions & 1 deletion dataretrieval/waterdata/nearest.py
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,8 @@ def _build_window_or_filter(targets: pd.DatetimeIndex, window_td: pd.Timedelta)
lowers = (targets - window_td).strftime(fmt)
uppers = (targets + window_td).strftime(fmt)
return " OR ".join(
f"(time >= '{lo}' AND time <= '{up}')" for lo, up in zip(lowers, uppers)
f"(time >= '{lo}' AND time <= '{up}')"
for lo, up in zip(lowers, uppers, strict=False)
)


Expand Down
19 changes: 15 additions & 4 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
name = "dataretrieval"
description = "Discover and retrieve water data from U.S. federal hydrologic web services."
readme = "README.md"
requires-python = ">=3.9"
requires-python = ">=3.10"
keywords = ["USGS", "water data"]
license = "CC0-1.0"
license-files = ["LICENSE.md"]
Expand All @@ -17,11 +17,22 @@ maintainers = [
{name = "Elise Hinman", email = "ehinman@usgs.gov"},
]
classifiers = [
"Development Status :: 5 - Production/Stable",
"Intended Audience :: Science/Research",
"Topic :: Scientific/Engineering :: Hydrology",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
]
dependencies = [
"httpx",
"pandas>=2.0.0,<4.0.0",
# Directly imported by ``waterdata`` (``anyio.from_thread.start_blocking_portal``),
# so declared here rather than relied on transitively via httpx.
"anyio>=4.0",
]
dynamic = ["version"]

Expand All @@ -35,7 +46,7 @@ dataretrieval = ["py.typed"]
# Minimal set the CI ``type-check`` job installs — just mypy + the package,
# not the whole test stack.
type-check = [
"mypy<2", # <2 so it can still target Python 3.9 (the project's floor)
"mypy",
]
test = [
"pytest > 5.0.0",
Expand Down Expand Up @@ -77,7 +88,7 @@ repository = "https://github.com/DOI-USGS/dataretrieval-python.git"
write_to = "dataretrieval/_version.py"

[tool.ruff]
target-version = "py39"
target-version = "py310"
extend-exclude = ["demos"]

[tool.ruff.lint]
Expand Down Expand Up @@ -115,7 +126,7 @@ docstring-code-line-length = 72
# libraries (pandas, geopandas) are treated as ``Any`` instead of
# requiring stub packages. Dropping that — via pandas-stubs/types-requests and
# per-module overrides — can follow.
python_version = "3.9" # the project's minimum supported version
python_version = "3.10" # the project's minimum supported version
files = ["dataretrieval"]
strict = true
ignore_missing_imports = true
Expand Down
5 changes: 5 additions & 0 deletions tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@
r"(?:QuotaExhausted|ServiceInterrupted):",
r"Connect(ion)?Error", # requests' ConnectionError + httpx' ConnectError
r"ReadTimeout|ConnectTimeout|Timeout",
# ``dataretrieval`` wraps connection-level failures (timeout / DNS / refused)
# in a typed ``NetworkError``; rerunfailures matches the crash line (the
# ``NetworkError``), not the chained raw httpx exception, so match the
# wrapper too -- otherwise a transient SSL/handshake timeout fails CI.
r"NetworkError",
]

#: Apply to a test module (``pytestmark = flaky_api``) or class (``@flaky_api``)
Expand Down
6 changes: 0 additions & 6 deletions tests/ngwmn_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,8 @@
behavior change still fails on the first run.
"""

import sys

import pytest
from pandas import DataFrame

if sys.version_info < (3, 10):
pytest.skip("Skip entire module on Python < 3.10", allow_module_level=True)

from dataretrieval import ngwmn
from dataretrieval.utils import BaseMetadata
from tests.conftest import flaky_api
Expand Down
13 changes: 1 addition & 12 deletions tests/nwis_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -359,7 +359,7 @@ class TestMetaData:
-----

- Originally based on GitHub Issue #73.
- Modified to site_info and variable_info as properties, not callables.
- Modified to expose site_info as a property, not a callable.
"""

def test_set_metadata_info_site(self):
Expand Down Expand Up @@ -416,17 +416,6 @@ def test_set_metadata_info_countyCd(self):
# assert that site_info is implemented
assert md.site_info

def test_variable_info_deprecated(self):
"""Test that variable_info raises a DeprecationWarning and returns None."""
response = mock.MagicMock()
md = NWIS_Metadata(response)
with pytest.warns(
DeprecationWarning,
match="Accessing variable_info via NWIS_Metadata is deprecated",
):
result = md.variable_info
assert result is None


class TestReadRdb:
"""Tests for the NWIS-specific _read_rdb wrapper.
Expand Down
4 changes: 2 additions & 2 deletions tests/utils_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -231,9 +231,9 @@ def test_init_with_response(self):
assert md.query_time is not None
assert md.header is not None

# Test NotImplementedError parameters
# site_info is abstract on BaseMetadata; only nwis/wqp implement it.
with pytest.raises(NotImplementedError):
_ = md.variable_info
_ = md.site_info


class Test_to_str:
Expand Down
4 changes: 0 additions & 4 deletions tests/waterdata_chunking_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@
import contextvars
import datetime
import http.server
import sys
import threading
import time
import warnings
Expand All @@ -32,9 +31,6 @@
import pandas as pd
import pytest

if sys.version_info < (3, 10):
pytest.skip("Skip entire module on Python < 3.10", allow_module_level=True)

from dataretrieval.exceptions import (
DataRetrievalError,
RateLimited,
Expand Down
4 changes: 0 additions & 4 deletions tests/waterdata_ratings_test.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,9 @@
import re
import sys
from urllib.parse import parse_qs, urlsplit

import pandas as pd
import pytest

if sys.version_info < (3, 10):
pytest.skip("Skip entire module on Python < 3.10", allow_module_level=True)

from dataretrieval.waterdata import get_ratings
from dataretrieval.waterdata.ratings import _build_filter

Expand Down
4 changes: 0 additions & 4 deletions tests/waterdata_test.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,12 @@
import datetime
import json
import sys
from unittest import mock

import numpy as np
import pandas as pd
import pytest
from pandas import DataFrame

if sys.version_info < (3, 10):
pytest.skip("Skip entire module on Python < 3.10", allow_module_level=True)

from dataretrieval.ogc.engine import _dialect
from dataretrieval.waterdata import (
get_channel,
Expand Down