# Usage

Most users will only need to call `careful.httpx.make_careful_client`.

::: careful.httpx.make_careful_client
    options:
      annotations_path: brief
      show_signature: false
      show_root_heading: true

## Throttling

If `requests_per_minute` is set, standard (non-retry) requests will automatically
sleep for a short period to target the given rate.

For example, at 30rpm, the sleep time on a fast request will be close to 2 seconds.

```python
client = make_careful_client(requests_per_minute=20)

for page in range(10):
  # will sleep ~3 seconds each time
  client.get(f"https://example.com?page={page}")
```

## Retries

If `retry_attempts` is set, responses will be passed to `should_retry`.
Responses that are rejected (return `True`) will be retried after a wait based on
`retry_wait_seconds`.
Each retry will wait twice as long as the one before.

```python
client = make_careful_client(retry_attempts=2, retry_wait_seconds=30)

# will try, wait 30s, try again, wait 60s, try again, then give up & return the 500
client.get("https://httpbin.org/status/500")
```

There are a few simple pre-written `should_retry` predicates you can use:

::: careful.httpx.retries.retry_default_rule
    options:
       heading_level: 3
       show_root_heading: true
::: careful.httpx.retries.retry_only_500s
    options:
       heading_level: 3
       show_root_heading: true
::: careful.httpx.retries.retry_all_400s_500s
    options:
       heading_level: 3
       show_root_heading: true

## Development Caching

!!! warning "Why _development_ caching?"

    This feature is named as a reminder that **this is not true HTTP caching**, which
    should take various headers into account. Look at libraries like [hishel](https://hishel.com) if that's what you are after.

The purpose of this feature is to allow you to cache all of your HTTP requests during development.
Often when writing a scraper or crawler, you wind up hitting the site you are working on more often than you'd like-- each time you iterate on your code you're likely making redundant requests to pages that haven't changed.

By caching all successful requests (configurable with the `should_cache` parameter),
you can easily re-run scrapers without making redundant HTTP requests.
This means much faster development & happier upstream servers.

To enable development caching, assign a [`MemoryCache`][careful.httpx.MemoryCache],
[`FileCache`][careful.httpx.FileCache], or [`SqliteCache`][careful.httpx.SqliteCache] to
the `cache_storage` property of a `scrapelib.Scraper`.

```python
client = make_careful_client(
  cache_storage=FileStorage("_cache")
)

# only one HTTP request is made
client.get("https://example.com")
client.get("https://example.com")
client.get("https://example.com")
client.get("https://example.com")
# on subsequent runs, zero will be made until _cache is cleared
```

## robots.txt

If `check_robots_txt` is set to `True`, then each request will be checked
against that domain's `robots.txt`.
(Each domain's `robots.txt` will be fetched once and cached.)

Evaluating `robots.txt` requires a *user agent*. `careful` will use the user agent
set on the `client` at the time of creation by default, but sometimes it is preferable
to use a custom user agent for these checks. To do so, set the `robots_txt_user_agent` parameter.

The default behavior when checking is enabled is that any attempt to fetch a denied page will
raise a `RobotExclusionError`. This behavior can be overriden by passing a function to `robots_txt_on_reject` that takes two parameters: `(url: str, rfp: RobotFileParser)`.

::: careful.httpx.RobotExclusionError
    options:
       heading_level: 3
       members: False
       show_root_heading: true

::: careful.httpx.robots.warn_robots_txt
    options:
       heading_level: 3
       members: False
       show_root_heading: true

### Note about multiple enhancements

When multiple features are applied, the order of wrapping ensures that:

- the cache is checked first, and bypasses throttling if hit
- retries use their own delays, but are not throttled separately

## Cache Storage Options

These options are available for `cache_storage`:

::: careful.httpx.MemoryCache
    options:
       heading_level: 3
       members: False
       show_root_heading: true

::: careful.httpx.FileCache
    options:
       heading_level: 3
       members: False
       show_root_heading: true

::: careful.httpx.SqliteCache
    options:
       heading_level: 3
       members: False
       show_root_heading: true

