Downloads a dataset from the Colorado Open Data Socrata API using either a human-readable catalog `key` or the official Socrata dataset `u_id` returned by [co_list_datasets()].
Usage
co_pull_dataset(
dataset,
limit = 10000,
filters = list(),
date = NULL,
from = NULL,
to = NULL,
date_field = NULL,
where = NULL,
order = NULL,
timeout_sec = 30,
clean_names = TRUE,
coerce_types = TRUE
)Arguments
- dataset
A single dataset `key` or Socrata dataset `u_id` from [co_list_datasets()]. For example, a key may look like `"denver_public_schools"`, while a u_id may look like `"e4mf-ggwf"`.
- limit
Number of rows to retrieve. Defaults to 10,000.
- filters
Optional named list of exact-match filters. Each list name should be a field name in the dataset, and each value should be the value or values to match. Vector values are translated into SQL-style `IN` conditions in the generated SoQL query. For example, `filters = list(grade_levels = c("9-12", "6-8"))` returns rows where `grade-levels` is either `"9-12"` or `"6-8"`.
- date
Optional single date used to match all records from that day. Requires `date_field`.
- from
Optional start date, inclusive. Requires `date_field`.
- to
Optional end date, exclusive. Requires `date_field`.
- date_field
Optional date or datetime column to use with `date`, `from`, or `to`. This must be supplied when any date filter is used. Users can identify available date columns by inspecting the dataset on the Colorado Open Data Portal or by pulling a small sample with `limit`.
- where
Optional raw SoQL `WHERE` clause for advanced filtering. SoQL is the Socrata Query Language used by Colorado Open Data. If `date`, `from`, or `to` are also supplied, their generated conditions are combined with `where` using `AND`.
- order
Optional raw SoQL `ORDER BY` clause, such as `"DATE_FIELD DESC"`.
- timeout_sec
Request timeout in seconds. Defaults to 30.
- clean_names
Logical. If `TRUE`, column names are converted to snake_case using [janitor::clean_names()]. Defaults to `TRUE`.
- coerce_types
Logical. If `TRUE`, the package attempts lightweight, heuristic-based type coercion after downloading the data. Columns are converted only when at least 95 percent of non-missing values can be parsed as the target type. This helps avoid unsafe conversions when source data are inconsistent.
Details
When a catalog `key` is supplied, `co_pull_dataset()` first retrieves the live Colorado Open Data catalog to look up the corresponding Socrata `u_id`, then sends a second request to download the dataset itself. Supplying a `u_id` directly is more stable and avoids ambiguity, while keys are provided for readability and classroom-friendly workflows.
Dataset keys are generated from dataset names using [janitor::make_clean_names()]. Because keys are derived from live catalog metadata, Socrata u_ids are the most stable identifiers.
`co_pull_dataset()` is designed for common catalog-based workflows. For arbitrary Socrata JSON endpoints that are not included in the package catalog, use [co_any_dataset()].
The `filters` argument is intended for simple exact-match filtering. For more complex conditions, use the `where` argument with raw SoQL syntax.
Internally, filter field names are wrapped in `TRIM()` when constructing SoQL queries to reduce mismatches caused by leading or trailing whitespace in source data.
Type coercion is intentionally conservative. When `coerce_types = TRUE`, the package attempts to infer common R column types from the API response, but columns with inconsistent values may remain character columns.
Datetime coercion is also conservative. Timezone offsets and sub-second precision may not always be preserved during automatic parsing, and columns with inconsistent datetime formats may remain character columns.
Examples
if (interactive() && curl::has_internet()) {
# Pull by human-readable key
co_pull_dataset("denver_public_schools", limit = 3)
# Pull by Socrata u_id
co_pull_dataset("e4mf-ggwf", limit = 3)
# Filter to one value
co_pull_dataset(
"e4mf-ggwf",
limit = 3,
filters = list(grade_levels = "ECE-5")
)
# Filter to multiple values
co_pull_dataset(
"e4mf-ggwf",
limit = 10,
filters = list(grade_levels = c("ECE-5", "K-5"))
)
# Date filtering
co_pull_dataset(
"e4mf-ggwf",
from = "2023-01-01",
to = "2024-01-01",
date_field = "last_verified",
limit = 100
)
# Advanced filtering with raw SoQL
co_pull_dataset(
"e4mf-ggwf",
where = "grade_levels = 'ECE-5' AND classification = 'Charter'",
order = "last_verified DESC",
limit = 100
)
}