Read Only the Columns You Need

A CSV stores a table row by row: each line holds one order with all its values, so even to get one column a reader goes through every line. Parquet stores a table column by column: each column is a block of its own, and the file notes where each block starts, so a reader can open only the blocks it needs and skip the rest. A data engineer needs this because tables at work often have dozens or hundreds of columns while a report needs a few, and reading them all wastes time and memory on every run.

In pandas: pd.read_parquet takes columns=, a list of column names. Only those columns are read; every row is still there.

view = pd.read_parquet("orders.parquet", columns=["product", "status"])
print(view.columns.tolist())   # ['product', 'status']

In DuckDB: DuckDB is a database that runs inside your Python program, with no server to start: con = duckdb.connect() opens one in memory. In a query, a file name in single quotes stands where a table name goes: FROM 'orders.parquet' reads the file in place, and DuckDB reads only the columns the query names. Write one clause per line, each a string piece that ends with a space:

query = (
    "SELECT product, MAX(unit_price) AS price "
    "FROM 'orders.parquet' "
    "GROUP BY product "
    "ORDER BY product "
)

The starting code writes orders.csv (1,497 order lines, 8 columns) to orders.parquet. The check prints how many columns the file has, what pandas read, your query's rows with show(con, query), and the columns DuckDB's plan read from the file:

columns in the file: 8
pandas read: ['category', 'quantity']
rows: 1497
category | items
gifts | 430
home | 843
kitchen | 794
office | 1186
(4 rows)
DuckDB read: ['category', 'quantity']

Your task: read only category and quantity from orders.parquet into small with columns=, open DuckDB in memory as con, and write a query that sums quantity per category as items, straight from the file, sorted by category.

Assignment
Follow the Coding Tutorial to read two of the file's eight columns, first with pandas, then with a DuckDB query.

Hint
Name only the columns you need: in pandas in the columns= list, in DuckDB in the query. The query has four clauses, one per line: SELECT, FROM, GROUP BY, ORDER BY.