Working with Data: Count, Compare, Find the Difference

← back to section

Everything so far was tools. Here are the three jobs they were picked up for.

Count what the screen does not show

The interface lists orders; the question is different: "what is the total of PAID orders for the day?" That number is not on the screen — but it is in the data.

live example

orders = [
    {"id": "ord-042", "status": "PAID", "total": 4990},
    {"id": "ord-043", "status": "CANCELLED", "total": 890},
    {"id": "ord-044", "status": "PAID", "total": 1780},
]

paid = [o for o in orders if o["status"] == "PAID"]
total = sum(o["total"] for o in paid)

print(f"Paid: {len(paid)}, total {total}")

sum() adds numbers from a sequence; min(), max() and len() live next to it, and those four cover most summaries.

Counting by group is a dict accumulator:

live example

orders = [{"status": "PAID"}, {"status": "CANCELLED"}, {"status": "PAID"}]

by_status = {}
for order in orders:
    status = order["status"]
    by_status[status] = by_status.get(status, 0) + 1

print(by_status)     # {'PAID': 2, 'CANCELLED': 1}

get(status, 0) holds it together: a new status has no counter yet, so instead of an error you get zero.

Practice here: Check a response field · Total for a report · Check an email format

Compare the total with the parts

The sum of the items must match the order total. Such a mismatch is invisible on the screen — only arithmetic finds it.

live example

# error: fails on purpose — the second order does not add up
orders = [
    {"id": "ord-042", "items": [{"qty": 1, "price": 4990}], "total": 4990},
    {"id": "ord-043", "items": [{"qty": 2, "price": 890}], "total": 1000},
]

def total_adds_up(order):
    computed = sum(i["qty"] * i["price"] for i in order["items"])
    return computed == order["total"]

broken = [o["id"] for o in orders if not total_adds_up(o)]
assert not broken, f"total does not add up for: {broken}"

What matters is the failure message:

live example

broken = ["ord-042", "ord-107"]

# wrong: you see how many broke, not which ones
print(f"total does not add up for {len(broken)} orders")

# right: identifiers let you open the order at once
print(f"total does not add up for: {broken}")

Careful with fractions: 0.1 + 0.2 is not exactly 0.3 in any language. Money is therefore counted in cents as integers, or compared with a tolerance:

live example

# wrong: fractions are not compared head-on
print(0.1 + 0.2 == 0.3)

# right: difference under a cent counts as equal
print(abs((0.1 + 0.2) - 0.3) < 0.01)

Practice here: Does the order total add up

Find what is missing and what is extra

Two exports: from the interface and from the database, from the old version and the new one. What is missing and what is extra? Lists make that awkward; sets do it in one line.

A set is a collection without order and without duplicates:

live example

before = {"ord-01", "ord-02", "ord-03"}
after = {"ord-02", "ord-03", "ord-04"}

print(before - after)      # {'ord-01'}  — missing
print(after - before)      # {'ord-04'}  — extra
print(before & after)      # in both

Hence the main reason not to compare lists:

live example

# wrong: different order, so not equal — though contents match
print(["ord-01", "ord-02"] == ["ord-02", "ord-01"])

# right: a set does not care about order
print({"ord-01", "ord-02"} == {"ord-02", "ord-01"})

From a list of dicts a set is built by the key field:

live example

database = [{"id": "ord-042"}, {"id": "ord-043"}]
export = [{"id": "ord-042"}, {"id": "ord-043"}]

in_database = {o["id"] for o in database}
in_export = {o["id"] for o in export}

missing = in_database - in_export
extra = in_export - in_database

assert not missing, f"missing from the export: {sorted(missing)}"
assert not extra, f"extra in the export: {sorted(extra)}"
print("contents match")

sorted() is not decoration: a set has no order, and without it the same discrepancy prints differently every time.

Practice here: What is missing and what is extra · Reconcile two exports

Compare content, not just membership

Matching identifiers is not enough: an order can be present with a different total. Then you compare fields:

live example

database = [{"id": "ord-042", "total": 4990}]
export = [{"id": "ord-042", "total": 4990}]

expected = {o["id"]: o for o in database}     # dict keyed by id

for order in export:
    reference = expected[order["id"]]
    assert order["total"] == reference["total"], (
        f"{order['id']}: export says {order['total']}, database says {reference['total']}")

print("totals match")

Building expected is a trick worth remembering: a list turns into a dict keyed by identifier, and lookups stop scanning.

Practice here: Compare a response with the expected one

In short

• sum(), len(), min(), max() cover most summaries; group counting is a dict accumulator with get(key, 0).
• Comparing the total with its parts catches mismatches invisible on the screen; put identifiers in the message, not counts.
• Money in floats is compared with a tolerance, or counted in cents.
• The difference between two sets is before - after and after - before; lists are no good for that — order matters to them.
• Turn a list into a dict by key when you need fast lookups.

What to read next

• pytest: your first real test — how this is packaged in a project.
• API autotests — where the second export comes from when a service returns the data.
• SQL for testers — the other half of the same work: the data comes from a database.