Week 1 · Ingest anything › Day 4

Day 4. Extracting data from email

1. Why this matters

In retail, restaurants and hotels, a surprising share of daily data still arrives as email attachments: store sales, supplier price lists, bank files, OTA statements. A reliable email extractor is one of the most practical skills you can show, and the easiest to get subtly wrong.

2. The anatomy of an email

An email is a tree of MIME parts. The top has headers (From, Subject, Date, Message-ID). Children are the text body, an HTML body, and attachments, each with a Content-Type and a filename, usually base64-encoded.

from email import policy
from email.parser import BytesParser
from pathlib import Path

def attachments(eml_path: Path):
    msg = BytesParser(policy=policy.default).parse(eml_path.open("rb"))
    meta = {"message_id": msg["Message-ID"], "from": str(msg["From"]),
            "subject": str(msg["Subject"]), "date": str(msg["Date"])}
    for part in msg.iter_attachments():
        name = part.get_filename()
        if name and name.lower().endswith((".csv", ".xlsx", ".xls")):
            yield meta, name, part.get_content()   # bytes for binary, str for text/csv

The course .eml files have no Message-ID (real ones do). Use a hash of the raw message bytes as your key when it's missing.

3. Three ways to reach a mailbox

A Graph call to list messages with attachments in a folder looks like this:

import msal, requests
app = msal.ConfidentialClientApplication(CLIENT_ID, authority=f"https://login.microsoftonline.com/{TENANT}",
                                         client_credential=CLIENT_SECRET)
token = app.acquire_token_for_client(scopes=["https://graph.microsoft.com/.default"])["access_token"]
H = {"Authorization": f"Bearer {token}"}
url = (f"https://graph.microsoft.com/v1.0/users/{MAILBOX}/mailFolders/inbox/messages"
       "?$filter=hasAttachments eq true and receivedDateTime ge 2026-09-01T00:00:00Z&$top=50")
while url:
    page = requests.get(url, headers=H, timeout=30).json()
    for m in page["value"]:
        atts = requests.get(f"https://graph.microsoft.com/v1.0/users/{MAILBOX}/messages/{m['id']}/attachments",
                            headers=H, timeout=30).json()["value"]
        # each fileAttachment has name, contentType and base64 contentBytes
    url = page.get("@odata.nextLink")              # follow pagination until it's gone

4. Rules that keep an email feed honest

• Allow-list senders ([email protected]) and parse identity from the sender and subject. Reject the rest into a review folder in your raw zone, never silently.
• Business date from the subject or the file content, not the email Date header. Emails sent at 00:30 are about yesterday; resends come days later.
• Dedup twice: by message key (same email fetched again) and by attachment hash (manager hits send twice).
• Don't change the mailbox unless the owner agrees. Marking read or moving to a "processed" folder seems tidy but breaks people who use the same box. Use a server-side cursor instead: "received after my last watermark".