Overview
Echtralex tracks how PRC ministry policy language shifts over time. That means aligning parallel English/Chinese transcripts from ministry press conferences, sentence by sentence: fully by hand, daily, up to four releases a day, averaging 12 minutes per transcript. Our team of five built the system that automates it, working directly with Echtralex founder Michael Horlick as our client. It shipped: it’s deployed to production and in active client use.
The measured result: 12 minutes down to 8.5 per transcript (~30%), faster than the manual workflow on 9 of 10 test transcripts, with typically just 1–2 manual corrections needed per ~40-sentence release. The client’s acceptance document graded the efficiency criterion as only partially met: the remaining time is the consultant still reading every transcript after alignment, a human step no pipeline was going to remove. I’d rather put the real number here than a rounder one.
What I owned: HTML extraction, end to end
Four ministry sites, four different page structures, and active scraper countermeasures to get past.
- Layered anti-bot fallback. Cloudscraper runs first (cheap and fast), with a Playwright real-browser fallback only when the cheap path fails. When English MOFCOM kept failing intermittently, I fixed it properly: I completed the browser fingerprint in the Playwright config and added retries for the flaky cases.
- Speaker-turn segmentation. The alignment subsystem needed paragraph-aligned input. But the client’s assumption that paragraphs align across languages turned out to be false, which broke the original design. HTML tags and a bare colon heuristic both fell short, so I landed on speaker detection instead: a colon heuristic validated by spaCy NLP (is the text before this colon actually a speaker?), which splits paragraphs on speaker turns. That’s what made the pipeline work across all four inconsistently-formatted sites.
- The silent-deletion bug. The Chinese segmentation regex was quietly dropping any fragment that didn’t match its pattern, so transcripts were getting altered with no error raised. I rewrote the expression so nothing ever gets discarded. It was my hardest bug of the semester: the system wasn’t wrong loudly, it was wrong silently.
- Tests that paid for themselves. I wrote the component’s unit tests even though I doubted their value at first. One of them caught that old MOFCOM pages store transcripts in iframes, which scraping was returning as nothing. I also automated the tedious part: a utility that generates alignment test fixtures by pulling gold-standard alignments straight from Echtralex’s live site.
The refactor with a before/after measurement
Mid-project I restructured HTML extraction: one file of per-ministry special cases became shared utilities plus per-ministry config files: about six hours of work. Then we measured it, using an outside developer who’d never seen the codebase as the control:
| Task | Before | After (me) | After (fresh eyes) |
|---|---|---|---|
| Add a new source | hours (team estimate) | 8 minutes | 31.82 minutes |
A second source took me 16 minutes and 35 lines of config. Walking away from a class project with a controlled measurement, including an outside tester, is a rare thing.
Beyond my component
- I found the model. My first pass at sentence alignment used BERT, and hit a wall: it couldn’t produce the output shape alignment needed. So I went looking, found LaBSE (Language-Agnostic BERT Sentence Embedding), demoed it, and updated the proposal. The team adopted it. A teammate implemented the alignment algorithm itself, while I built the accuracy test harness the results were measured against. We threw out a greedy matcher early, after errors cascaded down transcripts, and replaced it with look-behind/look-ahead pairing.
- Architecture and design calls that stuck. I argued the Lex Processor should be a module the Checking Service calls, not its own idle-waiting API (adopted). I pushed back on having my component query the database directly, so the interface became two arrays of URLs instead (adopted). I raised JWT protection for our endpoints, which the team then scheduled.
- Performance work outside my lane. I short-circuited the Checking Service’s pagination when a page held only already-seen IDs, and converted its listing scrape to a generator. I later found and fixed the regression where that short-circuit skipped genuinely new conferences.
Running the project
I was the client’s point of contact from the first meeting: scheduling every session, sending 3+ emails a week, and managing expectations when we fell behind. I introduced Jira and ran the board (I created 56 of our 95 issues), and ended up as the team’s de facto scrum master.
The moment I’m proudest of: when we were behind, someone proposed reframing our System Flexibility test (counting three of four ministry sources as “add-ons”) so it would pass. I argued that was cutting corners and wouldn’t survive scrutiny. The team backed me, and the honest version of that test became the Codebase Extensibility Test above, which is exactly what forced the modularization work. The harder criterion produced the project’s best engineering.
Result
A deployed system in daily client use, built by a team I helped keep honest and on schedule: 79 hours logged across the semester, about 48 of it hands-on development. The honest numbers (~30% faster, 9 of 10, one or two corrections per release) are the ones I’d rather defend.
