Two pipelines, one issue: how LawSignals tracks bills and federal rulemaking
LawSignals runs two ingestion pipelines into one practice-area schema: bills from 50-state scrapers, federal rules from the Federal Register. The architecture, and why they are separate.
Editor’s note, updated August 2026. This post originally described the second pipeline as news intelligence. That layer was withdrawn from every user-facing surface in August 2026 and nothing reads it. The post has been rewritten around the second pipeline as it actually exists now, Regulatory Watch, rather than left standing as a description of a removed system. The architectural argument it makes is unchanged, because it was never really about news.
The interesting architectural choice in LawSignals is not the semantic matching or the per-state coverage. It is the decision to run two independent ingestion pipelines that converge at one layer.
That choice, bill scraping as one pipeline and federal rulemaking as another, both writing into the same practice-area schema, is what lets one monthly issue carry both the pending legislation and the federal rules downstream of it. This post walks through the architecture and why it is built this way.
Why two pipelines, not one
Bills and federal rules look superficially similar. Both are primary legal documents. Both arrive over time. Both are filtered against practice areas.
In the dimensions that decide how you build for them, they are opposite:
| Dimension | State and federal bills | Federal Register documents |
|---|---|---|
| Sources | 50 states, DC and Congress, each different | One authoritative publication |
| Access | Scraped HTML, mixed APIs, aggregator feeds | A single documented, stable API |
| Schema stability | Drifts whenever a legislature redesigns | Stable |
| Dominant failure | A source breaks silently and coverage gaps | Volume: thousands of documents per month, nearly all irrelevant |
| Hard problem | Getting the data at all, reliably, from everywhere | Deciding which of it matters |
| Matching technique | Similarity scoring against a written scope | Model classification, because similarity is not enough |
The last two rows are the whole argument. The bill pipeline’s hard problem is ingestion; the Federal Register pipeline’s hard problem is triage. A single pipeline built for one does the other badly: either bill ingestion carries classification machinery it does not need, or federal rulemaking inherits a resilience apparatus built for sources that do not break.
The right answer is two pipelines sharing one downstream contract, the practice-area schema. Each can use the technique its data shape actually calls for.
Pipeline one: the bill scraper
The bill scraper pipeline ingests legislative data from all 50 states, DC, and the US Congress.
Sources. Three tiers, mixed:
- Tier 1 direct APIs for California, Texas, New York, Illinois, Florida, and Congress
- Tier 2 normalized aggregator feeds (Open States, LegiScan) for daily-cadence states
- Tier 3 dedicated scrapers for states with HTML-only or inconsistent feeds
What it captures. Per bill: metadata, sponsors, committee assignments, full text at every version, and the action timeline (introduction, referral, committee vote, floor vote, executive action).
What it does not capture. Committee hearing schedules. Practically no state publishes them as a first-class resource, and the ones that do publish them as PDFs on committee pages updated with under a day’s notice. Rather than ship a signal that is present for a handful of states and silently absent everywhere else, it is not a covered signal at all. A gap you know about is cheaper than a field that is empty for reasons nobody can see.
Cadence. Per state, per signal. Top-five states refresh on minute granularity for status changes. Mid-tier states refresh hourly to daily. The cadence is labeled per state in the dashboard, not hidden behind a uniform “real-time” claim.
Matching. Each bill is scored against the written scope of every tracker by meaning rather than by keyword, so a California bill saying “automated decision systems” reaches an AI practice area that never used the phrase.
Failure detection. Schema-drift monitors run continuously. When a state legislature redesigns its site, our scrapers fail loudly, not silently. Coverage gaps trigger pages, not next-week emails.
Output. Structured records into the practice-area schema. Each record carries provenance (source, fetch timestamp, scraper version) so downstream consumers can reason about freshness.
Pipeline two: Regulatory Watch
The second pipeline reads the Federal Register, and nothing else. State administrative codes are permanently outside the product, by decision rather than backlog.
Source. One: the Federal Register’s published API. Proposed rules, final rules and notices.
What it captures. Per document: title, abstract, full text, agency, document type, publication date, and the comment period with its closing date where one exists.
The hard part. Volume against relevance. The Federal Register publishes thousands of documents a month and nearly all of them are irrelevant to any given practice area. This is the opposite problem to the bill pipeline, where the difficulty is getting the data at all.
Classification, not similarity. Every document is classified against each tracker by a model rather than by a similarity score. This is the one place the architecture deliberately spends more compute than a cosine would cost, and the reason is a failure a similarity score cannot avoid: a vector cannot reliably separate an automated decision system rule from a countervailing duty determination on pasta imports, because the surface vocabulary overlaps and the subject does not. A sentence of context can. A number cannot.
Ordering. Output leads with the open comment deadline rather than the publication date, because the comment window is the one date on a rulemaking that expires and the one a client needs from you in time to act on.
Output. Records into the same practice-area schema as the bills, and into the same monthly issue.
The two pipelines run independently, and that is the point. The bill scrapers break regularly, because fifty legislatures redesign their sites on their own schedules. Regulatory Watch reads one stable API and essentially does not. Coupling them would mean a state scraper failure could delay federal rulemaking coverage, which would be a self-inflicted outage in the half of the system that has no reason to fail.
The convergence layer
Both pipelines write into the same practice-area schema. That schema is the contract:
- Tracker: the durable written scope of a practice area, plus the embedding derived from it. This is the thing both pipelines match against.
- Bill: a structured legislative record, scored against zero or more trackers.
- Regulatory document: a Federal Register record, classified against zero or more trackers.
- Event: a discrete signal (status change, new match, comment deadline approaching) that drives alerting and that is filtered by a materiality bar before it reaches anyone.
The report renderer, the alert engine and the search index all consume the schema. None of them knows which pipeline produced a given record, and none of them needs to.
That is what the loose coupling buys. A state scraper can be rewritten, and the classification prompt for federal documents can be changed, without either touching the other or touching the thing that reads both.
What this means for a reader
The user-facing consequence is the monthly issue. One document per practice area per period, carrying the pending legislation from pipeline one and the federal rulemaking from pipeline two, written as prose rather than presented as two lists to reconcile.
That is the whole reason for the convergence layer. If the two pipelines wrote into separate stores, the issue would have to join them at render time, and a renderer doing a join is a renderer that can produce a document with half its content missing when one side is late. Converging at ingestion instead means the issue is written from one consistent view or it is not written at all.
If you are evaluating tracking products, ask the vendor how their legislative data and their regulatory data are joined, and what happens to the output when one side fails. The answer tells you whether the unified view you saw in the demo is architectural or a screenshot.
Why we did not build one pipeline
We considered the unified pipeline first. It is conceptually cleaner, has fewer moving parts, and would have shipped faster.
It would also have been worse. The constraints bill ingestion needs, schema fidelity and per-source failure isolation across fifty-odd heterogeneous sites, are not the constraints federal rulemaking needs, which are classification quality and deadline extraction over one clean source. Forcing both into one pipeline means a worse bill experience to support rulemaking, or a worse rulemaking experience to support bills.
Splitting them was the slower build and the better product. We are not the first team to land on “specialised ingestion, unified storage, unified access”; most mature data platforms arrive there, because the alternative is a slow death by compromise.
A pipeline we removed
Worth recording, since this post used to describe it as the second pipeline: there was a news intelligence pipeline, matching trade press to bills and practice areas semantically. It worked, in the sense that the matching was accurate.
It was withdrawn from every user-facing surface in August 2026 anyway. The reason was not technical. A news article about a bill is a secondary account, usually written from a press release, often about a bill that does not exist yet or a version since amended. Placing it in the same feed as primary documents produced a signal that read as corroborated without being checkable, and the failure mode was a confident sentence in a client update sourced from a trade blog.
The pipeline still runs and its output table still fills. Nothing reads it. That is a deliberate state rather than an oversight: the cost of keeping it warm is small, and the decision to stop showing it to users was about what a reader is entitled to conclude from a match, not about whether the match was correct.
Running both pipelines in production
LawSignals runs this two-pipeline architecture in production across all 50 states, DC and Congress, with Regulatory Watch reading the Federal Register alongside it, converging on one monthly issue per practice area.
All model calls run on the platform key. Tracker scopes are model-drafted and validated by retrieval measurement, not attorney-reviewed.
If you are evaluating tracking platforms, book a demo and bring your practice areas.
Related reading: See regulatory enforcement tracking for exactly where the federal boundary sits, legislative tracking across every US jurisdiction, or the methodology for how the matching is measured.
