Fifty versions of my project
Abigail Haddad
Last year, I got asked about direct hire.
My former boss
Which agencies and positions are using it?
Me
I know how to do this. It’s not going to be fun.
There is no “direct hire” field. So:
direct[\s-]*hire\s*(?:authority|appointment)?
Military Treatment Facilities under DHA
DHA is also the Defense Health Agency.
7% of postings since 2017 mention it.
Here’s how I did it.
1
Collect
USAJOBS API, then scraping
2
Make rows
Phrases plus conditions
3
Connect
Agency, occupation, month
4
Publish
A public repo
5
Deliver
An answer
What is Congress trying to do to the Paperwork Reduction Act right now?
12 bills would amend it. Two passed the House.
1
Collect
Congress.gov, GovInfo, the Record
2
Make rows
Passages citing the Act
3
Connect
Bills, members, speeches, hearings
4
Publish
Hugging Face, daily
5
Deliver
A site per statute
Notice anything?
They’re all the same shape.
I have a lot of these—some for clients, many for myself.
1
Collect
Get it from the source
2
Make rows
Text, then fields
3
Connect
Match it to other data
4
Publish
Somewhere public
5
Deliver
A site, an email, a memo
A site. A newsletter. A chart in a memo.
The urgency tracker
A daily email of sole-source justifications
ICE hires by month
Here are some more questions I wanted to answer
- What is OPM putting out this week?
- Who is commenting on a proposed reg, and are they for it?
- How often do federal IT jobs require a degree?
- How much was awarded this year citing “urgency”?
- Which vendors sell to the most DC agencies?
- How does ICE’s surge-hiring attrition compare?
And so on.
I look at a lot of data
6,968 commits and pull requests this year.
I took nine days off.
“You have used 53 of 200 available DNS records.”
Cloudflare, about my website. Each record is another site.
Which vendors sell to the most DC agencies?
Dell: 81 agencies, 3,228 purchase orders.
1
Collect
DC’s contracts portal, weekly
2
Make rows
Fields as entered. No model.
3
Connect
Contracts, orders, payments, vendors
4
Publish
Hugging Face
5
Deliver
Vendor and renewal pages
How much was awarded this year citing “urgency”?
$32.8B
$6.7B
1
Collect
USAspending, monthly
2
Make rows
Awards coded urgent
3
Connect
Parent contract, J&A, protests
4
Publish
A CSV
5
Deliver
A site linking every award
Here are the hard parts
The same problems, every time.
Getting the documents
113-s-2000
api.congress.gov returns an error for this one bill.
Forever.
The model gets fields wrong
Incumbent’s contract number
30 · kept
Contract numbers covered
28 · kept
Funding
27 · kept
Number of prior extensions
25 · kept in part
Is it a bridge contract
23 · dropped
Need-by date
19 · dropped
Stated limit on the period
18 · dropped
What triggered the urgency
17 · dropped
Right out of 30 sampled notices, each checked against the document.
Here’s one it got wrong
The document
“the delivery date … is to be 8/1/2025 or best available date”
The model
Need-by date: 8/1/2025
“Or best available” is not a deadline.
Connecting one dataset to another
985 of 1,247
“floor speeches” about the Paperwork Reduction Act
are nobody talking.
Knowing what one row is
$105 trillion
The top number on my own federal budget site.
It’s nine years added together.
Keeping them running
My stuff breaks; but these days, it breaks more loudly.
My DoD scraper’s last 24 runs
Two nights running, this week.
429 Too Many Requests
Hugging Face, asking me to stop loading my own sites.
Up for adoption
I would like to give some of these away to a good home.
Federal J&A Documents
Born: January 2025
Size: 20,019 notices, about 11 GB
Eats: about $4.50 a month in LLM calls
Habits: wakes up daily, emails you what it found
Needs: an engineer, or someone engineer-like
I’ve seen this pattern fifty times.
I can spin yours up. You adopt it after.
What would it take for yours?
- What’s the question?
- Where is it published?
- What is one row?
- What does it connect to?
- What does a wrong answer cost?
Tell me what you’re trying to track.
abigailhaddad.com · adopt.abigailhaddad.com
What is OPM putting out this week?
Three documents one week. None the next.
1
Collect
Federal Register, memos, news
2
Make rows
A model summarizes each
3
Connect
Quotes checked against source
4
Publish
GitHub
5
Deliver
Weekly digest, RSS
Who is commenting on a proposed reg, and are they for it?
167,864 comments. 96% opposed.
1
Collect
A regulations.gov docket
2
Make rows
A model reads each comment
3
Connect
Form letters, rule sections
4
Publish
Labelled comments
5
Deliver
A report per rule
How often do federal IT jobs require a degree?
1% of 38,036 postings.
1
Collect
USAJOBS API plus scraping
2
Make rows
A model sorts each posting
3
Connect
Scrape checked against API
4
Publish
The classified table
5
Deliver
A site, by department
How often do workforce cuts or DOGE show up in sole-source justifications?
390 of 9,710. 41 name DOGE.
1
Collect
SAM.gov notices and PDFs
2
Make rows
Keywords, then a model
3
Connect
Quote, page, award
4
Publish
Hugging Face
5
Deliver
A site and a daily email
How did ICE surge-hiring attrition compare to previous hiring efforts?
27.5% gone inside a year. CBP lost 10–11%.
1
Collect
OPM’s monthly files
2
Make rows
Summed across every file
3
Connect
Departures to hires, by job
4
Publish
A public notebook
5
Deliver
A memo