Module · Data & Questions

The data each analysis needs, and the questions we construct to get it

HSG's analysis treats the disparity question the way the law requires it to be treated: as something to be tested, not assumed. Every technique below starts from a measured gap between an ownership group and the non-disadvantaged comparison group, then works to separate the part the City can legitimately price on the merits (firm size, age, revenue, bonding, prior experience, industry mix, how often a firm bids and how competitively) from the part that legitimate capability does not explain. Where an unexplained residual survives, the social-capital channels drawn from the Loury contact-stratification frame (information access, referral and vouching, mentorship, incumbency and repeat business, prime-sub ties, and credit and bonding relationships) are tested as candidate mediators, and verified business-owner accounts are matched cell-by-cell to corroborate what the numbers show. No technique presumes a verdict. Each one is built to answer whether a disparity traces to discrimination or to neutral factors, and to report exactly where that answer is robust versus where it depends on a defensible modeling choice. Boston's advertised scope also reaches beyond vendor utilization to the workforce participation of minority and female workers on City and large City-assisted construction projects; the primary data for that piece are certified payroll records on covered projects, including the compliance reporting the Boston Residents Jobs Policy already generates, analyzed under the same significance and no-presupposition discipline as the vendor-side techniques below. The matrix pairs each technique with the analytical question it answers, the data it needs and why, and the four kinds of questions the study will build: what to ask the City and Boston Public Schools to hand over, what to ask firms in the availability survey, what to ask owners in attributed interviews, and the analytical question the technique itself resolves. All sampling and data handling are written for City approval and to respect confidentiality.

Question-design and defensibility safeguards

These rules are what let the questions survive scrutiny. They govern every survey item and interview prompt below, so the instrument never embeds the conclusion it is meant to test.

No presupposition. Every conditional question is gated by a screen first, and the conclusion is never embedded in the stem.
Neutral, behavior-anchored response options, never a discrimination-primed option; closed items are paired with an open follow-up so the respondent supplies the attribution.
Mediation is reported as descriptive coefficient-movement accounting, not causal claim. A residual is described as differential returns and social-capital deprivation, never as proof of discrimination.
Pre-registered before any results are seen: the comparison-group definition, the Benjamini-Hochberg false-discovery-rate correction across the significance grid, the selection-model exclusion restriction, and the ownership coding bands.
Contested controls such as incumbency are reported both ways: with the control is the conservative lower bound, without it is the upper bound that treats the variable as itself a channel.
Ownership classification is reported under three codings as a sensitivity band: self-certified, survey-verified, and registry-verified.
Subcontract and team analyses are contingent on City data; the network layer leads with the well-populated co-bidding graph, with the prime-sub graph a documented supplement.
Every survey is delivered with a defensibility package: response rates by stratum, nonresponse analysis, design weights to the frame, and documented frame sourcing.
AI assists, it does not decide. The AI-assisted procurement-rules review and any AI-assisted coding surface candidate flags on a pre-registered rubric; a named investigator adjudicates every flag, counsel owns every legal conclusion, and a responsible-AI advisor governs the bias-testing and documentation of each AI step.

Technique by technique

Each method, the data it needs and why, and the questions it drives

Every technique starts from a measured gap and works to separate what legitimate firm capability explains from what it does not. None presumes a verdict.

Oaxaca-Blinder decomposition of the group gap in bid-win rate and award/subcontract dollars

Of the measured gap between an ownership group and the non-disadvantaged comparison group in bid-win rate and in award/subcontract dollars, how much is explained by differences in observable, legitimately-priced firm capability (size, age, revenue, bonding, prior City experience, NAICS mix, bid frequency, bid competitiveness) versus an unexplained residual, reported per group and per category without presuming the residual is discrimination?

Data needed, and why

Firm-by-solicitation bid/quote outcome panel with win/loss flag across 5 FY, so a win rate and its denominator exist (City bid tabs, quote logs, plan-holder and non-responsive records; bulk historical extract)
Award and payment dollars per firm per FY by category and method, from the City financial-system AP extract plus the Boston Public Schools disbursement extract, the internal counterparts of the Analyze Boston contract-award and discretionary-spending datasets (the dollars-paid utilization base and dollar outcome)
Subcontract-dollar records linking prime to sub (weak/at-risk; City compliance records plus prime payment affidavits)
Ownership-group classification per firm with a self-cert / survey-verified / registry-verified coding band (City Supplier Portal registration, City and Commonwealth SDO certification directories, SAM.gov, calibrated by survey)
Firm capability covariates for the explained block: firm age, employment size, receipts, primary and secondary NAICS, RGMA location (registration, Census ABS/CBP, D&B/Data Axle, survey)
Bonding capacity per firm so it sits in the explained block, not the residual (SBA Surety Bond Guarantee, prequalification fields, survey)
Prior City experience / incumbency count per firm, run both with and without it as a contested control (derived from the award spine)
Bid frequency per firm and bid-competitiveness measures where recorded (derived panel; bid-tab price columns, non-responsive flags, the latter weak/at-risk)
Procurement method and department per solicitation for category-level reporting (bid tabs and payment spine)

Questions to construct

DATA-REQUESTFor each of the past five fiscal years, please provide the complete bid and quote history for City and Boston Public Schools solicitations, including for every participating firm the solicitation number, category, procurement method, soliciting department, firm identifier, submitted bid amount where recorded, responsiveness determination, and whether the firm was awarded.
DATA-REQUESTPlease provide prime award and payment records and, separately, any subcontractor payment or compliance records for the past five fiscal years at the firm and fiscal-year level, including paid dollars, category, method, and award date, with a documented note where subcontractor data are incomplete.
DATA-REQUESTPlease provide vendor registration and certification records with each firm's reported ownership demographics, years in business, employment size, annual receipts, primary and secondary NAICS codes, and location, noting which fields are self-reported versus independently verified.
SURVEYIn what year did your firm begin operating, and how many years has it been bidding on government contracts in this region?
SURVEYWhat is the largest single contract value your firm is currently able to bond, and through what mechanism (surety bond line, SBA Surety Bond Guarantee, or none)?
SURVEYIn the past five years, how many times did your firm submit a bid or quote on a City of Boston or Boston Public Schools solicitation, and how many of those resulted in an award?
INTERVIEWThinking about a specific City or BPS solicitation your firm pursued, please describe what factors you believe most affected whether your firm was selected or not selected.
INTERVIEWWere there qualifications, experience floors, bonding levels, or other requirements on City solicitations that your firm met, did not meet, or found difficult to meet, and how did that affect your decision to bid?
ANALYTICALWhat share of the bid-win-rate gap and the award-dollar gap for each ownership group, in each category and method, is explained by observable firm capability differences, and what share remains as an unexplained residual, reported both with and without contested controls such as incumbency?

Gelbach decomposition attributing how much of the residual gap is mediated by each social-capital channel

Of the unexplained residual that survives capability controls, how much is statistically mediated by each of the six social-capital channels (information access, referral/vouching, mentorship, incumbency/repeat business, prime-sub ties, credit/bonding relationships), so the residual is characterized as differential returns and social-capital deprivation rather than left as an undifferentiated unknown, without labeling any channel as proof of discrimination?

Data needed, and why

S1 information-access measures per firm: how and when a firm learns of solicitations, use of debriefs and demand forecasts (survey and VAR interviews; City notification and debrief records)
S2 referral/vouching measures: whether a firm was referred or vouched for by an existing vendor, prime, or City contact (survey and VAR interviews)
S3 mentorship/know-how measures: access to mentor-protege, bid-writing or estimating help, capability-transferring teaming (survey and VAR interviews; City mentor-protege records if any)
S4 incumbency/repeat-business measures: count and recency of prior awards (derived from the award spine; strongest-measured mediator)
S5 prime-sub tie measures: subcontract relationships with active primes and tie count/strength (City subcontract/compliance records plus affidavits, weak/at-risk, supplemented by survey)
S6 credit/bonding-relationship measures: access to a banking and surety relationship, distinct from raw bonding capacity (survey and VAR; SBA SBG participation)
The shared capability X-block and group classification from technique 1, required for the order-invariant Gelbach base specification
Group-mean network position per channel and channel returns, to compute the mediated share per channel
Ownership-misclassification sensitivity codings, since several channels are survey-measured

Questions to construct

DATA-REQUESTPlease provide any records the City maintains on how vendors are notified of solicitations, including bidder-notification or interest lists by category, plan-holder lists, and records of debrief requests and demand-forecast distribution, for the past five fiscal years.
DATA-REQUESTPlease provide any subcontractor reporting, mentor-protege program participation, and prime payment-affidavit records that link firms to primes for the past five fiscal years, with a documented note where these linkages are incomplete or not collected.
SURVEYHow does your firm usually first learn that a City of Boston or BPS contracting opportunity is available, and at what point in the process do you typically become aware of it?
SURVEYWhen your firm has pursued a City opportunity, was anyone already working with the City, such as a prime contractor or an established vendor, who referred you, introduced you, or spoke on your behalf?
SURVEYDoes your firm have an established relationship with a bank or lender and with a surety provider that you can draw on for a contract, separate from the dollar amount you are able to bond?
INTERVIEWPlease describe, for a specific opportunity, how you found out about it, who if anyone connected you to the City or to a prime, and what relationships or sources of information you think made the most difference to the outcome.
INTERVIEWWhen your firm has needed bonding, financing, or a referral to compete for a City contract, what has that experience been like, and where did access come easily or prove difficult?
ANALYTICALOf the unexplained residual after capability controls, what statistically estimated share is mediated by each of the six social-capital channels for each ownership group and category, reported under each ownership-classification coding as a sensitivity band, with the residual labeled as differential returns and social-capital deprivation rather than proof of discrimination?

Capacity-controlled but-for regressions: win-probability probit/logit P(win|bid) and award-dollar Tobit / Cragg two-part model

Holding a firm's legitimate, race-neutral capacity constant (size, age, revenue, bonding, prior City experience, industry, bid frequency), does owner group still independently change its probability of winning once it has bid, and the dollars it is awarded? Between two otherwise-identical firms that both bid, does owner group move the outcome, or do measured capability differences fully explain the gap?

Data needed, and why

Bid-level outcome records linking each bid/offer/quote to its solicitation, firm, and a win/loss flag across 5 FY (City financial-system awards joined to bid tabs, quote logs, plan-holder lists; bulk extract)
Award and obligated/paid dollar amount per winning bid with contract and modification history, for the censored Tobit/two-part outcome (City financial system plus BPS records)
Verified owner-group classification per firm under a self-cert / survey-verified / registry-verified band (City registration, City and Commonwealth SDO certification, survey, SAM.gov)
Race-neutral capacity controls: years in business, employment size, receipts, bonding capacity, prior City experience count and dollars, certifications (Census ABS/CBP/PUMS, survey, SBA SBG, City records, SAM.gov)
Bidding-frequency / activity measure per firm, both a control and the exposure variable (bid tabs and plan-holder lists, cross-checked to Supplier Portal registration)
Procurement method, category, and department per solicitation as fixed-effect / stratification controls (City financial system plus BPS records; Supplier Portal)
Subcontract / prime-sub award and payment records for the subcontract-dollar version (weak/at-risk; degrades to prime-only)
A City-approved sampling and group-aggregation/power-analysis plan for thin cells (HSG methodology submitted for City approval)

Questions to construct

DATA-REQUESTPlease provide, for each of the last five fiscal years, a record of every solicitation, naming the solicitation number, procurement method, category, awarding department, and entity (City or BPS).
DATA-REQUESTPlease provide, for each solicitation, the list of every firm that submitted a bid, quote, or offer, with a flag indicating whether that firm was the awardee.
DATA-REQUESTPlease provide the award amount and all obligated and paid dollar amounts per awarded contract, including modifications, for each of the last five fiscal years from both the City financial system and the BPS payment system.
DATA-REQUESTPlease provide subcontractor award and payment records and prime payment affidavits where they exist, with the prime, the subcontractor, the dollar amount, and the parent solicitation.
SURVEYIn how many years has this firm been in business under current ownership?
SURVEYWhat was this firm's approximate gross revenue and full-time-equivalent employee count in the most recent year?
SURVEYWhat is the largest single contract value this firm is currently bonded to perform, and what is its aggregate bonding capacity, if any?
SURVEYDuring the last five years, approximately how many City or BPS solicitations did this firm submit a bid, quote, or offer on?
ANALYTICALHolding firm capacity constant, does owner group have an independent and statistically significant effect on the probability of winning conditional on having bid, and on the dollars awarded, reported per group-by-category-by-outcome cell with and without contested controls such as incumbency?

Heckman two-stage selection correction on the decision to bid, then outcome conditional on bidding

Are the firms that choose to bid systematically different from the available firms that do not, and once that selection is corrected, does owner group still affect outcomes? Is part of the measured disparity occurring upstream at the decision to bid at all, driven by neutral barriers such as awareness, bonding, and set-up cost versus discouragement, rather than at the win stage where most studies look?

Data needed, and why

A roster of available ready-willing-able RGMA firms that did NOT bid, not just bidders, to build the first-stage P(bid) equation (City-approved stratified availability survey, de-duped D&B/Data Axle frame, Supplier Portal registration origins)
At least one valid exclusion restriction that shifts bidding but not win merit (distance to Boston, prior awareness, registration status, bid-prep fixed cost), from the survey, registration timestamps, and geocoded location
Firm-level reasons for bidding or not, coded as neutral barrier versus discouragement (survey closed-set items corroborated by VAR interviews)
The same capacity block and owner-group classification as the win/dollar models, in both stages
City/BPS solicitation-awareness and outreach footprint per category (procurement outreach records and notification logs, corroborated by survey)
City/BPS standing eligibility requirements actually applied per category: bonding, insurance, experience floors, prequalification, prevailing-wage (purchasing policies, solicitation specs, prequalification records)
Plan-holder / document-pull records, the cleanest marker of aware-but-selected-out firms (plan-holder lists, Supplier Portal download logs; weak/at-risk)

Questions to construct

DATA-REQUESTPlease provide, for each solicitation in the last five fiscal years, the plan-holder or document-pull list of firms that obtained the solicitation documents, distinguished from the firms that ultimately submitted.
DATA-REQUESTPlease provide the vendor-notification and outreach records showing where each category of solicitation was advertised and which registered vendors were notified.
DATA-REQUESTPlease provide the standing eligibility requirements applied to each procurement category, including bonding thresholds, insurance minimums, prior-experience floors, prequalification, and prevailing-wage conditions.
SURVEYBefore being contacted for this study, how aware was this firm of City and BPS contracting opportunities in its line of work, on a scale from not at all aware to very aware?
SURVEYFor the most recent City or BPS opportunity this firm was eligible for, did it bid, and what were the main reasons it did or did not bid? Select all that apply: did not know about it, requirements such as bonding or insurance were out of reach, the contract was too large or bundled, the experience or prequalification requirement could not be met, the cost of preparing a bid was too high, prior experience with the process, expected the outcome would not be fair, chose to pursue other work, other.
SURVEYHow much staff time and out-of-pocket cost does it typically take this firm to prepare a City or BPS bid, and has that cost ever caused the firm to skip an opportunity it was qualified for?
INTERVIEWWalk me through a specific recent City or BPS opportunity your firm was qualified for but did not pursue. What happened, how did you learn of it or not, and what would have had to be different for you to bid?
ANALYTICALAfter correcting for non-random selection into bidding, does owner group significantly affect outcomes, and how much of the overall disparity arises at the decision-to-bid stage versus the win stage, with the upstream portion attributed to identified neutral barriers versus an unexplained discouragement residual?

Within-firm gap x procurement-method interaction with firm fixed effects

Within the same firm over time, is the group win/dollar gap larger in high-discretion, high-trust lanes (sole-source, small/micro-purchase, discretionary quote, prequalified lists) than in open sealed competition? If the gap concentrates in high-discretion lanes after differencing out time-invariant firm quality, that is the network signature the Loury frame predicts and is hard to dismiss as unobserved ability. It does not by itself prove discrimination; it locates where any disparity lives along the procurement-method dimension.

Data needed, and why

Firm-level panel of every bid/quote/award event across 5 FY keyed to a stable normalized vendor entity ID, so firm fixed effects can absorb time-invariant quality (City financial system plus BPS records joined to bid/quote/award records; entity de-duplication with human verification)
Procurement-method code on every event coded to a pre-registered low/mid/high discretion scale (purchasing policies and method fields; coding submitted for City approval; answers the advertised departmental-practices scope)
Outcome variables per event: bid-win indicator and award/subcontract dollars (bid tabs plus the City and BPS ledgers; subcontract dollars from affidavits where available)
Survey-verified ownership flags with self-cert and registry codings as a sensitivity band (City registration and certification, Commonwealth SDO directory, SAM.gov, corrected by survey)
Race-neutral capacity controls that VARY within firm over time: firm age at bid, current-year receipts, bonding capacity, prior-experience count, bid frequency, solicitation NAICS, plus contested incumbency both ways (ABS/CBP, survey, SBA SBG, City ledger)
Sufficient within-cell sample of firms that bid in BOTH a high- and a low-discretion lane by group x NAICS (derived panel; flagged as a feasibility limitation if thin)

Questions to construct

DATA-REQUESTPlease provide, for the five fiscal years, a record-level extract of every solicitation, bid, quote, and award for the City and BPS, each record tagged by procurement method (open competition, informal quote, small purchase, sole source, micro-purchase, emergency, prequalified or term contract), the awarded and paid dollar amount, the win or loss outcome, the soliciting department, and a vendor identifier consistent across all five years.
DATA-REQUESTPlease provide the City and BPS purchasing policies that define each procurement method, including the dollar thresholds and the level of buyer discretion permitted under each, so methods can be classified consistently.
SURVEYIn the past five years, has your firm submitted bids or quotes to the City of Boston or BPS through more than one type of procurement, for example both formal sealed solicitations and informal quote or small-purchase requests?
SURVEYFor each procurement type your firm has used with the City, how did your firm first learn that the opportunity was available?
INTERVIEWThink of a specific City or BPS opportunity that was awarded without a full open competition, such as a small purchase, a quote request, or a sole-source award. Walk me through how you became aware of it and how you were or were not invited to compete.
INTERVIEWWhen the City selects a vendor through an informal or discretionary process rather than open bidding, what would have made it easier for your firm to be considered?
ANALYTICALHolding the same firm fixed and controlling for time-varying capacity, does the estimated group difference in win probability and award dollars differ across low-, mid-, and high-discretion procurement methods, and is any such difference statistically significant per group and category?

Dynamic state-dependence: effect of a first City award on probability of a second, by group

After differencing out fixed firm quality, does winning a first City award raise a firm's probability of winning a second, and is that incumbency or learning premium smaller for minority- and women-owned firms than for comparable non-MWBE firms? A smaller second-award boost for a group is consistent with the Loury dynamic mechanism. The test is corroborative, not dispositive: awards build network capital and network capital wins awards, so reverse causation is acknowledged and the finding is labeled differential returns and social-capital deprivation, never proof of discrimination.

Data needed, and why

Firm-level panel with the full date-ordered City/BPS award history per firm across 5 FY, so first and subsequent awards can be sequenced (City financial system plus BPS records ordered by award/payment date on the normalized key)
An initial-conditions baseline: each firm's pre-period vendor status, registration date, and prior experience at window start, to separate true state dependence from unobserved heterogeneity (registration plus earliest historical award extract; weak/at-risk)
First-time-vendor and lost-contract / non-responsive records to define the correct at-risk set for a second award (bid tabs, non-responsive logs, lost-contract records; bulk extract)
Survey-verified ownership flags by firm with self-cert/registry band, since the test is by group (City registration and certification, Commonwealth SDO directory, SAM.gov, corrected by survey)
Time-varying capacity controls at the time of the first award plus the dollar size and method of that first award (City ledger; ABS/CBP, SBA SBG, survey)
Subcontract / prime-sub history per firm as a corroborating network-capital measure (compliance records and affidavits; weak/at-risk, reported under caveat)

Questions to construct

DATA-REQUESTPlease provide a date-ordered award and payment history for every vendor paid by the City or BPS during the five fiscal years, including each award date, awarding department, dollar amount, procurement method, and a vendor identifier consistent across years, plus award history predating the window where available so each vendor's first City award can be identified.
DATA-REQUESTPlease provide the records of firms that submitted a bid or quote but did not receive an award, including non-responsive and lost-contract determinations, for the five fiscal years.
SURVEYAfter your firm received its first contract or purchase order from the City of Boston or BPS, did the chance of winning additional City work change, and if so, in what way?
SURVEYFollowing your firm's first City award, did any of the following change: invitations to bid on later opportunities, introductions to prime contractors or other departments, your bonding capacity, or your access to past-performance references?
INTERVIEWTell me about the period right after your firm won its first contract with the City. What happened next in terms of being considered for additional work, and what helped or got in the way of winning a second contract?
INTERVIEWLooking back, do you attribute the path from your first City contract to later contracts more to your firm's performance and capacity, to relationships and referrals, or to something else?
ANALYTICALControlling for firm capacity and first-award characteristics and accounting for initial conditions, does receiving a first City award raise the probability of a second, and is that marginal effect significantly smaller for minority-owned or women-owned firms than for otherwise comparable firms, per group and category?

Social network analysis of prime-sub and co-bidding structure (bipartite plus co-bidding graph, ownership homophily/assortativity via E-I index and Newman, brokerage/centrality, MRQAP dyadic regression)

When teaming, subcontracting, and co-bidding relationships are mapped as a network, do firms connect along ownership lines (minority- and women-owned firms clustering with each other and away from well-connected primes), and does a firm's network position predict who wins City work after capability is controlled? Is access to the relationship structure that channels City contracts itself stratified by ownership group, which would be the present-day network signature the Loury frame predicts, or do relationships sort on neutral factors such as NAICS overlap, size, geography, and bonding capacity?

Data needed, and why

Prime-to-subcontractor payment and award records linking each prime contract to subs actually used and dollars paid (firm identity, ownership flags, NAICS, dollars, dates), the raw material for the bipartite graph (City/BPS financial-system subcontract modules if populated, contract files, required utilization reports; weak/at-risk)
Bid tabulations and plan-holder/quote logs across 5 FY listing every bidder, not only winners, to reconstruct the co-bidding graph as the fallback layer (City and BPS bid tabs, quote logs, plan-holder lists, non-responsive logs)
Team/joint-venture and partnering declarations submitted with bids, including for proposals that did not win (procurement files, SAM.gov entity relationships, Supplier Portal records)
Firm attribute table per node: ownership group, NAICS, size, location, years in business, bonding capacity, registration status, used as MRQAP covariates (City registration and certification, Census ABS/CBP, D&B/Data Axle, SBA SBG, survey)
Survey-reconstructed teaming and referral edges (firm-reported primes subbed to, firms teamed with, how work was learned of), the documented workaround for the weak subcontract layer (availability survey)
A record-completeness audit of subcontract/team coverage by year and department, to bound and disclose SNA coverage (City/BPS system metadata, data dictionary, extract logs; weak/at-risk)

Questions to construct

DATA-REQUESTFor each City and BPS contract awarded during the study period, please provide the subcontractors used and dollars paid to each, including name, address, ownership classification if recorded, NAICS, and payment dates, and identify the system of record and the date range for which these records are reliably populated.
DATA-REQUESTPlease provide complete bid tabulations and plan-holder or quote logs for every solicitation in the study period, listing all firms that submitted, not only the awardee, so firms appearing together in the same competitions can be identified.
DATA-REQUESTWhere bids included team, joint-venture, or mentor-protege arrangements, please provide the submitted documents that name the team members and their roles, including for proposals that were not selected, and a coverage statement by fiscal year and department for the share of contract actions and dollars for which subcontractor and team information is present.
SURVEYIn the past five years, which firms has your business worked with as a subcontractor or team member on a contract, public or private, and which firms has your business engaged as your subcontractors or team members?
SURVEYHas your business ever been invited by a prime contractor to join a team or provide a subcontract bid on City or BPS work, and approximately how many such invitations have you received in the past five years?
INTERVIEWIf you have sought to team with or subcontract under a prime contractor on public work and were unable to form that relationship, what reasons were given or what obstacles did you encounter?
INTERVIEWPlease describe how the teaming relationships you do have came about, for example through prior coworkers, trade associations, family or community ties, repeat business with the same firms, or formal matchmaking programs.
ANALYTICALAfter accounting for NAICS overlap, firm size, geographic proximity, and bonding capacity, do prime-sub and co-bidding ties form disproportionately between firms of the same ownership group, and does network position predict City award outcomes, or are the observed ties and outcomes explained by those neutral factors?

With-and-without-contested-controls reporting on the per-(group x category x outcome) significance grid

For each combination of ownership group, procurement category, and outcome (bid-win rate and award/subcontract dollars), is there a statistically significant disparity, and how sensitive is that finding to whether the model includes controls that may themselves be products of past exclusion, namely incumbency and prior City experience? Does a disparity hold up when everything including the contested variables is controlled (the conservative read), and does it grow when those variables are excluded (the read that treats them as channels of disadvantage), so the City can see exactly where the conclusion is robust versus modeling-dependent?

Data needed, and why

Firm-level outcome records by group, category, and outcome type: bids submitted, bids won, dollars awarded, and subcontract dollars where available, across 5 FY (City financial system plus BPS records, bid tabs, subcontractor-payment records)
Incumbency indicator with a clean date sequence: whether the firm had a prior City/BPS award before the action and how recently, entered both ways (derived from award history)
Prior-experience measures: count and dollar value of prior relevant contracts (City and, where available, other public or private) and years in business, entered both ways (award history, SAM.gov past performance, Supplier Portal registration, survey for non-City experience)
Uncontested capacity controls held constant in both specifications: firm size, bonding capacity, NAICS, and the category/method of each action, so only the contested variable changes (registration, Census ABS/CBP, SBA SBG, survey, method coding)
A consistent group and category taxonomy with cell-size counts plus a documented minimum-cell-size rule and a multiple-comparison adjustment such as false-discovery-rate (study design on City data; survey-verified flags; submitted for City approval)
Survey-verified ownership flags to validate the administrative classification that defines every grid row (availability survey)

Questions to construct

DATA-REQUESTPlease provide, for each firm and each contract action in the study period, the award and bid history needed to determine whether the firm was an incumbent or had prior City or BPS experience at the time, including award dates and dollar amounts, so prior-relationship status can be constructed in date order.
DATA-REQUESTPlease provide the procurement method and category for each solicitation and award (for example open bid, small purchase, sole source, cooperative contract, term contract) so outcomes can be tabulated by category.
DATA-REQUESTPlease provide your standard ownership and certification classification for each vendor, along with how and when that classification was recorded, so it can be reconciled with firm self-reported ownership.
SURVEYApproximately how many relevant contracts has your business performed in the past five years for public and for private clients, and how many years has your business been operating?
INTERVIEWPlease describe how prior experience or being an existing City vendor has affected your ability to win additional City work, in either direction, and how your business first obtained its first public contract.
ANALYTICALFor each combination of ownership group, procurement category, and outcome, is the measured disparity statistically significant both when incumbency and prior experience are controlled and when they are excluded, and where do the two specifications agree versus diverge?

AI-assisted review of the City's own procurement rules for internal conflicts and disparate impact (human-adjudicated, RAI-governed)

Do City of Boston and BPS procurement policies, manuals, and standing solicitation templates contain rules that conflict with one another, or that though facially neutral mechanically screen out ready-willing-able diverse firms more than the work requires, and can a large-language-model read of the full rules corpus surface those conflicts and candidate disparate-impact provisions faster and more completely than a manual read, for a named investigator and legal counsel to then verify?

Data needed, and why

The full City and BPS procurement policy corpus: purchasing manuals, administrative regulations, standing solicitation templates and boilerplate, prequalification rules, bonding and insurance schedules, and delegation-of-authority and method-threshold tables (City and BPS procurement offices)
The five-fiscal-year solicitation corpus already requested for the criterion audit, so a flagged rule can be tied to the solicitations that carry it (City and BPS solicitation archives)
Availability-survey and census firm-attribute distributions (firm age, bonding capacity, receipts by ownership) needed to estimate, for each flagged criterion, the share of each ownership group that could satisfy it (availability survey; Census ABS/CBP)
A pre-registered coding scheme mapping each rule to the 16-barrier framework and a four-fifths disparate-impact screen, so the AI pass produces candidate flags on a fixed rubric rather than open-ended judgments
A logged human-in-the-loop review record: every AI-surfaced conflict or disparate-impact flag is adjudicated by a named investigator, and every legal conclusion by counsel; the AI makes no finding

Questions to construct

DATA-REQUESTPlease provide the complete current and study-period procurement policy corpus for the City and BPS, including purchasing manuals, administrative regulations, standing solicitation templates and standard terms, prequalification and eligibility rules, bonding and insurance schedules, and the tables that set procurement-method thresholds and buyer discretion.
DATA-REQUESTWhere procurement rules changed during the five-year study period, please provide the dated versions so each rule can be attributed to the solicitations issued under it.
SURVEYIn your firm's experience, are there specific City or BPS requirements that seem to conflict with one another, or that appear unnecessary for the work, such that they made it harder to register, qualify, or bid? Please describe.
INTERVIEWWalk me through any City or BPS requirement that struck you as contradictory, outdated, or out of proportion to the job, and describe how it affected your firm's ability to compete.
ANALYTICALAcross the full procurement-rules corpus, which provisions conflict with one another, and which facially neutral provisions can be satisfied by a significantly smaller share of the available minority- or women-owned pool than of the comparison pool, once each AI-surfaced flag is verified by the investigator and, for legal conclusions, by counsel?

What we ask the City and Boston Public Schools to provide

The data request

The study stands or falls on the records the City and Boston Public Schools turn over. We name them explicitly and up front, so nothing is assumed.

01Complete bid and quote history for City and Boston Public Schools solicitations across the five most recent fiscal years, at the firm-by-solicitation level: solicitation number, category, procurement method, soliciting department and entity (City or BPS), firm identifier, submitted bid amount where recorded, responsiveness or non-responsive determination, and an awardee flag, delivered as a bulk historical extract.
02Plan-holder and document-pull lists per solicitation (including Supplier Portal download logs) distinguishing firms that obtained documents from firms that submitted, plus non-responsive and lost-contract determinations.
03Prime contract award and payment records from the City financial system and the BPS disbursement system for five fiscal years: paid and obligated dollars, category, procurement method, award date, department, and contract/modification history, at the firm and fiscal-year level, the internal counterparts of the Analyze Boston contract-award and discretionary-spending datasets.
04Subcontractor and second-tier payment records where the City tracks them, required utilization and compliance reports, and prime payment-affidavit confirmations linking prime to sub (name, address, ownership classification if recorded, NAICS, dollars, dates), with a documented note where these data are incomplete or not collected.
05Invoice-date and payment-date records (days-to-pay) for prime and, where captured, subcontractor payments, so payment timing can be tested by firm ownership rather than left as anecdote.
06Sheltered Market Program records since the program's 2022 adoption: contracts designated to the sheltered market, the certified vendors that competed and won, and dollar outcomes, so the program's effect can be measured directly.
07Team, joint-venture, and mentor-protege declarations submitted with bids, including for proposals that were not selected.
08A data dictionary and a coverage statement indicating, by fiscal year and department, the share of contract actions and dollars for which subcontractor and team information is present.
09Vendor registration and certification history files: reported ownership demographics, years in business, employment size, annual receipts, primary and secondary NAICS, business location, and City MBE/WBE/SLBE/SBE and Commonwealth SDO certification status with application, decision, and renewal dates, noting which fields are self-reported versus independently verified.
10Vendor-notification and outreach records showing where each category of solicitation was advertised, which registered vendors were notified, bidder-interest lists by category, debrief-request records, and demand-forecast distribution.
11Standing eligibility requirements actually applied per procurement category: bonding thresholds, insurance minimums, prior-experience floors, prequalification requirements, and prevailing-wage conditions, drawn from the purchasing policies and solicitation specifications.
12City and BPS purchasing policies defining each procurement method, including dollar thresholds and the level of buyer discretion permitted under each, plus the procurement method used on each transaction by department, to support a consistent low/mid/high discretion coding.
13A consistent vendor entity identifier usable across all five years (or the source fields needed for HSG entity de-duplication) so the same firm can be tracked across years, methods, and the prime and subcontract ledgers.
14Award history predating the five-fiscal-year window where available, to establish initial-conditions and identify each vendor's first City award for the state-dependence test.
15Mentor-protege or technical-assistance program participation records, if any such programs exist.
16Certified payroll and workforce-compliance records for covered City and City-assisted construction projects, including Boston Residents Jobs Policy compliance reporting, to support the workforce-participation side of the advertised scope.

Survey

Availability survey item bank

Firm-facing items measuring ready, willing, and able status, capability, bidding behavior, and the social-capital channels, with owner demographics for verification.

1Does your firm currently perform any of the following categories of goods, services, or construction? Select all that apply. [checklist of City-contracted NAICS/NIGP categories]
2In the past five years, has your firm performed work for any government, institutional, or large private client in these categories? Select the client types that apply. [None / Local government / State / Federal / School district or university / Large private firm]
3Would your firm be willing to bid on a City of Boston or Boston Public Schools contract in these categories under typical solicitation terms? [Yes / No / Depends on the specific opportunity]
4In what year was your firm established, and how many years has it been bidding on government contracts in this region? [numeric]
5Approximately how many full-time-equivalent employees did your firm have at the end of the last fiscal year? [numeric]
6Approximately what were your firm's total gross receipts in the last fiscal year? [banded numeric ranges]
7What is the largest single contract, by dollar value, your firm has completed in the last five years? [banded numeric ranges]
8What is the maximum bonding capacity your firm can currently obtain on a single project, if any, and through what mechanism? [None / Under 250k / 250k to 1M / 1M to 5M / Over 5M / Do not know], [surety bond line / SBA Surety Bond Guarantee / none]
9Has your firm ever been unable to pursue a contract opportunity because it could not obtain a required bond? [Yes / No / Not applicable, bonding not required for our work]
10In the last two years, has your firm sought external financing (a loan, line of credit, or surety credit) for working capital or to perform a contract? [Yes and obtained it / Yes and was denied or got less than requested / No, did not seek financing]
11Does your firm have an established relationship with a bank or lender and with a surety provider that you can draw on for a contract, separate from the dollar amount you are able to bond? [Yes both / Bank only / Surety only / Neither]
12How does your firm typically first learn about City or BPS contracting opportunities, and at what point in the process? Select all that apply. [City Supplier Portal postings / City website / Email from a buyer or department / A prime contractor or business contact / An industry association or chamber / A networking event / Word of mouth / We do not currently learn about them]
13How would you rate the timeliness and completeness of the information your firm receives about City and BPS opportunities before a bid is due? [More than adequate / Adequate / Inadequate / We do not receive it]
14When your firm pursues new work, how often does an introduction, recommendation, or referral from someone already working with the buyer play a role? [Always / Often / Sometimes / Rarely / Never]
15In the last five years, has a prime contractor invited your firm to join a bid team or subcontract on a City, BPS, or other public project, and approximately how many such invitations? [Yes more than once / Yes once / No], [numeric]
16When your firm acts as a prime, how do you most often identify subcontractors? Select all that apply. [Firms we have worked with before / Referrals from other contractors / Public directories or certified-firm lists / Open solicitation to subcontractors / We do not use subcontractors]
17Which firms has your business worked with as a subcontractor or team member in the past five years, public or private, and which firms has it engaged as your subcontractors or team members? [open list]
18Has your firm participated in a mentor-protege, technical-assistance, or bid-readiness program offered by any government or organization? [Yes currently / Yes in the past / No never / Was not aware any existed]
19Approximately how many separate City or BPS contracts or purchase orders has your firm been awarded in the last five years? [None / 1 / 2 to 5 / 6 to 20 / More than 20]
20If your firm has done City or BPS work, did your first award lead to additional awards from the City or BPS, and what changed afterward (invitations to bid, introductions to primes or departments, bonding capacity, past-performance references)? [Yes / No / Only one / None], [select all that apply]
21In the last five years, how many times has your firm submitted a bid, quote, or proposal to the City or BPS? [numeric]
22For the most recent City or BPS opportunity your firm was eligible for, did it bid, and what were the main reasons it did or did not? Select all that apply. [Did not learn about it in time / Contract too large or bundled / Bonding or insurance requirement out of reach / Experience or prequalification requirement could not be met / Cost of preparing a bid too high / Prior experience with the process / Did not expect a fair chance / Chose to pursue other work / Other]
23How much staff time and out-of-pocket cost does it typically take your firm to prepare a City or BPS bid, and has that cost ever caused the firm to skip an opportunity it was qualified for? [numeric / narrative], [Yes / No]
24Through which procurement methods has your firm most often won public work? Select all that apply. [Sealed competitive bid / Request for proposals / Small or micro purchase / Sole-source or single-source / Cooperative or piggyback contract / Have not won public work]
25Is your firm currently registered or certified under any of the following programs? Select all that apply. [City of Boston MBE/WBE certification / City of Boston SLBE or SBE certification / Massachusetts Supplier Diversity Office (SDO) certification / SBA 8(a) or other federal program / Other local or regional certification / None]
26Is your firm at least 51 percent owned and controlled by one or more individuals in any of the following groups? Select all that apply. This information is used only in aggregate for the study. [Woman / Black or African American / Hispanic or Latino / Asian American / Native American or Alaska Native / Native Hawaiian or Pacific Islander / Veteran or service-disabled veteran / None of these / Prefer not to answer]
27Approximately how far is your firm's primary place of business from the City of Boston, and in what city, county, and state is it located? [numeric / text]
28What is the highest level of formal education completed by the firm's primary owner? [optional; banded categories]

Interview

Attributed interview prompts

Owner-facing prompts that corroborate the statistical channels and distinguish discrimination from neutral factors, gated so they never presume a verdict.

1Tell me about your firm and the kinds of work it does. How did you come to start or run it?
2Walk me through the most recent time your firm pursued a City or BPS opportunity, from how you first heard about it to how it ended.
3How does your firm usually find out about contracting opportunities, and how far in advance? Has timing ever affected whether you could compete?
4When you have won or nearly won work, what role, if any, did knowing someone, an introduction, or a recommendation play? When you have lost, do you know who won and why?
5Please describe, for a specific opportunity, how you found out about it, who if anyone connected you to the City or to a prime, and what relationships or sources of information you think made the most difference to the outcome.
6Describe your experience working with or being invited by prime contractors on public projects. How are those teaming relationships formed in your experience, and if you sought one and could not form it, what obstacles did you encounter?
7Have you had access to a mentor, an experienced firm, or a program that helped you become bid-ready? What difference did it make, or what was missing?
8When your firm did not have prior City or BPS experience, how did that affect your ability to compete, and what happened after your first award, if you had one?
9Tell me about your experience with bonding, insurance, or financing requirements on public work. Were any of these a deciding factor in whether you bid or could perform?
10Have you ever decided not to bid on City or BPS work you were qualified for? Walk me through that decision, and what would have had to be different for you to bid.
11Thinking about how contracts are sized or specified, have you encountered requirements that made it hard for a firm like yours to qualify or compete? Which ones, and on what solicitation?
12Has the speed of payment after you completed public work ever affected your firm's cash flow or your willingness to take on more public contracts?
13Think of a specific City or BPS opportunity awarded without a full open competition, such as a small purchase, a quote request, or a sole-source award, and a large sealed bid. Comparing across these purchase types, do you notice any difference in how your firm fares? Describe an example.
14In your private-sector or commercial work, have you experienced anything similar to or different from what you have described about public contracting? How do they compare?
15Were there qualifications, experience floors, bonding levels, or other requirements on City solicitations that your firm met, did not meet, or found difficult to meet, and how did that affect your decision to bid?
16Of everything you have described, which one or two factors have most shaped your firm's ability to win and perform public work? Were they about your firm's capacity, about access and relationships, or about how you were treated?
17Is there anything I did not ask about that would help the City understand the obstacles or advantages your firm has experienced in its contracting?

How this maps to the RFP

Advertised scope: MWBE utilization as contractors through departmental procurement practices. The firm-by-solicitation bid/quote panel, the City financial-system and BPS payment extracts, and the procurement-method discretion coding (technique 5) measure utilization department by department and method by method, so the departmental-practices language of the scope is answered with department-level evidence rather than a citywide average.
Advertised scope: MWBE utilization as subcontractors. Subcontractor payment and compliance records, prime payment affidavits, and the network layer (technique 7) run the subcontract-dollar versions of techniques 1 and 3; where subcontract tracking is thin, the coverage statement documents the gap itself as a finding the City can act on.
Advertised scope: workforce participation of minority and female workers on City construction projects and large City-assisted construction projects. This piece is workforce-side rather than vendor-side: the primary data are certified payroll records on covered projects, including the compliance reporting the Boston Residents Jobs Policy already generates, analyzed under the same significance-grid, with/without-controls, and no-presupposition discipline the vendor techniques use.
Advertised scope: departmental procurement practices as a study object. The AI-assisted procurement-rules review (technique 9) and the standing-eligibility-requirements extract (technique 4) read the City's own manuals, templates, thresholds, and criteria for internal conflicts and candidate disparate-impact provisions, with every flag adjudicated by a named investigator and every legal conclusion owned by counsel.
Causal adjudication across the scope: the Oaxaca-Blinder and Gelbach decompositions (techniques 1 and 2), the Heckman selection correction (technique 4), and the with/without-contested-controls grid (technique 8) are framed to determine whether a measured disparity traces to neutral factors or survives as an unexplained residual, without presuming a verdict in either direction.
City-specific versus marketplace-wide: the interview prompt comparing public and commercial experience and the five-county regional survey frame locate whether a barrier is specific to City procurement or a marketplace-wide condition, which changes what the City can and should fix.
Two-phase structure: the availability-survey frame, the data-request inventory, and the pre-registered methodology are built as first-phase foundations that carry into the second phase, which the City's announcement describes as extensive data analysis and the final complete Disparity Study report, without rework in between.
Defensibility: consented, attributed, retained interview transcripts, pre-committed minimum-cell-size and multiple-comparison rules, and the disclosed certified-only floor on the portal's current figures keep the qualitative and quantitative records fit for evidentiary use under Croson's strong-basis-in-evidence standard.
Mapping basis: the City released the RFP documents on its Supplier Portal on June 29, 2026, with registration required. This mapping tracks the City's publicly advertised scope and will be re-keyed to the RFP's own section structure once the gated documents are reviewed.

See the method behind the questions

The approach module sets out how we move from a measured gap to its cause. The methodology module places the engine inside the legal framework a Massachusetts record must satisfy.

Our approach Methodology & legal framework