How it Works
TL;DR
polarobi builds the constraint matrix with polars, hands the Arrow buffers to Gurobi's C API in one call per family without copying them, and then registers the new elements with gurobipy, so the model stays a regular gurobipy model.
Where the Time Goes in gurobipy
Building a model with gurobipy from tabular data typically creates several Python objects for every row: a tuple from iter_rows(), a lookup in a tupledict, a LinExpr or a term in one, a quicksum over them and a name string from an f-string. For a few thousand rows that is irrelevant. For millions it dominates the time to build the model, often by far.
polarobi keeps the data in polars until it reaches Gurobi. Per row, it creates exactly one Python object: the gp.Var or gp.Constr handle that gurobipy needs to know about the element.
From Frames to Gurobi
Variables
add_vars resolves the bounds, objective coefficients, type and names into columns, then makes a single GRBaddvars call for the whole frame. Numeric columns are passed as they are in memory.
Constraints
add_constrs turns the terms and rows into the compressed sparse row (CSR) format Gurobi's bulk API takes:
- The terms are grouped by key with
group_by().agg(), which gives one list of variable indices and one of coefficients per key, keys in order of first appearance and terms in their order. - Rows, if given, are joined to the lists of their key, so a repeated key gets its lists again and a key without terms is dropped or gets empty lists.
All of that is polars, and the lists already are CSR: Arrow stores a list column as one flat array of items plus the offset of each row into it. The native module takes the offsets as the row starts and the items as the variable indices and coefficients, without reshaping anything, and passes them with one array each of senses and right-hand sides to a single GRBXaddconstrs call.
Before the call, the native module checks every variable index against the model's variable count, since Gurobi rejects an unknown variable without saying which. Gurobi also rejects a variable that appears twice in a row, while gurobipy sums such repeats first; the native module checks for repeats in one pass and, only if there are any, merges them into new arrays, summing coefficients left to right in the order of the terms frame.
Zero Copy
Numeric columns cross from polars to the native module through the Arrow PyCapsule interface. The native module reads polars' own buffers, at the same addresses polars uses, and passes pointers into them to Gurobi:
- Single-chunk columns are not copied at all, whatever their size.
- Multi-chunk columns are concatenated once, as Gurobi needs one contiguous array.
UInt32indices, the dtype ofidxcolumns, are reinterpreted as Cintin place once they are known to fit.- Names are copied, as C needs NUL-terminated strings.
Gurobi copies whatever it receives into its own storage, so the data is copied exactly once on its way in, by Gurobi.
Reading Attributes
get allocates one buffer, has Gurobi write the attribute values for all requested elements into it, and hands it to polars as an Arrow array. polars takes the buffer over without copying it.
Registering Handles with gurobipy
gurobipy keeps its own list of the model's variables and constraints next to Gurobi's model, and many of its functions, such as getVars() and getVarByName(), read from that list. Elements added through the C API alone are in the model but invisible to those functions.
So after each call, polarobi creates the gurobipy handles for the new elements and registers them exactly as gurobipy registers its own. The native module creates them, which takes about half the time of a Python loop over gurobipy's handle constructor; the constructor itself is gurobipy's and costs what it costs in Model.addVars. The handles are the reason a variable family still takes time proportional to its size, and the price of full compatibility with gurobipy.
This relies on private parts of gurobipy:
| Attribute | Use |
|---|---|
Model._cmodel | The GRBmodel* of the model |
Model._internal_vars, Model._livevars | gurobipy's variable list and count |
Model._constrs, Model._liveconstrs | gurobipy's constraint list and count |
Model._has_pending_deletions() | Whether removals await an update |
gp.Var(cmodel, index), gp.Constr(cmodel, index) | Handle constructors |
That is why polarobi supports a single gurobipy minor version per release, and why its test suite checks every model it builds against gurobipy itself (see Equivalence with gurobipy).
Binding to Gurobi
gurobipy ships the Gurobi library and loads it on import. polarobi never loads a second copy: two copies in one process would hold separate state, and the GRBmodel* owned by gurobipy would be foreign to the second one. On import, the native module binds to the library already loaded into the process, and checks that it reports the same version as gurobipy.
Performance
A reproducible benchmark builds the same allocation model with polarobi, in two common gurobipy styles and with gurobipy-pandas, on a Google Cloud VM. At ten million rows polarobi took 8.6 s, against 90.3 s for the faster gurobipy build and 150.7 s for gurobipy-pandas, with identical fingerprints; see Benchmarks for the figures, the machine, the model and how to run it yourself.