Sparse observables with extended alphabets #74

jakelishman · 2024-06-21T17:33:07Z

Summary

This describes a new SparseObservable object that would be added to qiskit.quantum_info for use in the Estimator interface, to address two major problems:

the Pauli alphabet of SparsePauliOp makes it impossible to efficiently represent all operators that can be efficiently measured by hardware.
as device qubit counts scale up, the number of non-identity terms in observable terms does not necessarily do so, making the SparsePauliOp representation inefficient.

Scope

This design doc is just about what the new class will look like. A key component of that is making sure that it can be used in Estimator implementers, but the exact mechanisms of how the public Estimator interface evolves to accept this are out-of-scope of this document.

This describes a new `SparseObservable` object that would be added to `qiskit.quantum_info` for use in the `Estimator` interface, to address two major problems: * the Pauli alphabet of `SparsePauliOp` makes it impossible to efficiently represent all operators that can be efficiently measured by hardware. * as device qubit counts scale up, the number of non-identity terms in observable terms does not necessarily do so, making the `SparsePauliOp` reprsentation inefficient. Co-authored-by: Ian Hincks <ian.hincks@ibm.com>

The only con (lack of no-copy deserialisation / CSR-array construction) can be alleviated by supplying an unsafe uninitialised Numpy view object onto an object created with a `with_capacity` constructor from Rust space. This way, the buffers are owned by Rust space, and the deserialisation / raw data can be written directly into the buffers.

####-sparse-observable.md

blakejohnson · 2024-06-21T20:06:30Z

####-sparse-observable.md

+* More complex code is needed from Rust space to handle mathematical manipulations efficiently in the happy path of "no Python-space parametrisation".
+
+Jake: personally I'd avoid this unless we have a really strong compelling use-case for giving it first-class support.
+A user can always work around this simply by splitting the terms of their sum into different broadcast axes in the estimator pub, then calculating the sums themselves, which gives them far more freedom.


I haven't seen strong use cases requiring parameterized observables, so I am fine leaving this out.

There are some use-cases around parameterized observables but I agree that I would not classify them as "strong", especially since there are alternative means by which to achieve the same end-result.

That said, I wonder if it will be possible to have some Python-level utilities (or maybe alternative classes) that could make working with parameterized observables a little bit easier. I am not sure what exactly that would look like, but am wondering if @jakelishman could comment on the feasibility of something like this.

blakejohnson · 2024-06-21T20:08:03Z

Looks like a strong proposal.

####-sparse-observable.md

Cryoris · 2024-06-24T14:15:16Z

####-sparse-observable.md

+
+A core function of `Estimator` is to group terms that can be measured within the same execution.
+There are many ways to do this, and we do not want to tie the observable to one particular implementation.
+Qiskit will provide a function `SparseObservable.measurement_bases(*observables)` that takes an arbitrary number of `SparseObservable` instances and returns a set of the measurement bases needed to measure all terms.


This method would then just give all required measurement basis w/o any kind of optimization, which is left to other algorithms, right?

Yeah, the purpose is to separate "grouping of measurement bases" (which primitives / users will likely want to configure) from "what are the measurement bases?".

Cryoris · 2024-06-24T14:19:28Z

####-sparse-observable.md

+
+```python
+class SparseObservableView:
+    num_bits: int


Maybe a too small comment for an RFC, but should this be num_qubits for consistency with the other observables & circuit?

The names and whatnot can be bikeshed, but this discrepancy was purposeful in the RFC, at least: there's interest from the primitives in not having this tied to qubits because of how they (eventually) plan to work dynamic circuits into the mixture.

That said, Ian and I talked about that fairly early on in the process, and it may well be something that's better done just by primitives-side documentation, and keep num_qubits as the term here, as you say, for consistency.

Calling them qubits is fine with me. The idea with dynamic circuits would be to attach observables to slices of measurement registers rather than to the entire circuit, with measurement terms prescribing instructions to insert before those registers. Therefore multiple qubits in an SparseObservable could in principle correspond to a single qubit, at the quantum programmer's discretion.

Since we are not there yet, and even when we get there it would be a point of minor concern, I'm not fussed.

If such a workflow is possible, it would certainly make sense to rename it! Until then IMO it might be a bit clearer to keep the existing names 😄

We can call this qubits in the actual implementation, no trouble.

Cryoris · 2024-06-24T14:32:43Z

####-sparse-observable.md

+There are several others that might be possible too.
+These operations could all fairly easily be supported:
+
+* Evolution of `SparseObservable` by another: this is completely doable, just naturally has quadratic complexity.


It comes up sometimes to compute the expectation value of the Hamiltonian squared in algos & applications (even in the estimator for the variance maybe?), so the would seem like an important feature

Adding the expectation value computation for a given state is certainly easy enough, the trick is probably just finding some sensible representation of the statevector; we don't have the concept of a "sparse statevector" in Qiskit at the moment, and this operator is intended for numbers of qubits that dense statevectors can't represent.

Hmm but if we square a Hamiltonian we might have to measure in new bases, e.g. if H = X -> H^2 = I we'd have to measure in Z-basis for the expval of H^2 (ok maybe a bit basic this example... 😄). Or maybe I didn't correctly understand your proposal 🤔

Oh sorry, I missed the word "squared" in your answer.

If we have some representation of a state, I think the expectation value of the square of a Hamiltonian might shake out neater in an API if we cast that problem to "evolve the (sparse) state by the operator, then take the inner product with itself"?

Hm I'm not sure you can do that since you'd actually have to measure in different bases.. but we can also discuss this later 🙂

Co-authored-by: Julien Gacon <gaconju@gmail.com>

mrossinek

I like the proposal a lot 👍

Just left two comments on aspects that stood out to me as noteworthy.

mrossinek · 2024-06-25T13:32:39Z

####-sparse-observable.md

+* More complex code is needed from Rust space to handle mathematical manipulations efficiently in the happy path of "no Python-space parametrisation".
+
+Jake: personally I'd avoid this unless we have a really strong compelling use-case for giving it first-class support.
+A user can always work around this simply by splitting the terms of their sum into different broadcast axes in the estimator pub, then calculating the sums themselves, which gives them far more freedom.


There are some use-cases around parameterized observables but I agree that I would not classify them as "strong", especially since there are alternative means by which to achieve the same end-result.

That said, I wonder if it will be possible to have some Python-level utilities (or maybe alternative classes) that could make working with parameterized observables a little bit easier. I am not sure what exactly that would look like, but am wondering if @jakelishman could comment on the feasibility of something like this.

mrossinek · 2024-06-25T13:35:51Z

####-sparse-observable.md

+These operations could all fairly easily be supported:
+
+* Evolution of `SparseObservable` by another: this is completely doable, just naturally has quadratic complexity.
+* Tidy-up structural compaction of the operator: summing all terms that share the same abstract operator, removing zeros at some specified tolerance.


I think this would be somewhat crucial, especially if we consider construction of operators to happen in an "iterative" manner by means of a sequence of mathematical operations.
The existing SparsePauliOp already has a part of its existing .simplify routine implemented in Rust, but performance still leaves room for improvement. I wonder how much this new operator completely implemented in Rust would benefit purely from the language advantage? It might be difficult to predict though 🤔

At the same time, could the CSR-like structure allow an alternative (read: more performant) approach for an implementation of .simplify?

Just thinking out loud here...

For background, the high-level algorithm of SparsePauliOp.simplify is basically:

start with an empty hashmap of pauli: coeff

for each term of the sum, add the term into the hashmap, summing the coeffs if there's a match

create a new SparsePauliOp with each term in the hashmap, if the coeff is sufficiently far from 0

The way we do that has a fair amount of fiddling around the edges that's making it less efficient than it could be, but asymptotically, the runtime complexity is already the best I can think of.

That said, the complexity still scales as $\mathcal O(\text{qubits}\times\text{terms})$. The CSR-like form would be effectively the same, and so have the same asymptotic complexity. As a rule of thumb, though, it would be faster when there's less memory used (i.e. individually sparse terms), simply because there's no real mathematical structure / algorithmic trickery going on here, and the limit is mostly the iteration speed over all stored data.

For operator construction, one thing (mostly unrelated to this comment) to highlight: the way I've written SparseObservable here, it's growable in place, which means for some operations (+ being a notable one), we can do it in-place and growing, which is very nice for iteration. evolve is harder to do in-place, though.

pedrorrivero

I really like this (much needed) RFC, awesome work @jakelishman @ihincks !

pedrorrivero · 2024-06-30T15:06:32Z

####-sparse-observable.md

+`SparseObservable` will support some set of mathematical operations.
+At a minimum, the following will be supported:
+
+* addition of two `SparseObservable`s
+* tensor product of two `SparseObservable`s
+* evolution of one `SparseObservable` by a Pauli string ($A' = P A P^\dagger$ for `SparseObservable` $A$ and Pauli $P$)
+* multiplication by complex scalars
+* structural equality of two `SparseObservable`s (structural not mathematical; it's highly inefficient to detect equality if the abstract terms form an over-complete spanning set).


Would it make sense to add the following?

Evolution over (closed group) basis rotations $H$, $S$ and $S^\dagger$

Qubit permutations (to easily/efficiently account for layout and routing during circuit transpilation)

Sure, apply_layout is easy enough to do. Longer term, the story around compilation of observables will probably change a bit in Qiskit to be more streamlined, but for now we'll keep consistency with SparsePauliOp on that.

Evolution is mathematically sound for many operators, the tricks are mostly around finding nice representations of more complex objects for the API surface.

pedrorrivero · 2024-06-30T15:29:13Z

####-sparse-observable.md

+Qiskit will provide a function `SparseObservable.measurement_bases(*observables)` that takes an arbitrary number of `SparseObservable` instances and returns a set of the measurement bases needed to measure all terms.
+A measurement basis is a Pauli string.
+
+Since the number of measurement bases will be (non-strictly) smaller than the total number of terms across all observables, and because only 2 bits of information per qubit is necessary to define the basis, there is not expected to be immediate memory concerns with a representation of this.
+Qiskit already has `PauliList` that can serve this purpose; it _could_ be bit-packed to use 8x less memory, but this can be done as a follow-up optimisation if it becomes a bottleneck.
+From this point, we can continue to use `PauliList.group_qubitwise_commuting`, or any other future grouping function.


I really like the design, however I don't fully see (from quick reading) how we avoid falling back into memory bottlenecks if we end up producing Pauli/PauliLists as the measurement bases.

Is it just from the fact that the observables will be transmitted in the new format and measurement bases only generated server side? What am I missing? 🤔

The only part of SparsePauliOp that causes serious immediate memory problems is that some things that are efficient to measure are not efficient to represent. For example, the all-zeros projector state takes linear complexity/space to measure (it's an all-Z basis) but needs $2^n$ terms in SparsePauliOp to represent. Since, for the measurement, you just need to know which basis to take your counts in, the information about "what to measure" (including mitigation) can be stored efficiently by PauliList, and you reconstruct the requested observables later from the original SparseObservable.

Bit-packing PauliList can reduce its memory usage by a factor of about 8, but that's just a scaling factor. We already know we must be able to do operations that are linear in the number of qubits (or how would we twirl?), and PauliList can represent all the measurement bases we realistically care about for error-mitigation purposes for the next while.

All that said, how a primitives implementation actually manages its error mitigation is entirely up to it, and this isn't fixed by any public interface. So any primitive can use whatever they like in the backend to do this task; the point about PauliList is mostly just showing that we don't immediately need any new object.

Thanks @jakelishman! I was thinking more on all the design decisions to avoid explicit identities, but I see your point that this is not the biggest concern 🙂

jakelishman and others added 6 commits June 20, 2024 12:17

Add specifics on construction

016dc9c

Leave Estimator integration out of scope

0d92659

Expand discussion of measurement bases

965eef2

Fix type of view objects

a6ff3ee

blakejohnson reviewed Jun 21, 2024

View reviewed changes

####-sparse-observable.md Outdated Show resolved Hide resolved

blakejohnson reviewed Jun 21, 2024

View reviewed changes

####-sparse-observable.md Outdated Show resolved Hide resolved

blakejohnson reviewed Jun 21, 2024

View reviewed changes

####-sparse-observable.md Outdated Show resolved Hide resolved

blakejohnson reviewed Jun 21, 2024

View reviewed changes

Fix typos

52eb1c1

Cryoris reviewed Jun 24, 2024

View reviewed changes

Fix markdown typo

c002014

Co-authored-by: Julien Gacon <gaconju@gmail.com>

mrossinek reviewed Jun 25, 2024

View reviewed changes

jakelishman mentioned this pull request Jun 26, 2024

WIP: Add base representation of SparseObservable Qiskit/qiskit#12671

Draft

pedrorrivero reviewed Jun 30, 2024

View reviewed changes

ihincks approved these changes Jul 4, 2024

View reviewed changes

blakejohnson approved these changes Jul 4, 2024

View reviewed changes

pedrorrivero approved these changes Jul 4, 2024

View reviewed changes

mrossinek approved these changes Jul 4, 2024

View reviewed changes

Cryoris approved these changes Jul 4, 2024

View reviewed changes

1ucian0 merged commit b04d53b into Qiskit:master Jul 4, 2024
1 check passed

jakelishman deleted the sparse-observable branch July 4, 2024 15:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sparse observables with extended alphabets #74

Sparse observables with extended alphabets #74

jakelishman commented Jun 21, 2024

blakejohnson Jun 21, 2024

mrossinek Jun 25, 2024

blakejohnson commented Jun 21, 2024

Cryoris Jun 24, 2024

jakelishman Jun 24, 2024

Cryoris Jun 24, 2024

jakelishman Jun 24, 2024

ihincks Jun 25, 2024

Cryoris Jun 27, 2024

jakelishman Jul 4, 2024

Cryoris Jun 24, 2024

jakelishman Jun 24, 2024

Cryoris Jun 27, 2024

jakelishman Jun 27, 2024

Cryoris Jul 4, 2024

mrossinek left a comment

mrossinek Jun 25, 2024

mrossinek Jun 25, 2024

jakelishman Jun 25, 2024

pedrorrivero left a comment

pedrorrivero Jun 30, 2024

jakelishman Jun 30, 2024

pedrorrivero Jun 30, 2024

jakelishman Jun 30, 2024

pedrorrivero Jun 30, 2024

Sparse observables with extended alphabets #74

Sparse observables with extended alphabets #74

Conversation

jakelishman commented Jun 21, 2024

Summary

Scope

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blakejohnson commented Jun 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrossinek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pedrorrivero left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment