diff --git a/Dockerfile b/Dockerfile index d9f1f5ce..84deb69f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,4 @@ -FROM golang:1.20 as build +FROM golang:1.21 as build WORKDIR /go/src/provider COPY go.mod go.sum ./ diff --git a/README.md b/README.md index 09f48819..95627622 100644 --- a/README.md +++ b/README.md @@ -15,10 +15,7 @@ A list of features include: * [`provider`](cmd/provider) CLI that can: * Run as a standalone provider daemon instance. * Generate and publish indexing advertisements directly from CAR files. - * Serve retrieval requests for the advertised content over GraphSync. - * list advertisements published by a provider instance - * verify ingestion of multihashes by an indexer node from CAR files, detached CARv2 indices or - from an index provider's advertisement chain. + * Serve retrieval requests for the advertised content over HTTP or HTTP over libp2p. * A Golang SDK to embed indexing integration into existing applications, which includes: * Programmatic advertisement for content via index provider [Engine](engine) with built-in chunking functionality @@ -30,6 +27,11 @@ A list of features include: * Index advertisement [`go-libipni/metadata`](https://pkg.go.dev/github.com/ipni/go-libipni/metadata) schema for retrieval over [graphsync](https://pkg.go.dev/github.com/ipni/go-libipni/metadata#GraphsyncFilecoinV1) and [bitswap](https://pkg.go.dev/github.com/ipni/go-libipni/metadata#Bitswap) +Use of the [ipni-cli](https://github.com/ipni/ipni-cli#ipni-cli) provides additional utility that is useful to check the functioning of an index-provider instance: + +* list advertisements published by a provider instance +* verify ingestion of multihashes by an indexer node from CAR files, detached CARv2 indices or from an index provider's advertisement chain. + ## Current status :construction: This implementation is under active development. @@ -40,14 +42,14 @@ The protocol implemented by this repository is the index provider portion of a l . The indexer node implementation can be found at [`storetheindex`](https://github.com/ipni/storetheindex) and [`go-libipni`](https://github.com/ipni/go-libipni). For more details on the ingestion protocol itself -see [Providing data to a network indexer](https://github.com/ipni/storetheindex/blob/main/doc/ingest.md) +see [IPNI Spec - Ingestion](https://github.com/ipni/specs/blob/main/IPNI.md#ingestion) . ## Install Prerequisite: -- [Go 1.19+](https://golang.org/doc/install) +- [Go 1.20+](https://golang.org/doc/install) To use the provider as a Go library, execute: @@ -56,18 +58,10 @@ go get github.com/ipni/index-provider ``` To install the latest `provider` CLI, run: - - ```shell go install github.com/ipni/index-provider/cmd/provider@latest ``` -Alternatively, download the executables directly from -the [releases](https://github.com/ipni/index-provider/releases). - ## Usage ### Running an standalone provider daemon @@ -97,7 +91,7 @@ to `http://localhost:3102`. You can then advertise content by importing/removing CAR files via the `provider` CLI, for example: ```shell -provider import car -l http://localhost:3102 -i +provider import car -i ``` Both CARv1 and CARv2 formats are supported. Index is regenerated on the fly if one is not present. @@ -121,17 +115,16 @@ Delegated Routing server is off by default. To enable it, add the following conf **Disclaimer: PUT /routing/v1 is currently not officially supported in Kubo. Please use it at your own risk. See [IPIP-378](https://github.com/ipfs/specs/pull/378) for the latest updates.** -Kubo supports HTTP delegated routing as of [v0.18.0](https://github.com/ipfs/kubo/releases/tag/v0.18.0). The following section contains configuration examples and a few tips to enable Kubo to advertise its CIDs to -IPNI systems like `cid.contact` using `index-provider`. Delegated Routing is still in the Experimental stage and configuration might change from version to version. +Kubo supports HTTP delegated routing as of [v0.18.0](https://github.com/ipfs/kubo/releases/tag/v0.18.0). The following section contains configuration examples and a few tips to enable Kubo to advertise its CIDs to IPNI systems like `cid.contact` using `index-provider`. Delegated Routing is still in the Experimental stage and configuration might change from version to version. This section serves as an inspiration for configuring your node to use IPNI, but for comprehensive information, refer to the [Kubo documentation](https://docs.ipfs.tech/install/command-line/). Here are some important points to consider: * `PUT /routing/v1` is currently not officially supported in Kubo. HTTP Delegated Routing supports only reads at the moment, not writes. Please use it at your own risk; * The `index-provider` delegated routing server should be running continuously as a "sidecar" to the Kubo node. While `index-provider` can be restarted safely, if it goes down, no new CIDs will flow from Kubo to IPNI. -* The latest version of Kubo (v0.18.+) with HTTP delegated routing support should be used as `index-provider` no longer supports Reframe. +* The latest version of Kubo with HTTP delegated routing support should be used since `index-provider` no longer supports Reframe. * Kubo advertises its data in snapshots, which means that all CIDs managed by Kubo are reprovided to the configured routers every 12/24 hours (configurable). This mechanism is similar to how the Distributed Hash Table (DHT) works. During the reproviding process, there may be significant communication between the involved processes. In between reprovides, Kubo also sends new individual CIDs to the configured routers. * Kubo requires `index-provider` only for publishing its CIDs to IPNI. Kubo can perform IPNI lookups natively without the need for a sidecar (refer to Kubo docs on `auto` routers). * `index-provider` must be publicly reachable. IPNI will try to establish connection into it to fetch Advertisement chains. If that can't be done CIDs will not appear in IPNI. -Ensure that your firewall is configured to allow incoming connections on the `ProviderServer` port specified in the `index-provider` configuration. +Ensure that your firewall is configured to allow incoming connections on the `ProviderServer` port specified in the `index-provider` configuration. Ensure that the index-provider is configured to advertise routable addresses in its announcements (where indexers get advertisements) and in its advertisements (where retrieval clients get content). To configure `index-provider` to expose the delegated routing server, use the following configuration: @@ -266,6 +259,10 @@ advertise content, see: * [`engine/example_test.go`](engine/example_test.go) +#### Configuration for Sublishing Advertisements + +See the [Publisher Configuratgion document](publisher-config.md) + #### Publishing advertisements with extended providers [Extended providers](https://github.com/ipni/storetheindex/blob/main/doc/ingest.md#extendedprovider) @@ -338,7 +335,7 @@ range of administrative operations. For example, the `provider` CLI can be used and advertise its content to the indexer nodes by executing: ```shell -provider import car -l http://localhost:3102 -i +provider import car -i ``` For usage description, execute `provider --help` @@ -389,8 +386,9 @@ advertisement. The cache expansion is logged in `INFO` level at `provider/engine ## Related Resources * [Indexer Ingestion IPLD Schema](https://github.com/ipni/go-libipni/blob/main/ingest/schema/schema.ipldsch) -* [Indexer Node Design](https://www.notion.so/protocollabs/Indexer-Node-Design-4fb94471b6be4352b6849dc9b9527825) -* [Providing data to a network indexer](https://github.com/ipni/storetheindex/blob/main/doc/ingest.md) +* [Indexer Ingestion JSON Schema](https://github.com/ipni/specs/blob/main/schemas/v1/openapi.yaml) +* [IPNI: InterPlanetary Network Indexer](https://github.com/ipni/specs/blob/main/IPNI.md#ipni-interplanetary-network-indexer) +* [`go-libipni` reference](https://pkg.go.dev/github.com/ipni/go-libipni) * [`storetheindex`](https://github.com/ipni/storetheindex): indexer node implementation * [`storetheindex` documentation](https://github.com/ipni/storetheindex/blob/main/doc/) * [`go-indexer-core`](https://github.com/filecoin-project/go-indexer-core): Core index key-value store diff --git a/cmd/provider/init.go b/cmd/provider/init.go index 9aad9976..3b97225a 100644 --- a/cmd/provider/init.go +++ b/cmd/provider/init.go @@ -2,6 +2,7 @@ package main import ( "errors" + "fmt" "io/fs" "os" @@ -16,7 +17,13 @@ var InitCmd = &cli.Command{ Action: initCommand, } -var initFlags = []cli.Flag{} +var initFlags = []cli.Flag{ + &cli.StringFlag{ + Name: "pubkind", + Usage: "Set publisher king in config. Must be one of 'http', 'libp2p', 'libp2phttp', 'dtsync'", + Value: "libp2p", + }, +} func initCommand(cctx *cli.Context) error { log.Info("Initializing provider config file") @@ -46,8 +53,15 @@ func initCommand(cctx *cli.Context) error { return err } - // Use values from flags to override defaults - // cfg.Identity = struct{}{} + pubkind := config.PublisherKind(cctx.String("pubkind")) + switch pubkind { + case "": + pubkind = config.Libp2pPublisherKind + case config.Libp2pPublisherKind, config.HttpPublisherKind, config.Libp2pHttpPublisherKind, config.DTSyncPublisherKind: + default: + return fmt.Errorf("unknown publisher kind: %s", pubkind) + } + cfg.Ingest.PublisherKind = pubkind return cfg.Save(configFile) } diff --git a/cmd/provider/internal/config/httppublisher.go b/cmd/provider/internal/config/httppublisher.go index 3fd931d6..db8a2e3e 100644 --- a/cmd/provider/internal/config/httppublisher.go +++ b/cmd/provider/internal/config/httppublisher.go @@ -7,11 +7,17 @@ import ( type HttpPublisher struct { // AnnounceMultiaddr is the address supplied in the announce message - // telling indexers the address to use to retrieve advertisements. If not - // specified, the ListenMultiaddr is used. + // telling indexers the address to use to retrieve advertisements. This + // configures the addresses to announce when using a Libp2pPublisher, + // HttpPublisher, or Libp2pHttpPublisher. + // + // If not specified, the ListenMultiaddr is used with HttpPubliser, the + // libp2p host address is used with Libp2pPublisher and both are used with + // Libp2pHttpPublisher. AnnounceMultiaddr string // ListenMultiaddr is the address of the interface to listen for HTTP - // requests for advertisements. + // requests for advertisements. Set this to "" to disable serving plain + // HTTP if only libp2phttp is wanted. ListenMultiaddr string } @@ -23,6 +29,9 @@ func NewHttpPublisher() HttpPublisher { } func (hs *HttpPublisher) ListenNetAddr() (string, error) { + if hs.ListenMultiaddr == "" { + return "", nil + } maddr, err := multiaddr.NewMultiaddr(hs.ListenMultiaddr) if err != nil { return "", err diff --git a/cmd/provider/internal/config/ingest.go b/cmd/provider/internal/config/ingest.go index 5645c8ed..b88d022c 100644 --- a/cmd/provider/internal/config/ingest.go +++ b/cmd/provider/internal/config/ingest.go @@ -11,8 +11,10 @@ const ( type PublisherKind string const ( - DTSyncPublisherKind PublisherKind = "dtsync" - HttpPublisherKind PublisherKind = "http" + DTSyncPublisherKind PublisherKind = "dtsync" + HttpPublisherKind PublisherKind = "http" + Libp2pPublisherKind PublisherKind = "libp2p" + Libp2pHttpPublisherKind PublisherKind = "libp2phttp" ) // Ingest configures settings related to the ingestion protocol. @@ -35,6 +37,9 @@ type Ingest struct { HttpPublisher HttpPublisher // PublisherKind specifies which dagsync.Publisher implementation to use. + // When set to "http", the publisher serves plain HTTP and libp2phttp. + // Libp2phttp is disabled by setting HttpPublisher.NoLibp2p to true, and + // plain HTTP is disabled by setting HttpPublisher.ListenMultiaddr to "". PublisherKind PublisherKind // SyncPolicy configures which indexers are allowed to sync advertisements @@ -49,7 +54,7 @@ func NewIngest() Ingest { LinkedChunkSize: defaultLinkedChunkSize, PubSubTopic: defaultPubSubTopic, HttpPublisher: NewHttpPublisher(), - PublisherKind: DTSyncPublisherKind, + PublisherKind: HttpPublisherKind, SyncPolicy: NewPolicy(), } } diff --git a/engine/engine.go b/engine/engine.go index f1fa39ca..00e3c31d 100644 --- a/engine/engine.go +++ b/engine/engine.go @@ -148,23 +148,50 @@ func (e *Engine) newPublisher() (dagsync.Publisher, error) { log.Info("Remote announcements disabled; all advertisements will only be stored locally.") return nil, nil case HttpPublisher: - var httpPub *ipnisync.Publisher - var err error - if e.pubHttpWithoutServer { - httpPub, err = ipnisync.NewPublisher(e.pubHttpListenAddr, e.lsys, e.key, - ipnisync.WithHeadTopic(e.pubTopicName), - ipnisync.WithHandlerPath(e.pubHttpHandlerPath), - ipnisync.WithServer(false)) - } else { - httpPub, err = ipnisync.NewPublisher(e.pubHttpListenAddr, e.lsys, e.key, - ipnisync.WithHeadTopic(e.pubTopicName), - ipnisync.WithServer(true)) - } + httpPub, err := ipnisync.NewPublisher(e.lsys, e.key, + ipnisync.WithHTTPListenAddrs(e.pubHttpListenAddr), + ipnisync.WithHeadTopic(e.pubTopicName), + ipnisync.WithHandlerPath(e.pubHttpHandlerPath), + ipnisync.WithStartServer(!e.pubHttpWithoutServer)) if err != nil { - return nil, fmt.Errorf("cannot create http publisher: %w", err) + return nil, fmt.Errorf("cannot create publisher: %w", err) + } + if len(e.pubHttpAnnounceAddrs) == 0 { + e.pubHttpAnnounceAddrs = append(e.pubHttpAnnounceAddrs, httpPub.Addrs()...) + log.Warn("HTTP publisher in use without address for announcements. Using publisher listen addresses, but external address may be needed.", "addrs", httpPub.Addrs()) } return httpPub, nil + case Libp2pPublisher: + libp2pPub, err := ipnisync.NewPublisher(e.lsys, e.key, + ipnisync.WithStreamHost(e.h), + ipnisync.WithHeadTopic(e.pubTopicName)) + if err != nil { + return nil, fmt.Errorf("cannot create publisher: %w", err) + } + if len(e.pubHttpAnnounceAddrs) == 0 { + e.pubHttpAnnounceAddrs = append(e.pubHttpAnnounceAddrs, libp2pPub.Addrs()...) + log.Warn("Libp2p publisher in use without address for announcements. Using libp2p host addresses, but external address may be needed.", "addrs", libp2pPub.Addrs()) + } + return libp2pPub, nil + case Libp2pHttpPublisher: + libp2phttpPub, err := ipnisync.NewPublisher(e.lsys, e.key, + ipnisync.WithStreamHost(e.h), + ipnisync.WithHTTPListenAddrs(e.pubHttpListenAddr), + ipnisync.WithHeadTopic(e.pubTopicName), + ipnisync.WithHandlerPath(e.pubHttpHandlerPath), + ipnisync.WithStartServer(!e.pubHttpWithoutServer)) + if err != nil { + return nil, fmt.Errorf("cannot create publisher: %w", err) + } + if len(e.pubHttpAnnounceAddrs) == 0 { + // No addresses explicitly specified, so use http and libp2p + // publisher listen addrs. + e.pubHttpAnnounceAddrs = append(e.pubHttpAnnounceAddrs, libp2phttpPub.Addrs()...) + log.Warn("Libp2p + HTTP publisher in use without address for announcements. Using HTTP listen and libp2p host addresses, but external addresses may be needed.", "addrs", libp2phttpPub.Addrs()) + } + return libp2phttpPub, nil case DataTransferPublisher: + log.Warn("Support ending for publishing IPNI data over data-transfer/graphsync, Disable this feature in configuration and test that indexing is working over libp2p.") if e.pubDT != nil { dtPub, err := dtsync.NewPublisherFromExisting(e.pubDT, e.h, e.pubTopicName, e.lsys, dtsync.WithAllowPeer(e.syncPolicy.Allowed)) if err != nil { @@ -218,7 +245,8 @@ func (e *Engine) createSenders() ([]announce.Sender, error) { func (e *Engine) announce(ctx context.Context, c cid.Cid) { var err error switch e.pubKind { - case HttpPublisher: + case HttpPublisher, Libp2pPublisher, Libp2pHttpPublisher: + // e.pubHttpAnnounceAddrs is always set in newPublisher. err = announce.Send(ctx, c, e.pubHttpAnnounceAddrs, e.senders...) case DataTransferPublisher: // TODO: It may be necessary to specify a set of external addresses to @@ -371,17 +399,14 @@ func (e *Engine) httpAnnounce(ctx context.Context, adCid cid.Cid, announceURLs [ case NoPublisher: log.Info("Remote announcements disabled") return nil + case HttpPublisher, Libp2pPublisher, Libp2pHttpPublisher: + // e.pubHttpAnnounceAddrs is always set in newPublisher. + msg.SetAddrs(e.pubHttpAnnounceAddrs) case DataTransferPublisher: // TODO: It may be necessary to specify a set of external addresses to // put into the announce message, instead of using the libp2p host's // addresses. msg.SetAddrs(e.h.Addrs()) - case HttpPublisher: - if len(e.pubHttpAnnounceAddrs) != 0 { - msg.SetAddrs(e.pubHttpAnnounceAddrs) - } else { - msg.SetAddrs(e.publisher.Addrs()) - } } // Create the http announce sender. diff --git a/engine/engine_test.go b/engine/engine_test.go index ef01ba53..7ab6018d 100644 --- a/engine/engine_test.go +++ b/engine/engine_test.go @@ -12,7 +12,6 @@ import ( "github.com/ipfs/go-cid" "github.com/ipfs/go-datastore" - dssync "github.com/ipfs/go-datastore/sync" leveldb "github.com/ipfs/go-ds-leveldb" "github.com/ipld/go-ipld-prime" cidlink "github.com/ipld/go-ipld-prime/linking/cid" @@ -21,8 +20,7 @@ import ( "github.com/ipld/go-ipld-prime/traversal/selector" selectorbuilder "github.com/ipld/go-ipld-prime/traversal/selector/builder" "github.com/ipni/go-libipni/announce/message" - "github.com/ipni/go-libipni/dagsync/dtsync" - "github.com/ipni/go-libipni/dagsync/p2p/protocol/head" + "github.com/ipni/go-libipni/dagsync/ipnisync" "github.com/ipni/go-libipni/ingest/schema" "github.com/ipni/go-libipni/metadata" "github.com/ipni/go-libipni/test" @@ -108,7 +106,7 @@ func TestEngine_PublishLocal(t *testing.T) { require.Equal(t, gotLatestAdCid, gotPublishedAdCid) } -func TestEngine_PublishWithDataTransferPublisher(t *testing.T) { +func TestEngine_PublishWithLibp2pHttpPublisher(t *testing.T) { ctx, cancel := context.WithTimeout(context.Background(), testTimeout) t.Cleanup(cancel) @@ -118,11 +116,17 @@ func TestEngine_PublishWithDataTransferPublisher(t *testing.T) { // Use test name as gossip topic name for uniqueness per test. topic := t.Name() - subHost, err := libp2p.New() + pubHost, err := libp2p.New() require.NoError(t, err) + t.Cleanup(func() { + pubHost.Close() + }) - pubHost, err := libp2p.New() + subHost, err := libp2p.New() require.NoError(t, err) + t.Cleanup(func() { + subHost.Close() + }) // DirectConnectTicks set to 5 here, because if the gossub for the subHost // does not start within n ticks then they never peer. So, if @@ -163,7 +167,7 @@ func TestEngine_PublishWithDataTransferPublisher(t *testing.T) { http.Error(w, err.Error(), 400) return } - // Since the message is coming from a dtsync publisher, the addresses + // Since the message is coming from a libp2p publisher, the addresses // should include a p2p ID. ais, err := peer.AddrInfosFromP2pAddrs(addrs...) if err != nil { @@ -186,7 +190,9 @@ func TestEngine_PublishWithDataTransferPublisher(t *testing.T) { } w.WriteHeader(http.StatusNoContent) })) - defer ts.Close() + t.Cleanup(func() { + ts.Close() + }) pubT, err := pubG.Join(topic) require.NoError(t, err) @@ -194,7 +200,7 @@ func TestEngine_PublishWithDataTransferPublisher(t *testing.T) { subject, err := engine.New( engine.WithDirectAnnounce(ts.URL), engine.WithHost(pubHost), - engine.WithPublisherKind(engine.DataTransferPublisher), + engine.WithPublisherKind(engine.Libp2pPublisher), engine.WithTopic(pubT), engine.WithTopicName(topic), engine.WithExtraGossipData(wantExtraGossipData), @@ -202,7 +208,9 @@ func TestEngine_PublishWithDataTransferPublisher(t *testing.T) { require.NoError(t, err) err = subject.Start(ctx) require.NoError(t, err) - defer subject.Shutdown() + t.Cleanup(func() { + subject.Shutdown() + }) subG, err := pubsub.NewGossipSub(ctx, subHost, pubsub.WithDirectConnectTicks(1), @@ -282,19 +290,20 @@ func TestEngine_PublishWithDataTransferPublisher(t *testing.T) { require.NoError(t, err) requireEqualDagsyncMessage(t, wantMessage, gotMessage) - gotRootCid, err := head.QueryRootCid(ctx, subHost, topic, pubHost.ID()) - require.NoError(t, err) - require.Equal(t, gotPublishedAdCid, gotRootCid) - - ds := dssync.MutexWrap(datastore.NewMapDatastore()) ls := cidlink.DefaultLinkSystem() store := &memstore.Store{} ls.SetReadStorage(store) ls.SetWriteStorage(store) - sync, err := dtsync.NewSync(subHost, ds, ls, nil, 0, 0) + sync := ipnisync.NewSync(ls, nil, ipnisync.ClientStreamHost(subHost)) + t.Cleanup(func() { + sync.Close() + }) + subjectInfo := peer.AddrInfo{ + ID: subject.Host().ID(), + } + syncer, err := sync.NewSyncer(subjectInfo) require.NoError(t, err) - syncer := sync.NewSyncer(subject.Host().ID(), topic) gotHead, err := syncer.GetHead(ctx) require.NoError(t, err) require.Equal(t, gotLatestAdCid, gotHead) @@ -453,10 +462,10 @@ func TestEngine_ProducesSingleChainForMultipleProviders(t *testing.T) { wantContextID1 := []byte("fish") wantContextID2 := []byte("bird") subject.RegisterMultihashLister(func(ctx context.Context, p peer.ID, contextID []byte) (provider.MultihashIterator, error) { - if string(contextID) == string(wantContextID1) && p == provider1id { return provider.SliceMultihashIterator(mhs1), nil - } else if string(contextID) == string(wantContextID2) && p == provider2id { + } + if string(contextID) == string(wantContextID2) && p == provider2id { return provider.SliceMultihashIterator(mhs2), nil } return nil, errors.New("not found") diff --git a/engine/options.go b/engine/options.go index 3852017d..f1e09e58 100644 --- a/engine/options.go +++ b/engine/options.go @@ -20,17 +20,27 @@ import ( ) const ( - // NoPublisher indicates that no announcements are made to the network and all advertisements - // are only stored locally. + // NoPublisher indicates that no announcements are made to the network and + // all advertisements are only stored locally. NoPublisher PublisherKind = "" - // DataTransferPublisher makes announcements over a gossipsub topic and exposes a - // datatransfer/graphsync server that allows peers in the network to sync advertisements. + // DataTransferPublisher exposes a datatransfer/graphsync server that + // allows peers in the network to sync advertisements. + // + // This option is being discontinued. Only provided as a fallback in case + // HttpPublisher is not working. DataTransferPublisher PublisherKind = "dtsync" - // HttpPublisher exposes a HTTP server that announces published advertisements and allows peers - // in the network to sync them over raw HTTP transport. + // HttpPublisher exposes an HTTP server that serves advertisements using an + // HTTP server. HttpPublisher PublisherKind = "http" + + // Libp2pPublisher serves advertisements using the engine's libp2p host. + Libp2pPublisher PublisherKind = "libp2p" + + // Libp2pHttpPublisher serves advertisements using both HTTP and libp2p + // servers. + Libp2pHttpPublisher PublisherKind = "libp2phttp" ) type ( @@ -223,8 +233,11 @@ func WithEntriesCacheCapacity(s int) Option { } } -// WithPublisherKind sets the kind of publisher used to announce new advertisements. -// If unset, advertisements are only stored locally and no announcements are made. +// WithPublisherKind sets the kind of publisher used to serve advertisements. +// If unset, advertisements are only stored locally and no announcements are +// made. This does not affect the methods used to send announcements of new +// advertisements, which are configured independent of this. +// // See: PublisherKind. func WithPublisherKind(k PublisherKind) Option { return func(o *options) error { @@ -233,10 +246,12 @@ func WithPublisherKind(k PublisherKind) Option { } } -// WithHttpPublisherListenAddr sets the net listen address for the HTTP publisher. -// If unset, the default net listen address of '0.0.0.0:3104' is used. +// WithHttpPublisherListenAddr sets the net listen address for the HTTP +// publisher. If unset, the default net listen address of '0.0.0.0:3104' is +// used. To disable plain HTTP and only serve libp2phttp, explicitly set this +// to "". // -// Note that this option only takes effect if the PublisherKind is set to HttpPublisher. +// This option only takes effect if the PublisherKind is set to HttpPublisher. // See: WithPublisherKind. func WithHttpPublisherListenAddr(addr string) Option { return func(o *options) error { @@ -280,7 +295,7 @@ func WithHttpPublisherAnnounceAddr(addr string) Option { } } -// WithTopicName sets toe topic name on which pubsub announcements are published. +// WithTopicName sets the topic name on which pubsub announcements are published. // To override the default pubsub configuration, use WithTopic. // // Note that this option only takes effect if the PublisherKind is set to DataTransferPublisher. diff --git a/go.mod b/go.mod index b02cf9aa..c64686c6 100644 --- a/go.mod +++ b/go.mod @@ -10,16 +10,16 @@ require ( github.com/ipfs/go-cid v0.4.1 github.com/ipfs/go-datastore v0.6.0 github.com/ipfs/go-ds-leveldb v0.5.0 - github.com/ipfs/go-graphsync v0.14.7 + github.com/ipfs/go-graphsync v0.14.8 github.com/ipfs/go-ipfs-blockstore v1.3.0 github.com/ipfs/go-log/v2 v2.5.1 - github.com/ipfs/kubo v0.21.0 + github.com/ipfs/kubo v0.22.0 github.com/ipld/go-car/v2 v2.11.0 github.com/ipld/go-codec-dagpb v1.6.0 github.com/ipld/go-ipld-adl-hamt v0.0.0-20220616142416-9004dbd839e0 github.com/ipld/go-ipld-prime v0.21.0 - github.com/ipni/go-libipni v0.4.0 - github.com/libp2p/go-libp2p v0.30.0 + github.com/ipni/go-libipni v0.5.0 + github.com/libp2p/go-libp2p v0.31.0 github.com/libp2p/go-libp2p-pubsub v0.9.3 github.com/mitchellh/go-homedir v1.1.0 github.com/multiformats/go-multiaddr v0.11.0 @@ -44,6 +44,8 @@ require ( github.com/google/pprof v0.0.0-20230821062121-407c9e7a662f // indirect github.com/gorilla/mux v1.8.0 // indirect github.com/gorilla/websocket v1.5.0 // indirect + github.com/hashicorp/go-cleanhttp v0.5.2 // indirect + github.com/hashicorp/go-retryablehttp v0.7.4 // indirect github.com/hashicorp/golang-lru/v2 v2.0.5 // indirect github.com/ipfs/go-bitfield v1.1.0 // indirect github.com/ipfs/go-ipfs-files v0.3.0 // indirect @@ -51,8 +53,8 @@ require ( github.com/multiformats/go-varint v0.0.7 // indirect github.com/onsi/ginkgo/v2 v2.11.0 // indirect github.com/quic-go/qpack v0.4.0 // indirect - github.com/quic-go/qtls-go1-20 v0.3.2 // indirect - github.com/quic-go/quic-go v0.38.0 // indirect + github.com/quic-go/qtls-go1-20 v0.3.3 // indirect + github.com/quic-go/quic-go v0.38.1 // indirect github.com/quic-go/webtransport-go v0.5.3 // indirect github.com/samber/lo v1.36.0 // indirect github.com/whyrusleeping/cbor-gen v0.0.0-20230418232409-daab9ece03a0 // indirect @@ -123,7 +125,6 @@ require ( github.com/libp2p/go-cidranger v1.1.0 // indirect github.com/libp2p/go-flow-metrics v0.1.0 // indirect github.com/libp2p/go-libp2p-asn-util v0.3.0 // indirect - github.com/libp2p/go-libp2p-gostream v0.6.0 // indirect github.com/libp2p/go-msgio v0.3.0 // indirect github.com/libp2p/go-nat v0.2.0 // indirect github.com/libp2p/go-netroute v0.2.1 // indirect diff --git a/go.sum b/go.sum index 91adcb27..87bf03f7 100644 --- a/go.sum +++ b/go.sum @@ -235,8 +235,14 @@ github.com/hannahhoward/go-pubsub v0.0.0-20200423002714-8d62886cc36e/go.mod h1:I github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I= github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ= +github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48= +github.com/hashicorp/go-hclog v0.9.2 h1:CG6TE5H9/JXsFWJCfoIVpKFIkFe6ysEuHirp4DxCsHI= +github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ= github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo= github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM= +github.com/hashicorp/go-retryablehttp v0.7.4 h1:ZQgVdpTdAL7WpMIwLzCfbalOcSUdkDZnpUv3/+BxzFA= +github.com/hashicorp/go-retryablehttp v0.7.4/go.mod h1:Jy/gPYAdjqffZ/yFGCFV2doI5wjtH1ewM9u8iYVjtX8= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc= @@ -279,8 +285,8 @@ github.com/ipfs/go-detect-race v0.0.1 h1:qX/xay2W3E4Q1U7d9lNs1sU9nvguX0a7319XbyQ github.com/ipfs/go-detect-race v0.0.1/go.mod h1:8BNT7shDZPo99Q74BpGMK+4D8Mn4j46UU0LZ723meps= github.com/ipfs/go-ds-leveldb v0.5.0 h1:s++MEBbD3ZKc9/8/njrn4flZLnCuY9I79v94gBUNumo= github.com/ipfs/go-ds-leveldb v0.5.0/go.mod h1:d3XG9RUDzQ6V4SHi8+Xgj9j1XuEk1z82lquxrVbml/Q= -github.com/ipfs/go-graphsync v0.14.7 h1:V90NORSdCpUHAgqQhApU/bmPSLOnwtSHM2v7R90k9Do= -github.com/ipfs/go-graphsync v0.14.7/go.mod h1:yT0AfjFgicOoWdAlUJ96tQ5AkuGI4r1taIQX/aHbBQo= +github.com/ipfs/go-graphsync v0.14.8 h1:NFFHquTNnwPi05tJhdpPj4CJMnqRBLxpZd+IfPRauf4= +github.com/ipfs/go-graphsync v0.14.8/go.mod h1:qyHjUvHey6EfKUDMQPwCuVkMOurRG3hcjRm+FaVP6bE= github.com/ipfs/go-hamt-ipld v0.1.1/go.mod h1:1EZCr2v0jlCnhpa+aZ0JZYp8Tt2w16+JJOAVz17YcDk= github.com/ipfs/go-ipfs-blockstore v1.3.0 h1:m2EXaWgwTzAfsmt5UdJ7Is6l4gJcaM/A12XwJyvYvMM= github.com/ipfs/go-ipfs-blockstore v1.3.0/go.mod h1:KgtZyc9fq+P2xJUiCAzbRdhhqJHvsw8u2Dlqy2MyRTE= @@ -333,8 +339,8 @@ github.com/ipfs/go-unixfs v0.4.5 h1:wj8JhxvV1G6CD7swACwSKYa+NgtdWC1RUit+gFnymDU= github.com/ipfs/go-unixfsnode v1.7.4 h1:iLvKyAVKUYOIAW2t4kDYqsT7VLGj31eXJE2aeqGfbwA= github.com/ipfs/go-unixfsnode v1.7.4/go.mod h1:PVfoyZkX1B34qzT3vJO4nsLUpRCyhnMuHBznRcXirlk= github.com/ipfs/go-verifcid v0.0.2 h1:XPnUv0XmdH+ZIhLGKg6U2vaPaRDXb9urMyNVCE7uvTs= -github.com/ipfs/kubo v0.21.0 h1:1+XKokeyatfI2Mri5iBn8Eplxf2F5ud0K5zLDg4tSSc= -github.com/ipfs/kubo v0.21.0/go.mod h1:LuK5ANXz/UhHp8jdt4mRwE16AHhbRqY68g/uyc5/VqU= +github.com/ipfs/kubo v0.22.0 h1:HxYkvtFqlF+qQMTxHW+xBhrIWykWm+WbEuQpw1d67mM= +github.com/ipfs/kubo v0.22.0/go.mod h1:Sn3hp55POjH9Ni0lEd/+smXpkZ0J1gKlm0Fx+E1LE60= github.com/ipld/go-car/v2 v2.11.0 h1:lkAPwbbTFqbdfawgm+bfmFc8PjGC7D12VcaLXPCLNfM= github.com/ipld/go-car/v2 v2.11.0/go.mod h1:aDszqev0zjtU8l96g4lwXHaU9bzArj56Y7eEN0q/xqA= github.com/ipld/go-codec-dagpb v1.6.0 h1:9nYazfyu9B1p3NAgfVdpRco3Fs2nFC72DqVsMj6rOcc= @@ -344,8 +350,8 @@ github.com/ipld/go-ipld-adl-hamt v0.0.0-20220616142416-9004dbd839e0/go.mod h1:od github.com/ipld/go-ipld-prime v0.21.0 h1:n4JmcpOlPDIxBcY037SVfpd1G+Sj1nKZah0m6QH9C2E= github.com/ipld/go-ipld-prime v0.21.0/go.mod h1:3RLqy//ERg/y5oShXXdx5YIp50cFGOanyMctpPjsvxQ= github.com/ipld/go-ipld-prime/storage/bsadapter v0.0.0-20230102063945-1a409dc236dd h1:gMlw/MhNr2Wtp5RwGdsW23cs+yCuj9k2ON7i9MiJlRo= -github.com/ipni/go-libipni v0.4.0 h1:zZ8OU2N0D4iYt0E9jInbDapeh9bG10b5sBgqvScflNw= -github.com/ipni/go-libipni v0.4.0/go.mod h1:LxH6NUmEVruK3FjV2bFWfXKougX7AIe7wVjvPqITrDI= +github.com/ipni/go-libipni v0.5.0 h1:UTVq4U9cReNz/rbeVeUvl+JTLexdfb7yjmBgouSteJ8= +github.com/ipni/go-libipni v0.5.0/go.mod h1:UnrhEqjVI2Z2HXlaieOBONJmtW557nZkYpB4IIsMD+s= github.com/ipsn/go-secp256k1 v0.0.0-20180726113642-9d62b9f0bc52 h1:QG4CGBqCeuBo6aZlGAamSkxWdgWfZGeE49eUOWJPA4c= github.com/ipsn/go-secp256k1 v0.0.0-20180726113642-9d62b9f0bc52/go.mod h1:fdg+/X9Gg4AsAIzWpEHwnqd+QY3b7lajxyjE1m4hkq4= github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus= @@ -393,12 +399,10 @@ github.com/libp2p/go-cidranger v1.1.0 h1:ewPN8EZ0dd1LSnrtuwd4709PXVcITVeuwbag38y github.com/libp2p/go-cidranger v1.1.0/go.mod h1:KWZTfSr+r9qEo9OkI9/SIEeAtw+NNoU0dXIXt15Okic= github.com/libp2p/go-flow-metrics v0.1.0 h1:0iPhMI8PskQwzh57jB9WxIuIOQ0r+15PChFGkx3Q3WM= github.com/libp2p/go-flow-metrics v0.1.0/go.mod h1:4Xi8MX8wj5aWNDAZttg6UPmc0ZrnFNsMtpsYUClFtro= -github.com/libp2p/go-libp2p v0.30.0 h1:9EZwFtJPFBcs/yJTnP90TpN1hgrT/EsFfM+OZuwV87U= -github.com/libp2p/go-libp2p v0.30.0/go.mod h1:nr2g5V7lfftwgiJ78/HrID+pwvayLyqKCEirT2Y3Byg= +github.com/libp2p/go-libp2p v0.31.0 h1:LFShhP8F6xthWiBBq3euxbKjZsoRajVEyBS9snfHxYg= +github.com/libp2p/go-libp2p v0.31.0/go.mod h1:W/FEK1c/t04PbRH3fA9i5oucu5YcgrG0JVoBWT1B7Eg= github.com/libp2p/go-libp2p-asn-util v0.3.0 h1:gMDcMyYiZKkocGXDQ5nsUQyquC9+H+iLEQHwOCZ7s8s= github.com/libp2p/go-libp2p-asn-util v0.3.0/go.mod h1:B1mcOrKUE35Xq/ASTmQ4tN3LNzVVaMNmq2NACuqyB9w= -github.com/libp2p/go-libp2p-gostream v0.6.0 h1:QfAiWeQRce6pqnYfmIVWJFXNdDyfiR/qkCnjyaZUPYU= -github.com/libp2p/go-libp2p-gostream v0.6.0/go.mod h1:Nywu0gYZwfj7Jc91PQvbGU8dIpqbQQkjWgDuOrFaRdA= github.com/libp2p/go-libp2p-pubsub v0.9.3 h1:ihcz9oIBMaCK9kcx+yHWm3mLAFBMAUsM4ux42aikDxo= github.com/libp2p/go-libp2p-pubsub v0.9.3/go.mod h1:RYA7aM9jIic5VV47WXu4GkcRxRhrdElWf8xtyli+Dzc= github.com/libp2p/go-libp2p-record v0.2.0 h1:oiNUOCWno2BFuxt3my4i1frNrt7PerzB3queqa1NkQ0= @@ -538,10 +542,10 @@ github.com/prometheus/procfs v0.11.1 h1:xRC8Iq1yyca5ypa9n1EZnWZkt7dwcoRPQwX/5gwa github.com/prometheus/procfs v0.11.1/go.mod h1:eesXgaPo1q7lBpVMoMy0ZOFTth9hBn4W/y0/p/ScXhY= github.com/quic-go/qpack v0.4.0 h1:Cr9BXA1sQS2SmDUWjSofMPNKmvF6IiIfDRmgU0w1ZCo= github.com/quic-go/qpack v0.4.0/go.mod h1:UZVnYIfi5GRk+zI9UMaCPsmZ2xKJP7XBUvVyT1Knj9A= -github.com/quic-go/qtls-go1-20 v0.3.2 h1:rRgN3WfnKbyik4dBV8A6girlJVxGand/d+jVKbQq5GI= -github.com/quic-go/qtls-go1-20 v0.3.2/go.mod h1:X9Nh97ZL80Z+bX/gUXMbipO6OxdiDi58b/fMC9mAL+k= -github.com/quic-go/quic-go v0.38.0 h1:T45lASr5q/TrVwt+jrVccmqHhPL2XuSyoCLVCpfOSLc= -github.com/quic-go/quic-go v0.38.0/go.mod h1:MPCuRq7KBK2hNcfKj/1iD1BGuN3eAYMeNxp3T42LRUg= +github.com/quic-go/qtls-go1-20 v0.3.3 h1:17/glZSLI9P9fDAeyCHBFSWSqJcwx1byhLwP5eUIDCM= +github.com/quic-go/qtls-go1-20 v0.3.3/go.mod h1:X9Nh97ZL80Z+bX/gUXMbipO6OxdiDi58b/fMC9mAL+k= +github.com/quic-go/quic-go v0.38.1 h1:M36YWA5dEhEeT+slOu/SwMEucbYd0YFidxG3KlGPZaE= +github.com/quic-go/quic-go v0.38.1/go.mod h1:ijnZM7JsFIkp4cRyjxJNIzdSfCLmUMg9wdyhGmg+SN4= github.com/quic-go/webtransport-go v0.5.3 h1:5XMlzemqB4qmOlgIus5zB45AcZ2kCgCy2EptUrfOPWU= github.com/quic-go/webtransport-go v0.5.3/go.mod h1:OhmmgJIzTTqXK5xvtuX0oBpLV2GkLWNDA+UeTGJXErU= github.com/raulk/go-watchdog v1.3.0 h1:oUmdlHxdkXRJlwfG0O9omj8ukerm8MEQavSiDTEtBsk= diff --git a/mirror/mirror.go b/mirror/mirror.go index 2939c32f..05f9b7ea 100644 --- a/mirror/mirror.go +++ b/mirror/mirror.go @@ -75,11 +75,13 @@ func New(ctx context.Context, source peer.AddrInfo, o ...Option) (*Mirror, error } } - // Create ipnisync publisher. - // - // TODO: When libp2phttp available, Listen on http over libp2p if - // httpListenAddr is not set. - m.pub, err = ipnisync.NewPublisher(m.httpListenAddr, m.ls, m.privKey, ipnisync.WithHeadTopic(m.topic), ipnisync.WithServer(true)) + // Create ipnisync publisher. If m.httpListenAddr has a value, then mirror + // will serve over HTTP on that address. If there is a libp2p Host, then + // the mirror will serve HTTP over libp2p using that Host. + m.pub, err = ipnisync.NewPublisher(m.ls, m.privKey, + ipnisync.WithHTTPListenAddrs(m.httpListenAddr), + ipnisync.WithStreamHost(m.h), + ipnisync.WithHeadTopic(m.topic)) if err != nil { return nil, err } diff --git a/mirror/mirror_env_test.go b/mirror/mirror_env_test.go index ce5997da..697cdd05 100644 --- a/mirror/mirror_env_test.go +++ b/mirror/mirror_env_test.go @@ -60,10 +60,14 @@ func (te *testEnv) startMirror(t *testing.T, ctx context.Context, opts ...mirror te.mirrorSyncLs.SetReadStorage(te.mirrorSyncLsStore) te.mirrorSyncLs.SetWriteStorage(te.mirrorSyncLsStore) - te.mirrorSync = ipnisync.NewSync(te.mirrorSyncLs, nil, nil) + te.mirrorSync = ipnisync.NewSync(te.mirrorSyncLs, nil) require.NoError(t, err) t.Cleanup(func() { te.mirrorSync.Close() }) - te.mirrorSyncer, err = te.mirrorSync.NewSyncer(te.mirrorHost.ID(), te.mirror.PublisherAddrs()) + pubInfo := peer.AddrInfo{ + ID: te.mirrorHost.ID(), + Addrs: te.mirror.PublisherAddrs(), + } + te.mirrorSyncer, err = te.mirrorSync.NewSyncer(pubInfo) require.NoError(t, err) } diff --git a/publisher-config.md b/publisher-config.md new file mode 100644 index 00000000..3cd694c8 --- /dev/null +++ b/publisher-config.md @@ -0,0 +1,90 @@ +# Publisher Configuration + +## Modes of Operation + +The content advertisement publisher is able to make content advertisements retrievable using five modes of operation: + +- HTTP served over libp2p (default) +- Plain HTTP using publisher's server +- Plain HTTP using external server +- HTTP served over libp2p and Plain HTTP together +- Data-transfer/graphsync (will be discontinued) + +Each of these modes of operation is enabled using a set of engine options, that can be specified via the engine API, or specified in a configuration file when using the command-line. + +### HTTP, Libp2p, Libp2p + HTTP, or DataTransfer Publisher + +The index-provider engine must be configured to use a libp2p publisher, HTTP publisher, HTTP + libp2p combined publisher, or a DataTransfer publisher. Specify what kind of publisher to use by calling the `WithPublisherKind` engine option and passing it one of the following values: +- `NoPublisher` indicates that no announcements are made to the network and all advertisements are only stored locally. +- `Libp2pPublisher` serves advertisements using the engine's libp2p host. +- `HttpPublisher` exposes an HTTP server that serves advertisements using an HTTP server. +- `Libp2pHttpPublisher` serves advertisements using both HTTP and libp2p servers. +- `DataTransferPublisher` exposes a data-transfer/graphsync server that allows peers in the network to sync advertisements. This option is being discontinued. Only provided as a fallback in case HttpPublisher and Libp2pHttpPublisher are not working. + +If `WithPublisherKind` is not provided a value, it defaults to `NoPublisher` and advertisements are only stored locally and no announcements are made. If configuring the command-line application, `WithPublisherKind` is configured by setting the `config.Ingest.PublisherKind` item in the configuration file to a value of "http", "libp2p", "libp2phttp, "dtsync", or "". + +For all publisher kinds, except the `DataTransfer` publisher, the `WithHttpPublisherAnnounceAddr` options sets the addresses that are announced to indexers, telling the indexers where to fetch advertisements from. If configuring the command-line application, `WithHttpPublisherAnnounceAddr is configured by specifying multiaddr strings in `config.Ingest.HttpPublisher.AnnounceMultiaddr`. + +In future index-provider releases support for the DataTransfer publisher kind will be removed. + +## Publishing vs Announcing + +When a new advertisement is made available by a publisher, the new advertisement's CID is generally announced to one or more indexers. An announcement message is constructed and is sent to indexers over libp2p gossip pubsub where it is received by any indexers subscribed to the topic on which the message was publisher. Or, the announcement message is send directly to specific indexers by HTTP. + +One, both, or none of these announcement methods may be used in conjunction with the publisher, without regard to what kind of publisher is configured. It is only necessary that the announcement message is created with one or more addresses that are usable by indexers to contact the publisher. The address(es) configured by `WithHttpPublisherAnnounceAddr` may depend on the type of publisher. Otherwise, the configuration of announcement senders is totally separate from the publisher. + +If using an HTTP server, then provide addresses that specify the "http" or "https" protocol, for example "/dns4/ipni.example.com/tcp/80/https" where is ipni.example.com is resolves to a public address and TLS termination is done for the endpoint. If using a Libp2p server then specify address(es) that your libp2p host can be contacted on without the "http". Specifying multiple addresses to announce is OK. If no addresses are specified, then the listening HTTP addresses are used if there is an HTTP publisher and the libp2p host addresses are used if there is a libp2p server. + +## Configure HTTP server `HttpPublisher` publisher kind + +The publisher can be configured to server HTTP using a plain (non-libp2p) HTTP server. This is configured by calling `WithPublisherKind(HttpPublisher)`. + +The publisher's HTTP listen address is configured using the engine option `WithHttpPublisherListenAddr`. If unset, the default net listen address of '0.0.0.0:3104' is used. + +The HTTP listen addresses only supports http, and https is not currently supported. A future release may support https, but this should generally not be necessary since the advertisement data transferred is signed and immutable, and if TLS is required then it can be terminated outside the index-provider. + +### Use HTTP server or Existing HTTP server + +The `HttpPublisher` kind can use either the publisher's HTTP server, or use an existing external HTTP server. The publisher's HTTP server is used by default. + +#### Publisher's HTTP server + +This is the default for the `HttpPublisher`. + +At least one HTTP listen address is required for the HTTP server to listen on. An HTTP listen address is supplied by the `WithHttpPublisherListenAddr` option. The corresponding config file item is `config.Ingest.HttpPublisher.ListenMultiaddr`. If left unspecified a default listen address is provided. This does not apply if using one of the http provider kinds. + +#### Existing HTTP server + +To avoid starting the publisher's HTTP server, call the `WithHttpPublisherWithoutServer` option passing it `true`. Use this when the publisher it to be used as an http handler with an existing server. There is no config file item for this as it is only available when an application, that supplies an HTTP server, is using the engine API. + +The engine function `GetPublisherHttpFunc()` returns the publisher's `ServeHTTP` function to use as a handler, with an external HTTP server, for handling advertisement and chain head requests. The publisher's HTTP handler expects the request URL path to start with the IPNI path `/ipni/v1/ad/`. If this is not present, then handler will not handle the request. + +### HTTP Request URL Path + +The publisher expects any HTTP request URL to contain the IPNI path `/ipni/v1/ad/`. Following the IPNI path is the requested resource, which is either the value `head` to request information about the advertisement chain head, an advertisement CID string to request an advertisement, or an advertisement multihash entry block CID to request multihash data. + +There is no need to specify the IPNI path anywhere, and the publisher always expects it to be in any incoming request URL that is to be handled by the publisher. Likewise, there is no need to specify this in announcement messages, or to indexers in any way. Indexers will implicitly add this path to any URL making requests to an IPNI publisher. + +The IPNI path in the URL may optionally be preceded by a user-defined path. This can be specified by the engine option `WithHttpPublisherHandlerPath`. It is useful when using an external server that serves the IPNI publisher under some existing path. If a handler path of "/foo/bar/" is specified, then the URL path for a chain head query will be "/foo/bar/ipni/v1/ad/head". The publisher will only handle requests that have a URL path starting with "/foo/bar/ipni/v1/ad/". When using an existing HTTP server, the publisher's handler function expect the full handler path and will return an error if it is asked to handle a request with a URL that does not contain the full path. + +## Configure HTTP over libp2p with `Libp2pPublisher' publisher kind + +This is the default mode of operation. It allows HTTP requests and responses to be communicated over libp2p. A libp2p stream host must be provided to use this mode of operation. + +When serving HTTP over libp2p, it is not necessary to specify a publisher HTTP listen address since the libp2p stream host is responsible for handling connections. Any publisher HTTP listen address is ignored. Announcement addresses, specified with `WithHttpPublisherAnnounceAddr`, do not need to include a `/http` component. + +### libp2p Stream Host + +The engine always has a libp2p stream host supplied to it with the `WithHost` option or created internally. This libp2p host is give to the engine's HTTP publisher. The private key associated with the libp2p host's Identity is given to the engine using the `WithPrivateKey` option, and is also given to the publisher. This allows advertisements to be signed. + +If using the command-line, the libp2p host Identity and private key are configured using the `config.Identiry.PeerID` and `config.Identity.PrivKey` configuration file items. The libp2p host's listen address is configured using the config file item `config.ProviderServer.ListenMultiaddr`. + +## Configure HTTP over libp2p and HTTP with `Libp2pHttpPublisher` publisher kind + +This is a combination of the `Libp2pPublisher` and `HttpPublisher` kinds, and the configurations for both apply. The transport used depends on which address a sync client connects to the publisher on. This configuration may be useful for using different protocols on different networks, or offering a choice of protocols to indexers. The sync client (indexer) will determine which protocol is used by sending an initial probe to the publisher address is it connecting to. + +## DataTransfer Publisher Configuration + +Publishing with data-transfer/graphsync is legacy configuration, and is only supported by the IPNI publisher for now as a fall-back in case there is some unforeseen problem with the new HTTP over libp2p. Publisher support for this will be dropped is future releases of the index-provider. + +A data-transfer publisher is configured by specifying the engine option `WithPublisherKind(DataTransferPublisher)`. When this option is specified, all of the `HttpPublisher` and `Libp2pPublisher` options are ignored.