Additional parameters for findroute* API calls (#1969)

Add options to ignore specific channels or nodes for
findRoute* APIs, and an option to specify a flat maximum
fee.

With these new parameters, it's now possible to do circular
rebalancing of your channels.

Co-authored-by: Roman Taranchenko <romantaranchenko@Romans-MacBook-Pro.local>
Co-authored-by: t-bast <bastuc@hotmail.fr>

docs/CircularRebalancing.md

@@ -0,0 +1,63 @@
+## How to perform circular rebalancing
+
+Circular rebalancing is a popular tool for managing liquidity between channels. This document describes an approach to
+rebalancing using Eclair.
+
+In this example we assume that there are 4 participants in the Lighting Network: `Alice`, `Bob`, `Carol`, and `Diana`. 
+
+```
+            1x1x1
+    Alice --------> Bob
+      ^              |
+4x4x4 |              | 2x2x2
+      |              v
+    Diana <------- Carol
+            3x3x3  
+```
+
+`Alice` has two channels `1x1x1` with outbound liquidity of 5M stats, and `4x4x4` with inbound liquidity of 5M sats.
+Now `Alice` wants to send some sats from channel `1x1x1` to channel `4x4x4` to be able to receive and forward payments 
+via both channels.
+
+First, `Alice` creates an invoice for the desired amount (eg. 1M sats):
+
+```shell
+eclair-cli createinvoice --description='circular rebalancing from 1x1x1 ro 4x4x4' \
+                         --amountMsat=1000000000
+```
+
+Eclair cannot send payments to self using `payinvoice` CLI command. Fortunately, `Alice` can use `sendtoroute` CLI 
+command to do so. 
+
+However, in this case `Alice` should provide a valid route from `1x1x1` to `4x4x4`. It's pretty straightforward to 
+build a route for our example network: `1x1x1` `->` `2x2x2` `->` `3x3x3` `->` `4x4x4`. `Alice` specifies the route using 
+`--shortChannelIds` parameter as a comma separated list of short channel IDs.  
+
+```shell
+eclair-cli sendtoroute --shortChannelIds=1x1x1,2x2x2,3x3x3,4x4x4 \
+                       --amountMsat=1000000000 \
+                       --invoice=<serialized invoice>
+```
+
+This command will send 1M sats from channel `1x1x1` to channel `4x4x4`.
+
+In real life its not always easy to find the most economically viable route manually, but Eclair is here to help.
+Similarly to `payinvoice`, `findroute` CLI command cannot find routes to self. But `Alice` can use a little trick with 
+`findroutebetweennodes`, which allows finding routes between arbitrary nodes. 
+
+To rebalance channels `1x1x1` and `4x4x4`, `Alice` wants to find a route from `Bob` (as a source node) to `Diana` (as a 
+target node). In our example there's at least one route from `Bob` to `Diana` via `Alice`, and in real life there can be 
+many more such routes, bacause `Alice` can have way more than two channels, so `Alice`'s node should be excluded from 
+path-finding using `--ignoreNodeIds` parameter:
+
+```shell
+eclair-cli findroutebetweennodes --sourceNodeId=<Bob`s node ID> \
+                                 --targetNodeId=<Diana`s node ID> \
+                                 --ignoreNodeIds=<Alice`s node ID`> \
+                                 --format=shortChannelId
+```
+
+Then `Alice` simply appends the outgoing channel ID to the beginning of the found route and the incoming channel ID to 
+the end: `1x1x1,<found route>,4x4x4`. In our example the found route is `2x2x2,3x3x3`, so the full route will be 
+`1x1x1,2x2x2,3x3x3,4x4x4`. `Alice` can use this route with `sendtoroute` command to perform rebalancing. 
+

docs/release-notes/eclair-vnext.md

@@ -51,6 +51,10 @@ Examples:
 This release contains many other API updates:
 
 - `deleteinvoice` allows you to remove unpaid invoices (#1984)
+- `findroute`, `findroutetonode` and `findroutebetweennodes` supports new output format `full` (#1969)
+- `findroute`, `findroutetonode` and `findroutebetweennodes` now accept `--ignoreNodeIds` to specify nodes you want to be ignored in path-finding (#1969)
+- `findroute`, `findroutetonode` and `findroutebetweennodes` now accept `--ignoreShortChannelIds` to specify channels you want to be ignored in path-finding (#1969)
+- `findroute`, `findroutetonode` and `findroutebetweennodes` now accept `--maxFeeMsat` to specify an upper bound of fees (#1969)
 
 Have a look at our [API documentation](https://acinq.github.io/eclair) for more details.
 

eclair-core/src/main/scala/fr/acinq/eclair/Eclair.scala

@@ -48,6 +48,7 @@ import scodec.bits.ByteVector
 
 import java.nio.charset.StandardCharsets
 import java.util.UUID
+import scala.collection.immutable.SortedMap
 import scala.concurrent.duration._
 import scala.concurrent.{ExecutionContext, Future, Promise}
 import scala.reflect.ClassTag
@@ -107,9 +108,9 @@ trait Eclair {
 
   def sendOnChain(address: String, amount: Satoshi, confirmationTarget: Long): Future[ByteVector32]
 
-  def findRoute(targetNodeId: PublicKey, amount: MilliSatoshi, pathFindingExperimentName_opt: Option[String], assistedRoutes: Seq[Seq[PaymentRequest.ExtraHop]] = Seq.empty, includeLocalChannelCost: Boolean = false)(implicit timeout: Timeout): Future[RouteResponse]
+  def findRoute(targetNodeId: PublicKey, amount: MilliSatoshi, pathFindingExperimentName_opt: Option[String], assistedRoutes: Seq[Seq[PaymentRequest.ExtraHop]] = Seq.empty, includeLocalChannelCost: Boolean = false, ignoreNodeIds: Seq[PublicKey] = Seq.empty, ignoreShortChannelIds: Seq[ShortChannelId] = Seq.empty, maxFee_opt: Option[MilliSatoshi] = None)(implicit timeout: Timeout): Future[RouteResponse]
 
-  def findRouteBetween(sourceNodeId: PublicKey, targetNodeId: PublicKey, amount: MilliSatoshi, pathFindingExperimentName_opt: Option[String], assistedRoutes: Seq[Seq[PaymentRequest.ExtraHop]] = Seq.empty, includeLocalChannelCost: Boolean = false)(implicit timeout: Timeout): Future[RouteResponse]
+  def findRouteBetween(sourceNodeId: PublicKey, targetNodeId: PublicKey, amount: MilliSatoshi, pathFindingExperimentName_opt: Option[String], assistedRoutes: Seq[Seq[PaymentRequest.ExtraHop]] = Seq.empty, includeLocalChannelCost: Boolean = false, ignoreNodeIds: Seq[PublicKey] = Seq.empty, ignoreShortChannelIds: Seq[ShortChannelId] = Seq.empty, maxFee_opt: Option[MilliSatoshi] = None)(implicit timeout: Timeout): Future[RouteResponse]
 
   def sendToRoute(amount: MilliSatoshi, recipientAmount_opt: Option[MilliSatoshi], externalId_opt: Option[String], parentId_opt: Option[UUID], invoice: PaymentRequest, finalCltvExpiryDelta: CltvExpiryDelta, route: PredefinedRoute, trampolineSecret_opt: Option[ByteVector32] = None, trampolineFees_opt: Option[MilliSatoshi] = None, trampolineExpiryDelta_opt: Option[CltvExpiryDelta] = None, trampolineNodes_opt: Seq[PublicKey] = Nil)(implicit timeout: Timeout): Future[SendPaymentToRouteResponse]
 
@@ -266,8 +267,8 @@ class EclairImpl(appKit: Kit) extends Eclair with Logging {
     }
   }
 
-  override def findRoute(targetNodeId: PublicKey, amount: MilliSatoshi, pathFindingExperimentName_opt: Option[String], assistedRoutes: Seq[Seq[PaymentRequest.ExtraHop]] = Seq.empty, includeLocalChannelCost: Boolean = false)(implicit timeout: Timeout): Future[RouteResponse] =
-    findRouteBetween(appKit.nodeParams.nodeId, targetNodeId, amount, pathFindingExperimentName_opt, assistedRoutes, includeLocalChannelCost)
+  override def findRoute(targetNodeId: PublicKey, amount: MilliSatoshi, pathFindingExperimentName_opt: Option[String], assistedRoutes: Seq[Seq[PaymentRequest.ExtraHop]] = Seq.empty, includeLocalChannelCost: Boolean = false, ignoreNodeIds: Seq[PublicKey] = Seq.empty, ignoreShortChannelIds: Seq[ShortChannelId] = Seq.empty, maxFee_opt: Option[MilliSatoshi] = None)(implicit timeout: Timeout): Future[RouteResponse] =
+    findRouteBetween(appKit.nodeParams.nodeId, targetNodeId, amount, pathFindingExperimentName_opt, assistedRoutes, includeLocalChannelCost, ignoreNodeIds, ignoreShortChannelIds, maxFee_opt)
 
   private def getRouteParams(pathFindingExperimentName_opt: Option[String]): Either[IllegalArgumentException, RouteParams] = {
     pathFindingExperimentName_opt match {
@@ -279,11 +280,15 @@ class EclairImpl(appKit: Kit) extends Eclair with Logging {
     }
   }
 
-  override def findRouteBetween(sourceNodeId: PublicKey, targetNodeId: PublicKey, amount: MilliSatoshi, pathFindingExperimentName_opt: Option[String], assistedRoutes: Seq[Seq[PaymentRequest.ExtraHop]] = Seq.empty, includeLocalChannelCost: Boolean = false)(implicit timeout: Timeout): Future[RouteResponse] = {
+  override def findRouteBetween(sourceNodeId: PublicKey, targetNodeId: PublicKey, amount: MilliSatoshi, pathFindingExperimentName_opt: Option[String], assistedRoutes: Seq[Seq[PaymentRequest.ExtraHop]] = Seq.empty, includeLocalChannelCost: Boolean = false, ignoreNodeIds: Seq[PublicKey] = Seq.empty, ignoreShortChannelIds: Seq[ShortChannelId] = Seq.empty, maxFee_opt: Option[MilliSatoshi] = None)(implicit timeout: Timeout): Future[RouteResponse] = {
     getRouteParams(pathFindingExperimentName_opt) match {
       case Right(routeParams) =>
-        val maxFee = routeParams.getMaxFee(amount)
-        (appKit.router ? RouteRequest(sourceNodeId, targetNodeId, amount, maxFee, assistedRoutes, routeParams = routeParams.copy(includeLocalChannelCost = includeLocalChannelCost))).mapTo[RouteResponse]
+        val maxFee = maxFee_opt.getOrElse(routeParams.getMaxFee(amount))
+        for {
+          ignoredChannels <- getChannelDescs(ignoreShortChannelIds.toSet)
+          ignore = Ignore(ignoreNodeIds.toSet, ignoredChannels)
+          response <- (appKit.router ? RouteRequest(sourceNodeId, targetNodeId, amount, maxFee, assistedRoutes, ignore = ignore, routeParams = routeParams.copy(includeLocalChannelCost = includeLocalChannelCost))).mapTo[RouteResponse]
+        } yield response
       case Left(t) => Future.failed(t)
     }
   }
@@ -475,4 +480,19 @@ class EclairImpl(appKit: Kit) extends Eclair with Logging {
     val pubKeyFromSignature = Crypto.recoverPublicKey(signature, signedBytes, recoveryId)
     VerifiedMessage(valid = true, pubKeyFromSignature)
   }
+
+  private def getChannelDescs(shortChannelIds: Set[ShortChannelId])(implicit timeout: Timeout): Future[Set[ChannelDesc]] = {
+    if (shortChannelIds.isEmpty){
+      Future.successful(Set.empty)
+    } else {
+      for {
+        channelsMap <- (appKit.router ? GetChannelsMap).mapTo[SortedMap[ShortChannelId, PublicChannel]]
+      } yield {
+        shortChannelIds.map { id =>
+          val c = channelsMap.getOrElse(id, throw new IllegalArgumentException(s"unknown channel: $id"))
+          ChannelDesc(c.ann.shortChannelId, c.ann.nodeId1, c.ann.nodeId2)
+        }
+      }
+    }
+  }
 }

eclair-core/src/main/scala/fr/acinq/eclair/json/JsonSerializers.scala

@@ -27,7 +27,7 @@ import fr.acinq.eclair.db.FailureType.FailureType
 import fr.acinq.eclair.db.{IncomingPaymentStatus, OutgoingPaymentStatus}
 import fr.acinq.eclair.payment.PaymentFailure.PaymentFailedSummary
 import fr.acinq.eclair.payment._
-import fr.acinq.eclair.router.Router.RouteResponse
+import fr.acinq.eclair.router.Router.{ChannelHop, Route}
 import fr.acinq.eclair.transactions.DirectedHtlc
 import fr.acinq.eclair.transactions.Transactions._
 import fr.acinq.eclair.wire.protocol._
@@ -260,15 +260,23 @@ object ColorSerializer extends MinimalSerializer({
   case c: Color => JString(c.toString)
 })
 
-object RouteResponseSerializer extends MinimalSerializer({
-  case route: RouteResponse =>
-    val nodeIds = route.routes.head.hops match {
-      case rest :+ last => rest.map(_.nodeId) :+ last.nodeId :+ last.nextNodeId
-      case Nil => Nil
-    }
-    JArray(nodeIds.toList.map(n => JString(n.toString)))
+// @formatter:off
+private case class RouteFullJson(amount: MilliSatoshi, hops: Seq[ChannelHop])
+object RouteFullSerializer extends ConvertClassSerializer[Route](route => RouteFullJson(route.amount, route.hops))
+
+private case class RouteNodeIdsJson(amount: MilliSatoshi, nodeIds: Seq[PublicKey])
+object RouteNodeIdsSerializer extends ConvertClassSerializer[Route](route => {
+  val nodeIds = route.hops match {
+    case rest :+ last => rest.map(_.nodeId) :+ last.nodeId :+ last.nextNodeId
+    case Nil => Nil
+  }
+  RouteNodeIdsJson(route.amount, nodeIds)
 })
 
+private case class RouteShortChannelIdsJson(amount: MilliSatoshi, shortChannelIds: Seq[ShortChannelId])
+object RouteShortChannelIdsSerializer extends ConvertClassSerializer[Route](route => RouteShortChannelIdsJson(route.amount, route.hops.map(_.lastUpdate.shortChannelId)))
+// @formatter:on
+
 // @formatter:off
 private case class PaymentFailureSummaryJson(amount: MilliSatoshi, route: Seq[PublicKey], message: String)
 private case class PaymentFailedSummaryJson(paymentHash: ByteVector32, destination: PublicKey, totalAmount: MilliSatoshi, pathFindingExperiment: String, failures: Seq[PaymentFailureSummaryJson])
@@ -487,7 +495,6 @@ object JsonSerializers {
     CommandResponseSerializer +
     InputInfoSerializer +
     ColorSerializer +
-    RouteResponseSerializer +
     ThrowableSerializer +
     FailureMessageSerializer +
     FailureTypeSerializer +

eclair-node/src/main/scala/fr/acinq/eclair/api/directives/ExtraDirectives.scala

@@ -46,7 +46,10 @@ trait ExtraDirectives extends Directives {
   val toFormParam: NameDefaultUnmarshallerReceptacle[TimestampSecond] = "to".as[TimestampSecond](timestampSecondUnmarshaller).?(Long.MaxValue unixsec)
   val amountMsatFormParam: NameReceptacle[MilliSatoshi] = "amountMsat".as[MilliSatoshi]
   val invoiceFormParam: NameReceptacle[PaymentRequest] = "invoice".as[PaymentRequest]
-  val routeFormat: NameUnmarshallerReceptacle[RouteFormat] = "format".as[RouteFormat](routeFormatUnmarshaller)
+  val routeFormatFormParam: NameUnmarshallerReceptacle[RouteFormat] = "format".as[RouteFormat](routeFormatUnmarshaller)
+  val ignoreNodeIdsFormParam: NameUnmarshallerReceptacle[List[PublicKey]] = "ignoreNodeIds".as[List[PublicKey]](pubkeyListUnmarshaller)
+  val ignoreShortChannelIdsFormParam: NameUnmarshallerReceptacle[List[ShortChannelId]] = "ignoreShortChannelIds".as[List[ShortChannelId]](shortChannelIdsUnmarshaller)
+  val maxFeeMsatFormParam: NameReceptacle[MilliSatoshi] = "maxFeeMsat".as[MilliSatoshi]
 
   // custom directive to fail with HTTP 404 (and JSON response) if the element was not found
   def completeOrNotFound[T](fut: Future[Option[T]])(implicit marshaller: ToResponseMarshaller[T]): Route = onComplete(fut) {

eclair-node/src/main/scala/fr/acinq/eclair/api/directives/RouteFormat.scala

@@ -16,38 +16,42 @@
 
 package fr.acinq.eclair.api.directives
 
+import akka.http.scaladsl.model.StatusCodes.OK
+import akka.http.scaladsl.model.{ContentTypes, HttpResponse}
+import fr.acinq.eclair.api.serde.JsonSupport._
+import fr.acinq.eclair.json.{JsonSerializers, RouteFullSerializer, RouteNodeIdsSerializer, RouteShortChannelIdsSerializer}
 import fr.acinq.eclair.router.Router.RouteResponse
+import org.json4s.Formats
 
 // @formatter:off
 sealed trait RouteFormat
 case object NodeIdRouteFormat extends RouteFormat
 case object ShortChannelIdRouteFormat extends RouteFormat
+case object FullRouteFormat extends RouteFormat
 // @formatter:on
 
 object RouteFormat {
 
   val NODE_ID = "nodeId"
   val SHORT_CHANNEL_ID = "shortChannelId"
+  val FULL = "full"
 
   def fromString(s: String): RouteFormat = s match {
     case NODE_ID => NodeIdRouteFormat
     case SHORT_CHANNEL_ID => ShortChannelIdRouteFormat
-    case _ => throw new IllegalArgumentException(s"invalid route format, possible values are ($NODE_ID, $SHORT_CHANNEL_ID)")
+    case FULL => FullRouteFormat
+    case _ => throw new IllegalArgumentException(s"invalid route format, possible values are ($NODE_ID, $SHORT_CHANNEL_ID, $FULL)")
   }
 
-  def format(route: RouteResponse, format_opt: Option[RouteFormat]): Seq[String] = format(route, format_opt.getOrElse(NodeIdRouteFormat))
-
-  def format(route: RouteResponse, format: RouteFormat): Seq[String] = format match {
-    case NodeIdRouteFormat =>
-      val nodeIds = route.routes.head.hops match {
-        case rest :+ last => rest.map(_.nodeId) :+ last.nodeId :+ last.nextNodeId
-        case Nil => Nil
-      }
-      nodeIds.toList.map(_.toString)
-    case ShortChannelIdRouteFormat =>
-      val shortChannelIds = route.routes.head.hops.map(_.lastUpdate.shortChannelId)
-      shortChannelIds.map(_.toString)
-  }
+  def format(route: RouteResponse, format_opt: Option[RouteFormat]): HttpResponse = format(route, format_opt.getOrElse(NodeIdRouteFormat))
 
+  def format(route: RouteResponse, format: RouteFormat): HttpResponse = {
+    val serializationFormats: Formats = format match {
+      case NodeIdRouteFormat => JsonSerializers.formats + RouteNodeIdsSerializer
+      case ShortChannelIdRouteFormat => JsonSerializers.formats + RouteShortChannelIdsSerializer
+      case FullRouteFormat => JsonSerializers.formats + RouteFullSerializer
+    }
+    HttpResponse(OK).withEntity(ContentTypes.`application/json`, serialization.write(route)(serializationFormats))
+  }
 }