Skip to content

Commit

Permalink
Merge branch 'thiagohora/OPIK-68_add_rate_limit' of https://github.co…
Browse files Browse the repository at this point in the history
…m/comet-ml/opik into thiagohora/OPIK-68_add_rate_limit
  • Loading branch information
thiagohora committed Sep 19, 2024
2 parents f9a537b + 56d21af commit 2e312b4
Show file tree
Hide file tree
Showing 78 changed files with 4,336 additions and 418 deletions.
68 changes: 65 additions & 3 deletions .github/workflows/sdk-e2e-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,73 @@ name: SDK E2E Tests
run-name: "SDK E2E Tests ${{ github.ref_name }} by @${{ github.actor }}"
on:
workflow_dispatch:
pull_request:
paths:
- 'sdks/python/**'
- 'apps/opik-backend/**'
push:
branches:
- 'main'
paths:
- 'sdks/python/**'
- 'apps/opik-backend/**'

jobs:
run-e2e:
name: SDK E2E Tests
name: SDK E2E Tests ${{matrix.python_version}}
runs-on: ubuntu-latest
defaults:
run:
working-directory: sdks/python

strategy:
fail-fast: false
matrix:
python_version: ["3.8", "3.9", "3.10", "3.11", "3.12"]

steps:
- name: echo
run: echo
- name: Checkout
uses: actions/checkout@v4.1.1

- name: Setup Python ${{matrix.python_version}}
uses: actions/setup-python@v5
with:
python-version: ${{matrix.python_version}}

- name: Install opik SDK
run: |
pip install -r tests/test_requirements.txt
pip install .
- name: Run latest opik server
run: |
cd ../../deployment/docker-compose
docker compose up -d
cd -
- name: Run tests
run: |
echo "Waiting for server to come up..."
sleep 30
export OPIK_URL_OVERRIDE=http://localhost:5173/api
pytest tests/e2e -vv
- name: Keep BE log in case of failure
if: failure()
run: |
docker logs opik-backend-1 > opik-backend_p${{matrix.python_version}}.log
- name: Attach BE log
if: failure()
uses: actions/upload-artifact@v4
with:
name: opik-backend-log-p${{matrix.python_version}}
path: sdks/python/opik-backend_p${{matrix.python_version}}.log

- name: Stop opik server
if: always()
run: |
cd ../../deployment/docker-compose
docker compose down
cd -
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -53,3 +53,6 @@ pip-log.txt

# charts
deployment/helm_chart/opik/charts
deployment/helm_chart/opik/values-cloud-test.yaml
deployment/helm_chart/opik/values-test.yaml
temp
44 changes: 29 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<h1 align="center" style="border-bottom: none">
<div>
<a href="https://www.comet.com/site/products/opik?utm_source=opik&utm_medium=github&utm_content=website_button"><picture>
<a href="https://www.comet.com/site/products/opik?utm_source=opik&utm_medium=github&utm_content=header_img"><picture>
<source media="(prefers-color-scheme: dark)" srcset="/apps/opik-documentation/documentation/static/img/logo-dark-mode.svg">
<source media="(prefers-color-scheme: light)" srcset="/apps/opik-documentation/documentation/static/img/opik-logo.svg">
<img alt="Comet Opik logo" src="/apps/opik-documentation/documentation/static/img/opik-logo.svg" width="200" />
Expand All @@ -27,7 +27,7 @@ Confidently evaluate, test and monitor LLM applications. 
</div>

<p align="center">
<a href="http://www.comet.com/site/products/opik"><b>Website</b></a> •
<a href="https://www.comet.com/site/products/opik?utm_source=opik&utm_medium=github&utm_content=website_button)"><b>Website</b></a> •
<a href="https://chat.comet.com"><b>Slack community</b></a> •
<a href="https://x.com/Cometml"><b>Twitter</b></a> •
<a href="https://www.comet.com/docs/opik/"><b>Documentation</b></a>
Expand Down Expand Up @@ -59,17 +59,22 @@ You can use Opik for:
<br>

## 🛠️ Installation

Opik is available as a fully open source local installation or using Comet.com as a hosted solution.
The easiest way to get started with Opik is by creating a free Comet account at [comet.com](https://www.comet.com/signup?from=llm?utm_source=opik&utm_medium=github&utm_content=install).

If you'd like to self-host Opik, you can do so by cloning the repository and starting the platform using Docker Compose:

```bash
# Clone the Opik repository
git clone https://github.com/comet-ml/opik.git

If you'd like to self-host Opik, you create a simple local version of Opik using::
# Navigate to the opik/deployment/docker-compose directory
cd opik/deployment/docker-compose

```bash
pip install opik-installer
# Start the Opik platform
docker compose up --detach

opik-server install
# You can now visit http://localhost:5173 on your browser!
```

For more information about the different deployment options, please see our deployment guides:
Expand All @@ -82,18 +87,23 @@ For more information about the different deployment options, please see our depl

## 🏁 Get Started

If you are logging traces to the Cloud Opik platform, you will need to get your API key from the user menu and set it as the `OPIK_API_KEY` environment variable:
To get started, you will need to first install the Python SDK:

```bash
export OPIK_API_KEY=<Your API key>
export OPIK_WORKSPACE=<You workspace, often the same as your username>
pip install opik
```

If you are using a local Opik instance, you don't need to set the `OPIK_API_KEY` or `OPIK_WORKSPACE` environment variable and isntead set the environment variable `OPIK_BASE_URL` to point to your local Opik instance:
Once the SDK is installed, you can configure it by running the `opik configure` command:

```bash
export OPIK_BASE_URL=http://localhost:5173
opik configure
```
This will allow you to configure Opik locally by setting the correct local server address or if you're using the Cloud platform by setting the API Key


> [!TIP]
> You can also call the `opik.configure(use_local=True)` method from your Python code to configure the SDK to run on the local installation.

You are now ready to start logging traces using the [Python SDK](https://www.comet.com/docs/opik/python-sdk-reference/?utm_source=opik&utm_medium=github&utm_content=sdk_link2).

Expand All @@ -103,19 +113,23 @@ The easiest way to get started is to use one of our integrations. Opik supports:

| Integration | Description | Documentation | Try in Colab |
| ----------- | ----------- | ------------- | ------------ |
| OpenAI | Log traces for all OpenAI LLM calls | [Documentation](https://www.comet.com//docs/opik/tracing/integrations/openai/?utm_source=opik&utm_medium=github&utm_content=openai_link) | [![Open Quickstart In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/openai.ipynb) |
| OpenAI | Log traces for all OpenAI LLM calls | [Documentation](https://www.comet.com/docs/opik/tracing/integrations/openai/?utm_source=opik&utm_medium=github&utm_content=openai_link) | [![Open Quickstart In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/openai.ipynb) |
| LangChain | Log traces for all LangChain LLM calls | [Documentation](https://www.comet.com/docs/opik/tracing/integrations/langchain/?utm_source=opik&utm_medium=github&utm_content=langchain_link) | [![Open Quickstart In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/langchain.ipynb) |
| LlamaIndex | Log traces for all LlamaIndex LLM calls | [Documentation](https://www.comet.com/docs/opik/tracing/integrations/llama_index?utm_source=opik&utm_medium=github&utm_content=llama_index_link) | [![Open Quickstart In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/llama-index.ipynb) |
| Predibase | Fine-tune and serve open-source Large Language Models | [Documentation](https://www.comet.com/docs/opik/tracing/integrations/predibase?utm_source=opik&utm_medium=github&utm_content=predibase_link) | [![Open Quickstart In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/predibase.ipynb) |
| Ragas | Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines | [Documentation](https://www.comet.com/docs/opik/tracing/integrations/ragas?utm_source=opik&utm_medium=github&utm_content=ragas_link) | [![Open Quickstart In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/ragas.ipynb) |

> [!TIP]
> If the framework you are using is not listed above, feel free to [open an issue](https://github.com/comet-ml/opik/issues) or submit a PR with the integration.
If you are not using any of the frameworks above, you can also using the `track` function decorator to [log traces](https://www.comet.com/docs/opik/tracing/log_traces/?utm_source=opik&utm_medium=github&utm_content=traces_link):

```python
from opik import track
import opik

opik.configure(use_local=True) # Run locally

@track
@opik.track
def my_llm_function(user_question: str) -> str:
# Your LLM code here

Expand Down
2 changes: 1 addition & 1 deletion apps/opik-backend/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,4 +70,4 @@ rateLimit:
enabled: ${RATE_LIMIT_ENABLED:-false}
generalEvents:
limit: ${RATE_LIMIT_GENERAL_EVENTS_LIMIT:-5000}
durationInSeconds: ${RATE_LIMIT_GENERAL_EVENTS_DURATION_SEC:-1}
durationInSeconds: ${RATE_LIMIT_GENERAL_EVENTS_DURATION_IN_SEC:-1}
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import com.comet.opik.infrastructure.bundle.LiquibaseBundle;
import com.comet.opik.infrastructure.db.DatabaseAnalyticsModule;
import com.comet.opik.infrastructure.db.IdGeneratorModule;
import com.comet.opik.infrastructure.db.NameGeneratorModule;
import com.comet.opik.infrastructure.ratelimit.RateLimitModule;
import com.comet.opik.infrastructure.redis.RedisModule;
import com.comet.opik.utils.JsonBigDecimalDeserializer;
Expand Down Expand Up @@ -60,7 +61,7 @@ public void initialize(Bootstrap<OpikConfiguration> bootstrap) {
.bundles(JdbiBundle.<OpikConfiguration>forDatabase((conf, env) -> conf.getDatabase())
.withPlugins(new SqlObjectPlugin(), new Jackson2Plugin()))
.modules(new DatabaseAnalyticsModule(), new IdGeneratorModule(), new AuthModule(), new RedisModule(),
new RateLimitModule())
new RateLimitModule(), new NameGeneratorModule())
.enableAutoConfig()
.build());
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ public record Experiment(
Experiment.View.Public.class, Experiment.View.Write.class}) UUID id,
@JsonView({Experiment.View.Public.class, Experiment.View.Write.class}) @NotBlank String datasetName,
@JsonView({Experiment.View.Public.class}) @Schema(accessMode = Schema.AccessMode.READ_ONLY) UUID datasetId,
@JsonView({Experiment.View.Public.class, Experiment.View.Write.class}) @NotBlank String name,
@JsonView({Experiment.View.Public.class, Experiment.View.Write.class}) String name,
@JsonView({Experiment.View.Public.class, Experiment.View.Write.class}) JsonNode metadata,
@JsonView({
Experiment.View.Public.class}) @Schema(accessMode = Schema.AccessMode.READ_ONLY) List<FeedbackScoreAverage> feedbackScores,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,9 @@ public static class Public {
}
}

public record ProjectPage(@JsonView( {
Project.View.Public.class}) int page,
public record ProjectPage(
@JsonView( {
Project.View.Public.class}) int page,
@JsonView({Project.View.Public.class}) int size,
@JsonView({Project.View.Public.class}) long total,
@JsonView({Project.View.Public.class}) List<Project> content)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
package com.comet.opik.api;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.databind.PropertyNamingStrategies;
import com.fasterxml.jackson.databind.annotation.JsonNaming;
import lombok.Builder;

import java.util.List;

@Builder(toBuilder = true)
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonNaming(PropertyNamingStrategies.SnakeCaseStrategy.class)
public record TraceCountResponse(
List<WorkspaceTraceCount> workspacesTracesCount) {
public static TraceCountResponse empty() {
return new TraceCountResponse(List.of());
}

@Builder(toBuilder = true)
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonNaming(PropertyNamingStrategies.SnakeCaseStrategy.class)
public record WorkspaceTraceCount(
String workspace,
int traceCount) {
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
package com.comet.opik.api.resources.v1.internal;

import com.codahale.metrics.annotation.Timed;
import com.comet.opik.api.TraceCountResponse;
import com.comet.opik.domain.TraceService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.media.Content;
import io.swagger.v3.oas.annotations.media.Schema;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.tags.Tag;
import jakarta.ws.rs.Consumes;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;
import jakarta.ws.rs.core.Response;
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;

@Path("/v1/internal/usage")
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
@Timed
@Slf4j
@RequiredArgsConstructor(onConstructor_ = @jakarta.inject.Inject)
@Tag(name = "System usage", description = "System usage related resource")
public class UsageResource {
private final @NonNull TraceService traceService;

@GET
@Path("/workspace-trace-counts")
@Operation(operationId = "getTracesCountForWorkspaces", summary = "Get traces count on previous day for all available workspaces", description = "Get traces count on previous day for all available workspaces", responses = {
@ApiResponse(responseCode = "200", description = "TraceCountResponse resource", content = @Content(schema = @Schema(implementation = TraceCountResponse.class)))})
public Response getTracesCountForWorkspaces() {
return traceService.countTracesPerWorkspace()
.map(tracesCountResponse -> Response.ok(tracesCountResponse).build())
.block();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import lombok.NonNull;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import reactor.core.publisher.Mono;
import reactor.core.scheduler.Schedulers;

Expand All @@ -31,6 +32,7 @@ public class ExperimentService {
private final @NonNull ExperimentDAO experimentDAO;
private final @NonNull DatasetService datasetService;
private final @NonNull IdGenerator idGenerator;
private final @NonNull NameGenerator nameGenerator;

public Mono<Experiment.ExperimentPage> find(
int page, int size, @NonNull ExperimentSearchCriteria experimentSearchCriteria) {
Expand Down Expand Up @@ -73,34 +75,35 @@ public Mono<Experiment> getById(@NonNull UUID id) {
public Mono<Experiment> create(@NonNull Experiment experiment) {
var id = experiment.id() == null ? idGenerator.generateId() : experiment.id();
IdGenerator.validateVersion(id, "Experiment");
var name = StringUtils.getIfBlank(experiment.name(), nameGenerator::generateName);

return getOrCreateDataset(experiment)
.onErrorResume(e -> handleDatasetCreationError(e, experiment).map(Dataset::id))
.flatMap(datasetId -> create(experiment, id, datasetId))
return getOrCreateDataset(experiment.datasetName())
.onErrorResume(e -> handleDatasetCreationError(e, experiment.datasetName()).map(Dataset::id))
.flatMap(datasetId -> create(experiment, id, name, datasetId))
.onErrorResume(exception -> handleCreateError(exception, id));
}

private Mono<UUID> getOrCreateDataset(Experiment experiment) {
private Mono<UUID> getOrCreateDataset(String datasetName) {
return Mono.deferContextual(ctx -> {
String userName = ctx.get(RequestContext.USER_NAME);
String workspaceId = ctx.get(RequestContext.WORKSPACE_ID);

return Mono.fromCallable(() -> datasetService.getOrCreate(workspaceId, experiment.datasetName(), userName))
return Mono.fromCallable(() -> datasetService.getOrCreate(workspaceId, datasetName, userName))
.subscribeOn(Schedulers.boundedElastic());
});
}

private Mono<Experiment> create(Experiment experiment, UUID id, UUID datasetId) {
var newExperiment = experiment.toBuilder().id(id).datasetId(datasetId).build();
return experimentDAO.insert(newExperiment).thenReturn(newExperiment);
private Mono<Experiment> create(Experiment experiment, UUID id, String name, UUID datasetId) {
experiment = experiment.toBuilder().id(id).name(name).datasetId(datasetId).build();
return experimentDAO.insert(experiment).thenReturn(experiment);
}

private Mono<Dataset> handleDatasetCreationError(Throwable throwable, Experiment experiment) {
private Mono<Dataset> handleDatasetCreationError(Throwable throwable, String datasetName) {
if (throwable instanceof EntityAlreadyExistsException) {
return Mono.deferContextual(ctx -> {
String workspaceId = ctx.get(RequestContext.WORKSPACE_ID);

return Mono.fromCallable(() -> datasetService.findByName(workspaceId, experiment.datasetName()))
return Mono.fromCallable(() -> datasetService.findByName(workspaceId, datasetName))
.subscribeOn(Schedulers.boundedElastic());
});
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
package com.comet.opik.domain;

import lombok.Builder;
import lombok.NonNull;

import java.security.SecureRandom;
import java.util.List;

@Builder
public class NameGenerator {

private final @NonNull SecureRandom secureRandom;

private final @NonNull List<String> adjectives;
private final @NonNull List<String> nouns;

public String generateName() {
var adjective = getRandom(adjectives);
var noun = getRandom(nouns);
var number = secureRandom.nextInt(0, 10000);
return "%s_%s_%s".formatted(adjective, noun, number);
}

private String getRandom(List<String> strings) {
int index = secureRandom.nextInt(0, strings.size());
return strings.get(index);
}
}
Loading

0 comments on commit 2e312b4

Please sign in to comment.