Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify Cortex README.md #1098

Merged
merged 27 commits into from
Sep 12, 2024
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
ac428aa
Draft the Platform readme
irfanpena Sep 5, 2024
3cca455
Update the Model library table
irfanpena Sep 5, 2024
2552745
nits
irfanpena Sep 5, 2024
0347b70
nits
irfanpena Sep 5, 2024
2a6adbf
Added the installation based on discord
irfanpena Sep 6, 2024
af7035f
nits
irfanpena Sep 6, 2024
6be6849
cortex-engine->cortex.cpp
irfanpena Sep 6, 2024
3b35dd9
use the current banner instead
irfanpena Sep 6, 2024
4b72283
Merge branch 'dev' into platform-readme
irfanpena Sep 6, 2024
da195cc
Simplify and update the cortex.cpp readme
irfanpena Sep 9, 2024
66d4d11
Merge branch 'platform-readme' of https://github.com/janhq/cortex.cpp…
irfanpena Sep 9, 2024
c4863ff
Update the CortexCPP readme
irfanpena Sep 9, 2024
8671572
Update Overview and Remove Cortex Platform readme
irfanpena Sep 9, 2024
af180e4
nits
irfanpena Sep 9, 2024
80a2735
Merge branch 'dev' into platform-readme
irfanpena Sep 9, 2024
f38f678
Separate installer for each version
irfanpena Sep 9, 2024
4d86d6f
Merge branch 'platform-readme' of https://github.com/janhq/cortex.cpp…
irfanpena Sep 9, 2024
425f641
Update the build from source steps
irfanpena Sep 9, 2024
7bcd47b
nits
irfanpena Sep 9, 2024
ba71e2b
nits
irfanpena Sep 9, 2024
fb14521
1337 -> 3928
irfanpena Sep 9, 2024
d73e133
Merge branch 'dev' into platform-readme
irfanpena Sep 10, 2024
ea84b2d
CortexCPP -> Cortex.cpp
irfanpena Sep 10, 2024
7c34758
Merge branch 'platform-readme' of https://github.com/janhq/cortex.cpp…
irfanpena Sep 10, 2024
01981b9
nits
irfanpena Sep 10, 2024
1d434cf
change the engine names
irfanpena Sep 11, 2024
a7e1de3
Updated per feedbacks, except the PORT
irfanpena Sep 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
177 changes: 65 additions & 112 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,39 +1,41 @@
# Cortex
# Cortex.cpp
<p align="center">
irfanpena marked this conversation as resolved.
Show resolved Hide resolved
<img alt="cortex-cpplogo" src="https://github.com/raw/janhq/cortex/dev/assets/cortex-banner.png">
</p>

<p align="center">
<a href="https://cortex.so/docs/">Documentation</a> - <a href="https://cortex.so/api-reference">API Reference</a>
- <a href="https://github.com/janhq/cortex/releases">Changelog</a> - <a href="https://github.com/janhq/cortex/issues">Bug reports</a> - <a href="https://discord.gg/AsJ8krTT3N">Discord</a>
irfanpena marked this conversation as resolved.
Show resolved Hide resolved
- <a href="https://github.com/janhq/cortex.cpp/releases">Changelog</a> - <a href="https://github.com/janhq/cortex.cpp/issues">Bug reports</a> - <a href="https://discord.gg/AsJ8krTT3N">Discord</a>
</p>

> ⚠️ **Cortex is currently in Development**: Expect breaking changes and bugs!
> ⚠️ **Cortex.cpp is currently in Development. This documentation outlines the intended behavior of Cortex, which may not yet be fully implemented in the codebase.**

## About
Cortex is a C++ AI engine that comes with a Docker-like command-line interface and client libraries. It supports running AI models using `ONNX`, `TensorRT-LLM`, and `llama.cpp` engines. Cortex can function as a standalone server or be integrated as a library.
Cortex.cpp is a Local AI engine that is used to run and customize LLMs. Cortex can be deployed as a standalone server, or integrated into apps like [Jan.ai](https://jan.ai/).

## Cortex Engines
Cortex supports the following engines:
- [`cortex.llamacpp`](https://github.com/janhq/cortex.llamacpp): `cortex.llamacpp` library is a C++ inference tool that can be dynamically loaded by any server at runtime. We use this engine to support GGUF inference with GGUF models. The `llama.cpp` is optimized for performance on both CPU and GPU.
irfanpena marked this conversation as resolved.
Show resolved Hide resolved
- [`cortex.onnx` Repository](https://github.com/janhq/cortex.onnx): `cortex.onnx` is a C++ inference library for Windows that leverages `onnxruntime-genai` and uses DirectML to provide GPU acceleration across a wide range of hardware and drivers, including AMD, Intel, NVIDIA, and Qualcomm GPUs.
- [`cortex.tensorrt-llm`](https://github.com/janhq/cortex.tensorrt-llm): `cortex.tensorrt-llm` is a C++ inference library designed for NVIDIA GPUs. It incorporates NVIDIA’s TensorRT-LLM for GPU-accelerated inference.
- [`cortex.llamacpp`](https://github.com/janhq/cortex.llamacpp)
- [`cortex.onnx`](https://github.com/janhq/cortex.onnx)
- [`cortex.tensorrt-llm`](https://github.com/janhq/cortex.tensorrt-llm)

## Installation
### MacOs
```bash
brew install cortex-engine
```
irfanpena marked this conversation as resolved.
Show resolved Hide resolved
### Windows
```bash
winget install cortex-engine
```
### Linux
```bash
sudo apt install cortex-engine
```
### Docker
**Coming Soon!**
To install Cortex, download the installer for your operating system from the following options:
- **Stable Version**
irfanpena marked this conversation as resolved.
Show resolved Hide resolved
- [Windows]()
- [Mac]()
- [Linux (Debian)]()
- [Linux (Fedora)]()
- **Beta Version**
- [Windows]()
- [Mac]()
- [Linux (Debian)]()
- [Linux (Fedora)]()
- **Nightly Version**
- [Windows]()
- [Mac]()
- [Linux (Debian)]()
- [Linux (Fedora)]()


### Libraries
- [cortex.js](https://github.com/janhq/cortex.js)
Expand All @@ -43,26 +45,23 @@ sudo apt install cortex-engine

To install Cortex from the source, follow the steps below:

irfanpena marked this conversation as resolved.
Show resolved Hide resolved
1. Clone the Cortex repository [here](https://github.com/janhq/cortex/tree/dev).
2. Navigate to the `platform` folder.
3. Open the terminal and run the following command to build the Cortex project:
1. Clone the Cortex.cpp repository [here](https://github.com/janhq/cortex.cpp).
2. Navigate to the `engine > vcpkg` folder.
3. Configure the vpkg:

```bash
npx nest build
cd vcpkg
./bootstrap-vcpkg.bat
vcpkg install
```

4. Make the `command.js` executable:

```bash
chmod +x '[path-to]/cortex/platform/dist/src/command.js'
```

5. Link the package globally:
4. Build the Cortex inside the `build` folder:

```bash
npm link
mkdir build
cd build
cmake .. -DBUILD_SHARED_LIBS=OFF -DCMAKE_TOOLCHAIN_FILE=path_to_vcpkg_folder/vcpkg/scripts/buildsystems/vcpkg.cmake -DVCPKG_TARGET_TRIPLET=x64-windows-static
```

5. Use Visual Studio with the C++ development kit to build the project using the files generated in the `build` folder.

## Quickstart
To run and chat with a model in Cortex:
Expand All @@ -72,12 +71,9 @@ cortex

# Start a model
cortex run [model_id]

# Chat with a model
cortex chat [model_id]
```
## Model Library
Cortex supports a list of models available on [Cortex Hub](https://huggingface.co/cortexso).
## Built-in Model Library
Cortex.cpp supports a list of models available on [Cortex Hub](https://huggingface.co/cortexso).

Here are example of models that you can use based on each supported engine:
### `llama.cpp`
Expand Down Expand Up @@ -122,87 +118,44 @@ Here are example of models that you can use based on each supported engine:
> **Note**:
> You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 14B models, and 32 GB to run the 32B models.

## Cortex CLI Commands
## Cortex.cpp CLI Commands

| Command Description | Command Example |
|------------------------------------|---------------------------------------------------------------------|
| **Start Cortex Server** | `cortex` |
| **Chat with a Model** | `cortex chat [options] [model_id] [message]` |
| **Embeddings** | `cortex embeddings [options] [model_id] [message]` |
| **Pull a Model** | `cortex pull <model_id>` |
| **Download and Start a Model** | `cortex run [options] [model_id]:[engine]` |
| **Get Model Details** | `cortex models get <model_id>` |
| **List Models** | `cortex models list [options]` |
| **Delete a Model** | `cortex models delete <model_id>` |
| **Start a Model** | `cortex models start [model_id]` |
| **Stop a Model** | `cortex models stop <model_id>` |
| **Update a Model** | `cortex models update [options] <model_id>` |
| **Get Engine Details** | `cortex engines get <engine_name>` |
| **Install an Engine** | `cortex engines install <engine_name> [options]` |
| **List Engines** | `cortex engines list [options]` |
| **Uninnstall an Engine** | `cortex engines uninstall <engine_name> [options]` |
| **Show Model Information** | `cortex ps` |
| **Update Cortex** | `cortex update [options]` |

> **Note**:
irfanpena marked this conversation as resolved.
Show resolved Hide resolved
> For a more detailed CLI Reference documentation, please see [here](https://cortex.so/docs/cli).
### Start Cortex Server
```bash
cortex
```
### Chat with a Model
```bash
cortex chat [options] [model_id] [message]
```
### Embeddings
```bash
cortex embeddings [options] [model_id] [message]
```
### Pull a Model
```bash
cortex pull <model_id>
```
> This command can also pulls Hugging Face's models.
### Download and Start a Model
```bash
cortex run [options] [model_id]:[engine]
```
### Get a Model Details
```bash
cortex models get <model_id>
```
### List Models
```bash
cortex models list [options]
```
### Remove a Model
```bash
cortex models remove <model_id>
```
### Start a Model
```bash
cortex models start [model_id]
```
### Stop a Model
```bash
cortex models stop <model_id>
```
### Update a Model Config
```bash
cortex models update [options] <model_id>
```
### Get an Engine Details
```bash
cortex engines get <engine_name>
```
### Install an Engine
```bash
cortex engines install <engine_name> [options]
```
### List Engines
```bash
cortex engines list [options]
```
### Set an Engine Config
```bash
cortex engines set <engine_name> <config> <value>
```
### Show Model Information
```bash
cortex ps
```

## REST API
Cortex has a REST API that runs at `localhost:1337`.
Cortex.cpp has a REST API that runs at `localhost:3928`.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check with @vansangpfiev @namchuai: There is a chance we will change the port number to something less collision-prone (i.e. more digits?)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's bring this into discussion. I'm ok with any port number decision.

### Pull a Model
```bash
curl --request POST \
--url http://localhost:1337/v1/models/{model_id}/pull
--url http://localhost:3928/v1/models/{model_id}/pull
```

### Start a Model
```bash
curl --request POST \
--url http://localhost:1337/v1/models/{model_id}/start \
--url http://localhost:3928/v1/models/{model_id}/start \
--header 'Content-Type: application/json' \
--data '{
"prompt_template": "system\n{system_message}\nuser\n{prompt}\nassistant",
Expand All @@ -224,7 +177,7 @@ curl --request POST \

### Chat with a Model
```bash
curl http://localhost:1337/v1/chat/completions \
curl http://localhost:3928/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "",
Expand All @@ -250,7 +203,7 @@ curl http://localhost:1337/v1/chat/completions \
### Stop a Model
```bash
curl --request POST \
--url http://localhost:1337/v1/models/mistral/stop
--url http://localhost:3928/v1/models/mistral/stop
```


Expand Down
142 changes: 0 additions & 142 deletions platform/README.md

This file was deleted.

Loading