Let's try to host REST Proxy in localhost and make it work with CC.
## DISCLAIMER: THIS IS A SUPER SIMPLE EXAMPLE, not really useful for production etc.
If you want a kubernetes example by Confluent
https://github.com/confluentinc/confluent-kubernetes-examples/tree/master/hybrid/ccloud-integration
- hosting Confluent Platform in my local machine, with Docker.
- Taking into advtange REST PROXY for sending messages to a topic in Confluent Cloud.
Good inspo from these links!
https://www.confluent.io/es-es/blog/a-comprehensive-rest-proxy-for-kafka/
https://github.com/confluentinc/kafka-rest
https://docs.confluent.io/platform/current/kafka-rest/api.html#rest-api-usage-examples
First clone the git repo, and enter in the folder:
git clone https://github.com/angietd94/confluent-cloud-rest-proxy.git
cd confluent-cloud-hybrid-demo
NOTE: first change the values in config.sh depending on your own account.
You can find the description on how to find all values in this repo:
https://github.com/angietd94/confluent-cloud-hybrid-demo
Where also it is explained how to launch the script in the "Second part: Launching the script! " but basically this is just to have CP running (AND THEN REST PROXY included).
## Detailed video of every step here:
---->
https://drive.google.com/file/d/1uh-MYKVAOFPCmMYWZnNLR3vbl6dsZeyX/view?usp=sharing
That basically is summaryzing:
chmod +x executable.sh
chmod +x config_files.sh
chmod +x config.sh
chmod +x ccloud-generate-cp-configs.sh
./executable.sh
- Let's create the topic (here rest_proxy_example, and put a simple schema AVRO, for example having a name and age of someone:
{
"doc": "Sample schema to help you get started.",
"fields": [
{
"name": "age",
"type": "int"
},
{
"doc": "The string is a unicode character sequence.",
"name": "name",
"type": "string"
}
],
"name": "sampleRecord",
"namespace": "com.mycorp.mynamespace",
"type": "record"
}
Is it all up and running in Docker? GOOD!
Let's try some curl commands...
For example:
curl http://localhost:8082/v3/clusters
curl http://localhost:8082/topics
curl http://localhost:8082/topics/rest_proxy_example | jq
A sample message that will work here:
curl -i -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" -H "Accept: application/vnd.kafka.v2+json" --url "http://localhost:8082/topics/rest_proxy_example" --data '{"value_schema": "{\"type\": \"record\", \"name\": \"sampleRecord\", \"doc\":\"Sample schema to help you get started.\", \"namespace\":\"com.mycorp.mynamespace\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}","records":[{"value":{"name":"angelica","age":29}}]}'
Comprehensive guide can be found in https://docs.confluent.io/platform/current/kafka-rest/api.html and other documentations. Summizing you will get errors like:
In this guide, check https://github.com/angietd94/confluent-cloud-rest-proxy/blob/main/restproxy_commands.sh .
curl -i -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" -H "Accept: application/vnd.kafka.v2+json" --url "http://localhost:8082/topics/rest_proxy_example" --data '{"value_schema": "{\"type\": \"record\", \"name\": \"sampleRecord\", \"doc\":\"Sample schema to help you get started.\", \"namespace\":\"com.mycorp.mynamespace\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"nombre\",\"type\":\"string\"}]}","records":[{"value":{"name":"angelica","age":29}}]}'
## Error 422 Example wrong data: name field wrong in the schema AND in the data
curl -i -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" -H "Accept: application/vnd.kafka.v2+json" --url "http://localhost:8082/topics/rest_proxy_example" --data '{"value_schema": "{\"type\": \"record\", \"name\": \"sampleRecord\", \"doc\":\"Sample schema to help you get started.\", \"namespace\":\"com.mycorp.mynamespace\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"nombre\",\"type\":\"string\"}]}","records":[{"value":{"nombre":"angelica","age":29}}]}'
<br>
curl -i -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" -H "Accept: application/vnd.kafka.v2+json" --url "http://localhost:8082/topics/rest_proxy_example" --data '{"value_schema": "{\"type\": \"record\", \"doc\":\"Sample schema to help you get started.\", \"namespace\":\"com.mycorp.mynamespace\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}","records":[{"value":{"age":29}}]}'
curl -i -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" -H "Accept: application/vnd.kafka.v2+json" --url "http://localhost:8082/topics/rest_proxy_example" --data '{"value_schema": "{\"type\": \"record\", \"name\": \"sampleRecord\", \"doc\":\"Sample schema to help you get started.\", \"namespace\":\"com.mycorp.mynamespace\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}","records":[{"value":{"nombre":30,"age":29}}]}'
curl -i -X POST -H "Content-Type: application/vnd.kafka.avro.v2+json" -H "Accept: application/vnd.kafka.v2+json" --url "http://localhost:8082/topics/rest_proxy_example" --data '{"value_schema": "{\"type\": \"record\", \"name\": \"sampleRecord\", \"doc\":\"Sample schema to help you get started.\", \"namespace\":\"com.mycorp.mynamespace\",\"fields\":[{\"name\":\"age\",\"type\":\"int\"},{\"name\":\"name\",\"type\":\"string\"}]}","records":[{"value":{"age":29}}]}'
In the other parts of the video I basically just made ChatGPT make the work by creating some mock data similar to this.
The first time is working fine and in the second you see some errors. Not really relevant just an example.