freedomofpress · legoktm · Sep 20, 2024
@@ -39,11 +39,13 @@ class Config:
         "gpg_domain": "QUBES_GPG_DOMAIN",
         "journalist_key_fingerprint": "SD_SUBMISSION_KEY_FPR",
         "download_retry_limit": "SD_DOWNLOAD_RETRY_LIMIT",
+        "proxy_vm_name": "SD_PROXY_VM_NAME",
     }
 
     journalist_key_fingerprint: str
     gpg_domain: str | None = None
     download_retry_limit: int = 3
+    proxy_vm_name: str = "sd-proxy"
 
     @classmethod
     def load(cls) -> "Config":

@@ -1,4 +1,3 @@
-import configparser
 import http
 import json
 import logging
@@ -113,17 +112,10 @@ def __init__(
         self.default_request_timeout = default_request_timeout or DEFAULT_REQUEST_TIMEOUT
         self.default_download_timeout = default_download_timeout or DEFAULT_DOWNLOAD_TIMEOUT
 
-        self.proxy_vm_name = DEFAULT_PROXY_VM_NAME
-        config_parser = configparser.ConfigParser()
-        try:
-            if os.path.exists("/etc/sd-sdk.conf"):
-                config_parser.read("/etc/sd-sdk.conf")
-                self.proxy_vm_name = config_parser["proxy"]["name"]
-        except Exception:
-            pass  # We already have a default name
-
-        # Load download retry limit from the config
-        self.download_retry_limit = Config.load().download_retry_limit
+        # Load configurable settings
+        config = Config.load()
+        self.proxy_vm_name = config.proxy_vm_name
+        self.download_retry_limit = config.download_retry_limit
 
     def _rpc_target(self) -> list:
         """In `development_mode`, check `cargo` for a locally-built proxy binary.

@@ -71,21 +71,13 @@ make test TESTS=tests/test_api.py::TestAPI::test_get_sources
 
 When tests are run, they replay recorded API request and response data instead of making actual API calls to a server. This is why tests can pass even when there is no server running. If the server ever changes its API or you want to add new tests that make API calls, then you'll need to record new request and response data by following the steps outlined below.
 
-**Note:** We have a CI test that does not use the recorded API request and response data in order to make sure we are testing the latest changes to the SDK against the latest server API (see `test-against-latest-api` in https://github.com/freedomofpress/securedrop-sdk/blob/main/.circleci/config.yml).
+**Note:** We have a CI check that does not use the recorded API request and response data in order to make sure we are testing the latest changes to the SDK against the latest server API (see `sdk-with-server` in `.github/workflows/sdk.yml`).
 
-We use [vcrpy](https://vcrpy.readthedocs.io/en/latest/) to record and replay API calls made over HTTP and a decorator called `@qrexec_datasaver` to record and replay API calls made over qrexec. Each request made from a test and its response from the server is stored in a "cassette" in the `data` directory. Tests replay these cassettes instead of making actual API calls to a server.
+We use [vcrpy](https://vcrpy.readthedocs.io/en/latest/) to record and replay API calls made over HTTP. Each request made from a test and its response from the server is stored in a "cassette" in `data` directories. Tests replay these cassettes instead of making actual API calls to a server.
 
-If you run the tests and see the following vcrpy warning, then you'll need to re-record cassettes because none of the existing cassettes contain the expected API call and we don't allow existing cassettes to be overwritten:
+## Generating new cassettes for API calls
 
-```
-Can't overwrite existing cassette ('<path-to-cassette-for-a-functional-test>') in your current record mode ('once').
-```
-
-The steps to generate new cassettes are split into two sections based on communication protocol: [Generating cassettes for API calls over HTTP](#generating-cassettes-for-api-calls-over-http) and [Generating cassettes for API calls over qrexec](#generating-cassettes-for-api-calls-over-qrexec).
-
-## Generating cassettes for API calls over HTTP
-
-1. Start the server in a docker container by running:
+1. Start the server in a container by running:
 
     ```bash
     NUM_SOURCES=5 LOADDATA_ARGS="--random-file-size 3" make dev
@@ -98,139 +90,15 @@ The steps to generate new cassettes are split into two sections based on communi
     ```
 
    If you are only adding a new test and not modifying existing ones, you can
-   skip this step, but you still need to remove the authentication setup during
-   cassette generation. Otherwise you will get 403 errors for API endpoints that
-   require a valid token. Remove the setup cassette by running:
+   skip this step
 
-   ```bash
-   rm data/test-setup.yml
-   ```
-
-   (You can reinstate the unmodified version later.)
-
-3. Generate new cassettes that make API calls over HTTP by running:
+3. Generate new cassettes by running:
 
     ```bash
     make test TESTS=tests/test_api.py
     ```
 Note: Some tests alter source and conversation data on the server so you may need to restart the server in between test runs.
 
-## Generating cassettes for API calls over qrexec
-
-In order to generate cassettes for tests that make API calls over qrexec, you'll need to run the server and proxy on a separate VM. If this is the first time you are generating cassettes, first follow the steps outlined in the [Test setup for qrexec communication](#test-setup-for-qrexec-communication) section, which will help you set up a new VM called `sd-dev-proxy`.
-
-Once your proxy VM is set up, follow these steps:
-
-1. Start the server in a docker container on `sd-dev-proxy` by running:
-
-    ```bash
-    NUM_SOURCES=5 LOADDATA_ARGS="--random-file-size 3" make dev
-    ```
-
-2. Delete the cassettes you wish to regenerate or just delete all JSON files by running:
-
-    ```bash
-    rm data/*.json
-    ```
-
-   If you are only adding a new test and not modifying existing ones, you can
-   skip this step, but you still need to remove the authentication setup during
-   cassette generation. Otherwise you will get 403 errors for API endpoints that
-   require a valid token. Remove the setup cassette by running:
-
-   ```bash
-   rm data/setup_method.json
-   ```
-
-   (You can reinstate the unmodified version later.)
-
-3. Make qrexec calls to the server and collect response data. To run all proxy
-   tests:
-
-    ```bash
-    make test TESTS=tests/test_apiproxy.py
-    ```
-
-**Note:** If you get a 403 error it may also be because the test is trying to
-reuse an old TOTP code, so wait for 60 seconds and try again. Some tests alter
-source and conversation data on the server so you should restart the server in
-between test runs.
-
-## Test setup for qrexec communication
-
-If this is the first time you are generating new cassettes that make API calls over qrexec, then you'll need to set up a new VM for running the server and proxy following these steps:
-
-1. Create a new AppVM based on the **debian-10** template called **sd-dev-proxy**.
-2. Install the lastest proxy package:
-
-    ```bash
-    wget https://apt.freedom.press/pool/main/s/securedrop-proxy/<latest-package>.deb
-    dpkg -i <latest-package>.deb
-    ```
-
-3. Create `/home/user/.securedrop_proxy/sd-proxy.yaml` with the following contents (assuming the VM you'll be running the SDK tests from is called **sd-dev**):
-
-    ```
-    host: 127.0.0.1
-    scheme: http
-    port: 8081
-    target_vm: sd-dev
-    dev: False
-    ```
-
-4. Install Docker.
-5. Clone `securedrop` on **sd-dev-proxy** and run the server in a Docker container:
-
-    ```bash
-    git clone https://github.com/freedomofpress/securedrop
-    cd securedrop
-    virtualenv .venv --python=python3
-    source .venv/bin/activate
-    pip install -r securedrop/requirements/python3/develop-requirements.txt
-    NUM_SOURCES=5 make dev
-    ```
-
-6. Open a terminal in **sd-dev** and create `/etc/sd-sdk.conf` with the following contents:
-
-```
-[proxy]
-name=sd-dev-proxy
-```
-
-7. Modify `/etc/qubes-rpc/policy/securedrop.Proxy` in **dom0** by adding the following line to the top of the file so that the sdk tests can make calls to the proxy:
-
-```
-sd-dev sd-dev-proxy allow
-```
-
-**NOTE:** You may want to switch back to the RPC configuration files in their as-provisioned state before a `make test` run in `dom0`, as this and the following change to the RPC policies will break the strict validation of the RPC policies that is one of those tests.
-
-8. Modify `/etc/qubes-rpc/policy/qubes.Filecopy` in **dom0** by adding the following line to the top of the file so that the proxy can send files over qrexec to the sdk:
-
-```
-sd-dev-proxy sd-dev allow
-```
-
-9. Verify qrexec communication between `sd-dev-proxy` and `sd-dev` is set up properly.
-
-    a. Run the server on `sd-dev-proxy` if it isn't already running:
-
-    ```bash
-    NUM_SOURCES=5 make dev
-    ```
-    b. With the main branch of this repo checked out on `sd-dev`, comment out the `@qrexec_datasaver` decorator above the `test_apiproxy.py::TestAPIProxy::setUp` method so that `test_api_auth` makes an actual API call over qrexec.
-    c. Run `test_api_auth`:
-
-    ```bash
-    make test TESTS=tests/test_apiproxy.py::TestAPIProxy::test_api_auth
-    ```
-
-    **Note:** If the test fails, run `journalctl -f` in **dom0** before trying again to see if communication between `sd-dev` and `sd-dev-proxy` is being denied. A successful log looks like this:
-
-    ```
-    Aug 28 15:45:13 dom0 qrexec[1474]: securedrop.Proxy: sd-dev -> sd-dev-proxy: allowed to sd-dev-proxy
-    ```
-
 # Releasing
 
 To make a release, you should:

@@ -18,7 +18,7 @@ def test_config_from_qubesdb():
     qubesdb = MagicMock()
     QubesDB = MagicMock()
     QubesDB.read = MagicMock()
-    QubesDB.read.side_effect = ["foobar", "foobar", "10"]
+    QubesDB.read.side_effect = ["foobar", "foobar", "10", "foobar"]
     qubesdb.QubesDB = MagicMock(return_value=QubesDB)
 
     with patch.dict("sys.modules", qubesdb=qubesdb):