Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bus: Expose shared memory on Unix systems #3219

Closed
wants to merge 1 commit into from

Conversation

bemug
Copy link
Contributor

@bemug bemug commented May 30, 2024

On Windows, the shared memory handle is closed on Destroy() and therefore is accessible at runtime. On Unixes the shared memory was unlinked right after its creation, effectively making the shared memory unavailable from the outside world.
This commit brings feature parity to Unix systems. Allows the shared memory to be accessed at runtime from an external process for RAM inspection.

@stenzek
Copy link
Owner

stenzek commented May 30, 2024

Sorry, I'm fundamentally against this.

  1. If DS crashes, then it leaves a file hanging around.
  2. It doesn't handle invalidating the recompiler code cache when you overwrite previously compiled blocks.

This is why I implemented PINE, which executes in the context of the CPU thread, and safely invalidates the rec as needed.

(I know Windows has this "functionality", but I'm planning on removing that as well, since it has the same safety problems, once consumers have a chance to switch over to PINE)

On Windows, the shared memory handle is closed on Destroy() and
therefore is accessible at runtime. On Unixes the shared memory was
unlinked right after its creation, effectively making the shared memory
unavailable from the outside world.
This commit brings feature parity to Unix systems.
Allows the shared memory to be accessed at runtime from an external
process for RAM inspection.
@bemug
Copy link
Contributor Author

bemug commented May 30, 2024

I was not aware of PINE's existence. If you tell me it can achieve the same things without these limitations then that sounds pretty good. Thank you.

Do you have any example code available somewhere to use PINE ?

@stenzek
Copy link
Owner

stenzek commented May 30, 2024

It's fairly straightforward, you just send over a 4-byte packet length, 1-byte command type, and variable args over a unix domain socket in $XDG_RUNTIME_DIR/duckstation.sock (on Linux), or 127.0.0.1:28001 (on Windows).

So e.g. to write 0x12345678 to address 0x80001000, you would send:

uint32(13) // 0D 00 00 00 - command length
uint8(6) // 06 - command type, MsgWrite32
uint32(0x80001000) // 00 10 00 80 - memory address
uint32(0x12345678) // 78 56 34 12 - data

and get back:

uint32(0) // 00 00 00 00 - reply length
uint8(0) // 00 - success (fail is 0xFF)

You don't need to handle the replies, can just ignore them, but you should read them out, otherwise your socket receive buffer will eventually fill, and the DS side will stop processing commands.

Full command list:

enum IPCCommand : u8
{
MsgRead8 = 0, /**< Read 8 bit value to memory. */
MsgRead16 = 1, /**< Read 16 bit value to memory. */
MsgRead32 = 2, /**< Read 32 bit value to memory. */
MsgRead64 = 3, /**< Read 64 bit value to memory. */
MsgWrite8 = 4, /**< Write 8 bit value to memory. */
MsgWrite16 = 5, /**< Write 16 bit value to memory. */
MsgWrite32 = 6, /**< Write 32 bit value to memory. */
MsgWrite64 = 7, /**< Write 64 bit value to memory. */
MsgVersion = 8, /**< Returns PCSX2 version. */
MsgSaveState = 9, /**< Saves a savestate. */
MsgLoadState = 0xA, /**< Loads a savestate. */
MsgTitle = 0xB, /**< Returns the game title. */
MsgID = 0xC, /**< Returns the game ID. */
MsgUUID = 0xD, /**< Returns the game UUID. */
MsgGameVersion = 0xE, /**< Returns the game verion. */
MsgStatus = 0xF, /**< Returns the emulator status. */
MsgUnimplemented = 0xFF /**< Unimplemented IPC message. */
};

bool PINEServer::PINESocket::HandleCommand(IPCCommand command, BinarySpanReader rdbuf)
{
// example IPC messages: MsgRead/Write
// refer to the client doc for more info on the format
// IPC Message event (1 byte)
// | Memory address (4 byte)
// | | argument (VLE)
// | | |
// format: XX YY YY YY YY ZZ ZZ ZZ ZZ
// reply code: 00 = OK, FF = NOT OK
// | return value (VLE)
// | |
// reply: XX ZZ ZZ ZZ ZZ
BinarySpanWriter reply;
switch (command)
{
case MsgRead8:
{
if (!rdbuf.CheckRemaining(sizeof(PhysicalMemoryAddress)) || !System::IsValid())
return SendErrorReply();
else if (!BeginReply(reply, sizeof(u8))) [[unlikely]]
return false;
const PhysicalMemoryAddress addr = rdbuf.ReadU32();
u8 res = 0;
reply << (CPU::SafeReadMemoryByte(addr, &res) ? IPC_OK : IPC_FAIL);
reply << res;
return EndReply(reply);
}
case MsgRead16:
{
if (!rdbuf.CheckRemaining(sizeof(PhysicalMemoryAddress)) || !System::IsValid())
return SendErrorReply();
else if (!BeginReply(reply, sizeof(u16))) [[unlikely]]
return false;
const PhysicalMemoryAddress addr = rdbuf.ReadU32();
u16 res = 0;
reply << (CPU::SafeReadMemoryHalfWord(addr, &res) ? IPC_OK : IPC_FAIL);
reply << res;
return EndReply(reply);
}
case MsgRead32:
{
if (!rdbuf.CheckRemaining(sizeof(PhysicalMemoryAddress)) || !System::IsValid())
return SendErrorReply();
else if (!BeginReply(reply, sizeof(u32))) [[unlikely]]
return false;
const PhysicalMemoryAddress addr = rdbuf.ReadU32();
u32 res = 0;
reply << (CPU::SafeReadMemoryWord(addr, &res) ? IPC_OK : IPC_FAIL);
reply << res;
return EndReply(reply);
}
case MsgRead64:
{
if (!rdbuf.CheckRemaining(sizeof(PhysicalMemoryAddress)) || !System::IsValid())
return SendErrorReply();
else if (!BeginReply(reply, sizeof(u64))) [[unlikely]]
return false;
const PhysicalMemoryAddress addr = rdbuf.ReadU32();
u32 res_low = 0, res_high = 0;
reply << ((!CPU::SafeReadMemoryWord(addr, &res_low) || !CPU::SafeReadMemoryWord(addr + sizeof(u32), &res_high)) ?
IPC_FAIL :
IPC_OK);
reply << ((ZeroExtend64(res_high) << 32) | ZeroExtend64(res_low));
return EndReply(reply);
}
case MsgWrite8:
{
// Don't do the actual write until we have space for the response, otherwise we might do it twice when we come
// back around.
if (!rdbuf.CheckRemaining(sizeof(PhysicalMemoryAddress) + sizeof(u8)) || !System::IsValid())
return SendErrorReply();
else if (!BeginReply(reply, 0)) [[unlikely]]
return false;
const PhysicalMemoryAddress addr = rdbuf.ReadU32();
const u8 value = rdbuf.ReadU8();
reply << (CPU::SafeWriteMemoryByte(addr, value) ? IPC_OK : IPC_FAIL);
return EndReply(reply);
}
case MsgWrite16:
{
if (!rdbuf.CheckRemaining(sizeof(PhysicalMemoryAddress) + sizeof(u16)) || !System::IsValid())
return SendErrorReply();
else if (!BeginReply(reply, 0)) [[unlikely]]
return false;
const PhysicalMemoryAddress addr = rdbuf.ReadU32();
const u16 value = rdbuf.ReadU16();
reply << (CPU::SafeWriteMemoryHalfWord(addr, value) ? IPC_OK : IPC_FAIL);
return EndReply(reply);
}
case MsgWrite32:
{
if (!rdbuf.CheckRemaining(sizeof(PhysicalMemoryAddress) + sizeof(u32)) || !System::IsValid())
return SendErrorReply();
else if (!BeginReply(reply, 0)) [[unlikely]]
return false;
const PhysicalMemoryAddress addr = rdbuf.ReadU32();
const u32 value = rdbuf.ReadU32();
reply << (CPU::SafeWriteMemoryWord(addr, value) ? IPC_OK : IPC_FAIL);
return EndReply(reply);
}
case MsgWrite64:
{
if (!rdbuf.CheckRemaining(sizeof(PhysicalMemoryAddress) + sizeof(u32)) || !System::IsValid())
return SendErrorReply();
else if (!BeginReply(reply, 0)) [[unlikely]]
return false;
const PhysicalMemoryAddress addr = rdbuf.ReadU32();
const u64 value = rdbuf.ReadU64();
reply << ((!CPU::SafeWriteMemoryWord(addr, Truncate32(value)) ||
!CPU::SafeWriteMemoryWord(addr + sizeof(u32), Truncate32(value >> 32))) ?
IPC_FAIL :
IPC_OK);
return EndReply(reply);
}
case MsgVersion:
{
const TinyString version = TinyString::from_format("DuckStation {}", g_scm_tag_str);
if (!BeginReply(reply, version.length() + 1)) [[unlikely]]
return false;
reply << IPC_OK << version;
return EndReply(reply);
}
case MsgSaveState:
{
if (!rdbuf.CheckRemaining(sizeof(u8)) || !System::IsValid())
return SendErrorReply();
const std::string& serial = System::GetGameSerial();
if (!serial.empty())
return SendErrorReply();
if (!BeginReply(reply, 0)) [[unlikely]]
return false;
std::string state_filename = System::GetGameSaveStateFileName(serial, rdbuf.ReadU8());
Host::RunOnCPUThread([state_filename = std::move(state_filename)] {
Error error;
if (!System::SaveState(state_filename.c_str(), &error, false))
ERROR_LOG("PINE: Save state failed: {}", error.GetDescription());
});
reply << IPC_OK;
return EndReply(reply);
}
case MsgLoadState:
{
if (!rdbuf.CheckRemaining(sizeof(u8)) || !System::IsValid())
return SendErrorReply();
const std::string& serial = System::GetGameSerial();
if (!serial.empty())
return SendErrorReply();
std::string state_filename = System::GetGameSaveStateFileName(serial, rdbuf.ReadU8());
if (!FileSystem::FileExists(state_filename.c_str()))
return SendErrorReply();
if (!BeginReply(reply, 0)) [[unlikely]]
return false;
Host::RunOnCPUThread([state_filename = std::move(state_filename)] {
Error error;
if (!System::LoadState(state_filename.c_str(), &error))
ERROR_LOG("PINE: Load state failed: {}", error.GetDescription());
});
reply << IPC_OK;
return EndReply(reply);
}
case MsgTitle:
{
if (!System::IsValid())
return SendErrorReply();
const std::string& name = System::GetGameTitle();
if (!BeginReply(reply, name.length() + 1)) [[unlikely]]
return false;
reply << IPC_OK << name;
return EndReply(reply);
}
case MsgID:
{
if (!System::IsValid())
return SendErrorReply();
const std::string& serial = System::GetGameSerial();
if (!BeginReply(reply, serial.length() + 1)) [[unlikely]]
return false;
reply << IPC_OK << serial;
return EndReply(reply);
}
case MsgUUID:
{
if (!System::IsValid())
return SendErrorReply();
const TinyString crc = TinyString::from_format("{:016x}", System::GetGameHash());
if (!BeginReply(reply, crc.length() + 1)) [[unlikely]]
return false;
reply << IPC_OK << crc;
return EndReply(reply);
}
case MsgGameVersion:
{
ERROR_LOG("PINE: MsgGameVersion not supported.");
return SendErrorReply();
}
case MsgStatus:
{
EmuStatus status;
switch (System::GetState())
{
case System::State::Running:
status = EmuStatus::Running;
break;
case System::State::Paused:
status = EmuStatus::Paused;
break;
default:
status = EmuStatus::Shutdown;
break;
}
if (!BeginReply(reply, sizeof(u32))) [[unlikely]]
return false;
reply << IPC_OK << static_cast<u32>(status);
return EndReply(reply);
}
default:
{
ERROR_LOG("PINE: Unhandled IPC command {:02X}", static_cast<u8>(command));
return SendErrorReply();
}
}
}

This is the test script I used when writing the implementation:

import socket
import binascii
import platform
import os
   
def get_socket():
    if (platform.system() == "Windows"):
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.connect(("127.0.0.1", 28011))
    else:
        sdir = os.getenv("XDG_RUNTIME_DIR")
        if sdir is None:
            sdir = os.getenv("TEMPDIR", "/tmp")
        sfilename = f"{sdir}/duckstation.sock"
        s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
        s.connect(sfilename)
    return s
    

def print_hex_ascii(data):
    def to_printable_ascii(byte):
        return chr(byte) if 32 <= byte <= 126 else '.'

    for i in range(0, len(data), 8):
        chunk = data[i:i+8]
        hex_part = ' '.join(f'{byte:02x}' for byte in chunk)
        ascii_part = ''.join(to_printable_ascii(byte) for byte in chunk)
        print(f"{hex_part:<23} {ascii_part}")

        
def test_query(data):
    s = None    
    try:
        s = get_socket()
        for i in range(4):
            s.sendall(data)
            print(f"Sent data: {binascii.hexlify(data).decode('utf-8')}")
            response = s.recv(4096)
            print("Response:")
            print_hex_ascii(response)
    
    except Exception as e:
        print(f"An error occurred: {e}")
    
    finally:
        if s is not None:
            s.close()

        
if __name__ == "__main__":
    data = b"\x05\x00\x00\x00\x08" * 1
    #data = b"\x09\x00\x00\x00\x02\x00\x00\x00\x00" * 4    
    test_query(data)

@stenzek stenzek force-pushed the master branch 4 times, most recently from 1c8d966 to 6021e43 Compare June 23, 2024 14:55
@stenzek stenzek force-pushed the master branch 3 times, most recently from dd53df7 to e0509eb Compare July 3, 2024 05:54
@stenzek stenzek force-pushed the master branch 5 times, most recently from 3b1521a to fae6b7a Compare July 14, 2024 11:32
@stenzek
Copy link
Owner

stenzek commented Aug 4, 2024

I've implemented this in 02fbfae, but with a number of changes:

  1. The option is opt-out, i.e. by default an anonymous mapping gets created. This is to reduce the risk of leaks if DuckStation crashes, and can't clean up after itself. Or, if you are debugging DuckStation, and don't let it exit itself.
  2. The crash handler makes a last-ditch effort to unlink the shared memory object before dumping the stack. This way if DS does crash, we won't leak ~100MB of memory each time.

Please note that as I've stated above, this is not safe for use when manipulating MIPS code externally. There is no page protection on the external mapping, so it will not invalidate JIT blocks, and you'll likely be executing stale code. It is only safe for manipulating/writing data.

@stenzek stenzek closed this Aug 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants