Generate compile_commands.json for Improved Language Server Support

[StoneLin0708]

Thanks to the LiteX team for this great project!

Issue:

The current LiteX build system does not generate compile_commands.json, limiting functionality of language servers such as clangd, and affecting features like code completion.

Questions:

1. Existing Solutions: Is there an existing solution for integrating language servers more effectively?
2. Interest in Enhancement: Would the community find it useful to have a feature that generates a correct compile_commands.json during the build?

Proposed Solutions:

*Note: Both methods require post-processing to remove target-specific flags (-march=, -mabi=), which are necessary for my clangd setup to function correctly.

I am considering two approaches and would appreciate community input on both:

1. Modify LiteX Builder: Modify litex.soc.integration.builder.Builder._generate_rom_software to perform a --dry-run, capture the commands, and do post-processing*. This method integrates directly into the build process without requiring additional dependencies but may require more maintenance.
2. Use Third-Party Tools: Use third-party tools like Bear to generate a compile database and refine the results through post-processing*. This approach retains the simplicity of using established tools.

Current Implementation:

I’ve implemented a working solution by modifying the LiteX Builder and using it in the build process of my project.

from litex.soc.integration.builder import Builder
builder = Builder(soc, output_dir=build_dir, csr_csv=build_dir / "csr.csv")
# patch _generate_rom_software() and capture the results later
make_compile_commands = patch_litex_builder_for_compile_commands(builder)

def init_mems(**kwargs):
    Path(build_dir.parent / "compile_commands.json").write_text(
        json.dumps(parse_compile_commands(make_compile_commands), indent=4)
    ) # parse and save the captured commands

soc.init_mems = init_mems
builder.build(build_name='mysoc')

Which approach would better suit our community’s needs, or are there other methods you’d suggest? Thanks for any feedback.

[enjoy-digital]

Thanks @StoneLin0708, what's complexity of your patch_litex_builder_for_compile_commands? If simple and useful for other users, we could try to integrate. If complex, we could rely on a third-party tool as the one you are suggesting.

[StoneLin0708]

Thanks for the feedback, @enjoy-digital

The patch_litex_builder_for_compile_commands is straightforward in my setup but might require additional testing and some error handling to ensure it works universally for other users. Here’s the core part of the patch that captures the make commands and takes dry run results for later parsing:

from litex.soc.integration.builder import Builder
import os
import subprocess

def patch_litex_builder_for_compile_commands(builder: Builder):
    make_compile_commands = []
    # most of the code is from litex.soc.integration.builder.Builder._generate_rom_software
    def _generate_rom_software(self: Builder, compile_bios=True):
        for name, src_dir in self.software_packages:
            if name == "bios" and not compile_bios:
                continue
            dst_dir = os.path.join(self.software_dir, name)
            makefile = os.path.join(src_dir, "Makefile")
            if self.compile_software:
                dry_command = ["make", "--dry-run", "V=1", "-C", dst_dir, "-f", makefile]
                make_compile_commands.extend(
                    subprocess.check_output(dry_command).decode('utf-8').splitlines()
                )
                subprocess.check_call(["make", "-C", dst_dir, "-f", makefile])

    builder._generate_rom_software = _generate_rom_software.__get__(builder)
    return make_compile_commands

Additionally, here's the parse_compile_commands function that processes the results to generate the compile commands format:

import re
from typing import List

def parse_compile_commands(make_compile_commands: List[str]) -> List:
    compile_commands = []
    make_cwd = None
    for i in make_compile_commands:
        if i.startswith('make: Entering directory'):
            m_output = re.search(r"'(.+)'", i)
            if m_output:
                make_cwd = m_output.group(1)
            else:
                make_cwd = None
            continue
        if i.startswith('make: Leaving directory'):
            make_cwd = None
            continue
        if not i.startswith('riscv64-unknown-elf-gcc'):
            continue
        # extract output file
        m_output = re.search(r'-o\s+(\S+)', i)
        # extract input file (with/without -c)
        m_input = re.search(r'(-c\s+)?(\S+\.[cS])', i)
        # add to compile_commands if both output and input are found
        if m_output and m_input:
            # remove output from command
            i = i.replace(m_output.group(0), '')
            # remove -march=... -mabi=... if present
            i = re.sub(r'-march=\S+', ' ', i)
            i = re.sub(r'-mabi=\S+', ' ', i)
            i = re.sub(r'\s+', ' ', i)
            compile_commands.append(
                {
                    'output': m_output.group(1),
                    'file': m_input.group(2),
                    'command': i.strip(),
                    'directory': make_cwd
                }
            )
    return compile_commands