Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/digest report 231212 #365

Merged
merged 2 commits into from
Dec 12, 2023
Merged

Feature/digest report 231212 #365

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c77908a
add report digest metadata, redo color scheme add concertina
leondz Dec 12, 2023
c4aa461
add garak uri, make config info collapsed
leondz Dec 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 29 additions & 3 deletions analyze/report_digest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/usr/bin/env python
"""Prints a report given a garak report.jsonl"""

from collections import defaultdict
import json
import sqlite3
import sys
Expand All @@ -15,8 +16,7 @@
module_template = templateEnv.get_template("analyze/templates/digest_module.jinja")
probe_template = templateEnv.get_template("analyze/templates/digest_probe.jinja")
detector_template = templateEnv.get_template("analyze/templates/digest_detector.jinja")

digest_content = header_template.render()
end_module = templateEnv.get_template("analyze/templates/end_module.jinja")


def map_score(score):
Expand All @@ -32,12 +32,36 @@ def map_score(score):
return 4


report_path = sys.argv[1]


evals = []
with open(sys.argv[1], "r") as reportfile:
setup = defaultdict(str)
with open(report_path, "r", encoding="utf-8") as reportfile:
for line in reportfile:
record = json.loads(line.strip())
if record["entry_type"] == "eval":
evals.append(record)
elif record["entry_type"] == "init":
garak_version = record["garak_version"]
start_time = record["start_time"]
run_uuid = record["run"]
elif record["entry_type"] == "start_run setup":
setup = record


digest_content = header_template.render(
{
"reportfile": report_path.split("/")[-1],
"garak_version": garak_version,
"start_time": start_time,
"run_uuid": run_uuid,
"setup": repr(setup),
"probespec": setup["plugins.probe_spec"],
"model_type": setup["plugins.model_type"],
"model_name": setup["plugins.model_name"],
}
)

conn = sqlite3.connect(":memory:")
cursor = conn.cursor()
Expand Down Expand Up @@ -119,6 +143,8 @@ def map_score(score):
)
# print(f"\t\tdetector: {detector} - {score:.1f}%")

digest_content += end_module.render()

conn.close()

digest_content += footer_template.render()
Expand Down
24 changes: 24 additions & 0 deletions analyze/templates/digest_footer.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,27 @@

<script>
var acc = document.getElementsByClassName("accordion");
var i;

for (i = 0; i < acc.length; i++) {
acc[i].addEventListener("click", function() {
/* Toggle between adding and removing the "active" class,
to highlight the button that controls the panel */
this.classList.toggle("active");

/* Toggle between hiding and showing the active panel */
var panel = this.nextElementSibling;
if (panel.style.display === "block") {
panel.style.display = "none";
} else {
panel.style.display = "block";
}
});
}</script>
</body>

<p style="height:400px; padding-top: 100px">
generated with <a href="https://garak.ai">garak</a>
</p>

</html>
75 changes: 56 additions & 19 deletions analyze/templates/digest_header.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,27 +1,64 @@
<!doctype html>
<!DOCTYPE html>
<html lang="en">

<head>
<meta charset="UTF-8" />
<title>garak report</title>
<meta name="description" content="" />
<style>
.defcon1 {background-color: #600; text-color: #000}
.defcon2 {background-color: #f00; text-color: #fff}
.defcon3 {background-color: #fb0; text-color: #fff}
.defcon4 {background-color: #ff0; text-color: #fff}
.defcon5 {background-color: #0f0; text-color: #fff}
h2 {padding-left: 20pt}
h3 {padding-left: 40pt}
h4 {padding-left: 60pt}
</style>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta charset="UTF-8" />
<style>
body {font-family: sans-serif}
.defcon1 {background-color: #DAB894; text-color: #000}
.defcon2 {background-color: #FFCAAF; text-color: #fff}
.defcon3 {background-color: #F1FFC4; text-color: #fff}
.defcon4 {background-color: #C6E2E9; text-color: #fff}
.defcon5 {background-color: #A7BED3; text-color: #fff}
h2 {padding-left: 20pt}
h3 {padding-left: 40pt}
h4 {padding-left: 60pt}
h2,h3,h4 {padding-top: 10px; padding-bottom: 10px}

/* Style the buttons that are used to open and close the accordion panel */
.accordion {
// background-color: #eee;
color: #444;
cursor: pointer;
padding: 18px;
width: 100%;
text-align: left;
border: none;
outline: none;
transition: 0.4s;
margin: 1pt;
}

/* Add a background color to the button if it is clicked on (add the .active class with JS), and when you move the mouse over it (hover) */
//.active, .accordion:hover {
// background-color: #ccc;
//}

/* Style the accordion panel. Note: hidden by default */
.panel {
padding: 0 18px;
background-color: white;
display: none;
overflow: hidden;
}
</style>
<title>garak report: {{reportfile}}</title>
<meta name="description" content="" />
</head>

<body>

<h1>garak run</h1>
<!-- <p>generator</p>
<p>generator config</p>
<p>run config</p>
<p>probe list</p> -->
<h1>garak run: {{reportfile}}</h1>
<button class="accordion">⚙️ view config</button>
<div style="border:solid black 1px; padding: 5px; margin: 5px" class="panel">
<h2>config</h2>
<p>filename: {{reportfile}}</p>
<p>garak version: {{garak_version}}</p>
<p>generator: {{model_type}}.{{model_name}}</p>
<p>started at: {{start_time}}</p>
<p>run config: {{setup}}</p>
<p>probe spec: {{probespec}}</p>
</div>

<h2>results</h2>
6 changes: 4 additions & 2 deletions analyze/templates/digest_module.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
<h2 class="defcon{{severity}}">module {{ module }}: {{ module_score }}</h2>

<button class="defcon{{severity}} accordion">➡️ module {{ module }}: {{ module_score }}</button>
<div class="panel">
{%if module_score != "100.0%"%}
<p>{{module}} scored the system a {{ module_score }} pass rate.</p>
<p>Probes in the <a href="https://reference.garak.ai/en/latest/garak.probes.{{module}}.html" target="_new">{{module}}</a> module scored the system a {{ module_score }} pass rate.</p>
{%endif%}
1 change: 1 addition & 0 deletions analyze/templates/end_module.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
</div>
Loading