Locate Elements on PDF

[juqing27]

@stephanrauh and community - Hi. I'm building an app that can programmatically add annotations to a pdf file, based on ngx-extended-pdf-viewer and another library highKite/pdfAnnotate. Previous discussions can be found here #2090 and here #2117 just for background. Also, to clarify and isolate the problems, the locating issue I describe here does not involve the highKite/pdfAnnotate library - it simply takes coordinates, size, color, etc. as input and draw the annotations. According to my check, it draws the annotations at correct locations on the pdf pages according to the input data. So the focus is how to find something in a pdf web viewer - ngx-extended-pdf-viewer in this case, report the coordinates in viewer and convert that to coordinates on pdf pages.

Please see snapshot below - I'm searching for "framework" as an example.

• The orange boxes are the programmatically generated annotations that are supposed to overlap the instances of "framework". However, they are relatively close but not right. I attached my view-to-page converting function below for further discussion.
• The pink spans are displayed by ngx-extended-pdf-viewer's native search function (with "highlight all" checked). I also turned on the text layer (light gray) for troubleshooting. As Stephan mentioned in the demo page, the text layer "is not perfect" and has a small offset too. But obviously it's closer than the orange boxes.

Questions -

1. I'm curious if the text layer (pink/light gray) can be improved. Asking because other pdf viewers or web browsers (chrome, Bluebeam, etc.) can more precisely highlight the search results these days. Wonder how they managed to achieve that if their searching is also based on the text layer...?
2. Even if not, it's still very helpful if I can get the exact location of current pink spans relative to the pdf page, and then use that to generate annotation (put orange boxes where pink spans are! :).
   - I do have a suspicion that the offset might be due to the difference between pdfPageElement.getBoundingClientRect() and theoretic page size in viewport. I tested with a 8.5in * 11in document and different zoom/viewport scale settings. As shown in the 2nd snapshot, the "theoretic" page width should be 8.5" * 72 DPI * Column B view scale. However, the actual width got from pdfPageElement.getBoundingClientRect().width (Column C) is always 17.6 more, no matter the viewport scale. This could also explain that the more I zoom in (like 400% as against to 50%), the closer the annotations are from the target, because the constant 17.6 becomes relatively insignificant compared to the increased viewport size. Is there a padding/margin that's causing this? However I really have no idea and don't want to mislead others if this is not the reason.

Snapshot 1 - offset of the annotations placed:

Snapshot 2 - chart of actual pdf page width in viewport vs. theoretical pdf page width in viewport:

pdf file used in test:
user-experience.pdf

my test-1.component.html:

<div class="container">
  <div class="content">
    <ngx-extended-pdf-viewer
      #ngxExtendedPdfViewer3
      [src]="pdfSrc"
      [textLayer]="true"
      (textLayerRendered)="highlightWords($event)"
      [page]="0"
      [showHandToolButton]="true"
      [handTool]="false"
      [minifiedJSLibraries]="!isLocalhost"
      (pdfLoaded)="onPDFLoaded()"
      (click)="onPdfClick($event)"
      [(zoom)]="zoom"
    ></ngx-extended-pdf-viewer>
  </div>
  <div class="sidebar">
    <div>Mouse coordinates relative to PDF: {{ mouseCoord }}</div>
    <mat-checkbox class="margin" [(ngModel)]="showTextLayer"
      >show the text layer</mat-checkbox
    >
    <button mat-raised-button color="primary" (click)="onCheckButtonClick()">
      CHECK!
    </button>
  </div>
</div>

test-1.component.ts:

import { Component, ViewEncapsulation } from '@angular/core';
import { FormsModule } from '@angular/forms';
import { SharedUtilsModule } from '../../shared/shared-utils/shared-utils.module';
import { NgxExtendedPdfViewerService, PageRenderedEvent, TextLayerRenderedEvent, pdfDefaultOptions } from 'ngx-extended-pdf-viewer';
import { isLocalhost } from '../common/utilities';
import { AnnotationFactory } from 'annotpdf';
import { SquareAnnotationObj } from 'annotpdf/lib/annotations/circle_square_annotation';

declare var PDFViewerApplication: any; // This line declares the global variable

@Component({
  selector: 'app-test-1',
  standalone: true,
  imports: [
    SharedUtilsModule,
    FormsModule,
  ],
  templateUrl: './test-1.component.html',
  styleUrl: './test-1.component.css',
  encapsulation: ViewEncapsulation.None,
})
export class Test1Component {
  
  public mouseCoord: string = 'tbd';

  private viewScale!:number;

  public zoom: any;

  public imageDataURL: string | undefined = undefined;

  public selectedTabIndex = 0;

  public rawAnnotations: Array<any> | null = null;

  private _showTextLayer = false;

  private alreadyRendered: Array<HTMLSpanElement> = [];

  public isLocalhost = isLocalhost();

  private annotationFactory!: AnnotationFactory;
  pdfSrc : string = '/assets/pdfs/user-experience.pdf';
  private isAnnotationFactoryInitialized: boolean = false;

  constructor(private pdfViewerService: NgxExtendedPdfViewerService) {
    pdfDefaultOptions.textLayerMode = 1;
    this.zoom = 100;
  }

  public onCheckButtonClick(): void {
    const annotationPromises = this.alreadyRendered.map(span => this.doMarkSearchTermsInSpan_1(span));

    Promise.all(annotationPromises).then(() => {
      this.downloadAnnotatedPdf();
      const updatedPdfBlob = this.annotationFactory.write();
      const blobUrl = URL.createObjectURL(new Blob([updatedPdfBlob]));
      this.pdfSrc = blobUrl;
    });
  }

  //THIS IS THE FUNCTION IN QUESTION - 
  //HOW TO BETTER CONVERT THE VIEWPORT LOCATION TO PAGE LOCATION?
  public doMarkSearchTermsInSpan_1(span: HTMLSpanElement): Promise<void> {
    return new Promise((resolve) => {
            const withMarks = span.innerText
                .split(' ')
                .map((t) => this.markOneSearchTerm(t))
                .join(' ');
            span.innerHTML = withMarks;
  
            const pageNumber = this.findPageNumber(span);
            const pdfPageElement = document.querySelector(`.pdfViewer .page[data-page-number="${pageNumber}"]`) as HTMLElement;
            const searchTermsPromises = Array.from(span.querySelectorAll('.search-term')).map((div: Element) => {
           
            //get element's location IN VIEWPORT  
            const rect = div.getBoundingClientRect();
            if (pdfPageElement) {
                const pdfViewer = PDFViewerApplication.pdfViewer;
                const currentPage = pdfViewer._pages[0];
                const viewport = currentPage.viewport;
                const scale = viewport.scale;
                this.viewScale = scale;
                const pageRect = pdfPageElement.getBoundingClientRect();

                //convert to location ON PDF PAGE
                rect.x -= (pageRect.left);
                rect.y -= (pageRect.top);     
                const scaledLeft = rect.x / scale;
                const scaledTop = rect.y / scale;
                const scaledWidth = rect.width / scale;
                const scaledHeight = rect.height / scale;
  
                //pdfAnnotation use bottom left as origin
                const adjustedTop = viewport.viewBox[3] - scaledTop - scaledHeight; // Subtract height to get the bottom coordinate

                //for debugging
                const annotContent = `x: ${scaledLeft.toFixed(2)}, y: ${scaledTop.toFixed(2)}, w: ${scaledWidth.toFixed(2)}, h: ${scaledHeight.toFixed(2)}`;                
                console.log(`***** page bouding rect width: ${pageRect.width.toFixed(2)} ` + 
                            `| Scale factor: ${scale.toFixed(2)} | Zoom: ${this.zoom}`);  
  
                return this.addSquareAnnotation_1(scaledLeft, adjustedTop, scaledWidth, scaledHeight, pageNumber, `Coordinates of ${div.textContent}`, annotContent);
              }
      
            });
  
            // Resolve the outer Promise once all inner Promises are resolved
            Promise.all(searchTermsPromises).then(() => resolve());
        
    });
  }

  private markOneSearchTerm(word: string): string {
    if (word.toString() == "framework" || word.toString() == "Framework" ){
      return `<div class="search-term">${word}</div>`;
    }    
    return word;
  }  

  addSquareAnnotation_1(scaledLeft: number, adjustedTop: number, scaledWidth: number, scaledHeight: number, pageN: number, coord: string, annotContent: string) {
  if (!this.annotationFactory) {
    console.error("Annotation factory is not initialized yet.");
    return;
  } 
  // Prepare the annotation properties
  const color = { r: 0, g: 0, b: 0 };
  const fill = {
    r: Math.floor(Math.random() * 256),
    g: Math.floor(Math.random() * 256),
    b: Math.floor(Math.random() * 256),
    a: 0.5 // Semi-transparent
  };
  const author = "ZGF Design Tech";  
console.log("page: ", pageN-1)
  // Create the annotation with adjusted coordinates
  let squareAnnot: SquareAnnotationObj =  this.annotationFactory.createSquareAnnotation(
    {
    page: pageN-1, // Adjust for zero-based index if needed
    rect: [scaledLeft, adjustedTop, scaledLeft + scaledWidth, adjustedTop + scaledHeight],
    contents: annotContent,
    author: coord,
    color: color,
    fill: {r:200, g:100, b:0},
    opacity: 0.5,
    subject: "special note",
    intent: "assist"}
  );  
  console.log("***Annotation added:", squareAnnot.id);
  }

  onPDFLoaded() {
    // Check if the PDF document and the PDFViewerApplication are available
    if (PDFViewerApplication?.pdfDocument) {
      console.log("pdfViewerViewport: ", PDFViewerApplication.pdfViewer._pages[0].viewport);

      // Check if the AnnotationFactory has already been initialized
      if (!this.isAnnotationFactoryInitialized) {
        PDFViewerApplication.pdfDocument.getData().then((data: Uint8Array) => {
          this.annotationFactory = new AnnotationFactory(data);
          this.isAnnotationFactoryInitialized = true; // Set the flag to true
        }).catch((error: any) => {
          console.error('Error getting PDF data:', error);
        });
      }
    } else {
      console.warn('PDF document is not available yet.');
    }
  }
  
  //for debugging - get coordinates of mouse click 
  public onPdfClick(event: MouseEvent): void {
    // Get the page element inside the PDF viewer
    const pageElement = document.querySelector('.pdfViewer .page') as HTMLElement;
    if (pageElement) {
      const pageRect = pageElement.getBoundingClientRect();
      // Calculate coordinates relative to the PDF page
      let x = event.clientX - pageRect.left;
      let y = event.clientY - pageRect.top;
  
      // Access the PDF.js viewer's viewport for the current page
      const pdfViewer = PDFViewerApplication.pdfViewer;
      const currentPage = pdfViewer._pages[pdfViewer.currentPageNumber - 1];
      const viewport = currentPage.viewport;
  
      // Use the viewport's scale to convert to PDF point coordinates
      x /= viewport.scale;
      y /= viewport.scale;
  
      // Convert from top-left origin to bottom-left origin used by PDF coordinate system
      y = viewport.viewBox[3] - y; // The height of the PDF page in points
  
      // Store the coordinates
      this.mouseCoord = "x raw: " + event.clientX + " | y raw: " + event.clientY + " | x: " + x.toString() + " | y: " + y.toString();
      console.log("Mouse coordinates relative to PDF:", this.mouseCoord);
    }
  } 

  private findPageNumber(element: Element): number {
    let parent = element.parentElement;
    while (parent && !parent.classList.contains('page')) {
      parent = parent.parentElement;
    }
    if (parent && parent.dataset['pageNumber']) {
      return parseInt(parent.dataset['pageNumber'], 10);
    }
    return 0; // or return a default value or throw an error if the page number is crucial
  }

  public get showTextLayer(): boolean {
    return this._showTextLayer;
  }

  public set showTextLayer(layer: boolean) {
    this._showTextLayer = layer;
    const divs = document.getElementsByClassName('textLayer');
    for (let i = 0; i < divs.length; i++) {
      const div = divs.item(i);
      if (div) {
        if (layer) {
          div.classList.add('show-text-layer');
        } else {
          div.classList.remove('show-text-layer');
        }
      }
    }
  }

  public highlightWords(event: TextLayerRenderedEvent): void {
    (event as any).source.textLayer.textDivs.forEach((span: HTMLSpanElement) => {
      this.alreadyRendered.push(span);
    });

    if (this.showTextLayer) {
      (event as any).source.textLayer.textDivs.forEach((span: HTMLSpanElement) => {
        span.classList.add('box');
      });
    }
  }

  async downloadAnnotatedPdf() {
  try {

    function getCurrentFormattedTime() {
      const now = new Date();
      const hours = String(now.getHours()).padStart(2, '0');
      const minutes = String(now.getMinutes()).padStart(2, '0');
      const seconds = String(now.getSeconds()).padStart(2, '0');
      return `${hours}${minutes}${seconds}`;
    }

    const currentTime = getCurrentFormattedTime();

    const filename = `output_${currentTime}_vs_${this.viewScale.toFixed(2)}.pdf`;
    this.annotationFactory.download(filename);
  } catch (error) {
    console.error('Error in downloading the annotated PDF:', error);
  }
  }
}

test-1.component.css:

.box {
    border: 2px solid red;
    border-radius: 8px;
  }
  
  ngx-extended-pdf-viewer div.show-text-layer > span {
    color: unset;
  }
  
  .long-word {
    background-color: blueviolet;
    display: inline;
  }

  .search-term {
    background-color: gold;
    display: inline;
  }

[Hely0n]

I'm in a similar situation. I process PDFs with PDFbox in the Backend, which returns TextPositions as coordinates but I can't find a way to draw those rectangles correctly in the frontend, as the coordinates in the PDFviewer don't match the ones from PDFbox, even if I try to calculate with the scales provided.

[stephanrauh]

As far as I know (and I may be wrong), you can get the size of the page using PDFViewerApplication.pdfViewer._pages[0].pdfPage.view (plus some type casting to make TypeScript happy).

If you want to convert PDF coordinates to HTML canvas coordinates, have a look at this code snippet: #2132 (comment).

[jcurtisKSM]

The two key parts to that code in #2132 are:

1. Getting the zoom from the PDF Viewer

this.currentZoomFactor = zoom;
const scale = this.zoom;

More on the zoom factor:

• https://github.com/stephanrauh/extended-pdf-viewer-showcase/blob/cd2abf56daeaf80e2fa8c028e8ef1cc4dd0335ba/src/app/extended-pdf-viewer/zoom/zoom.component.html#L75
• https://github.com/stephanrauh/extended-pdf-viewer-showcase/blob/cd2abf56daeaf80e2fa8c028e8ef1cc4dd0335ba/src/app/extended-pdf-viewer/zoom/zoom.component.ts

2. Setting the DPI and Scale

const dpi = 96; // Standard screen DPI
const scaledDpi = dpi * scale;

The points array that the functions I'm given start out as "inches" in a decimal format. I then take those and convert them into pixels by taking the standard DPI of a screen (96) and the current scale (currentZoomFactor).

[stephanrauh]

Sorry for responding late - for some reason unknown, your question slipped my attention.

The constant offset of 17.6 pixels (?) sounds like a CSS issue. Maybe the page is smaller than you expect it to be because there's a padding or margin reducing the available space. Can you check this? Does it get better if you set [showBorders]="false"?

As for improving the alignment of the text layer: I'm afraid I'm not the expert here. You're right, in theory it should be possible to generate a perfect text layer. The PDF viewer renders the page correctly, so it obviously knows the precise coordinates. It just fails to tell the HTML layer about it. I suspect that's difficult because the canvas is rendered in the worker thread, while the text layer is rendered in the main thread, but it's just guesswork. The only thing I know for sure is that the text layer has to use one of the fonts the browser knows, and that's usually not the font the PDF file uses.

If you'd like to try to improve the alignment of the text layer yourself, I'd point you to @ Timvandermeij. I'm not sure whether he's the expert, but he knows who the expert is, so he's probably a good contact. If you're ready to submit a pull request, just tell me, and I'll introduce you to Tim.