A collection of postprocessing helper functions and examples for Flat Data.
These examples and functions are written in Deno, a new language created by the same founders of Node.js and meant to improve on many aspects of Node.
Note: If you're noticing your scripts failing recently, try updating to
0.0.15
. More info here
When writing a Flat Data Action, you can specify a path to a postprocessing Deno script that can manipulate the data downloaded by Flat even further.
- name: Fetch data
uses: githubocto/flat@v2
with:
http_url: http://api.coindesk.com/v2/bpi/currentprice.json # The endpoint to fetch
downloaded_filename: btc-price.json # The http_url gets saved and renamed in our repository as btc-price.json
postprocess: postprocess.ts # A postprocessing javascript or typescript file written in Deno
This is an example of a postprocessing script. Notice the use of Deno.args[0]
to pass in the path of the downloaded_filename
.
// The Flat Data postprocessing libraries can be found at https://deno.land/x/flat/mod.ts
// Replace 'x' with latest library version
import { readJSON, writeJSON } from 'https://deno.land/x/flat@0.0.x/mod.ts'
const filename = Deno.args[0] // equivalent to writing `const filename = 'btc-price.json'`
const data = await readJSON(filename)
// pluck a specific key off and write it out to a new file
const newfile = `postprocessed_${filename}`
await writeJSON(newfile, data.path.to.something)
Can be found in the examples folder. Once you install Deno you can run these examples with:
deno run -A examples/csv/csv-example.ts
deno run -A examples/csv/arquero-example.ts
deno run -A --unstable examples/image/image-example.ts
deno run -A examples/json/json-example.ts
deno run -A examples/sheets/sheets-example.ts
deno run -A examples/xlsx/xlsx-example.ts
deno run -A --unstable examples/zip/zip-example.ts
Deno can run javascript or typescript files, so you can easily convert any of these examples to javascript and run them in the same way:
deno run -A examples/csv/csv-example.js
While our examples use a Deno file to run postprocessing tasks, you can also use Python as specified in this example: https://github.com/pierrotsmnrd/flat_data_py_example. Thank you @pierrotsmnrd!
You can also use bash as specified in this example: https://github.com/aborruso/flat_data_bash_example. By @aborruso
The Flat Data postprocessing library can be found at: https://deno.land/x/flat/mod.ts
You can import and use these helper functions directly, or treat them as a starting point for writing your own postprocessing scripts.
readCSV(path: string, options?: ParseOptions): Promise<Record<string, unknown>[]>
Args:
- path: path to a local CSV file
- options: options for parsing the CSV file
Usage:
const csv = await readCSV('./path/to/file.csv')
writeCSV(path: string, data: Record<string, unknown>[] | string, options?: Deno.WriteFileOptions)
Args:
- path: path to a local CSV file
- data: string or object array to store
- options: options for writing the CSV file
Usage:
const data = [
{ age: 70, name: 'Rick' },
{ age: 14, name: 'Smith' }
]
await writeCSV('./path/to/file.csv', data)
readTXT(path: string): string
Args:
- path: path to a local TXT file
Usage:
const text = await readTXT('./path/to/file.txt')
writeTXT(path: string, text: string, options?: Deno.WriteFileOptions): void
Args:
- path: path to a local TXT file
- text: text to write to file
- options: options for writing the TXT file
Usage:
await writeTXT('./path/to/file.txt', 'Content for the file')
readJSON(path: string): JSON
Args:
- path: path to a local JSON file
Usage:
const json = await readJSON('./path/to/file.json')
readJSONFromURL(url: string): JSON
Args:
- url: URL to a json file
Usage:
const json = await readJSON('www.url.com/file.json')
writeJSON(path: string, data: any, replacer?: any, space?: string | number): void
Args:
- path: path to a local JSON file
- data: data to store as JSON
- replacer: replacer function that transforms the results or an array of strings and numbers that acts as an approved list for selecting the object properties that will be stringified
- space: adds indentation, white space, and line break characters to the to the JSON text to make it easier to read.
Usage:
const data = { age: 40 }
await writeJSON('./path/to/file.json', data)
await writeJSON('./path/to/file-with-indentation', data, null, 2)
Our library relies on SheetJS, a library for parsing various spreadsheet formats. In addition to a simple readXLSX
function you can access the core xlsx
module by importing it directly.
import { xlsx, readXLSX } from 'https://deno.land/x/flat/mod.ts'
xlsx
provides many more utility functions.
readXLSX(path: string): XLSX.WorkBook
Args:
- path: path to a local XLSX file
Usage:
const workbook = await readXLSX('./path/to/file.xlsx')
const sheetData = workbook.Sheets[workbook.SheetNames[0]]
const csvString = await xlsx.utils.sheet_to_csv(sheetData)
We recommend using a library like imagescript for more advanced image manipulation. See an example here.
readImageFromFile(path: string): Promise<Uint8Array>
Args:
- path: path to a local image file
Usage:
const bytes = await readImageFromFile('./path/to/image.jpeg')
readImageFromURL(url: string): Promise<{ bytes: Uint8Array; name: string; }>
Args:
- url: url string to an image
Usage:
const image = await readImageFromURL('www.url.com/image.jpg')
const bytes = image.bytes
const name = image.name
writeImage(imageBytes: Uint8Array, path: string): void
Args:
- imageBytes: a byte array
- path: path and name to write the image file
Usage:
await writeImage(bytes, './path/to/image.jpeg')
unZipFromFile(
filePath: string,
destinationPath: string | null = "./",
options: any = {},
): Promise<string | false>
Args:
- filePath: a path to a local zip file
- destinationPath: a folder path to unzip the files
- options: option.includeFileName can be true or false
Usage:
const result = await unZipFromFile('./path/to/folder.zip', './unzip/path')
const output = result ? 'File unzipped successfully' : 'Error unzipping'
unZipFromURL(
fileURL: string,
destinationPath: string | null = "./",
options: any = {},
): Promise<string | false>
Args:
- filePath: a path to a local zip file
- destinationPath: a folder path to unzip the files
- options: option.includeFileName can be true or false
Usage:
const result = await unZipFromURL('www.url.com/file.zip', './unzip/path')
const output = result ? 'File unzipped successfully' : 'Error unzipping'
removeFile(path: string): void
Args:
- path: path to a local file to delete
Usage:
await removeFile('/path/to/file.x')
Run all the tests:
deno test -A --unstable tests/*
Run separate tests
deno test -A --unstable tests/csv-test.ts