-
Notifications
You must be signed in to change notification settings - Fork 287
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cue test
support
#209
Comments
Original reply by @rudolph9 in cuelang/cue#209 (comment) PLEASE NOTE: I am in the middle of reworking ipcf/t to align nicer with the way cuelang tests naturally get written. The current setup is a little cumbersome. I briefly introduced list support for defining asserts but reverted it as I found that use of disjunctions made I am also in the process of moving away from the rspec describe syntax. I've found it to be combersome and out of place in the context of cuelang. I'm working toward an an api similar to node-tap specifically the |
Original reply by @mpvl in cuelang/cue#209 (comment) Here is my take.
Some of this should be provided natively by the cue tool, other parts can be provided by a library. In CUE we can step it up a notch using CUE
Anyway, let's start simple by specifying a data representation that is future proof. I was thinking more along the line of allowing hierarchical tests where you assign a value to a $test field to trigger a certain test, such as:
and have the testing framework automatically expand this by filling out the test results and running trim as follows:
or something like that. Note that this is a fairly primitive (Go style) and would allow other frameworks to be mapped on top. Not user whether you would need to import a testing package or not. A fuzzer could even automatically populate the test cases, either statically or dynamically. For instance (rough sketch):
or something like
could tell the test tool to automatically create instances of the enclosing template. This may be useful for testing roundtrips between converters for different APIs. Testing those then becomes a matter of I used the Fuzzing may generate large tests sets and it may make sense to either not write them out (but reproduce them on the fly), or put them in a separate file or Many open questions remain. One I don't have a clear picture of is how to represent tool tests, for instance. |
Original reply by @mpvl in cuelang/cue#209 (comment) I think a good first question is, what kind of things do people want to test? |
Original reply by @rudolph9 in cuelang/cue#209 (comment)
It would be good to get others feedback than mine since my use-case for cue is somewhat unique but the kinds of things I've been testing are mostly just sanity checks. Does a regex do what I expect it to, is a closed struct actually a closed struct, is a number actually a number etc. The nature of cue likely doesn't warrant exotic test suites but good to have a mechanism to do some basic checks. |
Original reply by @rudolph9 in cuelang/cue#209 (comment) @mpvl Here is my latest update to It makes for a pretty clean workflow and is similar to the feel of golang testing:
I didn't use the Although, I could see using |
Original reply by @xinau in cuelang/cue#209 (comment) I like the idea of having a "simple" testing setup as @mpvl mentioned. My test use cases are similar to those of @rudolph9. But I think a more powerful approach to solve this problem is to implement a framework similar to haskells quickcheck (not sure if this falls into the fuzzer category) instead of describing valid and invalid cases. |
Signed-off-by: Paul Jolly <paul@myitcv.io>
any updates on |
We are experimenting with different best practices on how a test framework should look like. Details. We like the autoupdating and don't want to give that up. But this shows that there is merit to having some kind of invariants specified which each test that are not auto-updated. We want One CL that may point in that general direction, and another experiment we are conducting, is the refactoring of the builtin tests in: https://review.gerrithub.io/c/cue-lang/cue/+/556198. This CL organizes tests so that they are easier to write. But more importantly, it generates test output where input and output are collated together. A common nuisance in the current approach to generated test output is that one needs to jump back and forth to compare test input and output. This mitigates that issue considerably. Moreover, one can see how this approach is amenable to invariant injection. |
Originally opened by @rudolph9 in cuelang/cue#209
Cuelang has reserved
*_test.cue
similar to golang but does not yet have support for actually running tests.Unless there is a good reason to follow a different path it seems reasonable to model cuelang testing after golang testing
Ideally we would "eat our own dog food" for testing and not create new builtin function but rather express the logic of the test using cuelang.
An example of a testing package is https://github.com/ipcf/t.
The text was updated successfully, but these errors were encountered: