The assessment tool

ASys performs three different assessment phases:
  1. Compilation: To identify compilation errors.
  2. Analysis: To check whether the specified requirements are properly implemented.
  3. Testing: To identify runtime errors.

In phases 1 and 3, ASys is not limited to a set of predefined tests to check some provided finite input-output pairs. Contrarily, ASys can generate any arbitrary number of test cases. For this, the system inputs the solution of the exam (in Java), and iteratively generates them. Then, tests are used with a battery of Java exams, so that, the output of exams is checked against the output of the compiled solution. Tests implemented manually can also be used in the testing phase.

Phase 2 (the analysis phase) includes a Java metaprogramming library with assessment functions, and a DSL on top of the library for the specification of assessment templates. Because the DSL is compliant with Java, it is directly executable by an assessment engine that can automatically verify the properties in the template against a given exam.