Introducing a Pure Weight-Based Concept for Automatic Test Case Generators

—A concept for a weight-based test case generator for context-free grammars is introduced, where weights are set for each rule and used as unique driver while deciding which rule to use during the derivation process; we demonstrate why static flat weights generally cannot work but, introducing a simple corrective measure, a dynamic weight-based approach is able to produce random sentences while ensuring coverage and convergence; finally, analyzing the grammar from a new perspective, we describe how it is always possible to define constraints for the identification of balanced weights : balanced weights own the interesting characteristic of making it possible to drive the sentence generation, by a pure weight-based automatic test case generator, of extremely random but finite sentences able to stress the system being tested more extensively than the shortest derivation methods. An example of the application of balanced weights approach is included, about a simplified context-free grammar for a theoretical programming language, and possible future directions to strengthen the balanced approach are eventually outlined.


INTRODUCTION
TEST case generators for context-free grammars have been developed to offer an automatic way to obtain sentences to exercise systems during the test phase; automatic test case generators usually are more effective than manual process, and several tools have been successfully developed and applied in the industry for decades.
One of the required characteristics of test case generators is the coverage of all the syntax elements to be tested, meaning the usage of all corresponding rules of the underlying context-free grammar; options to exclude some syntax elements as well as to introduce controlled wrong sequences is welcome and in general test case generators offer options to configure these features.
Main concern while approaching automatic test case generation is represented by the convergence of the process, and methods to ensure the process will finish in a finite time producing the expected results have to be adopted; a milestone in such a direction is represented by the process described in [1] and refined in more recent papers [2], [3], [4]: other than the convergence, such a process also owns the interesting feature of generating sentences having the minimal length.
Another relevant characteristic of automatic test case generators (and very powerful for effective tests) is the capacity to produce random but syntactically valid sentences, where huge combinations of primitive syntax elements, which would probably not be thought by humans, are generated; the random feature, able to deeply and widely exercise the system being tested, is the primary goal addressed by this paper, while it is not the main characteristic of [1] which privileges the shortness of generated sentences and where the produced output is always the same anytime the process is repeated.
Where automatic test case generators differ from each other is the logic designed to select, between all rules expanding the same nonterminal, the rule to be used to substitute it. Within this paper all the presented approaches share the same logic, and this is simply and uniquely based on weights, introduced in Section 3.
A weight-based approach emphasizes the random feature (while releasing the goal to obtain shortest sentences) and represents an alternative against the most widely tools designed so far.
Beginning with a theoretical flat approach, where all productions have the same chance to be selected, we will add a penalty concept to make a weight-based approach a working solution; then, starting from a simple example and thereafter moving to more complex grammars, we will see how formal conditions which have to be satisfied by static balanced weights can be designed: definition of such conditions, making the penalty concept unnecessary, can be obtained by analyzing the embedded structure of productions and corresponding relationships between nonterminals from a new perspective.
The Balanced approach, described in Section 6, represents the final step in the weight-based proposals introduced in this paper, as it is able to indefinitely generate full random but always finite sentences, ensuring coverage and combined with a very simple logic in selecting the rule to be used during the generation cycles.

DEFINITIONS
According to [5] a context free grammar is a Type-2 grammar defined as a 4-tuple = ( , , , ), where: is a finite set; elements of are called the terminals of ; within this paper, the typeface used for terminals is bold;

2.
is a finite set, disjoint from ; elements of are called the nonterminals (or symbols) of ; within this paper, the typeface used for nonterminals is italic;

3.
is a finite relation from to ( ∪ ) * ; elements of are called the productions (or rules) of , and identified as ; a production is expressed here in the form → , where is an element of , ∈ ( ∪ ) * and i is named index of the rule;

4.
is an element of and is known as the start symbol of : no productions → exist where ∈ .
We define the function ℛ: → { | : → }; in other terms, ℛ( ) corresponds to the set containing the indexes of all productions expanding the nonterminal .
Moreover, we also define the function : → { ∈ | ∃ : → } ∪ { ( ) | ∈ ( )}; in other terms, ( ) represents the successors of , meaning the set of symbols contained in all the derivation trees which can be generated starting from nonterminal .
The relevance of such a definition is that non-recursive grammars generate finite languages, while recursive grammars generate infinite languages, and this has an impact on the application scope of the different weightbased approaches.
About recursion, we can have two different situations: 1) direct recursion; and 2) indirect recursion.
A symbol is direct recursive if a rule → exists; direct recursion is easy to discover by just looking at productions, one at a time.
To detect indirect recursion we need to look at more productions to discover all the successors: an indirect recursion exists when we can identify a cycle starting from a nonterminal and returning to the same, passing through distinct nonterminals; the length m of the cycle is the number of distinct nonterminals involved.
The following is an example having m=3 (involved nonterminals are A0, A1 and A2): Finally, we define symbols A1 and A2 as part of the same cluster if 1 ∈ ( 2 ) and 2 ∈ ( 1 ).

INTRODUCING WEIGHTS
In this paper, all approaches apply the same logic to identify which production has to be used at the time when a nonterminal needs to be replaced, and that logic is simply and uniquely based on weights.
A weight-based context-free grammar could be defined as a 5-tuple = ( , , , , ), where has a finite relation 1:1 with (wi identifies the weight set for production pi).
Weights, having values 0 ≤ wi ≤ 1, define the chance of each pi to be selected: in case pi and pj are candidate productions to expand the same nonterminal at some time of the generation cycle, having wi > wj means that pi has a greater chance to be selected than pj (but this does not mean pi will surely be the actual selected one, being the effective choice determined by the random selection mechanism); if wi = 0, pi will not be selected.
Moreover, considering all expanding the same symbol, within a weight-based approach the sum of corresponding wi must always be 1; using the function ℛ introduced in Section 2, this can be expressed as ∀ ∈ : ∑ = 1 ∈ℛ( ) .
An implementation of a random selection mechanism to identify which rule to apply for a weight-based context-free grammar can be designed using a function (e.g. rnd()), generating random real numbers r (0 ≤ r < 1) with uniform distribution; assuming that all the n rules expanding the same nonterminal are pk,…, pk+n-1, a possible algorithm is the following, which at the end identifies pi as rule to be applied: The definition of weights is completed by specifying their type, which can be static or dynamic: 1) static: are kept with their initial values during the overall genera-tion cycles; and 2) dynamic: could change while progressing with the generation cycle, depending on some conditions. As additional note, programming languages usually own a function to set an initial seed in order to exactly reproduce the same sequence of random numbers being generated: so, in case it would be interesting, setting a seed at the beginning of the generation cycles allows implementing a repeatable process, where the output is always the same, even with a weight-based generator.

FLAT
With Flat we mean a weight-based approach where all productions expanding the same nonterminal always have the same chance to be selected; this means that the Flat approach uses static weights, set to 1/n to all n rules expanding the same symbol.
The Flat approach is always effective in case of nonrecursive grammars (so, for finite languages), as in the following example, introduced in [6]: Starting from the nonterminal S, rules can be actually selected until no more nonterminals exist; the generation cycle is repeated from the beginning until all rules are used at least once. Assuming that the random selection mechanism is designed correctly, the overall process always terminates in a finite time.
Moving towards recursive grammars, we have situations where a Flat approach is safely applicable too; let us consider the following grammar, which is recursive due to p3: By definition, we have w2=w3=0.50 and the process converges as soon as to expand nonterminal id the production p2 is eventually selected instead of p3 (and this will happen, sooner or later, as w2 > 0). But, in general, the Flat approach does not work as it could lead to a never-ending process; this because convergence during sentence generations is threatened by recursive rules.
As example, we can empirically assess that the Flat approach is not safe for the following recursive grammar G0: According to the definition, w2=w3=w4=w5=0.50; based on that and assuming that the random selection mechanism always identifies p2 and p3 alternatively (the same for p4 and p5), we see that an infinite generation S ⇒ exp ⇒ term + exp ⇒ id + exp ⇒ id + term ⇒ id + ( exp ) ⇒ ... is started, using in sequence rules 1-2-4-3-5: at that point, we are back to the initial situation where nonterminal exp needs to be expanded, meaning that the process is definitely not able to converge. Section 6 will include a formal proposal to calculate the convergence index, measuring the capacity of a weightbased context-free grammar to have finite generation cycles, and we will demonstrate that the Flat approach cannot work for the above grammar.

FLAT WITH PENALTIES
A mitigation mechanism to address the weakness of the Flat approach is necessary to ensure the convergence for any context-free grammar using a weight-based approach: with Flat with Penalties we introduce the possibility to dynamically change initial weights by penalties, while generating sentences: the idea is to downgrade weights (i.e. the chance to be selected) of not smart rules as soon as they are used.
For each nonterminal, a rule is identified as smart if it has the shortest derivation length; the process to identify smart rules is very similar to the computation of SLEN as introduced in [1], with the only difference (useful here to address coverage) being for each nonterminal we could have multiple rules identified as smart 1 .
Flat with Penalties uses dynamic weights, initially set by using the same principle of the Flat approach 2 and, in case that at the end of a generation cycle not all rules have been used (or used less than the minimum number of wished occurrences), it resumes after all weights have been reset to their initial flat value.
1According to the official definition, SLEN is a single-dimension array: always one production only is identified as shortest, even if multiple productions with same length could exist; nevertheless, the additional data structures and the logic adopted to select rules make this irrelevant within [1] 2Choosing adequate initial weights could produce a more balanced output (without deep nesting primarily concentrated at the beginning of generated sentences), but this option is not addressed in this paper The penalty mechanism works through a configuration parameter m (0 < m < 1): every time a non-smart production pi is used, weight wi is reduced by k=min(wi,m); at the same time, to respect the weight definition, weight of all n smart rules expanding the same nonterminal is increased by k/n; no weights are changed if the selected production is a smart one.
Due to reductions, weight wi for a non-smart rule at some time could be 0, meaning rule pi will not be selected anymore during the current generation cycle.
No specific criteria are defined to identify the value for the configuration parameter m, but it is quite clear that the convergence (end of a generation cycle) is faster for higher values of m.
The same mechanism to select a rule for a weightbased context-free grammar introduced in Section 3 is used even with Flat with Penalties (all rules act as candidates, smart as well as non-smart), and the unique difference with Flat is because of penalties.
Considering the grammar G0 introduced in the previous section, the Flat with Penalties starts setting the same flat weight 0.50 to rules 2/3 and 4/5; about smart, Table I contains the corresponding values for each nonterminal: Then, assuming that the configuration parameter m is set to 0.05, once the rules 1-2-4-3-5 have been used, the weights will have values as in Table II: In case the random selection mechanism will identify again rules 2 and 5 to expand exp and term (at this stage the difference between w2 and w3, as well as between w4 and w5, is not so high), w2 and w5 will decrease again while w3 and w4 will rise up correspondingly; so, the probability of using a smart rule is increased every time a non-smart rule is selected, ensuring the convergence of the generation process.
So, just by using one internal data structure (smart) and a very simple process logic, the Flat with Penalties approach is effective and always produces random sentences in a finite time (a higher number of sentences might be required in or-der to cover all syntax elements compared with [1]).

BALANCED
As it was addressed in Section 4, the Flat approach safely works just for some recursive grammars, as static flat weights (same weight for all rules expanding the same nonterminal) cannot generally ensure the convergence. To resolve such an issue, the Flat with Penalties uses dynamic weights adopting a penalty mechanism.
The Balanced concept represents a step forward, as it defines static weights which always ensure convergence, while eliminating the need to use the penalty mechanism; given any consistent context-free grammar, balanced weights can always be identified and a pure weight-based test case generator can be safely used to obtain full random finite sentences.
The challenge here is how to find such balanced weights: the idea is to analyze the grammar from a new perspective in order to capture the contribution of each rule to change the total number of occurrences of nonterminals in the sentence being generated, and consequently to identify which conditions the weights have to fulfill to be balanced.

Delta
Before outlining how balanced weights can be found, we firstly introduce delta (Δ), a table where the element δi,A contains the resulting difference of occurrences of nonterminal A in the sentence being generated in case : → would be used; if ∉ { } ∪ , δi,A is not defined.
Values of Δ are identified by just observing each rule of PG independently of each other: for rule pi, δi,A (whenever is defined) is set to the total number of occurrences of nonterminal A within , reduced by 1 in case ∈ ℛ( ).
As example, referring to the grammar G0 introduced in Section 3, Table III shows how Δ looks like (δi,A='-' where δi,A is not defined): Then, using Δ, the contribution of rule i to change the number of total occurrences of symbol A in the sentence being generated can be calculated as wiδi,A.

Constraints, Basics and Advanced
Next step is the definition of constraints, conditions the weights have to satisfy to represent balanced weights; any set of weights which satisfies all the defined constraints represents a valid set of balanced weights able to ensure that the generation process always ends in a finite time.
Then, observing the source grammar from a particular perspective, we also identify advanced constraints which, for each symbol, establish relationships between weights and Δ: advanced constraints are the most significant components which qualify the Balanced approach.
At this point, let us see how advanced constraints can be defined, starting from a simple grammar G1: Corresponding values Δ1 are shown in Table IV (column for non-recursive symbols, as by definition the start symbol S, can be omitted from Δ as irrelevant): Graphically, G1 can be represented as in Fig. 1 (characteristics of such representation are described later): To define the advanced constraint for symbol A, we have to consider how it should be possible to make all of its occurrences, as introduced by direct recursive rules, disappear from the generation sentence: this, for grammars not having indirect recursive rules like G1, is simply obtained when the overall contribution of rules removing A is higher than the overall contribution of rules adding A.
So, in this simple scenario the advanced constraint is represented by condition (1): 3In case, for some reasons, we are not interested in coverage or wish to avoid using specific rules, basic constraints of type 2) could be wi≥0 or wi=0, respectively; in the paper we always assume wi>0 ∑ , < .
The fact that w3 is not part of the resulting advanced constraint means that productions having δi,A=0 (i.e. direct recursive rules having just 1 occurrence of expanding symbol on the right side, like p3) do not influence convergence at all. Therefore, the entire set of constraints for this grammar is composed by the following conditions: Differences between σA of different sets of balanced weights for symbol A express different speed of convergence, hence the ability to be faster in eliminating all occurrences of A; to introduce a measure of such an ability, we define the convergence index zA, where higher values mean greater convergence.
Writing (1) as x < k, the convergence index expresses the measure of the strength of the disequation, that is how much x is less than k; it can be defined by k-x and, in this simple scenario, we consequently have zA = -σA.
Convergence is safely ensured where z > 0; nevertheless, having a higher convergence index may not lead to better results: balanced weights having higher convergence index 4As advanced constraints are disequations, multiple solutions could exist usually require more generation cycles in order to also satisfy coverage.
Referring uniquely to sets 1), 2) and 3), the corresponding convergence indexes zA are, respectively, 0.10, 0.05 and 0.25. That means with set 3) the convergence will be the fastest (generally, as it is depending on the random mechanism), while using set 2) it will be the slowest.
They compute zA respectively, as -0.15, 0 and -0.45, meaning that, over time, with the first set the occurrences of symbol A in the sentence being generated will increase slightly, while with the last set they will increase greatly; cases where σA=zA=0 represent limit situations where the generation cycle never terminates, not being able statistically to definitely remove all the occurrences of symbol A in the generation string.
Nevertheless, due to random mechanism, it could happen that the generation cycle sometimes terminates by using unbalanced weights too, depending on the effective sequence the rules are selected.
Obviously grammars are in general not so simple like G1, so now we move a step forward by analyzing G2, which includes both direct and indirect recursions.
Corresponding values Δ2 are shown in Table V: Graphically, we can describe it as in Fig. 2: path from symbol A1 to A2 exists if a rule 1 → 2 exists, while paths to black circles represent a rule → where ∈ ; a graphical representation helps in understanding the logic behind the advanced constraints, as it easily allows to follow the flow from a nonterminal to the next one.
Focusing on symbol E of G2, dotted lines represent the path which can reintroduce it (through symbol F, when p7 is used). In such a similar situation, having (2) as constraint is not enough, as it does not take into account indirect recursive rules either, with the possibility to reintroduce the symbol E: to take care of such indirect contributions, we need to estimate how many symbols E are statistically reintroduced by walking through the recursive path E ⇒ T ⇒ F ⇒ E.
To evaluate such indirect contributions, the rally concept is introduced (for symbol A, denoted as λA): the rally specifies the average number of steps which are necessary to statistically exit from expanding an occurrence of a symbol, applying the weight-based selection mechanism.
The condition defining the advanced constraint for symbol E (part of an indirect recursive path) has to ensure that the capacity to make the symbol E disappear, by applying rules expanding E only, is enough to counteract the impact of the number of occurrences of symbol E introduced for effect of indirect recursive path E ⇒ T ⇒ F ⇒ E.
The first term of the left side of the constraint, λEσE, analyzes the effect of rules expanding E, including the direct recursive too.
The remaining terms of the left side, always greater than 0, analyze the indirect recursive path (defined as cycle) of G2, E ⇒ T ⇒ F ⇒ E, and estimate the combined effect of counting: 1. How many occurrences of symbol T are introduced in the string being generated during the λE steps, computed as ( 2 2, + 3 3, ); 2. For each T introduced by 1, how many occurrences of symbol F are introduced in the string being generated during the λT steps, computed as ( 4 4, + 5 5, ); 3. For each F introduced by 2, how many occurrences of symbol E are reintroduced, due to the λF steps, computed as ( 7 7, ).
In case an indirect recursive path should not exist, these additional terms do not exist either, and the resulting advanced constraint simplifies into λEσE < 0, which represents a valid generalization of previous σE < 0: indeed, λEσE is less than 0 if and only if σE < 0 (λE cannot be negative).
If we wanted now to shift the focus on symbol T of G2 (as well as on F), by applying the same logic we would obtain the same condition: indeed, considering a cycle, the advanced constraint is always the same for all symbols, which are part of the same cycle, no matter what position they have in the cycle.
The left side of the condition can easily be found by looking at the graphical representation, being the advanced constraint for a cycle simply obtained by the product of elements part of the cycle itself (ignoring direct recursive paths as already embedded in the corresponding λ), where: 1. Name of each nonterminal A is replaced by corresponding λA; 2. Indexes over a path ending with nonterminal A are replaced by sum of corresponding wiδi,A.
Therefore, balanced weights for this grammar have to satisfy the following set of constraints (the basics ones and the advanced ones): The most qualifying condition is represented by the advanced constraints and, due to its simplicity, for G2 it is quite easy to empirically find possible solutions, like 1) w2=0. 10 Similarly, about G0 introduced in Section 4, we can now formally assess that the Flat approach is not applicable either: using the same method described above, the resulting qualifying condition is Such a condition is not satisfied, as the left side is equal to 1: being convergence index z = 0, representing the limit condition for non-convergence (ensured where z > 0), it formally confirms the previous empirical deduction.
To make another up-shift in grammar complexity, let us consider a grammar having n clustered symbols (e.g. In this scenario we will have n advanced constraints, where the left side of each one is the sum of expressions defining the advanced constraint about each cycle, part of the same cluster, considered separately. As a last example, and concrete application of all the above definitions, we introduce now G3, a context-free grammar representing a subset of a typical programming language 5 : we will see how advanced constraints look like, we will identify valid balanced weights and, finally, we will show a sample of generated sentences. Values of ∆ for this grammar are shown in Table VI below (as usual, only recursive symbols have been included as columns, because relevant for the identification of advanced constraints, opposite to non-recursive ones): The grammar can graphically be represented as in Fig.  4, where dotted rectangles refer to the 2 embedded distinct clusters: γ1, composed by symbols {body, stm_list, stm, comp_stm}, and γ2, composed by symbols {exp, term, fac-tor}: 5Additional rules could be added, e.g. for further statements as well as operators (like "+", "/", ">" etc), but have been omitted to keep the example simple but representative to show how the Balanced works Cluster γ2 contains 1 cycle (exp ⇒ term ⇒ factor ⇒ exp), while cluster γ1 contains 3 cycles: Applying the logic as previously introduced for the definition of the advanced constraints where multiple clusters and/or cycles exist, for clusters γ1 and γ2 we obtain the following corresponding conditions, respectively: Then, using the values of ∆ from Table VI, substituting the rallies with corresponding expressions and, moreover, taking advantages of basics constraints 1), advanced constraints for γ1 and γ2 can eventually be, respectively, rewritten as: [ 5 + 7 • (2 5 10 + 11 )] 3 ⁄ < 1; 19 ∕ ( 14 16 ) < 1.
To identify valid solutions, other than empirically (not so easy when we have complex advanced constraints), it is always possible to utilize a byproduct of the Flat with Penalties approach: indeed, at the end of generation cycles, relative percentages of usage of each rule expanding the same symbol always represent safe values to be used as balanced weights.
Anyhow, starting from a valid set of balanced weights, manual changes can be made, for instance to drive the sen-tence generation to increase or to reduce the nesting level caused by recursive rules: in any case, after a manual modification, the new set needs to be re-validated to assess if all the basics as well as the advanced constraints are still satisfied.
The above weights satisfy all the conditions being the expressions on the left side representing advanced constraints for clusters γ1 and γ2 evaluate, respectively, to 0.84 and 0.65, both less than 1 (with corresponding convergence index of 0.16 and 0.35): this means that they are balanced and we can safely run an automatic pure weight-based test case generator to always obtain finite sentences.
Below, a real output of a pure weight-based test case generator applied to G3 with the mentioned balanced weights, where we can see its power in generating extensively random (but always finite) sentences; moreover, this example also shows how with the Balanced approach all the several rules of a grammar can be used within a single sentence.

CONCLUSIONS
A demonstration about effective applicability of an automatic test case generator for context-free grammar where the unique criterion to select a rule is based on weights has been provided. The weight-based approaches differentiate against well-known methods as they easily generate sentences in a very random way while not requiring internal data structures as well as complex logic to run.
Convergence of the generation cycles for weight-based methods can be safely ensured by using a penalty mechanism or using balanced weights: how to identify constraints defining balanced weights depending on the typical recursive patterns of a grammar has been presented.
An example of generated sentences produced by using the Balanced method has been provided for a context-free grammar representing a reduced portion for a theoretical programming language; the example provides a clear understanding on how the method is capable to generate large but finite full random sentences, while ensuring coverage and convergence: this feature represents a step forward in automatic test case generation as, with a very simple generation logic, it offers a greater chance to extensively exercise the internal data structures of system being tested compared with existing well-known methods.
Possible further investigation is the development of an automatic engine to define the advanced constraints in order to eliminate the manual analysis of the grammar structure.
Moreover, although we have seen how the Flat with Penalties approach can be exploited to identify a valid set of balanced weights, another interesting area of investigation could be represented by the usage of neural network models, in order to iterate over the defined constraints to eventually obtain, starting from flat weights, valid balanced solutions (weights satisfying all the constraints): in this respect, we notice an interesting isomorphism between the elements of a neural network, the input, the parameters, the activation function and the loss function with delta, the weights, the constraints and the convergence index.