Linear Temporal Public Announcement Logic: a new perspective for reasoning about the knowledge of multi-classifiers

In this note, a formal transition system model called LTPAL to extract knowledge in a classification process is suggested. The model combines the Public Announcement Logic (PAL) and the Linear Temporal Logic (LTL). In the model, first, we consider classifiers, which capture single-framed data. Next, we took classifiers for data-stream data input into consideration. Finally, we formalize natural language properties in LTPAL with a video-stream object detection sample.


Introduction
Nowadays, classification is inseparable from real-world applications. The presence of errors increase the necessity for better interpreting and understanding such algorithms. Due to the probabilistic nature of most of these approaches, the reasoning about results remains complicated. Previously, a model was developed by Dehkordi et al. to verify properties for classifiers. In which, the analysis of the input data and external information in a multi-agent scenario was considered [1]. The proposed method works fine for single-framed input data; however, it suffers from not being able to be used appropriately for semicontinuous data streams (i.e., video streams). More progress is also possible by finding the similarity of vicinity frames, as the information changes in such data streams are considered minor [2]. The process of data understanding can be divided into several steps. The first and foremost one is the object recognition from the input data, for which there are many solutions, often with great performances, [3]. The next step is knowledge extraction to understand the knowledge acquired in the initial step (see [4]), where objects are transformed into symbols. In this step, knowledge can be shared and aggregated via a modal logic approach [1]. Subsequently, the given rules, which are fed into the system, should be interpreted. This can be achieved by employing predefined protocols (i.e., "cat is an animal" or "bat and pigeon could not have appeared in a single data frame" as predefined protocols). The final step is to satisfy the specified formula, employing a collected set of knowledge. The last three steps will be highlighted further in this study.

Related Works
The reasoning is one of the most studied and accepted dimensions of AI (see [5], [6], [7], [8]). Furthermore, it inspired researchers to target Visual data understanding, question answering, and natural language queries [9], [10]. For instance, a question-answering (QA) reasoning system is developed by Bauer et al., where an algorithm was introduced to select the best paths of commonsense knowledge to get the whole inference required for QA [11]. The method works fine for specific applications, but it did not provide a general solution for QA problems. Based on the systematic analysis of popular knowledge resources and the knowledge-integration methods, Ma et al. developed a modelling solution [12]. The method, called "non-extractive commonsense QA", employs Con-ceptNet (Speer et al. [13]) and the most recently introduced ATOMIC (Sap et al. [14]). More recently, based on human intelligence, the "CoLlision Events for Video REpresentation and Reasoning" method (CLEVRER) was developed by Yi and coworkers [15], which is a reasoning system for video streams. In that method, unlike former studies, "causal structures" are also taken into account. Moreover, this model was able to answer four main varieties of questions: descriptive (e.g., "what color"), explanatory (e.g., "what's responsible for"), predictive (e.g., "what will happen next"), and counterfactual (e.g., "what if"). In that model, the solution for extracting features from the video frames is ResNet-50, a well-known classifier [16]. Thanks to the characteristics of the reasoning systems and the power of temporal logic in formalizing natural language, the model developed by Fong et al. gets more interested in reasoning systems [17]. From another perspective, the Metric Temporal Logic (MTL), an extension of the Linear Temporal Logic (LTL), was examined to handle the stochastic state information [18]. Concerning typical AI-based problems, the employment of modal logic would be attractive. Recently, Dehkordi et al. developed an epistemic-based model for the extraction and aggregation of knowledge from classifiers [1]. The model works perfectly for single framed data; however, this model did not address data stream inputs.

Motivation
In the model introduced by Dehkordi et al. [1], algorithms 1, 2, 3 were developed for knowledge sharing modelling in a multi-classifier scenario. In a network of trusted classifiers (which share their knowledge correctly), the process of knowledge extraction from one input for an agent is developed in algorithm 1. In other words, all output classes of the input, presented in the neighbourhood set of η ǫ (x) (here referred to as η(x)), were collected. η(x) is a set of inputs in ǫ vicinity (based on distance function d(x 0 , x)). Here, the classifier function N (x) yields the respecting output classes by η(x) input. Consequently, the knowledge of N -which contains the set of output classes of N (x)-is represented by K (as the knowledge set of the classifier). The output is robust if the size of the set of output classes of N (x) is one. Therefore, from the agent's perspective, the output concludes the robustness of x. An agent's perspective is knowledge, which is accessible by that agent.
Algorithm 2 aggregated all knowledge of agents (i.e., the possible worlds) from the perspective of each agent. After finding the intersection, if it contains just one class, the input is verified. Here, if the output is an empty set, the MAS's knowledge is inconsistent. In the case of more than one output class, agents are not capable of concluding the answer. However, they know the answer is among these classes.
In algorithm 3, all captured knowledge is aggregated, and the reasoning is satisfied.
Afterwards, this knowledge of single-framed data is modelled in a multiagent scenario (formal definitions will be presented in section 4).
This study aims to extend the expressiveness of the model developed by Dehkordi et al. [1] to present semi-continuous and multi-framed data. Besides, a flexible reasoning system is suggested to render the knowledge achieved by classifiers. Here, the model is applied to the existed classifiers to translate the information attained from the results. Furthermore, the ability of the model to handle multi-knowledge flow in a multi-classifier scenario will be assessed. For this purpose, various collaboration scenarios of classifiers (i.e., the investigation of knowledge of each agent, a group of agents, the knowledge distributed between agents, etc.) would be studied. The focus is on verifying the investigated formulas proposed for such a statement using the definition of "verified Algorithm 1 The Classifier Knowledge Calculator (CKC) function shall calculate the knowledge produced by a classifier for an input point, using neighbourhood function and manipulation function Let N (x) = c be the function of the considered a classifier and c be the result class 1: function CKC(N , x 0 , η, µ) 2: ⊲ N , x 0 , η, µ are a classifier, an input point, neighbourhood function, and manipulation function respectively 3: return 0, K 14: end function formula" mentioned in Dehkordi et al. [1]. Certain questions (in human language) are translated to the developed temporal formula (as properties) to check the adaption and applicability of the proposed model. By checking the satisfiability of the temporal formula, the answers will be presented. This note has been structured in the following manner: In section 2, the outline of the problem and the structure of the model is proposed. Then, formal verification of classifiers are defined in section 3. A PAL model is defined in section 4 to capture the information from single-framed data. Finally, in part 5, an extension of the model (called "Linear Temporal Public Announcement Logic", LTPAL) is defined by combining PAL with LTL. LTPAL is applied to capture semi-continuous data of multi-agent systems. In order to create a more convenient approach for defining verification's property, a procedure will be suggested to formalise natural language in LTPAL -which can be served as an application.

Problem Definition
Classification is a process of assigning a label (from a label set) to an input (vector). The solution comes with uncertainties in most classification approaches (i.e., neural networks). There are approaches in which multiple answers would be provided if the answer does not meet the minimum requirements (i.e., low probability or near boundary answers). Using the collection of answers, we could create a knowledge set. The knowledge set shows what are the possible answers using applied classifiers. In this study, we will develop a model Algorithm 2 The MAS Knowledge Aggregator (MASKA) function is going to aggregate the knowledge that is produced by a group of classifiers, for each input point using a neighbourhood function and manipulation function. ⊲ N S , x 0 , η, µ are a group of classifiers, an input point, neighbourhood function, and manipulation function respectively 3: K S ← set of all output classes 4: for all N ∈ N S do 5: is Robust, K ← CKC (N , x 0 , η, µ) 6: return 0, K S 15: end function to extract knowledge in a classification process. Then, we consider classifiers, which capture single-framed data. Next, we took classifiers for data-stream data input into consideration. Finally, the developed model will be applied to two scenarios: question-answering systems and video-stream object detection.
Note that, a set of agents (classifiers) also could refer to a group of agents (classifiers), so the phrase set and group could be used interchangeably. Herein, we will develop a reasoning model on the top of a preexisted intelligent system, to reason about the knowledge, in the following steps: 1. Classification is a process of getting an input vector of size x and assigning an element c from the set C to the vector x (c is the answer of x), in which c is the output class, and C is the set of all possible classes. First of all, we are going to collect all the intelligent system's possible answers. Therefore, because of the statistical nature of most classifiers and the fact that the accuracy is based on the architecture and training set, this kind of classifier could not guarantee complete accuracy. To overcome this problem, Huang et al. [19] presented an approach to defining robustness for each input (point-wise verification) in such models. In this approach, for each input x, a set of input points created by an arbitrary function η(x) would be fed into a classifier; if all answers converged into one class, the input point is named robust for the classifier. More precisely, Dehkordi et al. ⊲ N S , x 0 , η, µ are a group of classifiers, an input point, neighbourhood function, and manipulation function respectively 3: for all N ∈ N S do 5: is Robust, K ←CKC(N , x 0 , η, µ) 6: for all M ∈ All knowledge sources do 16: is Robust, K ← Announced knowledge 17: ⊲ The external knowledge must be written in the DEL formula, where possible worlds are ones in which the formula is satisfied. 18: return 0, K S 27: end function worlds). In this study, if the size of the set of all possible worlds equals one, the model would be verified the input concerning the defined properties. 2. Generally, the set of knowledge is directly driven by classifiers from the possible worlds. This set of knowledge for input and a system of intelligent agents contains all possible output classes, yet from a real-world perspective, each class of objects would contain more information (i.e., ontology rules). To reflect this, an intelligent agent should understand the object's category with sub-categories fed in as input rules. In this model step, predefined inferences would be extracted in a unified and formal manner for each possible object class. 3. The time-series data could not be considered until this step. To consider them, we developed a combination of temporal logic transition systems Gerth et al. [20] with the developed epistemic logic model. Let each frame of time-series data as an input. After executing the above steps on all frames, we have a set of extracted knowledge (in the Kripke model) for each single-framed data of the input time-series data. By placing the Kripke models in hierarchical order, a transition system could be created. This leads us to extract all possible sequences (named execution path) of knowledge represented in time-series data. 4. Then, we will develop an approach to use our model for a questionanswering application. Accordingly, to get an outcome from the developed model, it should provide answers to questions. First, using preexisted approaches, natural language statements will be translated into temporal formulas. This formula could be inferred as a property to be verified in a verification problem. Then, each formula will be investigated in corresponded paths (respecting the type of answer, which we will discuss in future sections, it could be all or some paths). Finally, the questions related to a satisfying formula will be answered "true" and others "false". 5. The verification of property in this transition system is also defined by modifying the verification definition in Huang et al. [19] and Dehkordi et al. [1]. Accordingly, the situation of a formula (equivalently a property) would be determined. In other words, it is checked whether it is a verified formula (verified in all paths), possible formula (verified in at least one path), or if adding some missing information (by adding information to the system it could be verified), it would be a possible/verified formula.
Here, we will do the mentioned steps as an example of a video stream and classifiers. In this case, classifiers could determine objects appearing during the video. First, all possible outcomes for each image known as a data-frame will be extracted by the model developed in section 4. Then, by collecting all input rules of objects (i.e., an input rule could be a Sub-feature such as "elephants" are "animals" so, in each possible situation an "elephant" detected, we infer that an "animal" is also detected), each possible outcome will be enhanced for each image in the video. Next, the transition system would be made by placing the model of all possible images in a hierarchical sequencing (see section 5). The Kripke model was created before this step. Consequently, and as an application, if any question was asked about the video, it would be investigated by translating questions into Linear Temporal Public Announcement Logic (LTPAL). The detail of the question-answering system will be shown in section 5.1. After that, for the sake of the performance, we will calculate the probability of each transition relation, following that, finding the most probable execution path. Figure 1 shows a transition system created for two-frame data. In this transition system, w 0,0 and w 3,0 are initial and final worlds, defined for simplicity. In the first layer of the transition system, the corresponding Kripke model M containing worlds w 1,0 , w 1,1 , and w 1,2 represent knowledge extracted by classifiers for the first input data. As it can be seen, classifiers provided three possibilities for output classes of this input data, which are fully connected by epistemic relations (solid lines). These relations demonstrate that these classifiers could not decide which world demonstrates the correct output. Similarly, M 2 was created using the second input data of the data stream. Here, dotted lines are transition relations, and the transition system T S could be defined. This figure provides numbers at the top of each transition relation w i,j → w i+1,k , which shows the calculated probability of knowledge change between world w i,j and w i+1,k . Herein, a model for two-framed data is illustrated, in which the most probable path is w 0,0 w 1,0 w 2,0 w 3,0 . This path could lead us to correct the misclassifications of classifiers.

Formal verification of Classifiers
Generally, formal verification is a mathematical process to ensure that a model fulfills some property function in an environment [21]. Thus, in order to define the verification of classifiers, we should ensure that the defined property function holds for the classifier.
Definition 1 A property set ρ for elements of a set X defines with a property function fρ : A single-object classifier is a classifier detecting a single object from an input. In some critical cases, single-object classifiers provide a set of answers because the confidence level of the classification process does not satisfy the minimum requirements. For example, in the YOLO classifier threshold could be defined; and the outputs would be a set of answers with the scores over the threshold (Redmon et al. [3]). In another study, a set of inputs was created with a neighborhood and manipulation of the input. In this study, the output set would be all outputs of the classifier for the input set (Huang et al. [19]). In this study, the property set would be denoted by ρ(X); and the set of answers provided by classifier G for input x considering the property ρ would be represented by the notation [ρ(x)] G . By definition, a single-object classifier should provide a single output class. Thus we define verification as follows: Here, "| A |" is the cardinality of the set A.
In other words, a property for a single-object classifier is verified when it is confident about a single answer (see [1]).

Definition 3 A property ρ(x) for a set of single-object classifiers
In a multi-agent scenario, a property for single-object classifiers is verified when multiple classifiers agree about a single answer (see [1]).
Note that we can assume a multi-object classifier as multiple single-object classifiers and the definitions are identical for each single-object classifier. Thus, we use the term classifier instead of the single-object classifier for simplicity.
The definition of verification in a single-frame classifier is based on the appearance of objects in that input frame, but for data-stream, we consider objects in some frames. For example, in a video stream, we could verify the appearance of a dog after a cat. In this case, it is enough to detect an input frame with a dog after seeing a cat's input frame. Thus, we define the property in the form of LTPAL in section 5.1.

Public Announcement Logic
In this section, we are going to introduce a logical model for interpreting singleframed data. The epistemic model is based on an extension of PAL that is mentioned in [1]. It will be applied to extract knowledge from single-framed data (i.e., an image).
Let us first introduce the syntax and semantics of the logic.
Definition 4 (Language of P AL) Let Ag = {1, . . . , n} be a finite set of agents. The syntax of the language PAL is as follows, in BNF: Where p is a propositional variable (atomic formula) is a pair of form (x, c), in which x denotes the input data and c denotes the target class and A ⊆ Ag is a subset of agents. We also notice that K i φ read as "i-th classifier knows φ" (i ∈ set of all classifiers). In other words, the i-th classifier was assured about the truth value of φ. Following that, D A φ read as "φ is a distributed knowledge in group A of classifiers". The formula D A φ holds exactly when, aggregation of knowledge of agents (equivalently ∩ i∈A R i ) in the group A, satisfies φ (if A = {i} then K i ≡ D A ). Finally, the formula [ψ]φ reads as "after a correct announcement of ψ, φ will hold". By employing this operator, it could investigate that which knowledge is missing.
Suppose G = {G 1 , . . . , G n } is a finite set of classifiers, X is the set of input points, and C is the set of all output classes. A PAL Kripke model is a tuple M = (W, R 1 , . . . , R n , V ), where W is a set of worlds (hereabouts, the set W represents all possible output results of the input data), R i (c) are defined as follows: The intended meaning of wR i w ′ relation is, the worlds w and w ′ cannot be epistemically distinguished by the i-th classifier. Finally, V : W −→ 2 P rop is the evaluation function specified that the knowledge is represented in any world, where P rop is the set of all atoms. V (c) and V (c) are defined as follows: We extend the evaluation function V to all formulas as follows: The intended meaning of satisfaction of an atomic formula (x, c) in a world w is "class c has appeared as respective classifier's output class for the input data x". We also mention that for every formula φ and ψ, φ ψ means that for every model M , and every point w ∈ M , M, w φ implies that M, w ψ.
Now we can show that a point is verified for classifiers exactly when its interpretation is valid in M. More details about the model and proofs could be found in Dehkordi et al. [1].
Here, algorithm 4 was developed to investigate the correctness of PAL formulas. Proof We only prove the if direction, the other direction is proved similarly.
"If" direction: The proof is by induction on the complexity of Φ. Suppose that M, w φ, and φ ∈PAL.

Linear Temporal Public Announcement Logic
In this section, to extract all possible sequences, named execution paths, of knowledge represented in time-series data.

Definition 5 (Language of LTPAL ) We introduce an LTL extension of PAL by the following grammar, in BNF:
Here, the temporal operators are XΦ (in the neXt data-frame Φ must be true) and [ΦU Ψ] (Ψ must remain true U ntil Φ becomes true). To define the Kripke semantic of this logic, assume that if φ is in form of K i ψ then 14: return ∧ w ′ PALS(ψ, M, w ′ );∀w ′ ∈ R i (w) 15: end if 16: if φ is in form of D A ψ then 17: return false 28: end function (W n−1 , R (n−1) 0 , . . . , R (n−1) k , V n−1 ), and M n = ({w n0 }, R n0 , . . . , R n k , V n ) are PAL models, where W i 's are mutually disjoint sets, and V 0 (w 00 ) = V n (w 00 ) = ∅.
We build a new model T S = (S, R, s 0 , s −1 , →, L), known as a transition system, in which S = n i=0 W i is the set of states, R = {R ij | 0 ≤ i ≤ n, 0 ≤ j ≤ k}, s 0 = w 00 and s −1 = w n0 are initial and final states, → k = W k × W k+1 , 0 ≤ i < n are transition relations, →= 0≤k<n → k , (w ki , w (k+1)j ) ∈→ k is denoted by → k ij , and the labeling function L : S −→ 2 P rop which assigning propositional letters to states is defined by L(w ij ) = V i (w ij ), for each w ij ∈ W i . To extend the labeling function to all LTPAL formulas, we first fix the following notations.
An executive path π i...j T S in the model T S is a sequence of worlds w i w i+1 . . . w j where w i → w i+1 → · · · → w j for w k ∈ W k , i ≤ k ≤ j. An executive path π i...j T S is called total and denoted by π T S , if w i = s 0 and w j = s −1 . The set of all total executive paths will be denoted by Π T S . Next, for φ ∈ PAL and Φ, Φ 1 , Φ 2 ∈ LTPAL, we define: In the first clause above, we notice that we look at the transition system T S over PAL formulas simply as a Kripke model. Moreover, algorithm 5 is developed to create a transition system from PAL models.
By the function TEMS developed in algorithm 6, temporal formulas could be investigated. Other temporal operators could be driven from the two operators next (XΦ), and until (U Φ) in the following way: The intended meaning of future (F Φ) is "eventually Φ becomes true", global (GΦ) is "Φ must remain true forever", release (ΦRΨ) is "Ψ remains true until and including when Φ becomes true, if Φ never becomes true, Ψ always remains true", and weak until (ΦW Ψ) is "Φ has to remain true at least until Ψ; if Ψ never holds, Φ must always remain true" [22], [23]. Proof We only prove the if direction, the other direction is proved similarly. The proof is by induction on the complexity of Φ. If direction: For any path formula, in which the length of path is lower than zero, lines 3 and 4 would be executed; and false will be returned. For any PAL formula, lines 6 to 8 would be executed. Similar to the semantics of LTPAL, we will check whether Φ is in P ALS(Φ, M, w) or not. Here, M is the i-th Kripke model of the Transition system T S.
Suppose that T S, π i...j T S Φ, and Φ ∈LTPAL is not a PAL formula. So, Φ would be in form of for all x 0 in X do 14: , M :=MASKS(C, x 0 , η) 15: for all w ∈ W and w ′ ∈ W ′ do if Φ is in form of Ψ 1 ∧ Ψ 2 then 15: return TEMS(Ψ 1 , T S, π i...j T S )∧TEMS(Ψ 2 , T S, π i...j T S ) 16: end if 17: if Φ is in form of XΨ then

Formalizing natural language properties in LTPAL
A straightforward interpretation of human language in modal logics (especially in LTL) is one of the most critical strengths of such logics (see [24], [25], and [26]). This kind of interpretation could help robots react to human orders [27].
In order to convert the question into a formula, we first need to extract all possible answers to the question. Then, after converting each possible answer to the LTL formula using the existing -and abovementioned-approaches, we will investigate the satisfaction of each answer. The developed model would lead us a PAL modifications on such LTL formulas. To explain that, let Φ(p 1 , . . . , p σ ) be an LTL translated formula and φ i , 1 ≤ i ≤ σ is a PAL formula. Then, Φ(φ 1 , . . . , φ σ ) is obtained from φ by substituting each of p i with φ i for all 1 ≤ i ≤ σ, respectively. For instance, for Φ(p 1 , p 2 ) = G(p 1 U Xp 2 ), φ 1 = ¬D A ¬p 1 , and φ 2 = K i p 2 , we have Φ(φ 1 , φ 2 ) = G((¬D A ¬p 1 )U (X(K i p 2 ))), which, for the transition system T S and the investigated execution path π T S , means that "p 1 should always be a possible answer until in the world which right after, p 2 is a robust knowledge for the i-th agent". By defining a way for introducing the definition of verified and possibility concerning a group of classifiers, answers could be labeled to assure the questioner, "how reliable is this answer". This would lead us to design a more proper system for critical applications. Moreover, by capturing missing knowledge, it could be caught which knowledge is mandatory for the system that is not discovered by classifiers or external, and which part of the system could be manipulated to obtain the missing information.
To define when a formula is verified, possible, or is missing information in a transition system, we first introduce the following notations.
¬K i ¬} we define ΩΦ as follows: (1) Note that, the transition system is created by aggregation of PAL models. In each PAL model, we had a word named w i , in which all atomic formulas in that model are true. By definition of transition relation, there is a path π T S = s 0 → w 1 → w 2 · · · → w n → s −1 .
Now, for Φ ∈ LTPAL, we say that: 1. Φ is verified for the group A of classifiers exactly when, we have T S, π T S D A Φ. 2. Φ is possible for the group A of classifiers exactly when, we have T S, π T S ¬D A ¬Φ. 3. A PAL formula ψ is called verified-missing information for a formula Φ in group A of classifiers exactly when, we have T S, π T S D A Φ and T S, π T S [ψ](D A Φ). 4. A PAL formula ψ is called possible-missing information for a formula Φ in group A of classifiers exactly when, we have T S, π T S ¬D A ¬Φ and T S, π T S [ψ](¬D A ¬Φ).
Example. (Representation of a Video) In the previous chapter, a PAL logic for representing image data was illustrated. In this example, assume that the given finite video stream contains n − 1 images, and models of epistemic information extraction for these images are assumed to be M 1 = (W 1 , R 10 , . . . , R 1 k , V 1 ), . . . , M n−1 = (W n−1 , R (n−1) 0 , . . . , R (n−1) k , V n−1 ). By adding two dummy models, M 0 and M n , the transition system T S = (S, R, s 0 , s −1 , →, L) could be created.
For clarification, we will describe a real-world situation: Assume a medical clean-room in an Operating theater, which is monitored by a camera. The recorded video will be fed into a set of classifiers A. The room will be cleaned with an air conditioner together with an ultra-violet (UV) LED. As it is known that the UV LED would damage humans, it must be turned off while a person is in the room. On the opposite, the air conditioner should work during the appearance of a person in that room. It also should be stopped in an empty room for saving electricity. Therefore, the first LTPAL formula (here referred to as protocol) is "shut down UV while any person monitored", and the second one is "shout down the air conditioner when no one is in the room". The classifiers should answer the vital question of "How robust are the protocols?".
Let p be "at least one human exists" and q be "the UV is on", and r be "air conditioner is on". We have following translations: • G(p ⇒ X¬q): which is the translation of "by observation of human, the UV should be shut down". • T S, π T S G(D A p ⇒ X(D A ¬q)): by the satisfaction of this formula the satisfaction of the property would be verified within group A.
• T S, π T S G(¬D¬p ⇒ X(¬Dq)): by the satisfaction of this formula, there is a scenario in which the formula holds. It means that this property is possible within group A. • T S, π T S G(K i p ⇒ X(K i ¬q)): by the satisfaction of this formula, this property would be robust for the i-th classifier.
• T S, π T S G(¬K i ¬p ⇒ X(¬K i q)): by the satisfaction of this formula, the i-th classifier, consider this property as a possible one.
Thus, the formula "T S, π T S G(¬D A ¬p ⇒ X(D A ¬q))" will ensure that the system enforces UV-let to turn off in humankind's possible existence. Accordingly, this system leads us to investigate whether protocols are followed or not.

Conclusions and future work
In order to provide an interpretation for given answers, and to notice reasoning in critical cases, an approach was designed to formalise acquired knowledge by any classification or input rules for single-frame data. Next, by introducing LTPAL, an extension of PAL, the flow of knowledge in time-series data was modeled. This model gives us a representation of acquired knowledge in a transition system structure. This model could examine the satisfaction of properties (LTPAL formulas) in order to solve the verification problem for classifiers. As an application, we explained a question/answering problem scenario, in which questions are properties to be verified. Consequently, we investigated the reliability of answers (i.e., possible or verified). Moreover, using this model could answer the question: "which missing knowledge could lead the system to the correct answer?". Thus, the developed model was suitable for interpreting and verifying multi-classifiers that aim to classify data streams.
We are going to extend the model to consider probabilities. Then, we can find the most probable scenario for each data stream to assist classifiers. Besides, we are going to define notations to model multi-object classifiers in a unified way. For example, in an input image, multiple objects could be distinguished by their location in the picture.

A Tool
This section will describe the developed tool, which is done by the Python programming language. This tool is aimed to create an LTPAL model for data-stream inputs. To do this, first, we will collect all knowledge gained from classifiers for specific input. This knowledge could be in two forms. The first is "predicted output classes" collected by group A of classifiers, known as internal knowledge and in the form of formulas provided by external resources, known as external knowledge. As mentioned, the internal knowledge is provided from a data-stream so that classifiers would provide output classes for each dataframe of the data-stream. Here, for output classes of each data-frame, a Kripke model would be developed based on the provided internal knowledge. After that, the model would be enriched using Sub-feature formulas. Consequently, possible worlds of the Kripke model would be calculated. These steps are formally described in Remark 1. of section 4. Thus far, for each data-frame of the data-stream, a Kripke model was provided. Using these Kripke models and adding them in sequential order, we could create a transition system as we described in section 5. Now, LTPAL formulas could be investigated over paths of the transition system. Due to the theory provided in section 5.1, the following statements would be held: • If the LTPAL formula satisfies over all paths of the transition system, this will be a verified formula for the respective group A of classifiers. • If the LTPAL formula satisfies over at least one path of the transition system, this will be a possible formula for the respective group A of classifiers. • If group A consists of just one agent i, and if the LTPAL formula satisfies over all paths of the transition system, this will be a verified (or robust ) formula for the respective classifier i. • If group A consists of just one agent i, and if the LTPAL formula satisfies over at least one path of the transition system, this will be a possible formula for the respective classifier i. • If just after changing input information, the LTPAL formula satisfies over all paths of the transition system, this information change will be verifiedmissing information for the formula in the respective group A of classifiers. • If just after changing input information, the LTPAL formula satisfies over at least one path of the transition system, this information change will be possible-missing information for the formula in the respective group A of classifiers. • If group A consists of just one agent i, and if just after changing input information, the LTPAL formula satisfies over all paths of the transition system, this information change will be verified-missing information for the formula in the respective classifier i. • If just after changing input information, the LTPAL formula satisfies over at least one path of the transition system, this information change will be possible-missing information for the formula in the respective classifier i.

A.1 Atomic Formula
As it is assumed, classifiers have a specified output domain, which means for a group of classifiers A, the aggregated knowledge of output domains of them would be assumed as the group's output domain C. The output domain defines all possible outcomes of classifiers for every input. Here, a pair (x, c) for each class c ∈ C from the output domain of a specific input x will be stored as the atomic formula. Besides, all pairs of the input x and sub-features {(x, c 1 ), . . . , (x, c k )}; (x, c) (x, c 1 ) ∧ · · · ∧ (x, c k ) of these output classes would be counted as atomic formulas. So, we created a python-class namely "Atom-icFormula" with the property "inputData", which is referred to the specific data-frame, the property "name", which is referred to the name of the output class, and "fathers", which referred to what this atomic formula deducted from (i.e., if (x, c) (x, c 1 )∧· · ·∧(x, c k ), (x, c) would be the "fathers" of (x, c i ) 1≤i≤k ). We created an instance for each pair of input x and output class c, and then we will create the fathers' property using all sub-features.

A.2 Kripke Model
In this section, we will describe the development of the Kripke model in the developed tool. To do this, we keep in mind that a PAL Kripke model is a tuple M = (W, R 1 , . . . , R n , V ). Therefore, we developed a python-class named "KripkeModel", which consists of W , R i ; 1 ≤ i ≤ n, and V as its properties.
To create a proper python-class instance following steps would be considered. First, all output classes considering the predefined neighbourhood provided by classifiers for a specific input data-frame would be collected. Second, the intersection of the classes provided by classifiers as the possible output classes would be calculated (see algorithm 3). The cardinality power-set of such output classes would be the number of possible worlds in this step. Here, the initial Kripke model could be created in the following manner: • Let the cardinality power-set of output classes as n, • W = {w 1 , ..., w n }, x, c ij )}, all corresponding atomic formulas (atomic formulas with the same "name") from the power-set of output classes, plus all of their "fathers" from the "AtomicFormula" class.
After creating the Kripke model M, the PAL formula ϕ could be investigated for each world. Herein, for all provided PAL formulas (external knowledge fed to input), and for all worlds w ∈ W , if M, w ϕ, the world is considered impossible and will be removed from the Kripke model M (see algorithm ??). In this process, we also remove all relations with one end in w and all evaluation function V (w). Therefore, the created Kripke model considered both internal knowledge and provided external knowledge (in the PAL formula).

A.3 Transition System
In this step, a list of Kripke models were provided, each one related to a dataframe. So, to create the transitions system, first, we created a python-class, named "TransitionSystem", which contains properties S, R, s 0 , s −1 , →, and L, for a transition system T S. This python-class contains a python-method "add kripke" that gets a "KripkeModel" object as input and appends it to the end of the "TransitionSystem". By calling this python-method for all Kripke models, the transition system would be created in sequential order. The next step is to investigate the LTPAL formula ϕ in this transition system. To do this, we should investigate "for all paths does ϕ satisfied?" to check whether ϕ is a verified formula and "there exists a path in which, ϕ satisfied?" to check whether ϕ is a possible formula. For both objectives, extraction of all paths is needed, so we did it in the "get all paths" python-method. By iterating on extracted paths, together with algorithm 6, we could check whether a formula is verified (if it satisfies in all paths) or possible (if it satisfies in at least one path).

B Extended Data: how to use the LTPAL tool
We developed the LTPAL tool to verify the properties of multiple classifiers. Here, properties would appear in the form of LTPAL formulas. Such properties could be investigated for single-framed data (i.e., images) or data-streams (i.e., videos).

B.1 Input
As said, the tool could apply to an ensemble of classifiers. Therefore, one should collect classifiers that are suitable for the purpose. Then, each input (and its neighbourhoods) should be fed into the classifiers. The output would be a list of detected objects for each data-frame, so there is a list of lists of objects for each data-stream. The input should consist: • LTPAL formulas (properties), • PAL formulas, • Classifiers' output class domain, • Number of frames (1 for single data-frame), • Classifiers' ids, • For each classifier, predictions for each frame should be written,