A Secure and Disambiguating Approach for Generative Linguistic Steganography

Segmentation ambiguity in generative linguistic steganography could induce decoding errors. One existing disambiguating way is removing the tokens whose mapping words are the prefixes of others in each candidate pool. However, it neglects probability distribution of candidates and degrades imperceptibility. To enhance steganographic security, meanwhile addressing segmentation ambiguity, we propose a secure and disambiguating approach for linguistic steganography. In this letter, we focus on two questions: (1) Which candidate pools should be modified? (2) Which tokens should be retained? Firstly, we propose a secure token-selection principle that the sum of selected tokens' probabilities is positively correlated to statistical imperceptibility. To meet both disambiguation and optimal security, we present a lightweight disambiguating approach that is finding out a maximum weight independent set (MWIS) in one candidate graph only when candidate-level ambiguity occurs. Experiments show that our approach outperforms the existing method in various security metrics, improving 25.7% statistical imperceptibility and 11.2% anti-steganalysis capacity averagely.


I. INTRODUCTION
L INGUISTIC steganography is the practice of concealing covert messages within natural-language texts, which are the most widespread in transmission mediums [1]. The primary target of steganography is to maintain the security and particularly imperceptibility of the covert transmission against potential adversaries [2]. The embedding process is described as: given one language model and a guidance key k, embedding algorithm f emb converts a covert message m and prompt (introductory content) p into steganographic text s, i.e. f emb (k, m, p) = s. Conversely, the extraction process is denoted as f ext (k, s, p) = m , where m is the covert message decoded from the steganographic text s, given the shared key k, the shared prompt p, and the shared language model. Relevant recent work [3], [4], based on current NLG techniques whose basic unit is one token, needs to identify each mapping word. Thereby, linguistic steganography commonly works in segmented languages or space-delimited languages, mainly in English. However, in unsegmented languages (involving Chinese and Japanese, or using language models with Manuscript  sub-word mechanisms [5], [6]) the concept of segmentation ambiguity [7] illustrates one case: m = m . Segmentation ambiguity at one generating step (with its candidate pool CP ) during extraction can be denoted as: where S is the remaining steganographic text to be decoded. ∀ w ∈ W as the prefix of S respectively has their corresponding covert message, thus inducing ambiguity. Taking Chinese generative language models as an example (a typical unsegmented language) in Fig. 1 (one piece)'} in the second generating step. Hence, the receiver decodes the texts into 2 cases, i.e. '00' and '01', as |W | = 2.
The existing disambiguating approach [7] removes the tokens whose mapping words are the prefix of others, thus ensuring the receiver finding only one word matching the steganographic text. Fig. 2(a) shows that the approach removes the word ' (one)' in the second candidate pool. However, its side effects are changing conditional probabilities for tokens, which can have considerable implications on the security and its leading factor, imperceptibility. Modifying the probability distribution of candidate pools could increase embedding deviation [8] and has proved to damage steganographic imperceptibility [9].
Further, another optional solution is retaining the only word ' (one)' as depicted in Fig. 2(b), even though it seems like modifying the candidate pool more significantly. Especially in large-size candidate pools, more token-selection options inspire our motivation to investigate which words (tokens) should be retained, so as to maximize steganographic imperceptibility while addressing segmentation ambiguity. In this letter, our contributions are three-fold: 1) We propose a secure token-selection principle of a positive correlation between the sum of selected tokens' probabilities and statistical imperceptibility. 2) We present a lightweight disambiguating approach that is finding out a maximum weight independent set (MWIS) in one candidate graph. Our approach is carried out only if candidate-level ambiguity occurs. 3) Our approach enhances various security metrics, including 25.7% higher statistical imperceptibility and 11.2% higher anti-steganalysis capacity, compared to the existing method, through quantitative experiments.

A. Candidate Pools of Generative Steganography
For tokens belonging to the whole vocabulary Σ, the original candidate pool for the predicted word is denoted as [c 1 , c 2 , . . ., c |Σ| ] with corresponding probabilities [p 1 , p 2 , . . ., p |Σ| ], where |Σ| j=1 p j = 1. However, the oversized candidate pool would inevitably bring about excessive overhead for disambiguation in each candidate pool. Thus, it is reasonable and necessary to carry out the traditional pre-processing approach as top-k or top-p truncating method.

B. Reduction of Disambiguation
In a top-k candidate pool CP k , prefix relationships are the necessary but not sufficient condition for ambiguity. Though ensuring no prefix relationship of candidate pools strictly addresses ambiguity, it processes each candidate pool.
Definition 1: Token-selection disambiguation is reduced as finding out CP a . CP a is a subset of CP k and contains no prefix relationship of any pair of words, and obviously where P re denotes the prefix relationship: The disambiguation could be formalized by constructing a candidate graph, G(V, E, w). V is the set of nodes representing all nodes of CP k . E is the set of directed edges connecting a pair of nodes, and Finally, w associates every positive weight w i with node v i in V , which is equivalent to each probability p i of token c i .
Definition 2: Token-selection disambiguation is defined as finding out an independent set V a of V in G. Each node of V a has no neighbour with each other.  . Example for graph definition in a top-8 candidate pool, where capital letters represent the characters for arbitrary languages. Through the disambiguating method adopted by [7], the nodes (words) in green boxes constitute an independent set of the candidate graph.

A. Secure Token-Selection Principle
Steganographic security is highly related to its imperceptibility [3] which determines its capacibility of escaping from adversaries' suspicions. Kullback-Leibler divergence (KL divergence) acts a leading factor to assess steganographic imperceptibility in the following two aspects: r KLD-t (KLD between texts) indicates KL divergence between the distribution of cover texts c and steganographic texts s, which is a reliable criterion to assess the statistical imperceptibility, according to the information-theoretic security [10]. KLD-t is not only induced by deviation between language models and real-environment and naturallanguage texts, but also stems from embedding algorithms (f emb ) [9]. r KLD-c (KLD between candidates) indicates KL divergence between distribution (CP a ) modified by f emb and CP of language model, which acts a leading trigger exacerbating KLD-t [9], [11]. Hence, KLD-c significantly affects KLD-t and can further influence imperceptibility and other security metrics.
Assumption 1: Maximizing the sum of probabilities in CP a will minimize KLD-c, further optimizing imperceptibility.
Proof: The probability distribution of CP a has been processed to P a = [ p a1 η a , p a2 η a , . . .] from [p a1 , p a2 , . . .] via normalization process, where η a = j p aj . The probability distribution CP is P = [p 1 , p 2 , . . ., p |Σ| ]. KLD-c is showed as the following mathematical formulation: Therefore, KLD-c in each generating step is dependent on η a through the above proof. The positive effects on imperceptibility and other security criteria by maximizing η a will be experimentally proved in Section IV.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.

B. Maximizing the Sum of the Selected Probabilities
This section attempts to construct the CP a with maximum η a , so as to optimize steganographic imperceptibility. Considering adjusting all candidate pools brings about not only unnecessary overhead but also diminishes η a . Hence, it is essential to identify a situation called candidate-level ambiguity that there is at least one prefix relationships between the token originally intended for output and others. Candidate pools without candidate-level ambiguity never cause segmentation ambiguity, thus in this case the optimal CP a is just CP k . Considering candidate-level ambiguity in Fig. 3, there is no prefix relationship with 'B' and others. If the word to be output is 'B', η a = η k = 0.99. In other cases, candidate-level ambiguity occurs, thus it is necessary to find out an independent set and then select a new word to be output.
Definition 3: Minimizing KL divergence of probability distributions of CP a and CP is defined as finding out a maximum weight independent set (MWIS) [12], [13] of the graph G after candidate-level ambiguity occurs.
A subset V a of V can be denoted by an indicator vector x = (x i ) ∈ {0, 1} n , where x i = 0 means node c i ∈ V a , and x i = 1 means node c i / ∈ V a . Hence, MWIS solution represented as x * is described as the following integer program:

C. MWIS-Based Disambiguating Algorithm
MWIS is an optimization problem, which has proved an NP-hard problem [14], leading to efficiency problems especially in one large-scale graph [15], [16]. To mitigate the heavy overhead, the efficiency of MWIS in the graph G could not only benefit from the limited candidate-pool size, but also be further simplified via finding out connected components. Algorithm 1 shows the disambiguating process for one candidate pool. MWIS solution in Line 7 our approach respectively attempts two types of algorithms: 1) Greedy Algorithm: Our MWIS solution attempts a greedy algorithm which has been proved its lower limit is [17], in each G i . Algorithm 2 provides more details, where Line 7 could be fulfilled by dynamic programming in linear time based on a Input: if no more one neighbours of v j i has been visited then 4: F ← F ∪ {v j i } 5: for each tree T m ∈ F do 6: Collect nodes respectively in layers L = [l 1 , l 2 , . . .] via a root; 7: Find the subset L ⊂ L, where layers of L are nonadjacent and its weight sum is the maximum, and add all nodes of L into S i 8: return M i fact that F is a forest, which ensures arbitrary two nodes in nonadjacent layers have no edge.
2) Enumeration Algorithm: Enumeration algorithm traverses all cases in each connected component G i . Though its time complexity is obviously O(2 |V i | ), it definitely obtains the optimal solution and may be still a fast way in small-scale connected components (when |V i | is limited).

D. Disambiguating Message Extraction
The disambiguating process of f ext is the inverse process of f emb but more complicated, and they both process candidate pools based on the same disambiguating approach (Algorithm 1). There are two cases during extraction: (1) If only one word in a candidate pool is the prefix of the remaining stegatext, this word is deemed as t o and the whole process of Algorithm 1 should be executed; (2) If more than one words, the segmentation ambiguity occurs thus the MWIS process (Line 3-8 of Algorithm 1) must be executed.  (BITS PER TOKEN), PERPLEXITY, KL DIVERGENCE (KLD-C AND  KLD-T), STEGANALYSIS ACCURACY (ACCURACY-1 AND ACCURACY-2), UNDER VARIOUS TOP-K CASES   TABLE II  RUNNING TIMES BETWEEN OUR APPROACH AND THE BASELINE METHOD [7] UNDER VARIOUS TOP-K CASES (IN SECONDS)

A. Experimental Setup
Model: Our experiments utilize a Chinese Pre-trained Language Model (CPM) [18] as an unsegmented-language and off-the-shelf model with the phrase-level tokenization and detokenization based on a https://github.com/yangjianxin1/ CPMdataset of 260,000 Chinese essays (1.06 GB). Its frequency of segmentation ambiguity is considerable to estimate disambiguating performance of our approach, compared to rebuilding the baseline method [7].
Coding rule: Our experiments adopt arithmetic coding [19] as the coding rule. It has high compression rate, which results in less damage to conditional probability distribution. In arithmetic coding, KLD-c primarily depends on adopted disambiguating approaches.
Implementation: Our experiments are implemented with Python 3.7.14 and based on torch 1.7.0, using Tesla T4 and CUDA 11.2 as acceleration. The covert message to be embedded is 64-bit pseudo-random bitstream. All the prompts for our experiments are empty strings ∅. For each top-k in each disambiguation approach, its experimental statistics stem from generating and extracting 10,000 stegatexts, except for top-64 and top-128 in enumeration MWIS (the number of stegatexts is 2000) due to much more time of process.

B. Metrics of Security
Fluency: Perplexity (PPL) [20] is a standard metric used to evaluate text quality and fluency in the filed of natural language generation (NLG).
Imperceptibility: KL divergences both in probability distributions of subset candidates with original candidates (KLD-c) and in generated sentences with cover sentences (KLD-t) can estimate statistical difference and imperceptibility.
Anti-Steganalysis: One steganalysis method is Sesy [21]. Another method is fine-tuning https://huggingface.co/bert-basechinesebert-base-chinese into a discriminator. The size of nonsteganographic sentences (from the dataset) and steganographic sentences for each top-k in each disambiguation approach are the same (10,000 or 2,000). Then they are split into training (60%), validation (20%) and test (20%) parts. Non-steganographic sentences are truncated so that they had comparable lengths with steganographic texts, to keep the fairness. The optimizer is Adam, learning rates are initially set as 10 −5 , batch size is 16, the number of training epoch is 10, and dropout is 0.1. The detection accuracy of Sesy is denoted as 'Accuracy-1' and the accuracy of the other is denoted as 'Accuracy-2'. Table I illustrates the average performances between our approach and the baseline method [7] including embedding rate (bits per token) and four security metrics, remaining within the top-8 to top-128 CP k range. There is a remarkable improvement in embedding capacity denoted by 12.4%-30.8% higher bits per token.

C. Results and Analysis
Overall, in security criteria, our approach via greedy or enumeration MWIS solution outperforms the baseline. Our approach obtains higher imperceptibility denoted by 17.8%-31.9% (averagely 25.7%) lower KLD-t, the KL divergence between steganographic texts and normal texts. Ours also achieves higher anti-steganalysis capacity indicated by 8.3%-16.3% (averagely 11.2% combining two methods) lower detection accuracy. In terms of texts' fluency and quality, perplexity decreases slightly. Besides, those statistics imply an opposite inclination of fluency (perplexity) of other three security metrics, which complies with perceptual-statistical imperceptibility conflicting effect [3], indicating the potential of maintaining both high embedding capacity and security.
Enumeration solution achieves overall superior embedding rates. When CP k is relatively narrow, greedy MWIS solution's performance is more closer to that of the theoretically optimal enumeration solution. Besides, Table II indicates the overhead of running time between two MWIS solutions and the baseline, thus the high-overhead demerit of enumeration is uncovered, resulting from its exponential time complexity. This table shows our shorter running time and lightweight overhead, and polynomial-time greedy solution is much more feasible for high-embedding-rate requirements. The running time in different methods does not increase or decrease monotonically with top-k, which could be primarily attributed to higher embedding rate as well as less generating steps.

V. CONCLUSION
In this letter, we propose a secure and disambiguating approach for generative linguistic steganography via MWIS-based algorithms. Compared to the previous method, our approach considerably improves statistical imperceptibility, embedding rate, and anti-steganalysis capacity, meanwhile diminishing overhead, running time. Further, our lightweight approach could be potentially applied to various disambiguating scenarios, including more unsegmented-language models and other subword-supporting language models.