Back

CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Low Resource With Contrastive Learning

Xiaoming LiuZhaohan ZhangYichen WangHang PuYu LanChao Shen

article2023EMNLP56 citations

Proposes a detector that integrates entity-based coherence graphs with hard-negative contrastive learning to accurately identify machine-generated text in data-scarce settings.

Listen

Rapid advances in text generative artificial intelligence models make it easy to generate vast volumes of human-like text, increasing the risks of automated spam, forged reviews, and widespread disinformation. Distinguishing machine-generated text from human-written text has become critical, yet standard automated detectors face significant operational hurdles. Most existing solutions treat documents simply as flat word sequences without modeling high-level discourse structure, and they perform poorly in data-scarce settings where only limited human-annotated examples are available.

The article develops and evaluates an automated detection model called COCO (Coherence-Enhanced Contrastive Learning). The core objective is to determine whether incorporating document coherence structures and contrastive learning techniques can significantly improve machine-generated text detection, particularly in low-resource environments.

To achieve this, the authors model discourse coherence using entity consistency across sentences, converting documents into entity graphs that capture relationships both within and between sentences. They then combine these structural graph representations with language model sequence embeddings inside a supervised contrastive learning framework. This framework employs a dynamic memory bank and an improved loss function designed to prioritize difficult training samples over simple negative examples. The model was evaluated across multiple benchmark datasets, including news and web text generated by models such as GROVER, GPT-2, GPT-3.5, and GPT-4, testing both low-resource scenarios (500 training examples) and full dataset settings.

The evaluation revealed several key findings. First, in limited-data environments, COCO outperformed leading baseline detectors, improving accuracy by 1.23% to 3.07% on challenging datasets like GROVER and GPT-2, while maintaining state-of-the-art results on full datasets. Second, human-written text exhibits significantly more complex structural coherence than machine-generated text; static geometric analyses revealed human texts have roughly 35% more entity nodes, 64% more relational edges, and higher structural entropy. Third, against common expectations, modern text generated by large-scale models like GPT-3.5 is consistently easier to detect (exceeding 97% to 99% accuracy across detectors) than text from older, smaller, or adversarially trained generators like GROVER. Statistical attribution analysis indicates this ease stems from advanced models producing generalized language patterns across continuous token spans.

These findings demonstrate that discourse-level coherence provides a reliable, explainable signal for content verification systems. Organizations deploying automated content moderation or intellectual property controls can leverage graph-based coherence modeling to build robust classifiers that do not require massive labeled datasets, reducing labeling and fine-tuning costs.

Organizations should adopt coherence-aware and contrastive detection architectures rather than relying solely on simple metric-based classifiers or unaugmented sequence models. Before large-scale deployment, operational teams must conduct pilot testing under extreme class imbalances, as detector performance drops significantly when human-written text accounts for less than 30% of the training pool. Further research should also focus on developing efficient, adaptive graph-generation methods for short texts, code, and evolving instruction-tuned models.

While the findings are well supported across diverse generative architectures, confidence should be tempered by real-world operational constraints. The detector experiences high computational overhead when building graphs for large datasets, exhibits limited applicability to short texts or source code where named entities are sparse, and remains vulnerable to severely skewed data distributions where machine-generated content vastly outnumbers human text.

Queen Mary University of LondonXi'an Jiaotong Universitycontrastive learninggenerative modellanguage modelmachine-generated text detectioncontrastive learningtext coherence
YichenZW/Coh-MGT-DetectionLiu et al (2023).pdf

No sufficiently relevant recommendations were found.

Cover for CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Low Resource With Contrastive Learning

Abstract

Machine-Generated Text (MGT) detection, a task that discriminates MGT from Human-Written Text (HWT), plays a crucial role in preventing misuse of text generative models, which excel in mimicking human writing style recently. The latest proposed detectors usually take coarse text sequences as input and fine-tune pre-trained models with standard cross-entropy loss. However, these methods fail to consider the linguistic structure of texts. Moreover, they lack the ability to handle the low-resource problem, which could often happen in practice considering the enormous amount of textual data online. In this paper, we present a coherence-based contrastive learning model named CoCo to detect the possible MGT under the low-resource scenario. To exploit the linguistic feature, we encode coherence information in the form of graph into the text representation. To tackle the challenges of low data resources, we employ a contrastive learning framework and propose an improved contrastive loss for preventing performance degradation brought by simple samples. The experiment results on two public datasets and two self-constructed datasets prove our approach outperforms the state-of-the-art methods significantly. Also, we surprisingly find that MGTs originated from up-to-date language models could be easier to detect than these from previous models, in our experiments. And we propose some preliminary explanations for this counter-intuitive phenomenon. All the codes and datasets are open-sourced.

Topics

contrastive learninggenerative modellanguage modeltext detectiontext documenttext representationtextual datatext datadetection algorithmsyntactic features

Keywords

machine-generated text detectioncontrastive learningtext coherencecoherence modelingcoherence encodercoherence graphslow-resource text detectionentity coherence

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 3 Methodology
  • 3.1 Coherence Graph Construction
  • 3.2 Supervised Contrastive Learning
  • 3.2.1 Model Overview
  • 3.2.2 Positive/Negative Pair Definition
  • 3.2.3 Encoder Design
  • 3.2.4 Dynamic Memory Bank
  • 3.2.5 Loss Function
  • 3.2.6 Momentum Update
  • 4 Experiments
  • 4.1 Datasets
  • 4.2 Comparison Models
  • 4.3 Performance Comparison
  • 4.4 Ablation Study
  • 4.5 Discussion
  • 4.5.1 Model Robustness to Perturbation
  • 4.5.2 Statistic Cues for Detectable Feature in GPT-3.5
  • 5 Conclusion
  • Limitations
  • Ethical Considerations
  • Acknowledgements
  • References
  • A Basic Statistics of Datasets
  • B Details of GPT-3.5 Dataset
  • B.1 Human Written Texts
  • B.2 Machine Generated Texts
  • C GPT-3.5 Dataset Generated by Different Prompts and Experiment Results
  • D Implementation Details
  • E More Comparison Experiments
  • F Effect of Hyper-Parameters
  • F.1 Contrastive Learning Parameters
  • F.2 Graph Parameters
  • G Ablation Study
  • H Case Study
  • H.1 Coherence Graph Difference
  • H.2 Token Importance in GPT-3.5 Detection
  • I Static Geometric Analysis on Coherence Graph
  • I.1 Degree Distribution
  • I.2 Aggregation
  • I.3 Core & Degeneracy
  • I.4 Entropy
  • J Exploration on Imbalanced Data
  • K Related Work: Graph-based Text Representation

Knowls

  1. Knowl 1 — Entity coherence graphs encode sentence structure

    definition

    COCO represents a document’s coherence as an undirected graph whose nodes are named entities extracted by an ELMo-based TagLM named-entity recognizer. It connects distinct entities that occur in the same sentence with an inner-sentence edge, and connects occurrences of the same entity across different sentences with an inter-sentence edge. These relations are intended to encode within-sentence structure and cross-sentence entity continuity, respectively. In an analysis of GROVER data, human-written texts (HWTs) and machine-generated texts (MGTs) had similar average token counts (463.2 versus 456.0), but the HWT graphs averaged 43.60 vertices and 107.4 edges, compared with 32.37 vertices and 65.44 edges for MGTs. Their average graph degrees were 2.980 for HWTs and 2.591 for MGTs, supporting the authors’ claim that entity-graph structure provides a distinguishable signal.

  2. Knowl 2 — Coherence Encoder Module fuses graph and sequence representations

    model/method

    COCO’s Coherence Encoder Module (CEM) combines entity-graph structure with the document’s RoBERTa sequence representation. For each entity, it initializes the graph-node vector by averaging the contextual RoBERTa embeddings of the tokens in that entity span. A two-layer relation-aware graph convolutional network then processes inner- and inter-sentence edges separately, using relation-specific transformations and summing their outputs; the authors use two layers to reduce overfitting in low-resource training. Within each sentence, transformed entity vectors are aggregated into a sentence vector. CEM applies scaled self-attention over the sentence vectors and feeds the attended sequence to an LSTM; the final LSTM state is the document’s coherence vector. Concatenating this vector with RoBERTa’s final-layer [CLS] vector yields the coherence-enhanced document representation. The representation therefore combines entity relations at graph level, sequential sentence coherence, and the original token-sequence encoding.

  3. Knowl 3 — Hard-negative reweighting improves supervised contrastive learning

    model/method

    COCO defines examples with the same HWT/MGT label as positive pairs and examples with different labels as negative pairs. For a query vector qiq_i and a key vector kjk_j, pair similarity is Sij=exp⁡(qi⊤kj/τ)S_{ij}=\exp(q_i^\top k_j/\tau), where τ>0\tau>0 is a temperature. The improved contrastive term gives each positive pair its similarity in the numerator and uses the sum of positive similarities plus weighted negative similarities in the denominator. A negative key knk_n receives a weight proportional to its query–key dot product relative to the mean dot product over that query’s negative keys, scaled by a coefficient β\beta; thus relatively similar, harder negatives receive more emphasis than easy negatives. The contrastive term is combined with binary cross-entropy classification loss as Ltotal=αLICL+(1−α)LCEL_{\mathrm{total}}=\alpha L_{\mathrm{ICL}}+(1-\alpha)L_{\mathrm{CE}}, where α∈[0,1]\alpha\in[0,1] balances instance-level contrastive learning and class-level discrimination. The authors report their best tested contrastive hyperparameters as α=0.6\alpha=0.6 and τ=0.2\tau=0.2.

  4. Knowl 4 — Momentum encoders and a FIFO memory bank supply contrastive pairs

    model/method

    COCO uses a momentum-contrast training design with a query encoder and a key encoder, both initialized identically and both implementing the coherence-enhanced text encoder. The query encoder and a linear classifier are updated by back-propagation from the combined contrastive and cross-entropy objective. The key encoder is updated by an exponential moving average of the query-encoder parameters rather than by direct gradient descent. A dynamic memory bank stores key representations and their labels, with capacity equal to the number of training examples; it is maintained as a first-in, first-out queue, replacing older entries with newly encoded keys. The bank supplies positive and negative examples beyond the current mini-batch, which the authors intend to make contrastive training more effective when labeled training data are scarce.

  5. Knowl 5 — Evaluation covers four datasets and balanced low-resource training

    experimental setup

    COCO was evaluated on the GROVER news dataset, the GPT-2 WebText-style dataset, and two self-constructed GPT-3.5 news subsets. GROVER pairs RealNews human texts with Grover-Mega generations; GPT-2 pairs WebText with GPT-2 XLM-1542M generations. The GPT-3.5 subsets use text-davinci-003 generations imitating news: the unmixed human texts come from The New York Times, while the mixed subset draws from multiple news sources. The training splits contain 5,000 HWT and 5,000 MGT examples for GROVER, 25,000 of each for GPT-2, 3,454 of each for GPT-3.5 unmixed, and 3,032 of each for GPT-3.5 mixed. Low-resource experiments use a randomly sampled, balanced set of 500 training examples; full-data experiments use the full training split. Model-based results are reported as means and standard deviations over 10 random seeds. The implementation uses RoBERTa-base, AdamW with learning rate 10−510^{-5} and weight decay 0.01, and batch size 8; the selected graph limits are 90 entity nodes and 45 sentences.

  6. Knowl 6 — COCO improves detection results across the evaluated datasets

    empirical result

    In the 500-example setting, COCO achieved the following accuracy/F1 results, compared with the supervised-contrastive CE+SCL baseline: GROVER, 0.6993 ± 0.0119 / 0.6125 ± 0.0159 versus 0.6870 ± 0.0142 / 0.5961 ± 0.0197; GPT-2, 0.8530 ± 0.0019 / 0.8410 ± 0.0018 versus 0.8355 ± 0.0046 / 0.8127 ± 0.0067; GPT-3.5 unmixed, 0.9889 ± 0.0044 / 0.9791 ± 0.0062 versus 0.9823 ± 0.0053 / 0.9703 ± 0.0070; and GPT-3.5 mixed, 0.9701 ± 0.0069 / 0.9735 ± 0.0086 versus 0.9628 ± 0.0077 / 0.9686 ± 0.0062. With full training data, COCO scored 0.8826 ± 0.0018 / 0.8265 ± 0.0036 on GROVER, 0.9457 ± 0.0004 / 0.9452 ± 0.0004 on GPT-2, 0.9972 ± 0.0015 / 0.9957 ± 0.0020 on GPT-3.5 unmixed, and 0.9932 ± 0.0019 / 0.9937 ± 0.0028 on GPT-3.5 mixed. It exceeded CE+SCL on both metrics in these full-data settings except that accuracy tied at 0.9932 on GPT-3.5 mixed. The paper also reports COCO’s advantage over the compared methods across its broader benchmark experiments.

  7. Knowl 7 — Ablations isolate gains from coherence, aggregation, and contrastive learning

    empirical result

    On a 1,000-example GROVER experiment, the plain RoBERTa detector scored 0.7697 accuracy and 0.6428 F1. Replacing entity-graph modeling with sentence nodes scored 0.7733 / 0.6379; adding entity coherence scored 0.7777 / 0.6463; adding the attention LSTM to coherence scored 0.7787 / 0.6471; adding traditional supervised contrastive learning to coherence plus the LSTM scored 0.7827 / 0.6609; and full COCO, with its improved contrastive loss, scored 0.7843 / 0.6684. The authors interpret these comparisons as evidence that entity coherence, sentence-level LSTM aggregation, contrastive learning, and hard-negative reweighting each contribute to performance. The sentence-node variant’s lower F1 than the plain model also supports their choice to represent entities rather than sentences as graph nodes.

  8. Knowl 8 — GPT-3.5-generated news was unusually detectable in these experiments

    empirical result

    Across the paper’s tests, GPT-3.5-generated news was easier for detectors to classify than the GROVER- and GPT-2-generated datasets. This held even in additional tests that varied generation prompts to provide keywords, summaries, or outlines and sought to reduce differences in text length and semantics. To investigate the mixed GPT-3.5 dataset, the authors used integrated-gradient token attributions from a fine-tuned RoBERTa classifier. The fraction of positively supporting consecutive token spans was higher for MGT than HWT at every tested span length: for 1-grams, 0.6659 versus 0.6377; 2-grams, 0.4250 versus 0.3630; 3-grams, 0.2883 versus 0.2076; 4-grams, 0.2019 versus 0.1372; and 5-grams, 0.1425 versus 0.0935. The three individual tokens with the highest reported productivity were according (0.6923 productivity; 0.3126 coverage), where (0.6842; 0.1998), and they (0.6316; 0.3837). The authors could not identify a semantic explanation for these individual words and note that each covered less than 0.4 of the data. They therefore hypothesize—not demonstrate—that detectability is more associated with recurring multi-token language patterns than with particular words, possibly because the advanced generator produces expressions that are readily learned by fine-tuned detectors.

  9. Knowl 9 — COCO remains competitive under token perturbations

    empirical result

    For a low-resource GROVER detector, the authors tested four 15% test-text perturbations: random token deletion, repetition of selected tokens, insertion of random vocabulary tokens, and replacement of tokens with randomly selected vocabulary tokens. Averaged across the original test set and the perturbation conditions, RoBERTa achieved 0.5984 accuracy and 0.5246 F1, while COCO achieved 0.6437 accuracy and 0.5425 F1. Relative to each model’s original result, the average accuracy changes were −0.0651 for RoBERTa and −0.0556 for COCO; average F1 changes were −0.0655 and −0.0700, respectively. Thus COCO retained higher average accuracy under these perturbations, although its average F1 decline was slightly larger than RoBERTa’s.

  10. Knowl 10 — Low-resource, imbalance, and input-type constraints limit deployment

    limitation

    The authors identify several limits to COCO’s applicability. Building an entity coherence graph for every document adds processing cost, particularly on larger datasets, and the approach is poorly suited to short texts, code, and mathematical proofs where such graphs are difficult to construct. In an exploration using a 10% portion of GROVER data, all tested models’ accuracy fell sharply when HWTs made up less than 30% of training data; at a 10% HWT share, most models achieved below 50% accuracy, near random performance. The authors also caution that changing generation methods, including instruction-based and human-in-the-loop systems, may alter the patterns that the current entity-relation representation detects. They identify improved sampling for severe class imbalance and more efficient, adaptive text features as directions for future work.

Coverage note — The appendix’s additional comparisons across eight generators and its detailed degree, aggregation, core-number, and entropy analyses were omitted because their central evidence is summarized by the reported coherence-graph statistics and benchmark results; prompt-construction examples and case-study visualizations were omitted as supporting illustrations.

References

  1. 1.David Ifeoluwa Adelani, Haotian Mai, Fuming Fang, Huy H Nguyen, Junichi Yamagishi, and Isao Echizen. 2020. Generating sentiment-preserving fake online reviews using neural language models and their human-and machine-based detection. In International Conference on Advanced Information Networking and Applications, pages 1341–1354. Springer.
  2. 2.Ernst Althaus, Nikiforos Karamanis, and Alexander Koller. 2004. Computing locally coherent discourses. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 399–406.
  3. 3.Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403.
  4. 4.Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and Arthur Szlam. 2019. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351.
  5. 5.Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373.
  6. 6.Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745.
  7. 7.Roi Blanco and Christina Lioma. 2011. Graph-based term weighting for information retrieval. Information Retrieval, 15:54–92.
  8. 8.Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared J Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  9. 9.Tiffany Tianhui Cai, Jonathan Frankle, David J Schwab, and Ari S Morcos. 2020. Are all negatives created equal in contrastive instance discrimination? arXiv preprint arXiv:2010.06682.
  10. 10.Qianben Chen, Richong Zhang, Yaowei Zheng, and Yongyi Mao. 2022. Dual contrastive learning: Text classification via label-aware data augmentation. arXiv preprint arXiv:2201.08702.
  11. 11.Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297.
  12. 12.Yutian Chen, Hao Kang, Vivian Zhai, Liangze Li, Rita Singh, and Bhiksha Ramakrishnan. 2023. Gpt-sentinel: Distinguishing human and chatgpt generated content. arXiv preprint arXiv:2305.07969.
  13. 13.Tianyu Gao, Adam Fisch, and Danqi Chen. 2021a. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816–3830.
  14. 14.Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021b. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910.
  15. 15.Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019. Gltr: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116.
  16. 16.Nir Grinberg, Kenneth Joseph, Lisa Friedland, Briony Swire-Thompson, and David Lazer. 2019. Fake news on twitter during the 2016 us presidential election. Science, 363(6425):374–378.
  17. 17.Barbara J Grosz and Candace L Sidner. 1986. Attention, intentions, and the structure of discourse. Computational linguistics, 12(3):175–204.
  18. 18.Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2021. Supervised contrastive learning for pre-trained language model fine-tuning. In International Conference on Learning Representations.
  19. 19.Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597.
  20. 20.Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738.
  21. 21.Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023. Mgtbench: Benchmarking machine-generated text detection. arXiv preprint arXiv:2303.14822.
  22. 22.Xiaochen Hou, Peng Qi, Guangtao Wang, Rex Ying, Jing Huang, Xiaodong He, and Bowen Zhou. 2021. Graph ensemble learning over multiple dependency trees for aspect-level sentiment classification. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2884–2894.
  23. 23.Eduard H Hovy. 1988. Planning coherent multisentential text. In Proceedings of the 26th annual meeting on Association for Computational Linguistics, pages 163–169.
  24. 24.Binxuan Huang and Kathleen M Carley. 2019. Syntax-aware aspect level sentiment classification with graph attention networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5469–5477.
  25. 25.Kung-Hsiang Huang, Preslav Nakov, Yejin Choi, and Heng Ji. 2022. Faking fake news for real fake news detection: Propaganda-loaded training data generation. ArXiv, abs/2203.05386.
  26. 26.Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808–1822.
  27. 27.Nikiforos Karamanis and Hisar Maruli Manurung. 2002. Stochastic text structuring using the principle of continuity. In Proceedings of the International Natural Language Generation Conference, pages 81–88.
  28. 28.Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
  29. 29.Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
  30. 30.Mirella Lapata. 2003. Probabilistic text structuring: Experiments with sentence ordering. In ACL, volume 3, pages 545–552. Citeseer.
  31. 31.Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880.
  32. 32.Xien Liu, Xinxin You, Xiao Zhang, Ji Wu, and Ping Lv. 2020. Tensor graph convolutional networks for text classification. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8409–8416.
  33. 33.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  34. 34.Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations.
  35. 35.Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, and Pascale Fung. 2021. Few-shot bot: Prompt-based learning for dialogue systems. ArXiv, abs/2110.08118.
  36. 36.Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. 2022. Coherence boosting: When your pretrained language model is not paying enough attention. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8214–8236, Dublin, Ireland. Association for Computational Linguistics.
  37. 37.Fragkiskos D Malliaros and Konstantinos Skianis. 2015. Graph-based term weighting for text categorization. In Proceedings of the 2015 IEEE/ACM international conference on advances in social networks analysis and mining 2015, pages 1473–1479.
  38. 38.William C Mann and Sandra A Thompson. 1987. Rhetorical structure theory: A theory of text organization. University of Southern California, Information Sciences Institute Los Angeles.
  39. 39.Chris Mellish, Alistair Knott, Jon Oberlander, and Mick O’Donnell. 1998. Experiments using stochastic search for text planning. In Proceedings of the 9th International General Workshop, pages 98–107. ACL Anthology.
  40. 40.Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 404–411.
  41. 41.Fatemehsadat Mireshghallah, Justus Mattern, Sicun Gao, Reza Shokri, and Taylor Berg-Kirkpatrick. 2023. Smaller language models are better black-box machine-generated text detectors. arXiv preprint arXiv:2305.09859.
  42. 42.Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. arXiv preprint arXiv:2301.11305.
  43. 43.Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4658–4664.
  44. 44.OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
  45. 45.Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  46. 46.Matthew E Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1756–1765.
  47. 47.Xiao Pu, Jingyu Zhang, Xiaochuang Han, Yulia Tsvetkov, and Tianxing He. 2023. On the zero-shot generalization of machine-generated text detectors. arXiv preprint arXiv:2310.05165.
  48. 48.Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  49. 49.Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551.
  50. 50.Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203.
  51. 51.Xiaohui Song, Longtao Huang, Hui Xue, and Songlin Hu. 2022. Supervised prototypical contrastive learning for emotion recognition in conversation. arXiv preprint arXiv:2210.08713.
  52. 52.Yixuan Su, Fangyu Liu, Zaiqiao Meng, Tian Lan, Lei Shu, Ehsan Shareghi, and Nigel Collier. 2022. Tacl: Improving bert pre-training with token-aware contrastive learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2497–2507.
  53. 53.Ruixiao Sun, Jie Yang, and Mehrdad Yousefzadeh. 2020. Improving language generation with sentence coherence objective. arXiv preprint arXiv:2009.06358.
  54. 54.Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR.
  55. 55.Enhua Tan, Lei Guo, Songqing Chen, Xiaodong Zhang, and Yihong Zhao. 2012. Spammer behavior analysis and detection in user generated content on social networks. In 2012 IEEE 32nd International Conference on Distributed Computing Systems, pages 305–314. IEEE.
  56. 56.Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  57. 57.Peter D Turney. 2002. Learning to extract keyphrases from text. arXiv preprint cs/0212013.
  58. 58.Adaku Uchendu, Jeffrey Cao, Qiaozhi Wang, Bo Luo, and Dongwon Lee. 2019. Characterizing man-made vs. machine-made chatbot dialogs. In TTO.
  59. 59.Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. 2020. Authorship attribution for neural text generation. In Conf. on Empirical Methods in Natural Language Processing (EMNLP).
  60. 60.Adaku Uchendu, Zeyu Ma, Thai Le, Rui Zhang, and Dongwon Lee. 2021. Turingbench: A benchmark environment for turing test in the age of neural text generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2001–2016.
  61. 61.Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/mesh-transformer-jax.
  62. 62.Feng Wang and Huaping Liu. 2021. Understanding the behaviour of contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2495–2504.
  63. 63.Max Welling and Thomas N Kipf. 2016. Semi-supervised classification with graph convolutional networks. In J. International Conference on Learning Representations (ICLR 2017).
  64. 64.Yuta Yanagi, Ryohei Orihara, Yuichi Sei, Yasuyuki Tahara, and Akihiko Ohsuga. 2020. Fake news detection with generated comments for news articles. In 2020 IEEE 24th International Conference on Intelligent Engineering Systems (INES), pages 85–90. IEEE.
  65. 65.Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32.
  66. 66.Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 7370–7377.
  67. 67.Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. Advances in neural information processing systems, 32.
  68. 68.Wanjun Zhong, Duyu Tang, Zenan Xu, Ruize Wang, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020. Neural deepfake detection with factual structure of text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2461–2470.

Citation

MLA
Liu, X., et al. “CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Low Resource With Contrastive Learning”. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 16167–88, https://doi.org/10.18653/v1/2023.emnlp-main.1005.
APA
Liu, X., Zhang, Z., Wang, Y., Pu, H., Lan, Y., & Shen, C. (2023). CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Low Resource With Contrastive Learning. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 16167–16188. https://doi.org/10.18653/v1/2023.emnlp-main.1005
Chicago
Liu, X., Z. Zhang, Y. Wang, H. Pu, Y. Lan, and C. Shen. 2023. “CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Low Resource With Contrastive Learning”. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 16167–88. https://doi.org/10.18653/v1/2023.emnlp-main.1005.
Harvard
Liu, X. et al. (2023) “CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Low Resource With Contrastive Learning”, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 16167–16188. Available at: https://doi.org/10.18653/v1/2023.emnlp-main.1005.
Vancouver
1. Liu X, Zhang Z, Wang Y, Pu H, Lan Y, Shen C (2023) CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Low Resource With Contrastive Learning. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp 16167–16188

BibTeX

@inproceedings{liu-etal-2023-coco,
    title = "{C}o{C}o: Coherence-Enhanced Machine-Generated Text Detection Under Low Resource With Contrastive Learning",
    author = "Liu, Xiaoming  and
      Zhang, Zhaohan  and
      Wang, Yichen  and
      Pu, Hang  and
      Lan, Yu  and
      Shen, Chao",
    editor = "Bouamor, Houda  and
      Pino, Juan  and
      Bali, Kalika",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.emnlp-main.1005/",
    doi = "10.18653/v1/2023.emnlp-main.1005",
    pages = "16167--16188"
}
Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: https://creativecommons.org/licenses/by/4.0/