Performance Evaluation of Some Clustering Algorithms and Validity Indices

Ujjwal MaulikSanghamitra Bandyopadhyay

article2002TPAMI1,422 citations

Establishes theoretical lower bounds for the I cluster validity index while evaluating its ability, alongside traditional metrics and simulated annealing, to determine the optimal number of clusters across synthetic and real-world datasets.

Listen

In modern data analysis and machine learning, discovering natural groupings within unlabeled data is a critical capability across many industries, from market segmentation to medical diagnostics. A persistent challenge in this area is determining exactly how many distinct groups exist within a dataset and verifying how reliably an algorithm has partitioned them. Choosing the wrong number of clusters or an ineffective grouping technique can lead to incorrect insights and flawed operational decisions.

The article aims to evaluate how accurately different cluster evaluation metrics and grouping algorithms identify the true underlying structure across various datasets. It also establishes a formal theoretical foundation for a newer cluster evaluation metric, known as Index I, to guarantee reliable identification of well-separated groupings.

To conduct this evaluation, the analysis assessed three distinct grouping algorithms—standard iterative centroid clustering, hierarchical linking, and an advanced probabilistic optimization technique based on simulated annealing. These algorithms were paired with four validation metrics across five benchmark datasets, spanning both synthetic tests and real-world cases such as crude oil classification and breast cancer diagnosis. The datasets varied in dimensionality from two to ten features and contained between two and ten target clusters.

The findings show that Index I consistently outperformed established traditional validation measures, achieving a 100% success rate in identifying the correct number of clusters across all tested datasets, regardless of the underlying algorithm used. In contrast, standard metrics like the Davies-Bouldin, Dunn, and Calinski-Harabasz indices frequently failed on complex or overlapping datasets. Furthermore, the simulated annealing optimization method consistently avoided poor, suboptimal groupings, providing superior final partitions compared to standard centroid approaches when the correct cluster count was supplied.

These results demonstrate that adopting more balanced evaluation metrics and optimization-based grouping algorithms significantly reduces the risk of misidentifying underlying data patterns. For organizations relying on automated classification, using robust metrics minimizes misclassification errors and prevents poor strategic choices caused by flawed data segmentation.

Organizations and practitioners are encouraged to integrate Index I as a standard validation benchmark when evaluating unlabeled data structures, paired with global optimization algorithms when high partition accuracy is required. Future work should focus on extending these evaluation frameworks to fuzzy grouping models where data points can belong to multiple categories simultaneously, as well as testing non-standard distance measures and algorithmic computational speeds on larger-scale data.

Cover for Performance Evaluation of Some Clustering Algorithms and Validity Indices

Abstract

In this article, we evaluate the performance of three clustering algorithms, hard K-Means, single linkage, and a simulated annealing (SA) based technique, in conjunction with four cluster validity indices, namely Davies-Bouldin index, Dunn's index, Calinski-Harabasz index, and a recently developed index I. Based on a relation between the index I and the Dunn's index, a lower bound of the value of the former is theoretically estimated in order to get unique hard K-partition when the data set has distinct substructures. The effectiveness of the different validity indices and clustering methods in automatically evolving the appropriate number of clusters is demonstrated experimentally for both artificial and real-life data sets with the number of clusters varying from two to ten. Once the appropriate number of clusters is determined, the SA-based clustering technique is used for proper partitioning of the data into the said number of clusters.

Table of Contents

  • 1 INTRODUCTION
  • 2 CLUSTERING ALGORITHMS
  • 3 CLUSTER VALIDITY INDICES
  • 4 EXPERIMENTAL RESULTS
  • 4.1 Data Sets and Implementation Parameters
  • 4.2 Determining the Number of Clusters
  • 4.3 Determining the Appropriate Clustering
  • 5 DISCUSSION AND CONCLUSIONS
  • ACKNOWLEDGMENTS
  • REFERENCES

Knowls

  1. Knowl 1 — Index I Cluster Validity Measure

    definition

    The cluster validity index I(K)I(K) evaluates the quality of a partition of a dataset X={x1,x2,…,xn}⊂RNX = \{x_1, x_2, \dots, x_n\} \subset \mathbb{R}^N into KK clusters with cluster centers z1,z2,…,zKz_1, z_2, \dots, z_K. It is defined as:

    I(K)=(1K×E1EK×DK)pI(K) = \left( \frac{1}{K} \times \frac{E_1}{E_K} \times D_K \right)^p

    where:

    • EK=∑k=1K∑j=1nukj∥xj−zk∥E_K = \sum_{k=1}^K \sum_{j=1}^n u_{kj} \|x_j - z_k\| is the total intra-cluster distance for KK clusters, where ukj∈{0,1}u_{kj} \in \{0, 1\} represents the membership of data point xjx_j in cluster CkC_k.
    • E1=∑j=1n∥xj−z1∥E_1 = \sum_{j=1}^n \|x_j - z_1\| is the total scatter of the dataset computed with respect to the overall dataset centroid z1=1n∑j=1nxjz_1 = \frac{1}{n} \sum_{j=1}^n x_j. E1E_1 is constant for a given dataset.
    • DK=max⁡i,j=1K∥zi−zj∥D_K = \max_{i, j = 1}^K \|z_i - z_j\| measures the maximum Euclidean separation between any pair of cluster centers.
    • pp is a power exponent used to control the contrast between different cluster configurations (chosen as p=2p = 2).

    The index balances three competing factors:

    1. 1K\frac{1}{K} penalizes over-partitioning by decreasing monotonically with KK.
    2. E1EK\frac{E_1}{E_K} rewards cluster compactness, increasing as within-cluster scatter EKE_K decreases.
    3. DKD_K rewards cluster separation, increasing with KK up to the maximum pairwise distance between points in the dataset.

    The optimal number of clusters K∗K^* corresponds to the global maximum of the index:

    K∗=arg⁡max⁡KI(K)K^* = \arg\max_{K} I(K)

  2. Knowl 2 — Theoretical Lower Bound on Index I for Well-Separated Clusters

    theoretical result

    Let X={x1,x2,…,xn}⊂RNX = \{x_1, x_2, \dots, x_n\} \subset \mathbb{R}^N be partitioned into KK clusters C1,C2,…,CKC_1, C_2, \dots, C_K with centers z1,z2,…,zKz_1, z_2, \dots, z_K and cluster cardinalities n1,n2,…,nKn_1, n_2, \dots, n_K such that ∑k=1Knk=n\sum_{k=1}^K n_k = n. Define the cluster-wise mean squared error for cluster kk as:

    τk=1nk∑i=1nk∥xi−zk∥2\tau_k = \frac{1}{n_k} \sum_{i=1}^{n_k} \|x_i - z_k\|^2

    and let τmin⁡=min⁡1≤k≤Kτk\tau_{\min} = \min_{1 \le k \le K} \tau_k.

    Under the assumption K≤nK \le \sqrt{n}, Index I(K)I(K) (with p=2p = 2) and Dunn's cluster validity index νD\nu_D satisfy the relationship:

    I(K)⋅1νD2≥τmin⁡nI(K) \cdot \frac{1}{\nu_D^2} \ge \frac{\tau_{\min}}{n}

    When the dataset contains well-separated substructures such that Dunn's index satisfies νD>1\nu_D > 1 (the theoretical condition guaranteeing a unique hard KK-partition), Index I(K)I(K) is bounded from below by:

    I(K)≥τmin⁡nI(K) \ge \frac{\tau_{\min}}{n}

    As clusters become increasingly compact and separated, νD→∞\nu_D \to \infty and Index I(K)I(K) grows without bound.

  3. Knowl 3 — Simulated Annealing-Based Clustering with Probabilistic Redistribution

    algorithm

    The simulated annealing (SA) clustering algorithm finds a crisp KK-partition of nn points by minimizing the total squared error energy E=∑k=1K∑x∈Ck∥x−zk∥2E = \sum_{k=1}^K \sum_{x \in C_k} \|x - z_k\|^2, using probabilistic redistribution of data points to escape local minima.

    Input: Dataset X={x1,x2,…,xn}⊂RNX = \{x_1, x_2, \dots, x_n\} \subset \mathbb{R}^N, number of clusters KK, initial temperature Tmax⁡=100T_{\max} = 100, minimum temperature Tmin⁡=0.001T_{\min} = 0.001, cooling factor α=0.05\alpha = 0.05, iterations per temperature NT=100N_T = 100.
    Output: Final cluster centers Z={z1,z2,…,zK}Z = \{z_1, z_2, \dots, z_K\} and crisp partition {C1,C2,…,CK}\{C_1, C_2, \dots, C_K\}.
    1. Initialize cluster centers Z={z1,z2,…,zK}Z = \{z_1, z_2, \dots, z_K\} as KK randomly chosen points from XX.
    2. Assign each point xj∈Xx_j \in X to cluster CkC_k where k=arg⁡min⁡l=1,…,K∥xj−zl∥2k = \arg\min_{l=1,\dots,K} \|x_j - z_l\|^2.
    3. Compute initial energy E=∑k=1K∑x∈Ck∥x−zk∥2E = \sum_{k=1}^K \sum_{x \in C_k} \|x - z_k\|^2.
    4. Set temperature T←Tmax⁡T \leftarrow T_{\max}.
    5. while T>Tmin⁡T > T_{\min} do:
         for iter ←1\leftarrow 1 to NTN_T do:
           Create candidate partition {C1′,…,CK′}\{C_1', \dots, C_K'\}:
           for each cluster CjC_j (j=1,…,Kj = 1, \dots, K) and each point xi∈Cjx_i \in C_j do:
             for each cluster CkC_k (k≠jk \ne j) do:
               Let Dij=∥xi−zj∥D_{ij} = \|x_i - z_j\| and Dik=∥xi−zk∥D_{ik} = \|x_i - z_k\|.
               Reassign xix_i to CkC_k with probability exp⁡(−max⁡(0,Dik−Dij)T)\exp\left( - \frac{\max(0, D_{ik} - D_{ij})}{T} \right).
           Update centers Z′={z1′,…,zK′}Z' = \{z_1', \dots, z_K'\} to the means of points in {C1′,…,CK′}\{C_1', \dots, C_K'\}.
           Compute candidate energy E′=∑k=1K∑x∈Ck′∥x−zk′∥2E' = \sum_{k=1}^K \sum_{x \in C_k'} \|x - z_k'\|^2.
           Compute acceptance probability Paccept=11+exp⁡(−E−E′T)P_{\text{accept}} = \frac{1}{1 + \exp\left( - \frac{E - E'}{T} \right)}.
           Draw random number r∼Uniform(0,1)r \sim \text{Uniform}(0, 1).
           if r<Pacceptr < P_{\text{accept}} then:
             Z←Z′Z \leftarrow Z', E←E′E \leftarrow E', {C1,…,CK}←{C1′,…,CK′}\{C_1, \dots, C_K\} \leftarrow \{C_1', \dots, C_K'\}.
         Cool temperature: T←T×(1−α)T \leftarrow T \times (1 - \alpha).
    6. return ZZ and {C1,C2,…,CK}\{C_1, C_2, \dots, C_K\}.

    The computational complexity of distance evaluation per iteration is O(KnN)O(K n N), scaling linearly with cluster count KK, dataset size nn, and feature dimensionality NN.

  4. Knowl 4 — Empirical Identification of Optimal Cluster Number Across Validity Indices

    data/table

    The performance of four validity indices—Davies-Bouldin (DBDB), Dunn's index (νD\nu_D), Calinski-Harabasz (CHCH), and Index II—in detecting the true number of clusters K∗K^* was evaluated across three clustering algorithms: K-Means, Single Linkage (S-Link), and Simulated Annealing (SA).

    Data Set Algorithm DBDB νD\nu_D CHCH II
    AD_10_2 (True K=10K=10) K-means 8 18 2 10
    S-Link 8 4 2 10
    SA 8 5 2 10
    AD_4_3N (True K=4K=4) K-means 4 2 4 4
    S-Link 4 2 4 4
    SA 4 2 4 4
    AD_2_10 (True K=2K=2) K-means 30 14 2 2
    S-Link 30 14 2 2
    SA 30 14 2 2
    Crude_Oil (True K=3K=3) K-means 2 6/7 2 3
    S-Link 3/4 9–13 2 3
    SA 4 2 2 3
    Cancer (True K=2K=2) K-means 2 2 2 2
    S-Link 2 2 2 2
    SA 2 2 2 2

    Index II correctly identified the true number of clusters K∗K^* across all five datasets for all three clustering algorithms. In comparison:

    • DBDB failed on AD_10_2 (K=8K=8), AD_2_10 (K=30K=30), and Crude_Oil (yielding K=2,3/4,4K=2, 3/4, 4).
    • νD\nu_D failed on AD_10_2, AD_4_3N, AD_2_10, and Crude_Oil across all algorithms.
    • CHCH failed on AD_10_2 (K=2K=2) and Crude_Oil (K=2K=2).
  5. Knowl 5 — Formulations of Baseline Cluster Validity Indices

    definition

    Three baseline cluster validity indices evaluated alongside Index II are defined as follows:

    1. Davies-Bouldin (DBDB) Index: DB=1K∑i=1KRi,Ri=max⁡j≠i{Si+Sjdij}DB = \frac{1}{K} \sum_{i=1}^K R_i, \quad R_i = \max_{j \ne i} \left\{ \frac{S_i + S_j}{d_{ij}} \right\} where Si=1∣Ci∣∑x∈Ci∥x−zi∥S_i = \frac{1}{|C_i|} \sum_{x \in C_i} \|x - z_i\| is within-cluster scatter and dij=∥zi−zj∥d_{ij} = \|z_i - z_j\| is inter-cluster center distance. Lower values indicate better partitions; optimal K∗K^* minimizes DBDB.

    2. Dunn's (νD\nu_D) Index: νD=min⁡1≤i≤K{min⁡1≤j≤K,j≠i{δ(Ci,Cj)max⁡1≤k≤KΔ(Ck)}}\nu_D = \min_{1 \le i \le K} \left\{ \min_{1 \le j \le K, j \ne i} \left\{ \frac{\delta(C_i, C_j)}{\max_{1 \le k \le K} \Delta(C_k)} \right\} \right\} where cluster diameter Δ(Ck)=max⁡x,y∈Ck∥x−y∥\Delta(C_k) = \max_{x, y \in C_k} \|x - y\| and set distance δ(Ci,Cj)=min⁡x∈Ci,y∈Cj∥x−y∥\delta(C_i, C_j) = \min_{x \in C_i, y \in C_j} \|x - y\|. Larger values indicate better partitions; optimal K∗K^* maximizes νD\nu_D.

    3. Calinski-Harabasz (CHCH) Index: CH=trace(B)/(K−1)trace(W)/(n−K)CH = \frac{\text{trace}(B) / (K - 1)}{\text{trace}(W) / (n - K)} where between-cluster scatter trace is trace(B)=∑k=1Knk∥zk−z∥2\text{trace}(B) = \sum_{k=1}^K n_k \|z_k - z\|^2, within-cluster scatter trace is trace(W)=∑k=1K∑i=1nk∥xi−zk∥2\text{trace}(W) = \sum_{k=1}^K \sum_{i=1}^{n_k} \|x_i - z_k\|^2, nkn_k is the size of cluster kk, nn is total points, and zz is the dataset centroid. Larger values indicate better partitions; optimal K∗K^* maximizes CHCH.

  6. Knowl 6 — Benchmark Datasets for Cluster Validity Evaluation

    data/table

    The experimental evaluation was conducted on five benchmark datasets comprising three synthetic and two real-world datasets spanning 2 to 10 dimensions and 2 to 10 clusters:

    Dataset Name Number of Points (nn) Number of Clusters (KK) Dimensions (NN) Points per Cluster
    AD_10_2 500 10 2 50 per cluster
    AD_4_3N 402 4 3 101, 100, 101, 100
    AD_2_10 1000 2 10 492, 508
    Crude_Oil 56 3 5 7, 11, 38
    Cancer 683 2 9 444 (Benign), 239 (Malignant)
    • AD_10_2 is a 2D synthetic dataset with 10 overlapping clusters.
    • AD_4_3N is a 3D synthetic dataset with 4 compact clusters.
    • AD_2_10 is a 10D synthetic dataset generated from triangular distributions over [0,2]10[0, 2]^{10} for class 1 and [1,3]×[0,2]9[1, 3] \times [0, 2]^9 for class 2 with equal prior probabilities.
    • Crude_Oil is an overlapping real-world dataset with 5 chemical properties.
    • Cancer is the 9-dimensional Wisconsin Breast Cancer dataset with 2 classes.
  7. Knowl 7 — Two-Stage Automated Clustering Framework

    model/method

    The unsupervised classification procedure resolves the cluster count and cluster assignment in two distinct stages:

    1. Automated Cluster Number Selection:

      • Cluster counts are evaluated in the range K∈[Kmin⁡,Kmax⁡]K \in [K_{\min}, K_{\max}], where Kmin⁡=2K_{\min} = 2 and Kmax⁡=⌊n⌋K_{\max} = \lfloor\sqrt{n}\rfloor.
      • Partitions UKU_K are evolved using a candidate algorithm (K-Means, Single Linkage, or Simulated Annealing), and validity index values V(K)V(K) are computed.
      • The optimal cluster number K∗K^* is identified by optimizing the validity index (K∗=arg⁡max⁡KI(K)K^* = \arg\max_K I(K) for Index II).
    2. Partition Generation:

      • Given K∗K^*, the simulated annealing clustering technique with probabilistic point redistribution is executed with K=K∗K = K^* to produce the final crisp partition UK∗U_{K^*}. This avoids suboptimal local minima inherent to iterative gradient-descent schemes like K-Means.
      • The resulting tuple ⟨UK∗,K∗⟩\langle U_{K^*}, K^* \rangle is output as the clustering solution.
  8. Knowl 8 — Upper Bound on Cluster Search Space

    assumption

    In evaluating cluster validity indices and establishing theoretical bounds relating Index II to Xie-Beni and Dunn's indices, the number of clusters KK for a dataset of size nn is assumed to be bounded by:

    K≤nK \le \sqrt{n}

    This constraint defines the maximum search limit Kmax⁡=⌊n⌋K_{\max} = \lfloor\sqrt{n}\rfloor for cluster validity search across all evaluated datasets.

  9. Knowl 9 — Crisp and Euclidean Distance Limitations

    limitation

    The formulations of Index II, the baseline indices, and the simulated annealing clustering algorithm possess several specific limitations:

    1. Crisp Partition Restriction: The validity measures and clustering methods operate on crisp membership matrices where ukj∈{0,1}u_{kj} \in \{0, 1\}; fuzzy partition extensions where memberships lie continuously in [0,1][0, 1] require separate formulation.
    2. Metric Dependency: The formulations assume a Euclidean distance metric and hyperspheroidal cluster geometries, which may not be well suited for manifold or non-Euclidean data distributions.
    3. Theoretical Convergence: Formal analytical comparisons regarding the relative convergence rates of the different validity indices remain uncharacterized.

Coverage note — None was omitted; all primary contributions including definitions, algorithms, theoretical bounds, experimental data, empirical results, assumptions, and limitations have been extracted.

References

  1. 1.J.T. Tou and R.C. Gonzalez, Pattern Recognition Principles. Reading: Addison-Wesley, 1974.
  2. 2.A.K. Jain and R.C. Dubes, Algorithms for Clustering Data. Prentice Hall, 1988.
  3. 3.H. Frigui and R. Krishnapuram, "A Robust Competitive Clustering Algorithm with Application in Computer Vision," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 1, pp. 450-465, Jan. 1999.
  4. 4.B.S. Everitt, Cluster Analysis, Halsted Press, third ed., 1993.
  5. 5.U. Maulik and S. Bandyopadhyay, "Genetic Algorithm Based Clustering Technique," Pattern Recognition, vol. 33, pp. 1455-1465, 2000.
  6. 6.G.W. Milligan and C. Cooper, "An Examination of Procedures for Determining the Number of Clusters in a Data Set," Psychometrika, vol. 50, no. 2, pp. 159-179, 1985.
  7. 7.M. Meil a and D. Heckerman, "An Experimental Comparison of Several Clustering and Initialization Methods," Proc. 14th Conf. Uncertainty in Artificial Intelligence, pp. 386-395, 1998.
  8. 8.C. Fraley and A.E. Raftery, "How Many Clusters? Which Clustering Method? Answers Via Model-Based Cluster Analysis," The Computer J., vol. 41, no. 8, pp. 578-588, 1998.
  9. 9.L.O. Hall, I.B. Ozyurt, and J. C. Bezdek, "Clustering with a Genetically Optimized Approach," IEEE Trans. Evolutionary Computation, vol. 3, no. 2, pp. 103-112, 1999.
  10. 10.D.L. Davies and D.W. Bouldin, "A Cluster Separation Measure," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 1, pp. 224-227, 1979.
  11. 11.J.C. Dunn, "A Fuzzy Relative of the ISODATA Process and Its Use in Detecting Compact Well-Separated Clusters," J. Cybernetics, vol. 3, pp. 32-57, 1973.
  12. 12.R.B. Calinski and J. Harabasz, "A Dendrite Method for Cluster Analysis," Comm. in Statistics, vol. 3, pp. 1-27, 1974.
  13. 13.S. Kirkpatrik, C. Gelatt, and M. Vecchi, "Optimization by Simulated Annealing," Science, vol. 220, pp. 671-680, 1983.
  14. 14.S. Bandyopadhyay, U. Maulik, and M.K. Pakhira, "Clustering Using Simulated Annealing with Probabilistic Redistribution," Int'l J. Pattern Recognition and Artificial Intelligence, vol. 15, no. 2, pp. 269-285, 2001.
  15. 15.X.L. Xie and G. Beni, "A Validity Measure for Fuzzy Clustering," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, pp. 841-847, 1991.
  16. 16.J.C. Dunn, "Well Separated Clusters and Optimal Fuzzy Partitions," J. Cybernetics, vol. 4, pp. 95-104, 1974.
  17. 17.R.A. Johnson and D.W. Wichern, Applied Multivariate Statistical Analysis. Prentice Hall, 1982.

Citation

MLA
Maulik, U., and S. Bandyopadhyay. “Performance Evaluation of Some Clustering Algorithms and Validity Indices”. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 12, 2002, pp. 1650–54, https://doi.org/10.1109/TPAMI.2002.1114856.
APA
Maulik, U., & Bandyopadhyay, S. (2002). Performance evaluation of some clustering algorithms and validity indices. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(12), 1650–1654. https://doi.org/10.1109/TPAMI.2002.1114856
Chicago
Maulik, U., and S. Bandyopadhyay. 2002. “Performance Evaluation of Some Clustering Algorithms and Validity Indices”. IEEE Transactions on Pattern Analysis and Machine Intelligence 24 (12): 1650–54. https://doi.org/10.1109/TPAMI.2002.1114856.
Harvard
Maulik, U. and Bandyopadhyay, S. (2002) “Performance evaluation of some clustering algorithms and validity indices”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(12), pp. 1650–1654. Available at: https://doi.org/10.1109/TPAMI.2002.1114856.
Vancouver
1. Maulik U, Bandyopadhyay S (2002) Performance evaluation of some clustering algorithms and validity indices. IEEE Transactions on Pattern Analysis and Machine Intelligence 24:1650–1654

BibTeX

@article{Maulik_2002, title={Performance evaluation of some clustering algorithms and validity indices}, volume={24}, ISSN={0162-8828}, url={http://dx.doi.org/10.1109/TPAMI.2002.1114856}, DOI={10.1109/tpami.2002.1114856}, number={12}, journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, publisher={Institute of Electrical and Electronics Engineers (IEEE)}, author={Maulik, U. and Bandyopadhyay, S.}, year={2002}, month=Dec, pages={1650–1654} }
Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF