research-article

Still Benchmarking After All These Years

Authors:

Ismail S. Bustany,

Patrick H. Madden,

Natarajan Viswanathan,

Stephen YangAuthors Info & Claims

ISPD '21: Proceedings of the 2021 International Symposium on Physical Design

Pages 47 - 52

https://doi.org/10.1145/3439706.3446885

Published: 21 March 2021 Publication History

Abstract

Circuit benchmarks for VLSI physical design have been growing in size and complexity, helping the industry tackle new problems and find new approaches. In this paper, we take a look back at how benchmarking efforts have shaped the research community, consider trade-offs that have been made, and speculate on what may come next.

References

[1]

G. E. Moore. Cramming more components onto integrated circuits. Electronics Magazine, 38(8):114--117, April 1965.

[2]

C. Mead and L. Conway. Introduction to VLSI Systems. Addison-Wesley, 1993.

[3]

B. Preas. Benchmarks for cell-based layout systems. In Proc. Design Automation Conf, pages 319 -- 320, 1987.

Digital Library

[4]

K. Kozminski. Benchmarks for layout synthesis - evolution and current status. In Proc. Design Automation Conf, pages 265 -- 270, 1991.

[5]

S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimization by simulated annealing. Science, 220(4598):671--680, May 1983.

[6]

J. Kleinhans, G. Sigl, F. Johannes, and K. Antreich. GORDIAN: VLSI placement by quadratic programming and slicing optimization. IEEE Trans. on Computer-Aided Design of Integrated Circuits and Systems, 10(3):356--365, 1991.

Digital Library

[7]

A. E. Dunlop and B. W. Kernighan. A procedure for placement of standard-cell VLSI circuits. IEEE Trans. on Computer-Aided Design of Integrated Circuits and Systems, CAD-4(1):92--98, January 1985.

Digital Library

[8]

P. H. Madden. Reporting of standard cell placement results. In Proc. Int. Symp. on Physical Design, pages 30--35, 2001.

Digital Library

[9]

C. J. Alpert. The ISPD98 circuit benchmark suite. In Proc. Int. Symp. on Physical Design, pages 80--85, 1998.

[10]

M. Wang, X. Yang, and M. Sarrafzadeh. Dragon2000: Standard-cell placement tool for large industry circuits. In Proc. Int. Conf. on Computer Aided Design, pages 260--263, 2000.

Digital Library

[11]

S. N. Adya and I. L. Markov. Consistent placement of macroblock using floorplanning and standard-cell placement. In Proc. Int. Symp. on Physical Design, pages 12--17, 2002.

[12]

A. E. Caldwell, Y. Cao, A. B. Kahng, F. Koushanfar, H. Lu, I. L. Markov, M. Oliver, D. Stroobandt, and D. Sylvester. GTX: The MARCO GSRC technology extrapolation system. In Proc. Design Automation Conf, pages 693--698, 2000.

Digital Library

[13]

GSRC. The GSRC bookshelf. http://www.gigascale.org/bookshelf.

[14]

G.-J. Nam, C. J. Alpert, P. Villarrubia, B. Winter, and M. Yildiz. The ISPD2005 placement contest and benchmark suite. In Proc. Int. Symp. on Physical Design, pages 216--220, 2005.

[15]

V. Yutsis, I. S. Bustany, D. Chinnery, J. R. Shinnerl, and W.-H. Liu. ISPD 2014 benchmarks with sub-45nm technology rules for detailed-routing-driven placement. In Proc. ISPD, pages 161--168, 2014.

[16]

I. S. Bustany, D. Chinnery, J. R. Shinnerl, and V. Yutsis. ISPD 2015 benhmarks with fence regions and routing blockages for detailed-routing-driven placement. In Proc. ISPD, pages 157--164, 2015.

[17]

N. K. Darav, A. Kennings, D. Westwick, and L. Behjat. High performance global placement legalization accounting for fence regions. In ICCAD, pages 514--519, 2015.

Digital Library

[18]

N. K. Darav, A. Kennings, A. Fakheri Tabrizi, D. Westwick, and L. Behjat. Eh?Placer: A high-performance modern technology-driven placer. ACM Trans. Design Automation of Electronic Systems, 21, July 2016.

Digital Library

[19]

C. Huang, C.-H. Chiou, K.-H. Tseng, and Y. Chang. Detailed-routing-driven analytical standard-cell placement. In PRoc. ASPDAC, pages 378--383, 2015.

[20]

V. Betz and J. Rose. VPR: A new packing, placement and routing tool for FPGA research. In Proceedings of the 7th International Workshop on Field-Programmable Logic and Applications, pages 213--222, 1997.

Digital Library

[21]

Stephen Yang, Aman Gayasen, Chandra Mulpuri, Sainath Reddy, and Rajat Aggarwal. Routability driven FPGA placement contest. In International Symposium on Physical Design (ISPD), pages 139--143, 2016.

Digital Library

[22]

Wuxi Li, Shounak Dhar, and David Z. Pan. UTPlaceF: A routability-driven fpga placer with physical and congestion aware packing. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD), 2017.

[23]

C.-W. Pui, G. Chen, W.-K. Chow, K.-C. Lam, J. Kuang, P. Tu, H. Zhang, E. F. Y. Young, and B. Yu. RippleFPGA: A routabilitydriven placement for large-scale heterogeneous FPGAs. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD), page 67:1--67:8, 2016.

Digital Library

[24]

R. Pattison, Z. Abuowaimer, S. Areibi, G. Grewal, and A. Vannelli. GPlace: A congestion-aware placement tool for ultrascale FPGAs. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD), page 68:1--68:7, 2016.

Digital Library

[25]

Stephen Yang, Chandra Mulpuri, Sainath Reddy, Meghraj Kalase, Srinivasan Dasasathyan, Mehrdad E. Dehkordi, Marvin Tom, and Rajat Aggarwal. Clock aware FPGA placement contest. In International Symposium on Physical Design (ISPD), pages 159--164, 2017.

Digital Library

[26]

Wuxi Li, Yibo Lin, Meng Li, Shounak Dhar, and David Z. Pan. UTPlaceF 2.0: A high-performance clock-aware FPGA placement engine. ACM Transactions on Design Automation of Electronic Systems (TODAES), December 2017.

[27]

Y.-C. Chen, S.-Y. Chen, and Y.-W. Chang. Efficient and effective packing and analytical placement for large-scale heterogeneous FPGAs. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD), page 647--654, 2014.

Digital Library

[28]

C.W Pui, G Chen, Y Ma, EFY Young, and B Yu. Clock-aware ultrascale FPGA placement with machine learning routability prediction. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pages 929--936, 2017.

[29]

Wuxi Li, Shounak Dhar, and David Z. Pan. UTPlaceF: A routability-driven FPGA placer with physical and congestion aware packing. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2016.

Digital Library

[30]

Wuxi Li, Meng Li, Jiajun Wang, and David Z. Pan. UTPlaceF 3.0: A parallelization framework for modern FPGA global placement. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD) .

[31]

Chang Xu, Gai Liu, Ritchie Zhao, Stephen Yang, Guojie Luo, and Zhiru Zhang. A parallel bandit-based approach for autotuning FPGA compilation. In International Symposium on FPGA, pages 157--166, 2017.

Digital Library

[32]

Chak-Wa Pui, Gengjie Chen, Yuzhe Ma, Evangeline F. Y. Young, and Bei Yu. Clock-aware ultrascale FPGA placement with machine learning routability prediction. In IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2017.

[33]

G.-J. Nam. ISPD 2006 Placement Contest: Benchmark suite and results. In Proc. Int. Symp. on Physical Design, pages 167--167, April 2006.

Digital Library

[34]

S. Dolgov, A. Volkov, L. Wang, and B. Xu. 2019 CAD Contest: LEF/DEF based global routing. In Proc. Int. Conf. on Computer Aided Design, pages 1--4, November 2019.

[35]

IEEE CEDA Design Automation Technical Committee. https://ieee-ceda.org/node/2591.

[36]

J. Jung, I. H.-R. Jiang, G.-J. Nam, V. N. Kravets, L. Behjat, and Y.-L. Li. OpenDesign Flow Database: The infrastructure for VLSI design and design automation research. In Proc. Int. Conf. on Computer Aided Design, pages 42:1--42:6, November 2016.

Digital Library

[37]

J. Jung, P.-Y. Lee, Y. Wu, N. K. Darav, I. H. Jiang, V. N. Kravets, L. Behjat, Y. Li, and G.-J. Nam. DATC RDF: Robust design flow database. In Proc. Int. Conf. on Computer Aided Design, pages 872--873, November 2017.

[38]

J. Jung, I. H.-R. Jiang, J. Chen, S.-T. Lin, Y.-L. Li, V. N. Kravets, and G.-J. Nam. DATC RDF: An academic flow from logic synthesis to detailed routing. In Proc. Int. Conf. on Computer Aided Design, pages 37:1--37:4, November 2018.

Digital Library

[39]

J. Jung, I. H.-R. Jiang, J. Chen, S.-T. Lin, Y.-L. Li, V. N. Kravets, and G.-J. Nam. DATC RDF: An open design flow from logic synthesis to detailed routing. In Proc. Workshop Open-Source EDA Tech. (WOSET), pages 6:1--6:4, November 2018.

[40]

J. Chen, I. H.-R. Jiang, J. Jung, A. B. Kahng, V. N. Kravets, Y.-L. Li, S.-T. Lin, and M. Woo. DATC RDF-2019: Towards a complete academic reference design flow. In Proc. Int. Conf. on Computer Aided Design, pages 1--6, November 2019.

[41]

J. Chen, I. H.-R. Jiang, J. Jung, A. B. Kahng, V. N. Kravets, Y.-L. Li, S.-T. Lin, and M. Woo. DATC RDF-2020: Strengthening the foundation for academicresearch in ic physical design. In Proc. Int. Conf. on Computer Aided Design, pages 1--6, November 2020.

Digital Library

[42]

DATC RDF Calibrations. https://github.com/ieee-ceda-datc/datc-rdf-calibrations.

[43]

T. Ajayi, V. A. Chhabria, M. Fogaca, S. Hashemi, A. Hosny, A. B. Khang, M. Kim, J. Lee, U. Mallappa, M. Neseem, G. Pradipta, S. Reda, M. Saligane, S. S Sapatnekar, C. Sechen, M. Shalan, W. Swartz, L. Wang, Z. Wang, M. Woo, and B. Xu. Toward an open-source digital flow: First learnings from the OpenROAD project. In Proc. Design Automation Conf, pages 76:1 -- 76:4, 2019.

Digital Library

Cited By

Khasawneh MMadden PBehjat LYang S(2022)What's So Hard About (Mixed-Size) Placement?Proceedings of the 2022 International Symposium on Physical Design10.1145/3505170.3511035(57-64)Online publication date: 13-Apr-2022
https://dl.acm.org/doi/10.1145/3505170.3511035
Almeida STabrizi AAghaeekiasaraee ENetto RFontana TGandhi UGuntzel JBehjat LMeinhardt C(2022)E-RVP: An Initial Design Rule Violation Predictor Using Placement Information2022 29th IEEE International Conference on Electronics, Circuits and Systems (ICECS)10.1109/ICECS202256217.2022.9970846(1-4)Online publication date: 24-Oct-2022
https://doi.org/10.1109/ICECS202256217.2022.9970846
Chen JJiang IJung JKahng AKim SKravets VLi YVaradarajan RWoo M(2021)DATC RDF-2021: Design Flow and Beyond ICCAD Special Session Paper2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD)10.1109/ICCAD51958.2021.9643553(1-6)Online publication date: 1-Nov-2021
https://doi.org/10.1109/ICCAD51958.2021.9643553

Index Terms

Still Benchmarking After All These Years
1. Hardware
  1. Electronic design automation

Recommendations

Benchmarking in digital circuit design automation

This paper focuses on benchmarking, which is the main experimental approach to the design method and EDA-tool analysis, characterization and evaluation. We discuss the importance and difficulties of benchmarking, as well as the recent research effort ...
From Performance to Dependability Benchmarking: A Mandatory Path
Performance Evaluation and Benchmarking

The work on performance benchmarking has started long ago. Ranging from simple benchmarks that target a very specific system or component to very complex benchmarks for complex infrastructures, performance benchmarks have contributed to improve ...
Benchmarking in digital circuit design
MINO'08: Proceedings of the 7th WSEAS International Conference on Microelectronics, Nanoelectronics, Optoelectronics

This paper focuses on benchmarking, which is the main experimental approach to the design method and EDA-tool analysis, characterization and evaluation. We discuss the importance and difficulties of benchmarking, as well as the recent research effort ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ISPD '21: Proceedings of the 2021 International Symposium on Physical Design

March 2021

159 pages

ISBN:9781450383004

DOI:10.1145/3439706

General Chair:
Jens Lienig
Dresden University of Technology, Germany
,
Program Chairs:
Laleh Behjat
University of Calgary, Canada
,
Stephen Yang
USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 March 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISPD '21

Sponsor:

ISPD '21: International Symposium on Physical Design

March 22 - 24, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 62 of 172 submissions, 36%

Upcoming Conference

ISPD '25

Sponsor:
sigda

International Symposium on Physical Design

March 16 - 19, 2025

Austin , TX , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
178
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)8

Reflects downloads up to 26 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Khasawneh MMadden PBehjat LYang S(2022)What's So Hard About (Mixed-Size) Placement?Proceedings of the 2022 International Symposium on Physical Design10.1145/3505170.3511035(57-64)Online publication date: 13-Apr-2022
https://dl.acm.org/doi/10.1145/3505170.3511035
Almeida STabrizi AAghaeekiasaraee ENetto RFontana TGandhi UGuntzel JBehjat LMeinhardt C(2022)E-RVP: An Initial Design Rule Violation Predictor Using Placement Information2022 29th IEEE International Conference on Electronics, Circuits and Systems (ICECS)10.1109/ICECS202256217.2022.9970846(1-4)Online publication date: 24-Oct-2022
https://doi.org/10.1109/ICECS202256217.2022.9970846
Chen JJiang IJung JKahng AKim SKravets VLi YVaradarajan RWoo M(2021)DATC RDF-2021: Design Flow and Beyond ICCAD Special Session Paper2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD)10.1109/ICCAD51958.2021.9643553(1-6)Online publication date: 1-Nov-2021
https://doi.org/10.1109/ICCAD51958.2021.9643553

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents