Welcome to the Darshan project

July 31st, 2009

This is the home page for  Darshan, a scalable HPC I/O characterization tool. Darshan is designed to capture an accurate picture of application I/O behavior, including properties such as patterns of access within files, with minimum overhead.  The name is taken from a Sanskrit word for “sight” or “vision”.

Darshan can be used to investigate and tune the I/O behavior of complex HPC applications.  In addition, Darshan’s lightweight design makes it suitable for full time deployment for workload characterization of large systems.  We hope that such studies will help the storage research community to better serve the needs of scientific computing.

Darshan was originally developed on the IBM Blue Gene series of computers deployed at the Argonne Leadership Computing Facility, but it is portable across a wide variety of platforms include the Cray XE6, Cray XC30, and Linux clusters.  Darshan routinely instruments jobs using up to 786,432 compute cores on the Mira system at ALCF.

You will find current news about the Darshan project posted below.   Additional documentation and details about the Darshan are available from the links at the top of this page.

Darshan events at SC17

November 9th, 2017
Comments Off on Darshan events at SC17

We encourage anyone interested in Darshan and related technology to attend the following events at SC17 in Denver:

Darshan pre-release 3.1.5-pre1 is now available

November 1st, 2017
Comments Off on Darshan pre-release 3.1.5-pre1 is now available

A new  pre-release version (3.1.5-pre1) of Darshan is available for download here. Please be aware that this pre-release is experimental and not recommended for full-time use in production yet. An official 3.1.5 release will be available in a few weeks.

The list of new features and bug fixes include:

  • Darshan’s MPI instrumentation method now intercepts both MPI and PMPI symbols to workaround MPI library bindings that are calling PMPI routines directly
    • Contributed in part by Chris Zimmer
  • darshan-util now includes an analysis tool for visualizing DXT traces: dxt_analyzer
    • Contributed by Alex Sim
  • Darshan now supports the DARSHAN_EXCLUDE_DIRS environment variable, which specifies a comma-separated list of paths to avoid instrumenting at runtime
    • Contributed by Cristian Simarro
  • Darshan’s regression test system now includes test harnesses for Cray systems @ the ALCF (Theta) and NERSC (Cori/Edison)
  • Bug fix in Darshan’s shutdown procedure leading to potentially negative timer values
  • Bug fix in Darshan’s autoconf scripts causing specified libbz2 locations to be ignored
    • Contributed by Glenn Lockwood

Please report any questions, issues, or concerns using the Darshan-users mailing list, or by opening an issue on the Darshan GitLab page.

Darshan version 3.1.4 release available

April 11th, 2017
Comments Off on Darshan version 3.1.4 release available

A new version of Darshan (3.1.4) is available for download here. This release addresses a couple of minor bugs in the previous Darshan release:

  • darshan-parser causing segmentation fault when parsing logs with DXT module data present
  • darshan-diff utility not producing output

Documentation for this version of Darshan can be found here.

Please let us know if you have any issues or any feedback you would like to share with us, either using the Darshan-users mailing list or the Darshan GitLab page.

Darshan 3.1.3 official release is now available

February 13th, 2017
Comments Off on Darshan 3.1.3 official release is now available

The official release for Darshan version 3.1.3 is now available for download here. We have tested this release extensively and believe it is stable and ready for use in production. On top of the numerous new features and bug fixes included in the darshan-3.1.3-pre1 release, the official version has one additional bug fix:

  • Instrumentation of mmap calls is now disabled for dynamically linked applications (i.e., applications that LD_PRELOAD darshan)
    • this avoids potential deadlock issues on Cray systems when using dynamically linked executables (reported by Cristian Simarro)

You can find updated documentation for darshan-runtime and darshan-util components here.

Please let us know if you have any questions, issues, or concerns using the Darshan-users mailing list, or by opening an issue on the Darshan GitLab page.

Darshan pre-release 3.1.3-pre1 is now available

January 31st, 2017
Comments Off on Darshan pre-release 3.1.3-pre1 is now available

A new Darshan pre-release (3.1.3-pre1) is now available for download here. Please be aware that this pre-release is experimental and not recommended for full-time use in production yet. We hope to have the official 3.1.3 release ready some time next week.

Here is the list of new features and bug fixes in this release:

  • New DxT (Darshan eXtended Tracing) instrumentation modules that provide fine-grained tracing data of read/write operations at the POSIX & MPI-IO layers
    • This tracing module is not enabled by default, and it is envisioned that it be enabled selectively at runtime for applications of interest
    • This code was contributed by Cong Xu and Intel’s High Performance Data Division (HPDD)
  • Darshan now captures command line arguments for Fortran applications
    • This fix was contributed by Cristian Simarro
  • Fix potential deadlock when using the Cray hugepages module and Darshan
    • Reported by Glenn Lockwood & Cristian Simarro
  • Fix potential segmentation fault when Darshan exceeds the maximum number of tracked access sizes/strides
    • Reported by Glenn Lockwood
  • Disable Darshan’s HDF5 module by default to avoid potential ABI compatibility issues with HDF5
    • New configure switches are provided to enable Darshan support for specific HDF5 library versions
    • Problem report and initial patch provided by Karl-Ulrich Bamberg
  • Add whitelist to Darshan’s list of file path exclusions to prevent I/O to/from Cray’s Datawarp service from being filtered out
    • Contributed by Glenn Lockwood

Updated documentation related to using the DxT tracing modules can be found in the source tree (‘darshan-runtime/docs’ for the Darshan runtime library & ‘darshan-util/docs’ for Darshan’s utilities). To summarize, DxT tracing can be enabled at runtime by exporting the ‘DXT_ENABLE_IO_TRACE’ environment variable. DxT also has a corresponding parser utility named ‘darshan-dxt-parser’ that can be used to parse trace data out of Darshan log files.

As always, let us know if you have any questions, issues, or concerns using the Darshan-users mailing list, or by opening an issue on the Darshan GitLab page.

Darshan at SC16 recap

December 7th, 2016
Comments Off on Darshan at SC16 recap

At SC16, Darshan was once again able to make a broad impact on the I/O characterization community as well as the HPC community as a whole. Please check this document for a complete recap of Darshan-related activities from this year’s program.

Darshan events at SC16

November 9th, 2016
Comments Off on Darshan events at SC16

Darshan will be featured in several community events at SC 2016; please stop by and visit us if you are interested in learning more about Darshan and other related topics:

  • Tutorial: Parallel I/O in Practice, by Robert Latham, Robert Ross, and Brent Welch.  Sunday, November 13th, 8:30am – 5pm, in room 355-B (requires tutorial registration).
  • Paper presentation: Modular HPC I/O Characterization with Darshan,
    by Shane Snyder, Philip Carns, Kevin Harms, Robert Ross, Glenn Lockwood and Nicholas Wright.  Sunday, November 13th, 3:30pm – 3:55pm, ESPT workshop in room 155-E (requires workshop registration).
  • Birds of a Feather session: Analyzing Parallel I/O, by Philip Carns and Julian Kunkel with guest presentations from Glenn Lockwood, Shane Snyder, Jakob Luettgau, Salem El Sayed, and Xiaosong Ma.  Wednesday, November 16th, 5:15pm – 7:00pm, in room 155-E.

 

Darshan 3.1.2 point release now available

November 4th, 2016
Comments Off on Darshan 3.1.2 point release now available

Darshan version 3.1.2 is a point release meant to address a bug in versions 3.1.0 and 3.1.1 that can cause crashes in applications (or other binaries) when LD_PRELOADing the Darshan library. The bug was caused by an error in the resolution of the address of the fopen64 function call, so any binary that calls this function is affected. This bug does not affect Darshan’s methods for instrumenting statically-linked executables.

The release tarball can be downloaded HERE.

Documentation for this version of Darshan can be found HERE.

Darshan 3.1.1 is now available for download

October 6th, 2016
Comments Off on Darshan 3.1.1 is now available for download

Darshan version 3.1.1 is now available for download. This release contains the following bug fix for the recent 3.1.0 release:

  • Fix bug in calculation of the MPIIO_F_WRITE_START_TIMESTAMP counter in the MPI-IO module that could lead to incorrect timestamps in certain cases

Please let us know if you have any issues, questions, or comments using either our mailing list or the Darshan GitLab page.

The release tarball can be downloaded HERE.

Documentation for this version of Darshan can be found HERE.

New Darshan 3.1.0 release now available

September 30th, 2016
Comments Off on New Darshan 3.1.0 release now available

Darshan version 3.1.0 is now available for release! This version contains a number of general enhancements to the code base:

  • addition of a stdio I/O library instrumentation module
    • this handles more in-depth instrumentation of file stream I/O functions like fopen(), fprintf(), fscanf(), etc.
    • this module also captures stats on the standard I/O streams (stdin, stdout, & stderr)
  • addition of a Lustre instrumentation module
    • this module provides Lustre striping details (e.g., stripe width, stripe size, list of OSTs a file is striped over)
  • addition of a new mmap-based logging mechanism that allows Darshan to generate output logs even in cases where applications don’t call MPI_Finalize()
    • Note that these logs are uncompressed and are per-process rather than per-job
  • addition of the darshan-merge utility to darshan-util to allow per-process logs generated by the mmap-based logging mechanism to be converted into Darshan’s traditional compressed per-job log files
  • modification of the POSIX module timestamp counters to also include a LAST_OPEN & FIRST_CLOSE counters to give more details on application I/O intervals

This version has been tested extensively on a number of platforms. As always, please let us know if you have any issues, questions, or comments using either our mailing list or the Darshan GitLab page.

The release tarball can be downloaded HERE.

Documentation for this version of Darshan can be found HERE.