New Darshan 3.1.0 release now available

Darshan version 3.1.0 is now available for release! This version contains a number of general enhancements to the code base:

  • addition of a stdio I/O library instrumentation module
    • this handles more in-depth instrumentation of file stream I/O functions like fopen(), fprintf(), fscanf(), etc.
    • this module also captures stats on the standard I/O streams (stdin, stdout, & stderr)
  • addition of a Lustre instrumentation module
    • this module provides Lustre striping details (e.g., stripe width, stripe size, list of OSTs a file is striped over)
  • addition of a new mmap-based logging mechanism that allows Darshan to generate output logs even in cases where applications don’t call MPI_Finalize()
    • Note that these logs are uncompressed and are per-process rather than per-job
  • addition of the darshan-merge utility to darshan-util to allow per-process logs generated by the mmap-based logging mechanism to be converted into Darshan’s traditional compressed per-job log files
  • modification of the POSIX module timestamp counters to also include a LAST_OPEN & FIRST_CLOSE counters to give more details on application I/O intervals

This version has been tested extensively on a number of platforms. As always, please let us know if you have any issues, questions, or comments using either our mailing list or the Darshan GitLab page.

The release tarball can be downloaded HERE.

Documentation for this version of Darshan can be found HERE.

Darshan 3.0.1 release now available

Darshan version 3.0.1 is now available for download. This version contains a number of bug fixes and enhancements to Darshan 3.0.0. For reference, the complete list of changes for this release is:

* bug fix in darshan logutil mount parsing code that was causing file paths to be matched to the first mount point with a common prefix rather than the one with the longest common prefix
* bug fix in the darshan-util bzip2 configure check that was accidentally overwriting Darshan’s LDFLAGS
* minor bug fixes to IO start time counters in all modules to set IO start time to the actual first start time rather than the first IO op to complete
* update darshan-util perl scripts to get perl bin from user’s path, rather than from /usr/lib (reported by Kay Thust)
* update Darshan’s fortran and cxx compiler wrapper generators to automatically detect MPICH library names on BG/Q
* bug fix for calculating Darshan’s agg_perf_by_slowest performance metric correctly
* add performance estimate info to darshan-job-summary graph

We have tested this release on multiple platforms, but please let us know if you have any issues or questions. You can report issues directly using Darshan’s GitLab page or by using the Darshan users mailing list.

The release tarball can be downloaded HERE.

Documentation for this version of Darshan can be found HERE.

Darshan 3.0.0 performance evaluation technical report

As part of our transition to the new modularized version of Darshan (3.0.0), we have published a technical report evaluating its performance on Edison, a production Cray XC30 system at NERSC. This technical report primarily serves as validation that Darshan’s performance remains suitable for full-time deployment on production HPC systems, but it also provides some additional background that may be useful in understanding the design changes made to Darshan’s runtime library and log file format. Our results indicate that Darshan’s new design is still lightweight enough to run full-time on production systems with little to no impact on the performance of instrumented applications.

A PDF for this technical report can be found HERE.

Darshan 3.0.0 official release is ready

Darshan 3.0.0, the new modularized implementation of Darshan, is now officially ready to be released! This is the first stable release of the Darshan 3.0 fork and builds off the experimental pre-releases we have been providing over the last few months.  We encourage users to consider upgrading to this version in the near future as it contains numerous enhancements over older versions: logging of full file paths, clearer parser and job summary output, addition of a BG/Q specific instrumentation module, future proofing for any upcoming instrumentation modules, etc.

The following changes have been made between this official release and the last pre-release (darshan-3.0.0-pre3):

  • fix bug causing compile errors when building external utilities that use the darshan-util API/library
  • update docs to give debugging tips for cases where Darshan logs are not generated
  • update darshan-runtime docs to give information on upgrading Darshan
  • fix shared library regression test script to check for potential errors with Darshan symbols rather than failing silently in these cases
  • bug fix for determining minimum non-zero counters in shared file reductions in all modules
  • loosen Darshan’s PMPI symbol check to prevent inadvertent disabling of Darshan for some MPICH builds
  • bug fix for resolving MPI_Gather and MPI_Barrier when LDPRELOADing Darshan’s shared libraries
  • add more helpful error handling when opening 2.x version log files
  • port darshan-diff utility over to new 3.0.0 log file format
  • fix numerous configure bugs on Cray systems
  • add synthetic benchmarking hooks for testing Darshan’s shutdown procedure

We have tested this version extensively on a number of platforms over the last few months and are satisfied with the new features and Darshan’s overall performance. Included in the darshan-runtime documentation (a link is provided below) is some information on how to best upgrade from a Darshan 2.x version to Darshan 3.0.0. As always, please let us know if you have any issues or questions (using either the Darshan users mailing list or by opening an issue on the Darshan GitLab page).

The release tarball can be downloaded HERE.

Documentation for this version of Darshan can be found HERE.

New point release available for Darshan 3.0.0

There is a new point release available for the modularized 3.0.0 version of Darshan. This point release contains the following changes:

  • add module-specific version fields to header to allow utilities to handle different versions of a module’s I/O data for backwards compatibility — NOTE: this breaks the log file parsing for logs obtained using Darshan-3.0.0-pre2 & Darshan-3.0.0-pre1
  • bug fix in regression test scripts for setting proper environment variables to use MPI profiling configuration for Fortran apps
  • bug fix in bzip2 log writing implementation in darshan-logutils
  • possible race conditions resolved in each module’s shutdown code
  • general code, comment, and documentation cleanup
  • addition of module-specific counter descriptions printed prior to parsing a modules I/O data in darshan-parser

As a reminder, this is still an experimental release and is not appropriate for production use at the moment, though we are getting close to an official release! We hope you can take the time to try the new version in preparation for the upcoming release, and look forward to any feedback you may have.

Documentation for this version of Darshan can be found HERE, under the Darshan 3.0.0 heading.

New Darshan 3.0.0 point release

A new point release is available for the new modularized version of Darshan (Darshan 3.0). This point release includes the following additions to the initial Darshan 3.0.0 pre-release:

  • the ability to configure Darshan’s max memory usage per instrumentation module at configure time or runtime
    • use `–with-mod-mem=` configure parameter to set at configure time
    • use DARSHAN_MODMEM environment variable to set at runtime
  • updated Ruby bindings for the darshan-util library to simplify Darshan log analysis using Ruby (contributed by Matthieu Dorier)

As a reminder, this version of Darshan is experimental and still going through alpha testing, and is thus not suitable for deployment on production systems, yet. We do encourage users to consider using/testing this version as we will be switching over to this branch in the future. Please feel free to provide us with questions, feedback, feature requests, etc.

Relevant docs:
– darshan-runtime installation and usage: darshan-runtime
– darshan-util installation and usage: darshan-util
– docs on new modularized architecture and how to add new instrumentation modules: darshan modularization

Publication on I/O workload modeling using Darshan

A publication on modeling HPC I/O workloads using a variety of workload sources, including Darshan logs, has been accepted at PMBS 2015. This work utilizes the CODES exascale storage system simulation toolkit to analyze the merits of using distinct workload sources (I/O traces, synthetic I/O descriptions, and Darshan I/O characterizations) as input to storage system simulations, as well as replays of I/O workloads on real HPC systems. Documentation on how to configure the CODES workload generator to use Darshan logs as input is given in the CODES repository.

This post will be updated with a link to the paper once the final version is made available online.

Darshan 3.0.0 pre-release

There is a pre-release available for download today for a new experimental version of Darshan! You can download it here.

Darshan 3.0.0 provides much of the same functionality of prior versions, but with the following new enhancements and features:
* hooks for developers to add their own instrumentation module
implementations to capture new I/O characterization data
– these instrumentation modules can be used to instrument new
I/O interfaces or gather system-specific parameters, for instance
* modularized log format allows new module-specific utilities to
access their I/O characterization data independently
– this new format also allows new counters to be added to existing
instrumentation modules without breaking existing utilities
* Darshan logs now contain a mapping of Darshan’s unique record
identifiers to full file names, instead of fix-sized file name
* a new instrumentation module for capturing BG/Q-specific parameters
(BG/Q environment is automatically detected at configure time)
* new darshan-parser and darshan-job-summary output to utilize the
new modularized log format

This version is not intended for use in a production environment, yet, so please just use for testing for now. We have had success in the cross-platform testing we have done so far, but more thorough testing is necessary before we are ready for to officially release the new modularized implementation. Issues can be reported directly to the mailing list or using the new issue tracking feature at the Darshan GitLab page (

As always, we welcome any feedback you may have!

Relevant docs:
– darshan-runtime installation and usage: darshan-runtime
– darshan-util installation and usage: darshan-util
– docs on new modularized architecture and how to add new instrumentation modules: darshan modularization