Darshan 2.2.6 release

Darshan 2.2.6 has been released.  This release includes performance enhancements as well as various minor bug fixes and compatibility improvements.
Changelog:
darshan-2.2.6

  • Bug fixes:
    • make sure to honor user-specified hints passed in at runtime via the DARSHAN_LOGHINTS env variable.
    • include fread and fwrite in darshan-job-summary.pl read and write counts
    • fix segmentation fault on invalid arguments to darshan-parser
  • Performance enhancements:
    • collect mount point information at rank 0 and broadcast to all
    • processes to avoid excess file system traffic on startup
    • change default MPI-IO hints for writing log file to romio_no_indep_rw=true and cb_nodes=4 to improve log creation performance
  • Install libdarshan-util and headers during installation process for darshan-util; patch contributed by Matthieu Dorier
  • Detect PMPI support at link time when using compiler scripts produced by the darshan-gen-* utilities. This avoids link problems when Darshan compiler wrappers are used with the ADIOS dummy MPI library; reported by Jingqing Mu
  • Support MPICH_{CC/CXX/F77} environment variables in compiler scripts produced by the darshan-gen-* utilities
  • Add instrumentation of startup time, enabled via DARSHAN_INTERNAL_TIMING environment variable
  • Rename cp-shutdown-bench test utility to darshan-shutdown-bench and enable benchmarking hooks in library by default so that darshan-shutdown-bench can be used with any Darshan installation
  • Remove deprecated –enable-st-dev-workaround configure option
  • Misc. documentation updates

Darshan-Ruby

Matthieu Dorier of ENS Cachan Brittany has developed and released Darshan-Ruby, an object-oriented extension to simplify Darshan log analysis in Ruby.  Darshan-Ruby uses the Darshan logutils API to efficiently access Darshan data without intermediate conversion into text format.
Please visit the Darshan-Ruby web site for more information.

"Production I/O Characterization on the Cray XE6" to be presented at CUG 2013

We will be presenting a new paper entitled “Production I/O Characterization on the Cray XE6” at this year’s CUG meeting, May 6-9, in Napa CA.  This presentation will include our experiences in evaluating and deploying Darshan on the Hopper system at NERSC.  We will also use data collected on Hopper to highlight example metrics to identify applications that can benefit from additional tuning.

Darshan 2.2.5 release

Darshan version 2.2.5 has been released.  The  highlights of this release include multiple bug fixes and improvements to block size detection.  We would also like to thank Yushu Yao for contributing support for Cray xt-asyncpe 5.12 (and higher) environments.
Changelog:
Darshan 2.2.5

  • Fix bug in mount point identification when –enable-stat-at-open option is not used. In Darshan 2.2.4, some file entries were recorded as using the “/” file system regardless of their location.
  • Update patches and documentation for Cray xt-asyncpe environment 5.12 or higher; contributed by Yushu Yao.
  • Update documentation to no longer recommend –enable-stat-at-open option on any systems
  • Fix unresolved symbols error when linking non-MPI programs with Cray compiler. Reported by Yushu Yao.
  • Use fstat() in place of stat() for detecting file information when –enable-stat-at-open option is used
  • Use statfs() at startup to detect default block size for supported file systems
  • Added –enable-group-readable-logs configure option, which will cause Darshan to to generate log files with the group read permission bit set. This option is useful in conjunction with deployments that set the setgid bit on log directories.

Technical report: Darshan performance on the Cray platform

We have posted a new technical report entitled “Performance Analysis of Darshan 2.2.3 on the Cray XE6 Platform” to the Darshan publications page.  This report includes experimental results from the Beagle system at the University of Chicago Computation Institute.
Previous publications already investigated Darshan overhead on the IBM Blue Gene platform.  However, beginning with Darshan version 2.2.3 we fully support the Cray XE6 platform as well.  We used this opportunity to revisit the Darshan efficiency measurements and confirm that Darshan still operates with minimal impact to production applications.

Darshan 2.2.4 release

Darshan 2.2.4 is now available on the download page.
If you are upgrading Darshan on a Cray platform, please re-patch your compiler scripts.  If you are upgrading Darshan on a Blue Gene platform (P or Q), please check the BG/P recipe in the installation documentation and note that the –enable-stat-at-open option is now recommended on those platforms.
Change log:
darshan-2.2.4

  • Disable extra stat() of newly opened files by default. This improves performance on shared files for some platforms. Reported by Yushu Yao.
    • prevents Darshan from reporting SIZE_AT_OPEN, FILE_ALIGNENT, and DARSHAN_FILE_NOT_ALIGNED values
    • features can still be enabled via –enable-stat-at-open option
  • Fix missing -lz in post ld flags reported by Yushu Yao. Fixes a link-time error for some corner-case applications.
  • Fix bug in Cray compiler script patches that was setting compiler flags incorrectly. Reported by Yushu Yao.
    • WARNING: if you are using a previous Darshan release (2.2.3 through 2.2.4-pre4) on a Cray platform, please re-patch your compiler scripts.
  • Update darshan-gen-* scripts to support the potential for additonal LDFLAGS or CFLAGS that appear before the object or source file in the compile or link commands. This fixes compatibility with some mvapich2 installations, reported by Dragos Constantin.

Darshan 2.2.3 release

Darshan 2.2.3 is now available on the download page.  This release features official support for the Cray XE6 and IBM BG/Q platforms.  Special thanks to Yushu Yao and Katie Antypas of NERSC for their contributions in Cray integration and testing.
Change log:

darshan-2.2.3
  • improved Cray XE6 support
    • support for GNU, PGI, Cray, Pathscale, and Intel compilers
    • patch adding Darshan capability to system compiler scripts
    • software module, including testing and features contributed by Yushu Yao and Katie Antypas of The National Energy Research Scientific Computing Center (NERSC)
    • improved documentation
  • minor bug fixes:
    • properly detect cxx library name when generating BG/Q compiler wrappers
    • improve timing mechanism for cp-shutdown-bench benchmark
    • improve hashing to avoid log file name collisions

Darshan 2.2.2 release

Darshan 2.2.2 is now available on the download page.  Change log:

darshan-2.2.2
  • bug fixes:
    • remove debugging message that was inadvertently included in MPI_File_sync() wrapper
    • fix potential hang if the –with-log-path-by-env argument was used at configure time but the environment variable was not set at run time
darshan-2.2.1
  • significant improvements to how counters are handled in multi-threaded applications
  • initial (rough) documentation for using Darshan in Cray programming environments with static linking
  • bug fixes:
    • escape special characters in mount point paths in darshan-job-summary.pl (reported by Mouhamed Gueye)
    • workarounds for various runtime problems with cuserid() and stat() in Cray environments
    • build problems with darshan-utils on some versions of OSX
    • accurate shared file statistics for libraries that use deferred opens

Darshan 2.2.0 release

Darshan 2.2.0 is now available for download.   The biggest change is that this release splits the code into separate components for instrumentation and analysis.  There are also several improvements in documentation, portability, and an assortment of bug fixes.
Changelog:

  • split darshan into separate packages:
    • darshan-runtime: for runtime instrumentation
    • darshan-utils: for processing darshan log files
  • changed default output file name for darshan-job-summary.pl to be based on input file name rather than summary.pdf
  • reorganized init and finalize routines so that can be linked separately (to allow for easier integration with other instrumentation tools)
  • add -cc, -cxx, -f77, -f90, and -fc support to compiler scripts generated by the darshan-gen-*.pl scripts
  • bug fixes:
    • potential MAX_BYTE overflow on 32 bit systems
    • incorrect pread and pwrite offset tracking
    • corrections to darshan-job-summary variance table
    • better runtime error handling if bzip or gnuplot tools are insufficient
    • improvements to time range in darshan-job-summary graphs
  • documentation:
    • improved documentation for both the darshan-runtime and darshan-util portions of Darshan can be found in the respective doc/ subdirectory for each