注册 登录  
 加关注
   显示下一条  |  关闭
温馨提示!由于新浪微博认证机制调整,您的新浪微博帐号绑定已过期,请重新绑定!立即重新绑定新浪微博》  |  关闭

阿弥陀佛

街树飘影未见尘 潭月潜水了无声 般若观照心空静...

 
 
 

日志

 
 
关于我

一直从事气象预报、服务建模实践应用。 注重气象物理场、实况场、地理信息、本体知识库、分布式气象内容管理系统建立。 对Barnes客观分析, 小波,计算神经网络、信任传播、贝叶斯推理、专家系统、网络本体语言有一定体会。 一直使用Java、Delphi、Prolog、SQL编程。

网易考拉推荐

Writing to a netCDF-4 file with compression (version 4.5)  

2015-05-13 21:29:02|  分类: netcdf |  标签: |举报 |字号 订阅

  下载LOFTER 我的照片书  |

To write to netCDF-4, you must install the netCDF-4 C library on your machine.

The main use of netCDF-4 is to get the performance benefits from compression, and possibly from chunking (why it matters). By default, the Java library will write chunked and compressed netcdf-4 files, using the default chunking algorithm. To have your own control of chunking and compression, you must create a Nc4Chunking object and pass it into NetcdfFileWriter.createNew():

Nc4Chunking chunker = Nc4Chunking factory(Strategy type, int deflateLevel, boolean shuffle);
NetcdfFileWriter.Version version = NetcdfFileWriter.Version.netcdf4;

FileWriter2 writer = new ucar.nc2.FileWriter2(ncfileIn, filenameOut, version, chunker);
...
NetcdfFile ncfileOut = writer.write();
ncfileIn.close();
ncfileOut.close();
====================================================================

Chunking Strategy (version 4.5)

When writing NetCDF-4 files, one must decide on how the variables are to be chunked. In the NetCDF-Java library this is done through the use of a Nc4Chunking strategy. The possibilities currently are:

  • standard : this strategy is used by default (see below for description)
  • grib: this strategy matches how GRIB files are stored: the chunking uses only the two rightmost dimensions, which for GRIB is the x and y dimension. Use this strategy when converting GRIB files to NetCDF-4, in order to optimize conversion writing time, and to get similar access speed.
  • custom: you may write your own inplementation of ucar.nc2.write.Nc4Chunking and pass it into FileWriter2 or NetcdfFileWriter. This gives you complete control over chunking.

Both standard and grib strategies allow you to override individual variable chunking if you want by setting the variable's _ChunkSizes attribute.

By default, the Java library will write chunked and compressed NetCDF-4 files, using the default chunking algorithm. You may pass in a null for the chunking parameter to use the default.

Default chunking strategy

For each Variable:

  1. Look for a variable attribute named "_ChunkSizes", whose value is a vector of integer chunk sizes, one for each dimension. If it exists, use it.
  2. If the variable does not have an unlimited dimension:
    • it will be chunked if the total size in bytes > Nc4ChunkingDefault.minVariableSize
    • chunk size will be fillFastest( variable.shape, Nc4ChunkingDefault.defaultChunkSize)
  3. If the variable has one or more unlimited dimensions, it will be chunked, and the chunk size will be calculated as:
    1. set unlimited dimensions to length one, then compute fillFastest( variable.shape, Nc4ChunkingDefault.defaultChunkSize)
    2. if the resulting chunk size is greater than Nc4ChunkingDefault.minChunksize, use it
    3. if not, set the unlimited dimension chunk sizes so that the resulting chunksize is close to Nc4ChunkingDefault.minChunksize. If there are N unlimited dimensions, take the Nth root, ie evenly divide the chunk size among the unlimited dimensions.

The fillFastest( int[] shape, maxSize) algorithm fills the fastest varying (rightmost) dimensions first, until the chunkSize is as close to maxSize as possible without exceeding. The net effect is that the chunkSizes will be close to Nc4ChunkingDefault.defaultChunkSize, with a minimum of Nc4ChunkingDefault.minChunksize, and favoring read access along the fast dimensions. Any variable with an unlimited dimension will use at least Nc4ChunkingDefault.minChunksize bytes (approx, but if compressing, unused space should be mostly eliminated).

Current default values (these can be overidden by the user):

  • minVariableSize = 65K
  • defaultChunkSize = 256K
  • minChunksize = 8K

By default, compression (deflate level = 5) and the shuffle filter will be used. The user can override these by:

// set deflate > 0 to compress
// set shuffle to true for the shuffle filter 
Nc4Chunking chunker = Nc4Chunking factory(Nc4Chunking.Strategy.standard, int deflateLevel, boolean shuffle);
  评论这张
 
阅读(289)| 评论(0)
推荐 转载

历史上的今天

在LOFTER的更多文章

评论

<#--最新日志,群博日志--> <#--推荐日志--> <#--引用记录--> <#--博主推荐--> <#--随机阅读--> <#--首页推荐--> <#--历史上的今天--> <#--被推荐日志--> <#--上一篇,下一篇--> <#-- 热度 --> <#-- 网易新闻广告 --> <#--右边模块结构--> <#--评论模块结构--> <#--引用模块结构--> <#--博主发起的投票-->
 
 
 
 
 
 
 
 
 
 
 
 
 
 

页脚

网易公司版权所有 ©1997-2017