注册 登录  
 加关注
   显示下一条  |  关闭
温馨提示!由于新浪微博认证机制调整,您的新浪微博帐号绑定已过期,请重新绑定!立即重新绑定新浪微博》  |  关闭

阿弥陀佛

街树飘影未见尘 潭月潜水了无声 般若观照心空静...

 
 
 

日志

 
 
关于我

一直从事气象预报、服务建模实践应用。 注重气象物理场、实况场、地理信息、本体知识库、分布式气象内容管理系统建立。 对Barnes客观分析, 小波,计算神经网络、信任传播、贝叶斯推理、专家系统、网络本体语言有一定体会。 一直使用Java、Delphi、Prolog、SQL编程。

网易考拉推荐

BigDL Configuration  

2017-11-30 18:11:26|  分类: BigDL |  标签: |举报 |字号 订阅

  下载LOFTER 我的照片书  |

BigDL Configuration

BigDL uses Java properties to control its behavior. Here's the list of these properties.

How to set the properties

Spark

If you run BigDL on Apache Spark, you can set the properties by passing spark-submit options. Here's an example:

# Say you want to set property FOO to value BAR

spark-submit ...
    --conf 'spark.executor.extraJavaOptions=-DFOO=BAR' # Set that property for executor process
    --conf 'spark.driver.extraJavaOptions=-DFOO=BAR'   # Set that property for driver process
    ...

Local Java/Scala program

If you run BigDL in a local Java/Scala program, you can set the properties by passing JVM parameters. Here's an example:

# Say you want to set property FOO to value BAR
java -cp xxx.jar -DFOO=BAR your.main.class.name

Available Properties

Category Property Default value Description
Logging bigdl.utils.LoggerFilter.disable false Disable redirecting logs of Spark and BigDL to a file.
bigdl.utils.LoggerFilter.logFile Current_Working_Directory/bigdl.log Where is the redirecting log.
bigdl.utils.LoggerFilter.enableSparkLog true Enable redirecting Spark logs to logFile. Set it to false when you don't want to see Spark logs in the redirecting log file.
Mode bigdl.localMode false Whether BigDL is running as a local Java/Scala program.
Multi-threading bigdl.coreNumber half of the virtual core number How many cores BigDL use on your machine. It is only used when bigdl.localMode is set to true. If hyper thread is enabled on your machine, DO NOT set it larger than half of the virtual core number.
bigdl.Parameter.syncPoolSize 4 Thread pool size for syncing parameter between executors.
Distributed Training bigdl.network.nio true Whether use NIO as BlockManager backend in Spark 1.5. If it is set to false, user can specify spark.shuffle.blockTransferService to change the BlockManager backend. ONLY used when running on Spark 1.5.
bigdl.failure.retryTimes 5 How many times to retry when there's failure in distributed Training.
bigdl.failure.retryTimeInterval 120 Unit is second. How long recount the retry times.
bigdl.check.singleton false Whether to check if multiple partition run on a same executor, which is bad for performance.
Tensor bigdl.tensor.fold 1000 How many elements in a tensor to determine it is a large tensor, and thus print only part of it.
  评论这张
 
阅读(8)| 评论(0)
推荐 转载

历史上的今天

在LOFTER的更多文章

评论

<#--最新日志,群博日志--> <#--推荐日志--> <#--引用记录--> <#--博主推荐--> <#--随机阅读--> <#--首页推荐--> <#--历史上的今天--> <#--被推荐日志--> <#--上一篇,下一篇--> <#-- 热度 --> <#-- 网易新闻广告 --> <#--右边模块结构--> <#--评论模块结构--> <#--引用模块结构--> <#--博主发起的投票-->
 
 
 
 
 
 
 
 
 
 
 
 
 
 

页脚

网易公司版权所有 ©1997-2018