java keytool证书工具使用结
Keytool 是一个Java数据证书的管理工? ,Keytool密钥（keyQ和证书QcertificatesQ存在一个称为keystore的文件中在keystore里，包含两种数据:密钥实体QKey entityQ?密钥Qsecret keyQ或者是U钥和配对公钥（采用非对U加密）可信ȝ证书实体Qtrusted certificate entriesQ?只包含公?
JDK中keytool常用参数说明Q?span style="color: #ff0000;">不同版本有差异，详细可参见【附录】中的官Ҏ链?/span>Q?
- -genkey 在用户主目录中创Z个默认文?#8221;.keystore”,q会产生一个mykey的别名，mykey中包含用L公钥、私钥和证书(在没有指定生成位|的情况?keystore会存在用Ll默认目?
- -alias 产生别名 每个keystore都关联这一个独一无二的aliasQ这个alias通常不区分大写
- -keystore 指定密钥库的名称(产生的各cM息将不在.keystore文g?
- -keyalg 指定密钥的算?(?RSA DSAQ默认gؓQDSA)
- -validity 指定创徏的证书有效期多少?默认 90)
- -keysize 指定密钥长度 Q默?1024Q?/li>
- -storepass 指定密钥库的密码(获取keystore信息所需的密?
- -keypass 指定别名条目的密?U钥的密?
- -dname 指定证书发行者信?其中Q?“CN=名字与姓?OU=l织单位名称,O=l织名称,L=城市或区域名 U?ST=州或省䆾名称,C=单位的两字母国家代码”
- -list 昄密钥库中的证书信?keytool -list -v -keystore 指定keystore -storepass 密码
- -v 昄密钥库中的证书详l信?/li>
- -export 别名指定的证书导出到文?keytool -export -alias 需要导出的别名 -keystore 指定keystore -file 指定导出的证书位|及证书名称 -storepass 密码
- -file 参数指定导出到文件的文g?/li>
- -delete 删除密钥库中某条?keytool -delete -alias 指定需删除的别 -keystore 指定keystore – storepass 密码
- -printcert 查看导出的证书信?keytool -printcert -file g:\sso\michael.crt
- -keypasswd 修改密钥库中指定条目口o keytool -keypasswd -alias 需修改的别?-keypass 旧密?-new 新密?-storepass keystore密码 -keystore sage
- -storepasswd 修改keystore口o keytool -storepasswd -keystore g:\sso\michael.keystore(需修改口o的keystore) -storepass pwdold(原始密码) -new pwdnew(新密?
- -import 已{数字证书导入密钥?keytool -import -alias 指定导入条目的别?-keystore 指定keystore -file 需导入的证?/li>
- 生成证书
- 查看证书
- 证书导出
- 附录资料
~省情况下，-list 命o打印证书?nbsp;MD5 指纹。而如果指定了 -v 选项Q将以可L式打印证书，如果指定?nbsp;-rfc 选项Q将以可打印的编码格式输书?/p>
- jdk1.4.2 Qhttp://docs.oracle.com/javase/1.4.2/docs/tooldocs/windows/keytool.html
- jdk1.6 Qhttp://docs.oracle.com/javase/6/docs/technotes/tools/windows/keytool.html
- jdk1.7 Qhttp://docs.oracle.com/javase/7/docs/technotes/tools/windows/keytool.html
]]>
]]>
GenymotionLAndroid模拟器中q行最快的Q但是服务器在国外，Android镜像下蝲h那个速度׃惌了?/strong>
Add new device后下载速度太慢了，Ҏp|
解决Ҏ如下Q?/strong>
Ҏ一Q?/strong>
1、设|HTTP代理Q在Setting->NetworkQ自p|HTTP proxy和PortQ?/strong>
Ҏ二：
1、找C载链接，直接用迅h下来
遇到下蝲p|或者下载太慢，win+R打开q行框，输入 %appdata%Q?再点M一步（Alt+↑ Q，扑ֈlocal文g多w?/span>Genymobile,打开 查看里面的genymotion.log文gQ?/span>
扑ֈcM下面的文?/strong>
[Genymotion] [Debug] Downloading file
"http://files2.genymotion.com/dists/4.1.1/ova/genymotion_vbox86p_4.1.1_151117_133208.ova"
http://file........ova q个虚拟镜像地址直接用迅h速版下蝲Q或者用迅LU下载等功能很快能完成下?/strong>
2、把下蝲的文件复制到C:\Users\用户ȝ?span style="margin:0px; padding:0px; border:0px">\AppData\Local\Genymobile\Genymotion\ova 中覆盖里面以随机数命名的对应镜像?/span>实际上就是刚才看?/strong>genymotion软g刚刚点击下蝲的那个镜像，
3?/span>重新按照刚刚下蝲操作GUI的下载步骤，你会发现对应的镜像已l可以用了不需要下载了Q验证安装后即会昄在设备列表中?/span>
点击start Q启动模拟器Q开始?span style="border-style:initial; border-color:initial">
]]>
本经验目前在已有搜狗输入?for Linux和Sublime Text 3的情况下安装成功?/p>
void gtk_im_context_set_client_window (GtkIMContext *context,
GdkWindow *window)
{
GtkIMContextClass *klass;
g_return_if_fail (GTK_IS_IM_CONTEXT (context));
klass = GTK_IM_CONTEXT_GET_CLASS (context);
if (klass->set_client_window)
klass->set_client_window (context, window);
g_object_set_data(G_OBJECT(context),"window",window);
if(!GDK_IS_WINDOW (window))
return;
int width = gdk_window_get_width(window);
int height = gdk_window_get_height(window);
if(width != 0 && height !=0)
gtk_im_context_focus_in(context);
}
上一步的代码~译成共享库libsublime-imfix.soQ命?/p>
cd ~
gcc -shared -o libsublime-imfix.so sublime_imfix.c `pkg-config --libs --cflags gtk+-2.0` -fPIC
如果q行不成功，可能是某些库没有安装Q执行下边的命o来安装缺q?br />
sudo apt-get install build-essential
sudo apt-get install libgtk2.0-dev
然后?em>libsublime-imfix.so拯?em>sublime_text所在文件夹
sudo mv libsublime-imfix.so /opt/sublime_text/
修改sublime-text-2.desktop注意Qsublime_text.desktop不同版本有所不同Q请调整己安装版本的路径sudo vim /usr/share/applications/sublime_text.desktop
Version=1.0
Type=Application
Name=Sublime Text
GenericName=Text Editor
Comment=Sophisticated text editor for code, markup and prose
Exec=/usr/bin/subl %F #q里修改执行路径?usr/bin/subl,subl文g刚才已经修改q，大家应该记得
Terminal=false
MimeType=text/plain;
Icon=sublime-text
Categories=TextEditor;Development;
StartupNotify=true
Actions=Window;Document;
[Desktop Action Window]
Name=New Window
Exec=/usr/bin/subl -n #q里修改执行路径?usr/bin/subl,subl文g刚才已经修改q，大家应该记得
OnlyShowIn=Unity;
[Desktop Action Document]
Name=New File
Exec=/usr/bin/subl new_file #q里修改执行路径?usr/bin/subl,subl文g刚才已经修改q，大家应该记得
OnlyShowIn=Unity;
打开“控制中心”-》打开“主菜?#8221;-?#8220;应用E序”树k目录中找?#8220;~程”Q找?#8220;sublime text”Q双M攚w边的命o?div>/usr/bin/subl %F
]]>
在上一博?a >《模?HTTP h?/a>中，我们分别介绍了两U方法来q行 HTTP 的模拟请求：HttpURLConnection ?HttpClient Q到目前为止q两U方法都工作的很好，基本上可以实现我们需要的 GET/POST Ҏ的模拟。对于一个爬虫来_能发?HTTP hQ能获取面数据Q能解析|页内容Q这相当于已l完?80% 的工作了。只不过对于剩下的这 20% 的工作，q得p我们另外 80% 的时? :-)
在这博客里Q我们将介绍剩下 20% 的工作中最为重要的一：如何?Java 中?HTTP 代理Q代理也是爬虫技术中的重要一V你如果要大规模的爬别h|页上的内容Q必然会对h家的|站造成影响Q如果你太拼了，׃遭h查封。要防止别h查封?们，我们要么自qE序分布到大量机器上去，但是对于资金和资源有限的我们来说q是很奢侈的Q要么就使用代理技术，从网上捞一批代理，免费的也好收费的 也好Q或者购C批廉L VPS 来搭q代理服务器。关于如何搭q代理服务器，后面有时间的话我再写一关于这个话题的博客。现在有了一大批代理服务器之后，可以用我们这 博客所介绍的技术了?/p>
一、简单的 HTTP 代理
我们先从最单的开始，|上有很多免费代理，直接上百度搜?“免费代理” 或?“HTTP 代理” p扑ֈ很多Q虽然网上能扑ֈ大量的免费代理，但它们的安全性已l有很多文章讨论q了Q也有专门的文章Ҏq行调研的，譬如q篇文章Q我在这里就不多作说明，如果你的爬虫爬取的信息ƈ没有什么特别的隐私问题Q可以忽略之Q如果你的爬虫涉及一些例如模拟登录之cȝ功能Q考虑到安全性，我徏议你q是不要使用|上公开的免费代理，而是搭徏自己的代理服务器比较靠谱Q?/p>
1.1 HttpURLConnection 使用代理
HttpURLConnection ?openConnection() Ҏ可以传入一?Proxy 参数Q如下：
1 2 3 | Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("127.0.0.1", 9876));URL obj = new URL(url);HttpURLConnection con = (HttpURLConnection) obj.openConnection(proxy); |
OK 了，p么简单！
不仅如此Q我们注意到 Proxy 构造函数的W一个参Cؓ枚Dcd Proxy.Type.HTTP Q那么很昄Q如果将其修改ؓ Proxy.Type.SOCKS 卛_以?SOCKS 代理?/p>
1.2 HttpClient 使用代理
׃ HttpClient 非常灉|Q?HttpClient 来连接代理有很多不同的方法。最单的Ҏ莫过于下面这P
1 2 3 4 | HttpHost proxy = new HttpHost("127.0.0.1", 9876, "HTTP");CloseableHttpClient httpclient = HttpClients.createDefault();HttpGet request = new HttpGet(url);CloseableHttpResponse response = httpclient.execute(proxy, request); |
和上一中使用 HttpClient 发送请求的代码几乎一P只是 httpclient.execute() Ҏ多加了一个参敎ͼW一参数?HttpHost cdQ我们这里设|成我们的代理即可?/p>
q里要注意一点的是，虽然q里?new HttpHost() 和上面的 new Proxy() 一P也是可以指定协议cd的，但是遗憾的是 HttpClient 默认是不支持 SOCKS 协议的，如果我们使用下面的代码：
1 | HttpHost proxy = new HttpHost("127.0.0.1", 1080, "SOCKS"); |
会直接报协议不支持异常Q?/p>
org.apache.http.conn.UnsupportedSchemeException: socks protocol is not supported
如果希望 HttpClient 支持 SOCKS 代理Q可以参看这里：How to use Socks 5 proxy with Apache HTTP Client 4? 通过 HttpClient 提供?ConnectionSocketFactory cL实现?/p>
虽然使用q种方式很简单，只需要加个参数就可以了，但是其实?HttpClient 的代码注释，如下Q?/p>
1 2 3 4 5 6 7 | /** @param target the target host for the request.* Implementations may accept <code>null</code>* if they can still determine a route, for example* to a default target or by inspecting the request.* @param request the request to execute*/ |
可以看到W一个参?target q不是代理，它的真实作用?执行h的目标主?/strong>Q这个解释有Ҏp，什么叫?执行h的目标主?/strong>Q代理算不算执行h的目标主?/strong>呢？因ؓ按常理来Ԍ执行h的目标主?/strong> 应该是要h URL 对应的站Ҏ寏V如果不的话，Z么这里将 target 讄成代理也能正常工作？q个我也不清楚，q需要进一步研I下 HttpClient 的源码来深入了解下?/p> 除了上面介绍的这U方式（自己写的Q不推荐使用Q来使用代理之外QHttpClient 官网q提供了几个CZQ我其作ؓ推荐写法记录在此?/p> W一U写法是使用 RequestConfig c?/a>Q如下： W二U写法是使用 RoutePlanner c?/a>Q如下： 我们在调?HTTP 爬虫E序Ӟ常常需要切换代理来试Q有时候直接用系l自带的代理配置是一U简单的Ҏ。以前在?.Net 目ӞE序默认使用 Internet |络讄中配的代理，遗憾的是Q我q里说的pȝ代理配置指的 JVM pȝQ而不是操作系l，我还没找到简单的Ҏ来让 Java E序直接使用 Windows pȝ下的代理配置?/p> 管如此Q系l代理用v来还是很单的。一般有下面两种方式可以讄 JVM 的代理配|： Java 中的 可以通过下面的方式来分别讄 HTTP 代理QHTTPS 代理?SOCKS 代理Q?/p> q里有三点要说明Q?/p> 可以使用 上面两种Ҏ都可以设|系l，下面要怎么在程序中自动使用pȝ代理呢？ 对于 CloseableHttpClient httpclient = HttpClients.createDefault(); HttpGet request = new HttpGet(url);request.setConfig( RequestConfig.custom() .setProxy(new HttpHost("45.32.21.237", 8888, "HTTP")) .build()); CloseableHttpResponse response = httpclient.execute(request);HttpHost proxy = new HttpHost("127.0.0.1", 9876, "HTTP");DefaultProxyRoutePlanner routePlanner = new DefaultProxyRoutePlanner(proxy); CloseableHttpClient httpclient = HttpClients.custom() .setRoutePlanner(routePlanner) .build();HttpGet request = new HttpGet(url);CloseableHttpResponse response = httpclient.execute(request);二、用系l代理配|?/h2>
2.1 System.setProperty
System cM仅仅是用来给我们 System.out.println() 打印信息的，它其实还有很多静态方法和属性可以用。其?System.setProperty() 是比较常用的一个?/p> // HTTP 代理Q只能代?HTTP hSystem.setProperty("http.proxyHost", "127.0.0.1");System.setProperty("http.proxyPort", "9876");// HTTPS 代理Q只能代?HTTPS hSystem.setProperty("https.proxyHost", "127.0.0.1");System.setProperty("https.proxyPort", "9876");// SOCKS 代理Q支?HTTP ?HTTPS h// 注意Q如果设|了 SOCKS 代理׃要设 HTTP/HTTPS 代理System.setProperty("socksProxyHost", "127.0.0.1");System.setProperty("socksProxyPort", "1080");socksProxyHost ?socksProxyPort 中间没有数?/li>// 同时支持代理 HTTP/HTTPS hSystem.setProperty("proxyHost", "127.0.0.1");System.setProperty("proxyPort", "9876");2.2 JVM 命o行参?/h4>
System.setProperty() Ҏ来设|系l代理，也可以直接将q些参数通过 JVM 的命令行参数来指定。如果你使用的是 Eclipse Q可以按下面的步骤来讄Q?/p> -DproxyHost=127.0.0.1 -DproxyPort=9876
2.3 使用pȝ代理
HttpURLConnection cL_E序不用做Q何变动，它会默认使用pȝ代理。但?HttpClient 默认是不使用pȝ代理的，如果惌它默认用系l代理，可以通过 SystemDefaultRoutePlanner ?ProxySelector 来设|。示例代码如下：SystemDefaultRoutePlanner routePlanner = new SystemDefaultRoutePlanner(ProxySelector.getDefault());CloseableHttpClient httpclient = HttpClients.custom() .setRoutePlanner(routePlanner) .build();HttpGet request = new HttpGet(url); CloseableHttpResponse response = httpclient.execute(request);参?/h2>
]]>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.2</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<artifactSet>
<includes>
<include>org.apache.activemq:activemq-mqtt</include>
</includes>
</artifactSet>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>com.duxiu.demo.app.ApplicationKt</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
此配|方式将所有的依赖包的源码都解压打包进?
如果是war?会将整个站点解压打包q去.
同时打包q程中如果有classpath和依赖的jar包有相同的类,会将classpath里的cL换掉依赖包里的类
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<!--<descriptors>
<descriptor>assembly.xml</descriptor>
</descriptors>-->
<!--<finalName>employees-app-${project.version}</finalName>-->
<archive>
<manifest>
<mainClass>com.duxiu.demo.app.ApplicationKt</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>appassembler-maven-plugin</artifactId>
<version>1.10</version>
<configuration>
<!-- 生成linux, Windows两种q_的执行脚?-->
<platforms>
<platform>windows</platform>
<platform>unix</platform>
</platforms>
<!-- 根目?-->
<assembleDirectory>${project.build.directory}/mall</assembleDirectory>
<!-- 打包?/span>jarQ以?/span>maven依赖?/span>jar攑ֈq个目录里面 -->
<repositoryName>lib</repositoryName>
<!-- 可执行脚本的目录 -->
<binFolder>bin</binFolder>
<!-- 配置文g的目标目?-->
<configurationDirectory>conf</configurationDirectory>
<!-- 拯配置文gC面的目录?-->
<copyConfigurationDirectory>true</copyConfigurationDirectory>
<!-- 从哪里拷贝配|文?/span> (默认src/main/config) -->
<configurationSourceDirectory>src/main/resources</configurationSourceDirectory>
<!-- lib目录?/span>jar的存放规则，默认?/span>${groupId}/${artifactId}的目录格式，flat表示直接?/span>jar攑ֈlib目录 -->
<repositoryLayout>flat</repositoryLayout>
<encoding>UTF-8</encoding>
<logsDirectory>logs</logsDirectory>
<tempDirectory>tmp</tempDirectory>
<programs>
<program>
<id>mall</id>
<!-- 启动c?-->
<mainClass>com.duxiu.demo.app.ApplicationKt</mainClass>
<jvmSettings>
<extraArguments>
<extraArgument>-server</extraArgument>
<extraArgument>-Xmx2G</extraArgument>
<extraArgument>-Xms2G</extraArgument>
</extraArguments>
</jvmSettings>
</program>
</programs>
</configuration>
</plugin>
打包应用E序,q会生成bat或sh可执行文?br />
<plugin>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<id>move-main-class</id>
<phase>compile</phase>
<configuration>
<tasks>
<move todir="${project.build.directory}/${project.artifactId}-${version}/com/duxiu/demo/app">
<fileset dir="${project.build.directory}/classes/com/duxiu/demo/app">
<include name="*.class" />
</fileset>
</move>
</tasks>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
打包的时候将包里的某个文件移动到指定的位|?img src ="http://www.dentisthealthcenter.com/wangxinsh55/aggbug/431252.html" width = "1" height = "1" />
]]>
q样做的最l效果就是修改了目的运行方式。原先的q行方式是以tomcatZ心，由tomcat来启动和l止目Q现在是由我们的启动E序 Z心，由启动程序来负责启动和终止项目。就相当于现在流行的csE序一P有单独的启动脚本Q在启动时进行环境预初始化，更新E序以及其它操作Q待完成 之后再进行最l的目启动?/p>
q篇主要讲解如何使用embeded tomcat在代码中q行启动和终止。网上的一般文章均为tomca5.x来做Q这里用了最新的tomcat7Q因为tomcat7为embeded开 发，单独发布了org.apache.tomcat.embed包，以进行独立的embed开发。以下是相应的maven?/p>
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 | <dependency> <groupId>org.apache.tomcat.embed</groupId> <artifactId>tomcat-embed-core</artifactId> <version>7.0.2</version> </dependency> <dependency> <groupId>org.apache.tomcat</groupId> <artifactId>tomcat-util</artifactId> <version>7.0.2</version> </dependency> <dependency> <groupId>org.apache.tomcat.embed</groupId> <artifactId>tomcat-embed-jasper</artifactId> <version>7.0.2</version> </dependency> <dependency> <groupId>org.apache.tomcat.embed</groupId> <artifactId>tomcat-embed-logging-juli</artifactId> <version>7.0.2</version> </dependency> |
使用了embed包中的core包，以及用于~译jsp的jasper包，然后是工L以及q行上场记录的logging-juli包。开始写代码Q?/p>
1 2 3 4 5 6 7 | //讄工作目录 String catalina_home = "d:/"; Tomcat tomcat = new Tomcat(); tomcat.setHostname("localhost"); tomcat.setPort(startPort); //讄工作目录,其实没什么用,tomcat需要用这个目录进行写一些东?/code> tomcat.setBaseDir(catalina_home); |
上面使用了TomcatcLq行启动c，在tomcat7以前均是使用一个叫EmbedcLq行启动Q在tomcat7之后Qembedc被不徏 议用，而徏议用新的TomcatcLq行启动了。然后设|主机名Q端口，再设|一个工作目录。这个工作目录可以是L目录Q主要是tomcat需要这 个目录来记录一些东西，比如记录word信息Q日志信息（如果配置了日志的话）Q以及时文件存储等?/p>
1 2 3 4 5 6 7 8 | //讄E序的目录信?/code> tomcat.getHost().setAppBase("e:/"); // Add AprLifecycleListener StandardServer server = (StandardServer) tomcat.getServer(); AprLifecycleListener listener = new AprLifecycleListener(); server.addLifecycleListener(listener); //注册关闭端口以进行关?/code> tomcat.getServer().setPort(shutdownPort); |
上面的代码，首先讄我们的项目程序所在的appbaseQ即N目代码的地方。在通常的tomcat配置中，q个目录一般是webapps。接 着讄一个listenerQ这个listener主要是负责启动一些比如html native支持E序以及ipv6{信息配|（可以忽略Q。接着是配|一个关闭的注册端口Q当向这个端口发送信息时Q就可以辑ֈ关闭tomcat的目?Q后面会Ԍ?/p>
1 2 3 4 5 6 7 8 9 | //加蝲上下?/code> StandardContext standardContext = new StandardContext(); standardContext.setPath("/aa");//contextPath standardContext.setDocBase("aa");//文g目录位置 standardContext.addLifecycleListener(new Tomcat.DefaultWebXmlListener()); //保证已经配置好了?/code> standardContext.addLifecycleListener(new Tomcat.FixContextListener()); standardContext.setSessionCookieName("t-session"); tomcat.getHost().addChild(standardContext); |
我们单独使用了一个Context来ؓq个hostd上下文，tomcat本n提供一个方法tomcat.addWebҎ来添加项目包Q不q?׃q里需要单独设|一个tomcat的sessionNameQ所以用与与tomcat.addWeb实现cM的方法来d一个项目包?br /> 以上代码中有两个需要注意的listenerQ一个是DefaultWebXmlListenerQ这个是由tomcat加蝲一些默认的配置?息，比如jspServletQ以及一些繁复的mime/type信息Q加上这个，׃需要我们自己去写这么多的配|，因ؓ每个目都需要这些。这个配|?与tomcat目录下的conf/web.xml中的配置一P只不q这里是代码化了。第二个是FixContextListenerQ这个主要是在项?部v完后Q将q个上下文设|ؓconfiguredQ表C已l配|好了（不然Qtomcat启动时会报错Q即相应上下文还未配|好Q?br /> 配置OK了之后，是启动tomcat了：
1 2 | tomcat.start(); tomcat.getServer().await(); |
启动tomcatQƈ让tomcat在关闭端口上监听。如果没有最后一句，E序直接结束，保证监听之后Qtomcat一直监听关闭事Ӟ待有关闭事g之后才结束当前程序。所以如果想要关闭当前的tomcatQ只需要向关闭端口发送一些信息即可：
1 2 3 4 5 6 7 8 9 | private static void shutdown() throws Exception { Socket socket = new Socket("localhost", shutdownPort); OutputStream stream = socket.getOutputStream(); for(int i = 0;i < shutdown.length();i++) stream.write(shutdown.charAt(i)); stream.flush(); stream.close(); socket.close(); } |
q样卛_辑ֈ关闭tomcat的目的?/p>
实际上看整个目代码Q项目代码的q行Q就是一个配|一个基的server.xmlQ即tomcat目录下的 conf/server.xml)Q先配置q行端口Q关闭监听端口；然后配置q行的host以及d一个上下文contextQ最后就开始运行ƈ开始监 听。对照这个程序，再看一下server.xml中的配置信息Q就很容易明白以上这D代码了?/p>
]]>
]]>
哪些配置文gQ?/h1>
在安装好Kerberos的Y件之后，会用到几个配|文Ӟ例如
+ /etc/krb5.conf
+ /var/kerberos/krb5kdc/kdc.conf
配置文g的说?/h1>/etc/krb5.conf
可以用命?code>man krb5.conf来查看关于该配置文g的说?/p>
先看一下该文g的模板：
[logging]default = FILE:/var/log/krb5libs.logkdc = FILE:/var/log/krb5kdc.logadmin_server = FILE:/var/log/kadmind.log[libdefaults]default_realm = EXAMPLE.COMdns_lookup_realm = falsedns_lookup_kdc = falseticket_lifetime = 24hrenew_lifetime = 7dforwardable = true[realms]EXAMPLE.COM = {kdc = example.comadmin_server = example.com}[domain_realm].example.com = EXAMPLE.COMexample.com = EXAMPLE.COM
关于几个重要配置的说明
+ [realms].kdc : the name of the host running a KDC for that realm.
+ [realms].admin_server : identifies the host where the administration server is running. Typically this is the Master Kerberos server.
+ [domain_realm] : provides a translation from a hostname to the Kerberos realm name for the service provided by that host.
]]>
Kerberos
Kerberos is a network authentication system based on the principal of a trusted third party. The other two parties being the user and the service the user wishes to authenticate to. Not all services and applications can use Kerberos, but for those that can, it brings the network environment one step closer to being Single Sign On (SSO).
This section covers installation and configuration of a Kerberos server, and some example client configurations.
Overview
If you are new to Kerberos there are a few terms that are good to understand before setting up a Kerberos server. Most of the terms will relate to things you may be familiar with in other environments:
-
Principal: any users, computers, and services provided by servers need to be defined as Kerberos Principals.
-
Instances: are used for service principals and special administrative principals.
-
Realms: the unique realm of control provided by the Kerberos installation. Usually the DNS domain converted to uppercase (EXAMPLE.COM).
-
Key Distribution Center: (KDC) consist of three parts, a database of all principals, the authentication server, and the ticket granting server. For each realm there must be at least one KDC.
-
Ticket Granting Ticket: issued by the Authentication Server (AS), the Ticket Granting Ticket (TGT) is encrypted in the user's password which is known only to the user and the KDC.
-
Ticket Granting Server: (TGS) issues service tickets to clients upon request.
-
Tickets: confirm the identity of the two principals. One principal being a user and the other a service requested by the user. Tickets establish an encryption key used for secure communication during the authenticated session.
-
Keytab Files: are files extracted from the KDC principal database and contain the encryption key for a service or host.
To put the pieces together, a Realm has at least one KDC, preferably two for redundancy, which contains a database of Principals. When a user principal logs into a workstation, configured for Kerberos authentication, the KDC issues a Ticket Granting Ticket (TGT). If the user supplied credentials match, the user is authenticated and can then request tickets for Kerberized services from the Ticket Granting Server (TGS). The service tickets allow the user to authenticate to the service without entering another username and password.
Kerberos Server
Installation
Before installing the Kerberos server a properly configured DNS server is needed for your domain. Since the Kerberos Realm by convention matches the domain name, this section uses the example.com domain configured in the section called “Primary Master”.
Also, Kerberos is a time sensitive protocol. So if the local system time between a client machine and the server differs by more than five minutes (by default), the workstation will not be able to authenticate. To correct the problem all hosts should have their time synchronized using the Network Time Protocol (NTP). For details on setting up NTP see the section called “Time Synchronisation with NTP”.
The first step in installing a Kerberos Realm is to install the krb5-kdc and krb5-admin-server packages. From a terminal enter:
sudo apt-get install krb5-kdc krb5-admin-server
You will be asked at the end of the install to supply a name for the Kerberos and Admin servers, which may or may not be the same server, for the realm.
Next, create the new realm with the kdb5_newrealm utility:
sudo krb5_newrealm
Configuration
The questions asked during installation are used to configure the /etc/krb5.conf file. If you need to adjust the Key Distribution Center (KDC) settings simply edit the file and restart the krb5-kdc daemon.
-
Now that the KDC running an admin user is needed. It is recommended to use a different username from your everyday username. Using the kadmin.local utility in a terminal prompt enter:
sudo kadmin.local
Authenticating as principal root/admin@EXAMPLE.COM with password. kadmin.local:addprinc steve/adminWARNING: no policy specified for steve/admin@EXAMPLE.COM; defaulting to no policy Enter password for principal "steve/admin@EXAMPLE.COM": Re-enter password for principal "steve/admin@EXAMPLE.COM": Principal "steve/admin@EXAMPLE.COM" created. kadmin.local:quitIn the above example steve is the Principal, /admin is an Instance, and @EXAMPLE.COM signifies the realm. The "every day" Principal would be steve@EXAMPLE.COM, and should have only normal user rights.
![[Note]](https://help.ubuntu.com/libs/admon/note.png)
Replace EXAMPLE.COM and steve with your Realm and admin username.
-
Next, the new admin user needs to have the appropriate Access Control List (ACL) permissions. The permissions are configured in the
/etc/krb5kdc/kadm5.aclfile:steve/admin@EXAMPLE.COM *
This entry grants steve/admin the ability to perform any operation on all principals in the realm.
-
Now restart the krb5-admin-server for the new ACL to take affect:
sudo /etc/init.d/krb5-admin-server restart
-
The new user principal can be tested using the kinit utility:
kinit steve/admin
steve/admin@EXAMPLE.COM's Password:After entering the password, use the klist utility to view information about the Ticket Granting Ticket (TGT):
klist
Credentials cache: FILE:/tmp/krb5cc_1000 Principal: steve/admin@EXAMPLE.COM Issued Expires Principal Jul 13 17:53:34 Jul 14 03:53:34 krbtgt/EXAMPLE.COM@EXAMPLE.COMYou may need to add an entry into the
/etc/hostsfor the KDC. For example:192.168.0.1 kdc01.example.com kdc01
Replacing 192.168.0.1 with the IP address of your KDC.
-
In order for clients to determine the KDC for the Realm some DNS SRV records are needed. Add the following to
/etc/named/db.example.com:_kerberos._udp.EXAMPLE.COM. IN SRV 1 0 88 kdc01.example.com. _kerberos._tcp.EXAMPLE.COM. IN SRV 1 0 88 kdc01.example.com. _kerberos._udp.EXAMPLE.COM. IN SRV 10 0 88 kdc02.example.com. _kerberos._tcp.EXAMPLE.COM. IN SRV 10 0 88 kdc02.example.com. _kerberos-adm._tcp.EXAMPLE.COM. IN SRV 1 0 749 kdc01.example.com. _kpasswd._udp.EXAMPLE.COM. IN SRV 1 0 464 kdc01.example.com.
Replace EXAMPLE.COM, kdc01, and kdc02 with your domain name, primary KDC, and secondary KDC.
See Chapter 7, Domain Name Service (DNS) for detailed instructions on setting up DNS.
Your new Kerberos Realm is now ready to authenticate clients.
Secondary KDC
Once you have one Key Distribution Center (KDC) on your network, it is good practice to have a Secondary KDC in case the primary becomes unavailable.
-
First, install the packages, and when asked for the Kerberos and Admin server names enter the name of the Primary KDC:
sudo apt-get install krb5-kdc krb5-admin-server
-
Once you have the packages installed, create the Secondary KDC's host principal. From a terminal prompt, enter:
kadmin -q "addprinc -randkey host/kdc02.example.com"
After, issuing any kadmin commands you will be prompted for your username/admin@EXAMPLE.COM principal password.
-
Extract the keytab file:
kadmin -q "ktadd -k keytab.kdc02 host/kdc02.example.com"
-
There should now be a
keytab.kdc02in the current directory, move the file to/etc/krb5.keytab:sudo mv keytab.kdc02 /etc/krb5.keytab
If the path to the
keytab.kdc02file is different adjust accordingly.Also, you can list the principals in a Keytab file, which can be useful when troubleshooting, using the klist utility:
sudo klist -k /etc/krb5.keytab
-
Next, there needs to be a
kpropd.aclfile on each KDC that lists all KDCs for the Realm. For example, on both primary and secondary KDC, create/etc/krb5kdc/kpropd.acl:host/kdc01.example.com@EXAMPLE.COM host/kdc02.example.com@EXAMPLE.COM
-
Create an empty database on the Secondary KDC:
sudo kdb5_util -s create
-
Now start the kpropd daemon, which listens for connections from the kprop utility. kprop is used to transfer dump files:
sudo kpropd -S
-
From a terminal on the Primary KDC, create a dump file of the principal database:
sudo kdb5_util dump /var/lib/krb5kdc/dump
-
Extract the Primary KDC's keytab file and copy it to
/etc/krb5.keytab:kadmin -q "ktadd -k keytab.kdc01 host/kdc01.example.com" sudo mv keytab.kdc01 /etc/kr5b.keytab
Make sure there is a host for kdc01.example.com before extracting the Keytab.
-
Using the kprop utility push the database to the Secondary KDC:
sudo kprop -r EXAMPLE.COM -f /var/lib/krb5kdc/dump kdc02.example.com
There should be a SUCCEEDED message if the propagation worked. If there is an error message check
/var/log/syslogon the secondary KDC for more information.You may also want to create a cron job to periodically update the database on the Secondary KDC. For example, the following will push the database every hour:
# m h dom mon dow command 0 * * * * /usr/sbin/kdb5_util dump /var/lib/krb5kdc/dump && /usr/sbin/kprop -r EXAMPLE.COM -f /var/lib/krb5kdc/dump kdc02.example.com
-
Back on the Secondary KDC, create a stash file to hold the Kerberos master key:
sudo kdb5_util stash
-
Finally, start the krb5-kdc daemon on the Secondary KDC:
sudo /etc/init.d/krb5-kdc start
The Secondary KDC should now be able to issue tickets for the Realm. You can test this by stopping the krb5-kdc daemon on the Primary KDC, then use kinit to request a ticket. If all goes well you should receive a ticket from the Secondary KDC.
Kerberos Linux Client
This section covers configuring a Linux system as a Kerberos client. This will allow access to any kerberized services once a user has successfully logged into the system.
Installation
In order to authenticate to a Kerberos Realm, the krb5-user and libpam-krb5 packages are needed, along with a few others that are not strictly necessary but make life easier. To install the packages enter the following in a terminal prompt:
sudo apt-get install krb5-user libpam-krb5 libpam-ccreds auth-client-config
The auth-client-config package allows simple configuration of PAM for authentication from multiple sources, and the libpam-ccreds will cache authentication credentials allowing you to login in case the Key Distribution Center (KDC) is unavailable. This package is also useful for laptops that may authenticate using Kerberos while on the corporate network, but will need to be accessed off the network as well.
Configuration
To configure the client in a terminal enter:
sudo dpkg-reconfigure krb5-config
You will then be prompted to enter the name of the Kerberos Realm. Also, if you don't have DNS configured with Kerberos SRV records, the menu will prompt you for the hostname of the Key Distribution Center (KDC) and Realm Administration server.
The dpkg-reconfigure adds entries to the /etc/krb5.conf file for your Realm. You should have entries similar to the following:
[libdefaults] default_realm = EXAMPLE.COM ... [realms] EXAMPLE.COM = } kdc = 192.168.0.1 admin_server = 192.168.0.1 }
You can test the configuration by requesting a ticket using the kinit utility. For example:
kinit steve@EXAMPLE.COM Password for steve@EXAMPLE.COM: When a ticket has been granted, the details can be viewed using klist:
klist Ticket cache: FILE:/tmp/krb5cc_1000 Default principal: steve@EXAMPLE.COM Valid starting Expires Service principal 07/24/08 05:18:56 07/24/08 15:18:56 krbtgt/EXAMPLE.COM@EXAMPLE.COM renew until 07/25/08 05:18:57 Kerberos 4 ticket cache: /tmp/tkt1000 klist: You have no tickets cached Next, use the auth-client-config to configure the libpam-krb5 module to request a ticket during login:
sudo auth-client-config -a -p kerberos_example
You will should now receive a ticket upon successful login authentication.
Resources
-
For more information on Kerberos see the MIT Kerberos site.
-
The Ubuntu Wiki Kerberos page has more details.
-
O'Reilly's Kerberos: The Definitive Guide is a great reference when setting up Kerberos.
-
Also, feel free to stop by the #ubuntu-server IRC channel on Freenode if you have Kerberos questions.
]]>
1.下蝲krb5-1.9
http://web.mit.edu/kerberos/dist/krb5/1.9/krb5-1.9-signed.tar
2.解压
tar -xvf krb5-1.9.signed.tar
生成krb5-1.9.tar.gz 和krb5-1.9.tar.gz.asc
l箋解压tar zxvf krb5-1.9.tar.gz
3.~译
cd krb5-1.9/src
./configure
make
make install
4.配置/etc/krb5.conf
q个是Kerberos最主要的配|文Ӟ而且一定要攑֜/etc?nbsp;
- [libdefaults]
- default_realm = 360BUY.COM
- [realms]
- 360BUY.COM = {
- kdc = m1.360buy.com
- admin_server = m1.360buy.com
- default_domain =360buy.com
- }
- [logging]
- kdc = FILE:/data/logs/krb5/krb5kdc.log
- admin_server = FILE:/data/logs/krb5/kadmin.log
- default = FILE:/data/logs/krb5/krb5lib.log
[libdefaults]中的defalt_realm表示在不l出域的时候，默认采用q个
[logging]中的是指定日志的位置
[realms]是最重要的也是Kerberos中最隄概念。，UCؓkerberos域，表示KDC所辖的范_可以和DNS域名一P也可以不一?nbsp;
5.配置/usr/local/var/krb5kdc/kdc.conf
׃上面安装时没有选择安装目录Q所以默认的安装位置?usr/local/var/krb5kdc
- [kdcdefaults]
- kdc_ports=750,88
- [realm]
- 360BUY.COM ={
- database_name=/usr/local/var/krb5kdc/principal
- admin_keytab=/usr/local/var/krb5kdc/kadm5.keytab
- acl_file=/usr/local/var/krb5kdc/kadm5.acl
- key_stash_file=/usr/local/var/krb5kdc/.k5.360BUY.COM
- kdc_ports=750,88
- max_life=10h 0m 0s
- max_renewable_life=7d 0h 0m 0s
- }
6.创徏一个kerberos数据?/span>
- /usr/local/sbin/kdb5_util create -r 360BUY.COM -s
会要求创建数据库的密码?nbsp;
q且创徏/usr/local/var/krb5kdc/principal保存数据库文?nbsp;
7.dkerberos
- /usr/local/sbin/kadmin.local
1Q查看用?nbsp;
listprincs
2Q添加用?nbsp;
addprinc admin/admin@360BUY.COM
3Q删除用?nbsp;
delprinc
4Q创建keytab文g,
- ktadd -k /usr/local/var/krb5kdc/kadm5.keytab kadmin/admin kadmin/changepw
可以用kadd来增加用L权限
注意kadm5.keytab的\径要与kdc.conf中的路径一?nbsp;
8.重启krb5kdc和kadmindq程
/usr/local/sbin/kadmind
/usr/local/sbin/krb5kdc
9.更改/etc/hosts文g
d对应的host
192.168.101.201 m1.360buy.com kdc
192.168.101.202 m2.360buy.com client
q且需要修改对应的hostname
10.在KDC服务器上试据h
/usr/local/sbin/kadmin.local
kadmin.local:addprinc winston@360BUY.COM
提示创徏密码Q然后退?nbsp;
su winston
$ kinit winston@360BUY.COM
提示输入刚刚创徏的密?nbsp;
$ klist 查看自己甌的票?nbsp;
11.在Client端安装kerberos
同样需要编译，但是只需要配|文?etc/krb5.conf
内容和服务器的一?nbsp;
12.试KDC服务器申L?/span>
su winston
$ kinit winston@360BUY.COM
提示输入刚刚创徏的密?nbsp;
addprinc -randkey hdfs/sl.360buy.com@360BUY.COM
ktadd -norandkey -k hdfs.keytab hdfs/s1.360buy.com host/master.360buy.com
]]>
]]>
public staticQ对应的整数是二进制的Q?001Q也是9。如下：
…… | native | transient | volatile | synchronized | final | static | protected | private | public |
| 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 |
]]>
正确的日期分Z?/h3>
mysql优化器支持以下两U内|的日期函数q行分区Q?/p>
- TO_DAYS()
- YEAR()
看个例子Q?/p>
- mysql> CREATE TABLE part_date3
-
-> ( c1 int default NULL, -
-> c2 varchar(30) default NULL, -
-> c3 date default NULL) engine=myisam -
-> partition by range (to_days(c3)) -
-> (PARTITION p0 VALUES LESS THAN (to_days('1995-01-01')), -
-> PARTITION p1 VALUES LESS THAN (to_days('1996-01-01')) , -
-> PARTITION p2 VALUES LESS THAN (to_days('1997-01-01')) , -
-> PARTITION p3 VALUES LESS THAN (to_days('1998-01-01')) , -
-> PARTITION p4 VALUES LESS THAN (to_days('1999-01-01')) , -
-> PARTITION p5 VALUES LESS THAN (to_days('2000-01-01')) , -
-> PARTITION p6 VALUES LESS THAN (to_days('2001-01-01')) , -
-> PARTITION p7 VALUES LESS THAN (to_days('2002-01-01')) , -
-> PARTITION p8 VALUES LESS THAN (to_days('2003-01-01')) , -
-> PARTITION p9 VALUES LESS THAN (to_days('2004-01-01')) , -
-> PARTITION p10 VALUES LESS THAN (to_days('2010-01-01')), -
-> PARTITION p11 VALUES LESS THAN MAXVALUE ); - Query OK, 0 rows affected (0.00 sec)
以to_days()函数分区成功Q我们分析一下看看：
- mysql> explain partitions
-
-> select count(*) from part_date3 where -
-> c3> date '1995-01-01' and c3 <date '1995-12-31'\G - *************************** 1. row ***************************
-
id: 1 -
select_type: SIMPLE -
table: part_date3 -
partitions: p1 -
type: ALL - possible_keys: NULL
-
key: NULL -
key_len: NULL -
ref: NULL -
rows: 808431 -
Extra: Using where - 1 row in set (0.00 sec)
可以看到Q?a >mysql优化器这ơ不负众望，仅仅在p1分区q行查询。在q种情况下查询，真的能够带来提升查询效率么？下面分别对这ơ徏立的part_date3和之前分区失败的part_date1做一个查询对比：
- mysql> select count(*) from part_date3 where
-
-> c3> date '1995-01-01' and c3 <date '1995-12-31'; - +----------+
- | count(*) |
- +----------+
- |
805114 | - +----------+
- 1 row in set (4.11 sec)
-
- mysql> select count(*) from part_date1 where
-
-> c3> date '1995-01-01' and c3 <date '1995-12-31'; - +----------+
- | count(*) |
- +----------+
- |
805114 | - +----------+
- 1 row in set (40.33 sec)
可以看到Q分区正的话queryp旉?U，而分区错误则p旉40U（相当于没有分区）Q效率有90Q的提升Q所以我们千万要正确的用分区功能，分区后务必用explain验证Q这h能获得真正的性能提升?/p>
注意Q?/p>
在mysql5.1中徏立分的语句中Q只能包含下列函敎ͼ
ABS()
CEILING() and FLOOR() Q在使用q?个函数的建立分区表的前提是用函数的分区键是INTcdQ，例如
mysql> CREATE TABLE t (c FLOAT) PARTITION BY LIST( FLOOR(c) )( -> PARTITION p0 VALUES IN (1,3,5), -> PARTITION p1 VALUES IN (2,4,6) -> );; ERROR 1491 (HY000): The PARTITION function returns the wrong typemysql> CREATE TABLE t (c int) PARTITION BY LIST( FLOOR(c) )( -> PARTITION p0 VALUES IN (1,3,5), -> PARTITION p1 VALUES IN (2,4,6) -> ); Query OK, 0 rows affected (0.01 sec)
DAY()
DAYOFMONTH()
DAYOFWEEK()
DAYOFYEAR()
DATEDIFF()
EXTRACT()
HOUR()
MICROSECOND()
MINUTE()
MOD()
MONTH()
QUARTER()
SECOND()
TIME_TO_SEC()
TO_DAYS()
WEEKDAY()
YEAR()
YEARWEEK()
]]>
4 3 | I have a need to create a server farm that can handle 5+ million connections, 5+ million topics (one per client), process 300k messages/sec. I tried to see what various message brokers were capable so I am currently using two RHEL EC2 instances (r3.4xlarge) to make lots of available resources. So you do not need to look it up, it has 16vCPU, 122GB RAM. I am nowhere near that limit in usage. I am unable to pass the 600k connections limit. Since there doesn't seem to be any O/S limitation (plenty of RAM/CPU/etc.) on either the client nor the server what is limiting me? I have edited /etc/security/limits.conf as follows: I have edited /etc/sysctl.conf as follows: For Apollo: export APOLLO_ULIMIT=20000000 For ActiveMQ: I created 20 additional private addresses for eth0 on the client, then assigned them: ip addr add 11.22.33.44/24 dev eth0 I am FULLY aware of the 65k port limits which is why I did the above.
| ||||||||||||
|
2 | ANSWER: While doing this I realized that I had a misspelling in my client setting within /etc/sysctl.conf file for: net.ipv4.ip_local_port_range I am now able to connect 956,591 MQTT clients to my Apollo server in 188sec. More info: Trying to isolate if this is an O/S connection limitation or a Broker, I decided to write a simple Client/Server. The server: The Client: With 21 IPs, I would expect 65535-1024*21 = 1354731 to be the boundary. In reality I am able to achieve 1231734 So the socket/kernel/io stuff is worked out. I am STILL unable to achieve this using any broker. Again just after my client/server test this is the kernel settings. Client: Server: |
]]>
]]>
Spark history Server产生背景
以standaloneq行模式ZQ在q行Spark Application的时候，Spark会提供一个WEBUI列出应用E序的运行时信息Q但该WEBUI随着Application的完?成功/??而关闭，也就是说QSpark Applicationq行?成功/p|)后，无法查看Application的历史记录；
Spark history Server是Z应对q种情况而生的Q通过配置可以在Application执行的过E中记录下了日志事g信息Q那么在Application执行 l束后，WEBUIp重新渲染生成UI界面展现Application在执行过E中的运行时信息Q?/p>
Sparkq行在yarn或者mesos之上Q通过spark的history server仍然可以重构Z个已l完成的Application的运行时参数信息Q假如Applicationq行的事件日志信息已l记录下来）Q?/p>
配置&使用Spark History Server
以默认配|的方式启动spark history serverQ?/p>
cd $SPARK_HOME/sbin start-history-server.sh
报错Q?/p>
starting org.apache.spark.deploy.history.HistoryServer, logging to /home/spark/software/source/compile/deploy_spark/sbin/../logs/spark-spark-org.apache.spark.deploy.history.HistoryServer-1-hadoop000.out failed to launch org.apache.spark.deploy.history.HistoryServer: at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:44) ... 6 more
需要在启动时指定目录：
start-history-server.sh hdfs://hadoop000:8020/directory
hdfs://hadoop000:8020/directory可以配置在配|文件中Q那么在启动history-server时就不需要指定，后箋介绍怎么配置Q?/p>
注：该目录需要事先在hdfs上创建好Q否则history-server启动报错?/strong>
启动完成之后可以通过WEBUI讉KQ默认端口是18080Qhttp://hadoop000:18080
默认界面列表信息是空的，下面截图是我跑了几次spark-sql试后出现的?/p>
history server相关的配|参数描q?/strong>
1Q?spark.history.updateInterval
默认|10
以秒为单位，更新日志相关信息的时间间?/p>
2Qspark.history.retainedApplications
默认|50
在内存中保存Application历史记录的个敎ͼ如果过q个|旧的应用E序信息被删除Q当再次讉K已被删除的应用信息时需要重新构建页面?/p>
3Q?span style="color: #ff0000;">spark.history.ui.port
默认|18080
HistoryServer的web端口
4Qspark.history.kerberos.enabled
默认|false
是否使用kerberos方式d讉KHistoryServerQ对于持久层位于安全集群的HDFS上是有用的，如果讄为trueQ就要配|下面的两个属?/p>
5Qspark.history.kerberos.principal
默认|用于HistoryServer的kerberosM名称
6Qspark.history.kerberos.keytab
用于HistoryServer的kerberos keytab文g位置
7Qspark.history.ui.acls.enable
默认|false
授权用户查看应用E序信息的时候是否检查acl。如果启用，只有应用E序所有者和spark.ui.view.acls指定的用户可以查看应用程序信?否则Q不做Q何检?/p>
8Q?span style="color: #ff0000;">spark.eventLog.enabled
默认|false
是否记录Spark事gQ用于应用程序在完成后重构webUI
9Q?span style="color: #ff0000;">spark.eventLog.dir
默认|file:///tmp/spark-events
保存日志相关信息的\径，可以是hdfs://开头的HDFS路径Q也可以是file://开头的本地路径Q都需要提前创?/p>
10Q?span style="color: #ff0000;">spark.eventLog.compress
默认|false
是否压羃记录Spark事gQ前提spark.eventLog.enabled为trueQ默认用的是snappy
以spark.history开头的需要配|在spark-env.sh中的SPARK_HISTORY_OPTSQ以spark.eventLog开头的配置在spark-defaults.conf
我在试q程中的配置如下Q?/p>
spark-defaults.conf
spark.eventLog.enabled true spark.eventLog.dir hdfs://hadoop000:8020/directory spark.eventLog.compress true
spark-env.sh
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=7777 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs://had oop000:8020/directory"
参数描述Q?/p>
spark.history.ui.port=7777 调整WEBUI讉K的端口号?777
spark.history.fs.logDirectory=hdfs://hadoop000:8020/directory 配置了该属性后Q在start-history-server.sh时就无需再显C的指定路径
spark.history.retainedApplications=3 指定保存Application历史记录的个敎ͼ如果过q个|旧的应用E序信息被删除
调整参数后启动start-history-server.sh
start-history-server.sh 讉KWEBUIQ?http://hadoop000:7777
在用spark history server的过E中产生的几个疑问：
疑问1Qspark.history.fs.logDirectory和spark.eventLog.dir指定目录有啥区别Q?/strong>
l测试后发现Q?/p>
spark.eventLog.dirQApplication在运行过E中所有的信息均记录在该属性指定的路径下；
spark.history.fs.logDirectoryQSpark History Server面只展C指定路径下的信息Q?/p>
比如Qspark.eventLog.dir刚开始时指定的是hdfs://hadoop000:8020/directoryQ而后修改成hdfs://hadoop000:8020/directory2
那么spark.history.fs.logDirectory如果指定的是hdfs://hadoop000:8020/directoryQ就只能昄目录下的所有Applicationq行的日志信息；反之亦然?/p>
疑问2Qspark.history.retainedApplications=3 貌似没生效？Q？Q？Q?/strong>
The History Server will list all applications. It will just retain a max number of them in memory. That option does not control how many applications are show, it controls how much memory the HS will need.
注意Q该参数q不是也面中显C的application的记录数Q而是存放在内存中的个敎ͼ内存中的信息在访问页面时直接d渲染既可Q?/p>
比如说该参数配置?0个，那么内存中就最多只能存?0个applicaiton的日志信息，当第11个加入时Q第一个就会被t除Q当再次讉KW?个application的页面信息时?span style="font-size: 14px; line-height: 1.5;">需要重新读取指定\径上的日志信息来渲染展示面?nbsp;
详见官方文Qhttp://spark.apache.org/docs/latest/monitoring.html
]]>
今天在测试spark-sqlq行在yarn上的q程中，无意间从日志中发C一个问题：
spark-sql --master yarn
14/12/29 15:23:17 INFO Client: Requesting a new application from cluster with 1 NodeManagers 14/12/29 15:23:17 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 14/12/29 15:23:17 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 14/12/29 15:23:17 INFO Client: Setting up container launch context for our AM 14/12/29 15:23:17 INFO Client: Preparing resources for our AM container 14/12/29 15:23:17 INFO Client: Uploading resource file:/home/spark/software/source/compile/deploy_spark/assembly/target/scala-2.10/spark-assembly-1.3.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar -> hdfs://hadoop000:8020/user/spark/.sparkStaging/application_1416381870014_0093/spark-assembly-1.3.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar 14/12/29 15:23:18 INFO Client: Setting up the launch environment for our AM container
再开启一个spark-sql命o行，从日志中再次发现Q?/p>
14/12/29 15:24:03 INFO Client: Requesting a new application from cluster with 1 NodeManagers 14/12/29 15:24:03 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 14/12/29 15:24:03 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 14/12/29 15:24:03 INFO Client: Setting up container launch context for our AM 14/12/29 15:24:03 INFO Client: Preparing resources for our AM container 14/12/29 15:24:03 INFO Client: Uploading resource file:/home/spark/software/source/compile/deploy_spark/assembly/target/scala-2.10/spark-assembly-1.3.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar -> hdfs://hadoop000:8020/user/spark/.sparkStaging/application_1416381870014_0094/spark-assembly-1.3.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar 14/12/29 15:24:05 INFO Client: Setting up the launch environment for our AM container
然后查看HDFS上的文gQ?/p>
hadoop fs -ls hdfs://hadoop000:8020/user/spark/.sparkStaging/
drwx------ - spark supergroup 0 2014-12-29 15:23 hdfs://hadoop000:8020/user/spark/.sparkStaging/application_1416381870014_0093 drwx------ - spark supergroup 0 2014-12-29 15:24 hdfs://hadoop000:8020/user/spark/.sparkStaging/application_1416381870014_0094
每个Application都会上传一个spark-assembly-x.x.x-SNAPSHOT-hadoopx.x.x-cdhx.x.x.jar的jar包，影响HDFS的性能以及占用HDFS的空间?/p>
在Spark文(http://spark.apache.org/docs/latest/running-on-yarn.html)中发?span style="color: #ff0000;">spark.yarn.jar属性，spark-assembly-xxxxx.jar存放在hdfs://hadoop000:8020/spark_lib/?/p>
在spark-defaults.confd属性配|：
spark.yarn.jar hdfs://hadoop000:8020/spark_lib/spark-assembly-1.3.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar 再次启动spark-sql --master yarn观察日志Q?/p>
14/12/29 15:39:02 INFO Client: Requesting a new application from cluster with 1 NodeManagers 14/12/29 15:39:02 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 14/12/29 15:39:02 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 14/12/29 15:39:02 INFO Client: Setting up container launch context for our AM 14/12/29 15:39:02 INFO Client: Preparing resources for our AM container 14/12/29 15:39:02 INFO Client: Source and destination file systems are the same. Not copying hdfs://hadoop000:8020/spark_lib/spark-assembly-1.3.0-SNAPSHOT-hadoop2.3.0-cdh5.0.0.jar 14/12/29 15:39:02 INFO Client: Setting up the launch environment for our AM container 观察HDFS上文?/p>
hadoop fs -ls hdfs://hadoop000:8020/user/spark/.sparkStaging/application_1416381870014_0097
该Application对应的目录下没有spark-assembly-xxxxx.jar了，从而节省assembly包上传的q程以及HDFSI间占用?/p>
我在试q程中遇CcM如下的错误：
Application application_xxxxxxxxx_yyyy failed 2 times due to AM Container for application_xxxxxxxxx_yyyy
exited with exitCode: -1000 due to: java.io.FileNotFoundException: File /tmp/hadoop-spark/nm-local-dir/filecache does not exist
?tmp/hadoop-spark/nm-local-dir路径下创建filecache文g夹即可解x错问题?/p>
]]>
wrote a blog post about how LinkedIn uses Apache Kafka as a central publish-subscribe log for integrating data between applications, stream processing, and Hadoop data ingestion.
To actually make this work, though, this "universal log" has to be a cheap abstraction. If you want to use a system as a central data hub it has to be fast, predictable, and easy to scale so you can dump all your data onto it. My experience has been that systems that are fragile or expensive inevitably develop a wall of protective process to prevent people from using them; a system that scales easily often ends up as a key architectural building block just because using it is the easiest way to get things built.
I've always liked the benchmarks of Cassandra that show it doing a million writes per second on three hundred machines onEC2 and Google Compute Engine. I'm not sure why, maybe it is a Dr. Evil thing, but doing a million of anything per second is fun.
In any case, one of the nice things about a Kafka log is that, as we'll see, it is cheap. A million writes per second isn't a particularly big thing. This is because a log is a much simpler thing than a database or key-value store. Indeed our production clusters take tens of millions of reads and writes per second all day long and they do so on pretty modest hardware.
But let's do some benchmarking and take a look.
Kafka in 30 seconds
To help understand the benchmark, let me give a quick review of what Kafka is and a few details about how it works. Kafka is a distributed messaging system originally built at LinkedIn and now part of the Apache Software Foundation and used by a variety of companies.
The general setup is quite simple. Producers send records to the cluster which holds on to these records and hands them out to consumers:
The key abstraction in Kafka is the topic. Producers publish their records to a topic, and consumers subscribe to one or more topics. A Kafka topic is just a sharded write-ahead log. Producers append records to these logs and consumers subscribe to changes. Each record is a key/value pair. The key is used for assigning the record to a log partition (unless the publisher specifies the partition directly).
Here is a simple example of a single producer and consumer reading and writing from a two-partition topic.
This picture shows a producer process appending to the logs for the two partitions, and a consumer reading from the same logs. Each record in the log has an associated entry number that we call the offset. This offset is used by the consumer to describe it's position in each of the logs.
These partitions are spread across a cluster of machines, allowing a topic to hold more data than can fit on any one machine.
Note that unlike most messaging systems the log is always persistent. Messages are immediately written to the filesystem when they are received. Messages are not deleted when they are read but retained with some configurable SLA (say a few days or a week). This allows usage in situations where the consumer of data may need to reload data. It also makes it possible to support space-efficient publish-subscribe as there is a single shared log no matter how many consumers; in traditional messaging systems there is usually a queue per consumer, so adding a consumer doubles your data size. This makes Kafka a good fit for things outside the bounds of normal messaging systems such as acting as a pipeline for offline data systems such as Hadoop. These offline systems may load only at intervals as part of a periodic ETL cycle, or may go down for several hours for maintenance, during which time Kafka is able to buffer even TBs of unconsumed data if needed.
Kafka also replicates its logs over multiple servers for fault-tolerance. One important architectural aspect of our replication implementation, in contrast to other messaging systems, is that replication is not an exotic bolt-on that requires complex configuration, only to be used in very specialized cases. Instead replication is assumed to be the default: we treat un-replicated data as a special case where the replication factor happens to be one.
Producers get an acknowledgement back when they publish a message containing the record's offset. The first record published to a partition is given the offset 0, the second record 1, and so on in an ever-increasing sequence. Consumers consume data from a position specified by an offset, and they save their position in a log by committing periodically: saving this offset in case that consumer instance crashes and another instance needs to resume from it's position.
Okay, hopefully that all made sense (if not, you can read a more complete introduction to Kafka here).
This Benchmark
This test is against trunk, as I made some improvements to the performance tests for this benchmark. But nothing too substantial has changed since the last full release, so you should see similar results with 0.8.1. I am also using our newly re-written Java producer, which offers much improved throughput over the previous producer client.
I've followed the basic template of this very nice RabbitMQ benchmark, but I covered scenarios and options that were more relevant to Kafka.
One quick philosophical note on this benchmark. For benchmarks that are going to be publicly reported, I like to follow a style I call "lazy benchmarking". When you work on a system, you generally have the know-how to tune it to perfection for any particular use case. This leads to a kind of benchmarketing where you heavily tune your configuration to your benchmark or worse have a different tuning for each scenario you test. I think the real test of a system is not how it performs when perfectly tuned, but rather how it performs "off the shelf". This is particularly true for systems that run in a multi-tenant setup with dozens or hundreds of use cases where tuning for each use case would be not only impractical but impossible. As a result, I have pretty much stuck with default settings, both for the server and the clients. I will point out areas where I suspect the result could be improved with a little tuning, but I have tried to resist the temptation to do any fiddling myself to improve the results.
I have posted my exact configurations and commands, so it should be possible to replicate results on your own gear if you are interested.
The Setup
For these tests, I had six machines each has the following specs
- Intel Xeon 2.5 GHz processor with six cores
- Six 7200 RPM SATA drives
- 32GB of RAM
- 1Gb Ethernet
The Kafka cluster is set up on three of the machines. The six drives are directly mounted with no RAID (JBOD style). The remaining three machines I use for Zookeeper and for generating load.
A three machine cluster isn't very big, but since we will only be testing up to a replication factor of three, it is all we need. As should be obvious, we can always add more partitions and spread data onto more machines to scale our cluster horizontally.
This hardware is actually not LinkedIn's normal Kafka hardware. Our Kafka machines are more closely tuned to running Kafka, but are less in the spirit of "off-the-shelf" I was aiming for with these tests. Instead, I borrowed these from one of our Hadoop clusters, which runs on probably the cheapest gear of any of our persistent systems. Hadoop usage patterns are pretty similar to Kafka's, so this is a reasonable thing to do.
Okay, without further ado, the results!
Producer Throughput
These tests will stress the throughput of the producer. No consumers are run during these tests, so all messages are persisted but not read (we'll test cases with both producer and consumer in a bit). Since we have recently rewritten our producer, I am testing this new code.
Single producer thread, no replication
| 821,557 records/sec |
| (78.3 MB/sec) |
For this first test I create a topic with six partitions and no replication. Then I produce 50 million small (100 byte) records as quickly as possible from a single thread.
The reason for focusing on small records in these tests is that it is the harder case for a messaging system (generally). It is easy to get good throughput in MB/sec if the messages are large, but much harder to get good throughput when the messages are small, as the overhead of processing each message dominates.
Throughout this benchmark, when I am reporting MB/sec, I am reporting just the value size of the record times the request per second, none of the other overhead of the request is included. So the actually network usage is higher than what is reported. For example with a 100 byte message we would also transmit about 22 bytes of overhead per message (for an optional key, size delimiting, a message CRC, the record offset, and attributes flag), as well as some overhead for the request (including the topic, partition, required acknowledgements, etc). This makes it a little harder to see where we hit the limits of the NIC, but this seems a little more reasonable then including our own overhead bytes in throughput numbers. So, in the above result, we are likely saturating the 1 gigabit NIC on the client machine.
One immediate observation is that the raw numbers here are much higher than people expect, especially for a persistent storage system. If you are used to random-access data systems, like a database or key-value store, you will generally expect maximum throughput around 5,000 to 50,000 queries-per-second, as this is close to the speed that a good RPC layer can do remote requests. We exceed this due to two key design principles:
- We work hard to ensure we do linear disk I/O. The six cheap disks these servers have gives an aggregate throughput of 822 MB/sec of linear disk I/O. This is actually well beyond what we can make use of with only a 1 gigabit network card. Many messaging systems treat persistence as an expensive add-on that decimates performance and should be used only sparingly, but this is because they are not able to do linear I/O.
- At each stage we work on batching together small bits of data into larger network and disk I/O operations. For example, in the new producer we use a "group commit"-like mechanism to ensure that any record sends initiated while another I/O is in progress get grouped together. For more on understanding the importance of batching, check out this presentation by David Patterson on why "Latency Lags Bandwidth".
If you are interested in the details you can read a little more about this in our design documents.
Single producer thread, 3x asynchronous replication
| 786,980 records/sec |
| (75.1 MB/sec) |
This test is exactly the same as the previous one except that now each partition has three replicas (so the total data written to network or disk is three times higher). Each server is doing both writes from the producer for the partitions for which it is a master, as well as fetching and writing data for the partitions for which it is a follower.
Replication in this test is asynchronous. That is, the server acknowledges the write as soon as it has written it to its local log without waiting for the other replicas to also acknowledge it. This means, if the master were to crash, it would likely lose the last few messages that had been written but not yet replicated. This makes the message acknowledgement latency a little better at the cost of some risk in the case of server failure.
The key take away I would like people to have from this is that replication can be fast. The total cluster write capacity is, of course, 3x less with 3x replication (since each write is done three times), but the throughput is still quite good per client. High performance replication comes in large part from the efficiency of our consumer (the replicas are really nothing more than a specialized consumer) which I will discuss in the consumer section.
Single producer thread, 3x synchronous replication
| 421,823 records/sec |
| (40.2 MB/sec) |
This test is the same as above except that now the master for a partition waits for acknowledgement from the full set of in-sync replicas before acknowledging back to the producer. In this mode, we guarantee that messages will not be lost as long as one in-sync replica remains.
Synchronous replication in Kafka is not fundamentally very different from asynchronous replication. The leader for a partition always tracks the progress of the follower replicas to monitor their liveness, and we never give out messages to consumers until they are fully acknowledged by replicas. With synchronous replication we just wait to respond to the producer request until the followers have replicated it.
This additional latency does seem to affect our throughput. Since the code path on the server is very similar, we could probably ameliorate this impact by tuning the batching to be a bit more aggressive and allowing the client to buffer more outstanding requests. However, in spirit of avoiding special case tuning, I have avoided this.Three producers, 3x async replication
| 2,024,032 records/sec |
| (193.0 MB/sec) |
Our single producer process is clearly not stressing our three node cluster. To add a little more load, I'll now repeat the previous async replication test, but now use three producer load generators running on three different machines (running more processes on the same machine won't help as we are saturating the NIC). Then we can look at the aggregate throughput across these three producers to get a better feel for the cluster's aggregate capacity.
Producer Throughput Versus Stored Data
One of the hidden dangers of many messaging systems is that they work well only as long as the data they retain fits in memory. Their throughput falls by an order of magnitude (or more) when data backs up and isn't consumed (and hence needs to be stored on disk). This means things may be running fine as long as your consumers keep up and the queue is empty, but as soon as they lag, the whole messaging layer backs up with unconsumed data. The backup causes data to go to disk which in turns causes performance to drop to a rate that means messaging system can no longer keep up with incoming data and either backs up or falls over. This is pretty terrible, as in many cases the whole purpose of the queue was to handle such a case gracefully.
Since Kafka always persists messages the performance is O(1) with respect to unconsumed data volume.
To test this experimentally, let's run our throughput test over an extended period of time and graph the results as the stored dataset grows:
This graph actually does show some variance in performance, but no impact due to data size: we perform just as well after writing a TB of data, as we do for the first few hundred MBs.
The variance seems to be due to Linux's I/O management facilities that batch data and then flush it periodically. This is something we have tuned for a little better on our production Kafka setup. Some notes on tuning I/O are available here.
Consumer Throughput
Okay now let's turn our attention to consumer throughput.
Note that the replication factor will not effect the outcome of this test as the consumer only reads from one replica regardless of the replication factor. Likewise, the acknowledgement level of the producer also doesn't matter as the consumer only ever reads fully acknowledged messages, (even if the producer doesn't wait for full acknowledgement). This is to ensure that any message the consumer sees will always be present after a leadership handoff (if the current leader fails).
Single Consumer
| 940,521 records/sec |
| (89.7 MB/sec) |
For the first test, we will consume 50 million messages in a single thread from our 6 partition 3x replicated topic.
Kafka's consumer is very efficient. It works by fetching chunks of log directly from the filesystem. It uses the sendfile API to transfer this directly through the operating system without the overhead of copying this data through the application. This test actually starts at the beginning of the log, so it is doing real read I/O. In a production setting, though, the consumer reads almost exclusively out of the OS pagecache, since it is reading data that was just written by some producer (so it is still cached). In fact, if you run I/O stat on a production server you actually see that there are no physical reads at all even though a great deal of data is being consumed.
Making consumers cheap is important for what we want Kafka to do. For one thing, the replicas are themselves consumers, so making the consumer cheap makes replication cheap. In addition, this makes handling out data an inexpensive operation, and hence not something we need to tightly control for scalability reasons.
Three Consumers
| 2,615,968 records/sec |
| (249.5 MB/sec) |
Let's repeat the same test, but run three parallel consumer processes, each on a different machine, and all consuming the same topic.
As expected, we see near linear scaling (not surprising because consumption in our model is so simple).
Producer and Consumer
| 795,064 records/sec |
| (75.8 MB/sec) |
The above tests covered just the producer and the consumer running in isolation. Now let's do the natural thing and run them together. Actually, we have technically already been doing this, since our replication works by having the servers themselves act as consumers.
All the same, let's run the test. For this test we'll run one producer and one consumer on a six partition 3x replicated topic that begins empty. The producer is again using async replication. The throughput reported is the consumer throughput (which is, obviously, an upper bound on the producer throughput).
As we would expect, the results we get are basically the same as we saw in the producer only case—the consumer is fairly cheap.
Effect of Message Size
I have mostly shown performance on small 100 byte messages. Smaller messages are the harder problem for a messaging system as they magnify the overhead of the bookkeeping the system does. We can show this by just graphing throughput in both records/second and MB/second as we vary the record size.
We can see that with the 10 byte messages we are actually CPU bound by just acquiring the lock and enqueuing the message for sending—we are not able to actually max out the network. However, starting with 100 bytes, we are actually seeing network saturation (though the MB/sec continues to increase as our fixed-size bookkeeping bytes become an increasingly small percentage of the total bytes sent).
End-to-end Latency
| 2 ms (median) |
| 3 ms (99th percentile) |
| 14 ms (99.9th percentile) |
We have talked a lot about throughput, but what is the latency of message delivery? That is, how long does it take a message we send to be delivered to the consumer? For this test, we will create producer and consumer and repeatedly time how long it takes for a producer to send a message to the kafka cluster and then be received by our consumer.
Note that, Kafka only gives out messages to consumers when they are acknowledged by the full in-sync set of replicas. So this test will give the same results regardless of whether we use sync or async replication, as that setting only affects the acknowledgement to the producer.
Replicating this test
If you want to try out these benchmarks on your own machines, you can. As I said, I mostly just used our pre-packaged performance testing tools that ship with Kafka and mostly stuck with the default configs both for the server and for the clients. However, you can see more details of the configuration and commands here.
]]>
本文针对Kafka性能斚wq行单分析，首先单介l一下Kafka的架构和涉及到的名词Q?/p>
1. TopicQ用于划分Message的逻辑概念Q一个Topic可以分布在多个Broker上?/p>
2. PartitionQ是Kafka中横向扩展和一切ƈ行化的基Q每个Topic都至被切分?个Partition?/p>
3. OffsetQ消息在Partition中的~号Q编号顺序不跨Partition?/p>
4. ConsumerQ用于从Broker中取?消费Message?/p>
5. ProducerQ用于往Broker中发?生Message?/p>
6. ReplicationQKafka支持以Partition为单位对Messageq行冗余备䆾Q每个Partition都可以配|至?个Replication(当仅1个Replication时即仅该Partition本n)?/p>
7. LeaderQ每个Replication集合中的Partition都会选出一个唯一的LeaderQ所有的dh都由Leader处理。其他Replicas从Leader处把数据更新同步到本圎ͼq程cM大家熟悉的MySQL中的Binlog同步?/p>
8. BrokerQKafka中用Broker来接受Producer和Consumer的请求，q把Message持久化到本地盘。每个Cluster当中会选DZ个Broker来担任ControllerQ负责处理Partition的Leader选DQ协调Partitionq移{工作?/p>
9. ISR(In-Sync Replica)Q是Replicas的一个子集，表示目前Alive且与Leader能够“Catch-up”的Replicas集合。由于读写都是首先落到Leader上，所以一般来说通过同步机制从Leader上拉取数据的Replica都会和Leader有一些gq?包括了gq时间和延迟条数两个l度)QQ意一个超q阈值都会把该Replicat出ISR。每个Partition都有它自q立的ISR?/p>
以上几乎是我们在使用Kafka的过E中可能遇到的所有名词，同时也无一不是最核心的概忉|lgQ感觉到从设计本w来_Kafkaq是_z的。这ơ本文围lKafka优异的吞吐性能Q逐个介绍一下其设计与实现当中所使用的各?#8220;黑科技”?/p>
Broker
不同于Redis和MemcacheQ{内存消息队列，Kafka的设计是把所有的Message都要写入速度低容量大的硬盘，以此来换取更强的存储能力。实际上QKafka使用盘q没有带来过多的性能损失Q?#8220;规规矩矩”的抄了一?#8220;q道”?/p>
首先Q说“规规矩矩”是因为Kafka在磁盘上只做Sequence I/OQ由于消息系l读写的Ҏ性，qƈ不存在什么问题。关于磁盘I/O的性能Q引用一lKafka官方l出的测试数?Raid-5Q?200rpm)Q?/p>
Sequence I/O: 600MB/s
Random I/O: 100KB/s
所以通过只做Sequence I/O的限Ӟ规避了磁盘访问速度低下Ҏ能可能造成的媄响?/p>
接下来我们再聊一聊Kafka是如?#8220;抄近道的”?/p>
首先QKafka重度依赖底层操作pȝ提供的PageCache功能。当上层有写操作Ӟ操作pȝ只是数据写入PageCacheQ同时标记Page属性ؓDirty。当L作发生时Q先从PageCache中查找，如果发生~页才进行磁盘调度，最l返回需要的数据。实际上PageCache是把可能多的空闲内存都当做了磁盘缓存来使用。同时如果有其他q程甌内存Q回收PageCache的代价又很小Q所以现代的OS都支持PageCache?/p>
使用PageCache功能同时可以避免在JVM内部~存数据QJVM为我们提供了强大的GC能力Q同时也引入了一些问题不适用与Kafka的设计?/p>
· 如果在Heap内管理缓存，JVM的GCU程会频J扫描HeapI间Q带来不必要的开销。如果Heapq大Q执行一ơFull GC对系l的可用性来说将是极大的挑战?/p>
· 所有在在JVM内的对象都不免带有一个Object Overhead(千万不可视)Q内存的有效I间利用率会因此降低?/p>
· 所有的In-Process Cache在OS中都有一份同LPageCache。所以通过缓存只攑֜PageCacheQ可以至让可用~存I间d?/p>
· 如果Kafka重启Q所有的In-Process Cache都会失效Q而OS理的PageCache依然可以l箋使用?/p>
PageCacheq只是第一步，KafkaZq一步的优化性能q采用了Sendfile技术。在解释Sendfile之前Q首先介l一下传l的|络I/O操作程Q大体上分ؓ以下4步?/p>
1. OS 从硬盘把数据d内核区的PageCache?/p>
2. 用户q程把数据从内核区Copy到用户区?/p>
3. 然后用户q程再把数据写入到SocketQ数据流入内核区的Socket Buffer上?/p>
4. OS 再把数据从Buffer中Copy到网卡的Buffer上，q样完成一ơ发送?/p>
整个q程q历两ơContext SwitchQ四ơSystem Call。同一份数据在内核Buffer与用户Buffer之间重复拯Q效率低下。其??两步没有必要Q完全可以直接在内核区完成数据拷贝。这也正是Sendfile所解决的问题，l过Sendfile优化后，整个I/Oq程变成了下面q个样子?/p>
通过以上的介l不隄出，Kafka的设计初h一切努力在内存中完成数据交换，无论是对外作Z整个消息pȝQ或是内部同底层操作pȝ的交互。如果Producer和Consumer之间生和消费进度上配合得当Q完全可以实现数据交换零I/O。这也就是我Z么说Kafka使用“盘”q没有带来过多性能损失的原因。下面是我在生环境中采到的一些指标?/p>
(20 Brokers, 75 Partitions per Broker, 110k msg/s)
此时的集只有写Q没有读操作?0M/s左右的Send的流量是Partition之间q行Replicate而生的。从recv和writ的速率比较可以看出Q写盘是使用Asynchronous+Batch的方式，底层OS可能q会q行盘写顺序优化。而在有Read Requestq来的时候分ZU情况，W一U是内存中完成数据交换?/p>
Send量从^?0M/s增加C到^?0M/sQ而磁盘Read只有不超q?0KB/s。PageCache降低盘I/O效果非常明显?/p>
接下来是M些收C一D|_已经从内存中被换出刷写到盘上的老数据?/p>
其他指标q是老样子，而磁盘Read已经飚高?0+MB/s。此时全部的数据都已l是走硬盘了(对硬盘的序dOS层会q行Prefill PageCache的优?。依然没有Q何性能问题?/p>
Tips
1. Kafka官方q不通过Broker端的log.flush.interval.messages和log.flush.interval.ms来强制写盘，认ؓ数据的可靠性应该通过Replica来保证，而强制Flush数据到磁盘会Ҏ体性能产生影响?/p>
2. 可以通过调整/proc/sys/vm/dirty_background_ratio?proc/sys/vm/dirty_ratio来调优性能?/p>
a. 脏页率超q第一个指标会启动pdflush开始Flush Dirty PageCache?/p>
b. 脏页率超q第二个指标会阻塞所有的写操作来q行Flush?/p>
c. Ҏ不同的业务需求可以适当的降低dirty_background_ratio和提高dirty_ratio?/p>
Partition
Partition是Kafka可以很好的横向扩展和提供高ƈ发处理以及实现Replication的基?/p>
扩展性方面。首先，Kafka允许Partition在集内的Broker之间LUdQ以此来均衡可能存在的数据倾斜问题。其ơ，Partition支持自定义的分区法Q例如可以将同一个Key的所有消息都路由到同一个Partition上去?同时Leader也可以在In-Sync的Replica中迁UR由于针Ҏ一个Partition的所有读写请求都是只由Leader来处理，所以Kafka会尽量把Leader均匀的分散到集群的各个节点上Q以免造成|络量q于集中?/p>
q发斚w。Q意Partition在某一个时d能被一个Consumer Group内的一个Consumer消费(反过来一个Consumer则可以同时消费多个Partition)QKafka非常z的Offset机制最化了Broker和Consumer之间的交互，qKafkaq不会像同类其他消息队列一P随着下游Consumer数目的增加而成比例的降低性能。此外，如果多个Consumer恰y都是消费旉序上很相q的数据Q可以达到很高的PageCache命中率，因而Kafka可以非常高效的支持高q发L作，实践中基本可以达到单机网卡上限?/p>
不过QPartition的数量ƈ不是多好QPartition的数量越多，q_到每一个Broker上的数量也就多。考虑到Broker宕机(Network Failure, Full GC)的情况下Q需要由Controller来ؓ所有宕机的Broker上的所有Partition重新选DLeaderQ假设每个Partition的选D消?0msQ如果Broker上有500个PartitionQ那么在q行选D?s的时间里Q对上述Partition的读写操作都会触发LeaderNotAvailableException?/p>
再进一步，如果挂掉的Broker是整个集的ControllerQ那么首先要q行的是重新d一个Broker作ؓController。新d的Controller要从Zookeeper上获取所有Partition的Meta信息Q获取每个信息大?-5msQ那么如果有10000个Partitionq个旉׃辑ֈ30s-50s。而且不要忘记q只是重新启动一个Controllerp的时_在这基础上还要再加上前面说的选DLeader的时?-_-!!!!!!
此外Q在Broker端，对Producer和Consumer都用了Buffer机制。其中Buffer的大是l一配置的，数量则与Partition个数相同。如果Partition个数q多Q会DProducer和Consumer的Buffer内存占用q大?/p>
Tips
1. Partition的数量尽量提前预分配Q虽然可以在后期动态增加PartitionQ但是会冒着可能破坏Message Key和Partition之间对应关系的风险?/p>
2. Replica的数量不要过多，如果条g允许量把Replica集合内的Partition分别调整C同的Rack?/p>
3. 一切努力保证每ơ停Broker旉可以Clean ShutdownQ否则问题就不仅仅是恢复服务所需旉长，q可能出现数据损坏或其他很诡异的问题?/p>
Producer
Kafka的研发团队表C在0.8版本里用Java重写了整个ProducerQ据说性能有了很大提升。我q没有亲自对比试用过Q这里就不做数据Ҏ了。本文结扩展阅读里提C一套我认ؓ比较好的对照l，有兴的同学可以试一下?/p>
其实在Producer端的优化大部分消息系l采取的方式都比较单一Q无非也化零ؓ整、同步变异步q么几种?/p>
Kafkapȝ默认支持MessageSetQ把多条Message自动地打成一个Group后发送出去，均摊后拉低了每次通信的RTT。而且在组lMessageSet的同Ӟq可以把数据重新排序Q从爆发式的随机写入优化成较ؓq稳的线性写入?/p>
此外Q还要着重介l的一ҎQProducer支持End-to-End的压~。数据在本地压羃后放到网l上传输Q在Broker一般不解压(除非指定要Deep-Iteration)Q直x息被Consume之后在客L解压?/p>
当然用户也可以选择自己在应用层上做压羃和解压的工作(毕竟Kafka目前支持的压~算法有限，只有GZIP和Snappy)Q不q这样做反而会意外的降低效率！Q！Q?Kafka的End-to-End压羃与MessageSet配合在一起工作效果最佻I上面的做法直接割裂了两者间联系。至于道理其实很单，压羃法中一条基本的原理“重复的数据量多Q压~比高”。无关于消息体的内容Q无关于消息体的数量Q大多数情况下输入数据量大一些会取得更好的压~比?/p>
不过Kafka采用MessageSet也导致在可用性上一定程度的妥协。每ơ发送数据时QProducer都是send()之后p为已l发送出MQ但其实大多数情况下消息q在内存的MessageSet当中Q尚未发送到|络Q这时候如果Producer挂掉Q那׃出现丢数据的情况?/p>
Z解决q个问题QKafka?.8版本的设计借鉴了网l当中的ack机制。如果对性能要求较高Q又能在一定程度上允许Message的丢失，那就可以讄request.required.acks=0 来关闭ackQ以全速发送。如果需要对发送的消息q行认Q就需要设|request.required.acks??1Q那??1又有什么区别呢Q这里又要提到前面聊的有关Replica数量问题。如果配|ؓ1Q表C消息只需要被Leader接收q确认即可，其他的Replica可以q行异步拉取无需立即q行认Q在保证可靠性的同时又不会把效率拉得很低。如果设|ؓ-1Q表C消息要Commit到该Partition的ISR集合中的所有Replica后，才可以返回ackQ消息的发送会更安全，而整个过E的延迟会随着Replica的数量正比增长，q里需要根据不同的需求做相应的优化?/p>
Tips
1. Producer的线E不要配|过多，其是在Mirror或者Migration中用的时候，会加剧目标集Partition消息乱序的情?如果你的应用场景Ҏ息顺序很敏感的话)?/p>
2. 0.8版本的request.required.acks默认?(?.7)?/p>
Consumer
Consumer端的设计大体上还是比较常规的?/p>
· 通过Consumer GroupQ可以支持生产者消费者和队列讉K两种模式?/p>
· Consumer API分ؓHigh level和Low level两种。前一U重度依赖ZookeeperQ所以性能差一些且不自由，但是省心。第二种不依赖Zookeeper服务Q无Z自由度和性能上都有更好的表现Q但是所有的异常(Leaderq移、Offset界、Broker宕机{?和Offset的维护都需要自行处理?/p>
· 大家可以x下不日发布的0.9 Release。开发h员又用Java重写了一套Consumer。把两套API合ƈ在一P同时L了对Zookeeper的依赖。据说性能有大q度提升哦~~
Tips
强烈推荐使用Low level APIQ虽然繁琐一些，但是目前只有q个API可以对Error数据q行自定义处理，其是处理Broker异常或由于Unclean ShutdownD的Corrupted DataӞ否则无法Skip只能{着“坏消?#8221;在Broker上被Rotate掉，在此期间该Replica会一直处于不可用状态?/p>
扩展阅读
Sendfile: https://www.ibm.com/developerworks/cn/java/j-zerocopy/
So what’s wrong with 1975 programming: https://www.varnish-cache.org/trac/wiki/ArchitectNotes
Benchmarking: https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
]]>
Redis有一pd的命令，特点是以NXl尾QNX是Not eXists的羃写，如SETNX命o应该理解ؓQSET if Not eXists。这pd的命令非常有用，q里讲用SETNX来实现分布式锁?/p>
用SETNX实现分布式锁
利用SETNX非常单地实现分布式锁。例如：某客L要获得一个名字foo的锁Q客L使用下面的命令进行获取：
SETNX lock.foo <current Unix time + lock timeout + 1>
- 如返?Q则该客L获得锁，把lock.foo的键D|ؓ旉DC键已被锁定，该客L最后可以通过DEL lock.foo来释放该锁?/li>
- 如返?Q表明该锁已被其他客L取得Q这时我们可以先q回或进行重试等Ҏ完成或等待锁时?/li>
解决死锁
上面的锁定逻辑有一个问题：如果一个持有锁的客Lp|或崩溃了不能释放锁，该怎么解决Q我们可以通过锁的键对应的旉x判断q种情况是否发生了，如果当前的时间已l大于lock.foo的|说明该锁已失效，可以被重C用?/p>
发生q种情况Ӟ可不能简单的通过DEL来删除锁Q然后再SETNX一ơ，当多个客L到锁超时后都会试去释攑֮Q这里就可能出现一个竞态条?让我们模拟一下这个场景：
- C0操作时了，但它q持有着锁，C1和C2dlock.foo查时间戳Q先后发现超时了?/li>
- C1 发送DEL lock.foo
- C1 发送SETNX lock.foo q且成功了?/li>
- C2 发送DEL lock.foo
- C2 发送SETNX lock.foo q且成功了?/li>
q样一来，C1QC2都拿C锁！问题大了Q?/p>
q好q种问题是可以避免DQ让我们来看看C3q个客户端是怎样做的Q?/p>
- C3发送SETNX lock.foo 惌获得锁，׃C0q持有锁Q所以Redisq回lC3一?
- C3发送GET lock.foo 以检查锁是否时了，如果没超Ӟ则等待或重试?/li>
- 反之Q如果已时QC3通过下面的操作来试获得锁：
GETSET lock.foo <current Unix time + lock timeout + 1> - 通过GETSETQC3拿到的时间戳如果仍然是超时的Q那p明，C3如愿以偿拿到锁了?/li>
- 如果在C3之前Q有个叫C4的客L比C3快一步执行了上面的操作，那么C3拿到的时间戳是个未超时的|q时QC3没有如期获得锁，需要再ơ等待或重试。留意一下，管C3没拿到锁Q但它改写了C4讄的锁的超时|不过q一炚w常微的误差带来的媄响可以忽略不计?/li>
注意Q?/strong>Z让分布式锁的法更稳键些Q持有锁的客L在解锁之前应该再查一ơ自q锁是否已l超Ӟ再去做DEL操作Q因为可能客L因ؓ某个耗时的操作而挂P操作完的时候锁因ؓ时已经被别得，q时׃必解锁了?/p> Ҏ上面的代码，我写了一段Fake代码来描qC用分布式锁的全过E： 是的Q要惌D逻辑可以重用Q用python的你马上想CDecoratorQ而用Java的你是不是也惛_了那谁？AOP + annotationQ行Q怎样舒服怎样用吧Q别重复代码p?/p>CZ伪代?/h3>
背景
?很多互联|品应用中Q有些场景需要加锁处理，比如Q秒杀Q全局递增IDQ楼层生成等{。大部分的解x案是ZDB实现的，Redis为单q程单线E模 式，采用队列模式ƈ发访问变成串行访问，且多客户端对Redis的连接ƈ不存在竞争关pR其ơRedis提供一些命令SETNXQGETSETQ可以方 便实现分布式锁机制?/p>
Redis命o介绍 GETSET命o GET命o DEL命o 兵贵_，不在多。分布式锁，我们׃靠这四个命o。但在具体实玎ͼq有很多l节Q需要仔l斟酌，因ؓ在分布式q发多进E中QQ何一点出现差错，都会D死锁Qhold住所有进E?/p> 加锁实现 SETNX 可以直接加锁操作Q比如说Ҏ个关键词foo加锁Q客L可以试 如果q回1Q表C客L已经获取锁，可以往下操作，操作完成后，通过 命o来释N?br />如果q回0Q说明foo已经被其他客L上锁Q如果锁是非堵塞的，可以选择q回调用。如果是堵塞调用调用Q就需要进入以下个重试循环Q直x功获得锁或者重试超时。理x好的，现实是残L。仅仅用SETNX加锁带有竞争条g的，在某些特定的情况会造成死锁错误?/p> 处理死锁 ?上面的处理方式中Q如果获取锁的客L端执行时间过长，q程被kill掉，或者因为其他异常崩溃，D无法释放锁，׃造成死锁。所以，需要对加锁要做?效性检。因此，我们在加锁时Q把当前旉戳作为value存入此锁中，通过当前旉戛_Redis中的旉戌行对比，如果过一定差|认ؓ锁已l时 效，防止锁无限期的锁下去Q但是，在大q发情况Q如果同时检锁失效Qƈ单粗暴的删除死锁Q再通过SETNX上锁Q可能会D竞争条g的生，卛_个客 L同时获取锁?/p> C1获取锁，q崩溃。C2和C3调用SETNX上锁q回0后，获得foo.lock的时间戳Q通过比对旉戻I发现锁超时?br />C2 向foo.lock发送DEL命o?br />C2 向foo.lock发送SETNX获取锁?br />C3 向foo.lock发送DEL命oQ此时C3发送DELӞ其实DEL掉的是C2的锁?br />C3 向foo.lock发送SETNX获取锁?/p> 此时C2和C3都获取了锁，产生竞争条gQ如果在更高q发的情况，可能会有更多客户端获取锁。所以，DEL锁的操作Q不能直接用在锁超时的情况下，q好我们有GETSETҎQ假设我们现在有另外一个客LC4Q看看如何用GETSET方式Q避免这U情况生?/p> C1获取锁，q崩溃。C2和C3调用SETNX上锁q回0后，调用GET命o获得foo.lock的时间戳T1Q通过比对旉戻I发现锁超时?br />C4 向foo.lock发送GESET命oQ?br />GETSET foo.lock <current unix time> 如果T1=T2Q说明C4获得旉戟?br />如果T1!=T2Q说明C4之前有另外一个客LC5通过调用GETSET方式获取了时间戳QC4未获得锁。只能sleep下，q入下次循环中?/p> 现在唯一的问题是QC4讄foo.lock的新旉戻I是否会对锁生媄响。其实我们可以看到C4和C5执行的时间差值极，q且写入foo.lock中的都是有效旉错，所以对锁ƈ没有影响?br />?了让q个锁更加强壮，获取锁的客户端，应该在调用关键业务时Q再ơ调用GETҎ获取T1Q和写入的T0旉戌行对比，以免锁因其他情况被执行DEL?外解开而不知。以上步骤和情况Q很Ҏ从其他参考资料中看到。客L处理和失败的情况非常复杂Q不仅仅是崩溃这么简单，q可能是客户端因为某些操作被d 了相当长旉Q紧接着 DEL 命o被尝试执?但这旉却在另外的客L手上)。也可能因ؓ处理不当Q导致死锁。还有可能因为sleep讄不合理，DRedis在大q发下被压垮?最为常见的问题q有 GETq回nil时应该走那种逻辑Q?/span> W一U走时逻辑 W二U情况走循环走setnx逻辑 ?U逻辑貌似都是OKQ但是从逻辑处理上来_W一U情况存在问题。当GETq回nil表示Q锁是被删除的，而不是超Ӟ应该走SETNX逻辑加锁。走W一 U情늚问题是，正常的加锁逻辑应该走SETNXQ而现在当锁被解除后，走的是GETSTQ如果判断条件不当，׃引v死锁Q很悲催Q我在做的时候就到 了，具体怎么到的看下面的问?/p> GETSETq回nil时应该怎么处理Q?/span> C1和C2客户端调用GET接口QC1q回T1Q此时C3|络情况更好Q快速进入获取锁Qƈ执行DEL删除锁，C2q回T2(nil)QC1和C2都进入超时处理逻辑?br />C1 向foo.lock发送GETSET命oQ获取返回值T11(nil)?br />C1 比对C1和C11发现两者不同，处理逻辑认ؓ未获取锁?br />C2 向foo.lock发送GETSET命oQ获取返回值T22(C1写入的时间戳)?br />C2 比对C2和C22发现两者不同，处理逻辑认ؓ未获取锁?/p> ?时C1和C2都认为未获取锁，其实C1是已l获取锁了，但是他的处理逻辑没有考虑GETSETq回nil的情况，只是单纯的用GET和GETSET值就?ҎQ至于ؓ什么会出现q种情况Q一U是多客LӞ每个客户端连接Redis的后Q发出的命oq不是连l的Q导致从单客L看到的好像连l的命oQ到 Redis server后，q两条命令之间可能已l插入大量的其他客户端发出的命oQ比如DEL,SETNX{。第二种情况Q多客户端之间时间不同步Q或者不是严?意义的同步?/p> 旉戳的问题 我们看到foo.lock的valuegؓ旉戻I所以要在多客户端情况下Q保证锁有效Q一定要同步各服务器的时_如果各服务器_旉有差异。时间不一致的客户端，在判断锁时Q就会出现偏差，从而生竞争条件?br />锁的时与否Q严g赖时间戳Q时间戳本n也是有精度限Ӟ假如我们的时间精度ؓU，从加锁到执行操作再到解锁Q一般操作肯定都能在一U内完成。这L话，我们上面的CASEQ就很容易出现。所以，最好把旉_ֺ提升到毫U。这L话，可以保证毫秒U别的锁是安全的?/p> 分布式锁的问?/span> 1Q必要的时机制Q获取锁的客L一旦崩溃，一定要有过期机Ӟ否则其他客户端都降无法获取锁Q造成死锁问题?br />2Q分布式锁，多客L的时间戳不能保证严格意义的一致性，所以在某些特定因素下，有可能存在锁串的情况。要适度的机Ӟ可以承受概率的事g产生?br />3Q只对关键处理节点加锁，良好的习惯是Q把相关的资源准备好Q比如连接数据库后，调用加锁机制获取锁，直接q行操作Q然后释放，量减少持有锁的旉?br />4Q在持有锁期间要不要CHECK锁，如果需要严g赖锁的状态，最好在关键步骤中做锁的CHECK查机Ӟ但是Ҏ我们的测试发玎ͼ在大q发Ӟ每一ơCHECK锁操作，都要消耗掉几个毫秒Q而我们的整个持锁处理逻辑才不?0毫秒Q玩客没有选择做锁的检查?br />5Qsleep学问Qؓ了减对Redis的压力，获取锁尝试时Q@环之间一定要做sleep操作。但是sleep旉是多是门学问。需要根据自qRedis的QPSQ加上持锁处理时间等q行合理计算?br />6Q至于ؓ什么不使用Redis的mutiQexpireQwatch{机Ӟ可以查一参考资料，找下原因?/p> 锁测试数?/span> 未用sleep ab -n1000 -c100 'http://sandbox6.wanke.etao.com/test/test_sequence.php?tbpm=t' 使用Sleep?/strong> 单次执行h?br /> 我们看到Q和不用sleep机制Ӟ性能相当。当时用相同的压条件进行压~时 获取锁的旉明显变长Q而锁的释放时间明昑֏短，仅是不采用sleep机制的一半。当然执行时间变成就是因为，我们在执行过E中Q重新创建数据库q接Q导致时间变长的。同时我们可以对比下Redis的命令执行压力情?nbsp; 参考资?/span> http://www.worlduc.com/FileSystem/18/2518/590664/9f63555e6079482f831c8ab1dcb8c19c.pdf
使用Redis实现分布式锁Q有两个重要函数需要介l?br />
SETNX命oQSET if Not eXistsQ?/span>
语法Q?br />SETNX key value
功能Q?br />当且仅当 key 不存在，?key 的D?value Qƈq回1Q若l定?key 已经存在Q则 SETNX 不做M动作Qƈq回0?/p>
语法Q?br />GETSET key value
功能Q?br />给?key 的D?value Qƈq回 key 的旧?(old value)Q当 key 存在但不是字W串cdӞq回一个错误，当key不存在时Q返回nil?/p>
语法Q?br />GET key
功能Q?br />q回 key 所兌的字W串|如果 key 不存在那么返回特D?nil ?/p>
语法Q?br />DEL key [KEY …]
功能Q?br />删除l定的一个或多个 key ,不存在的 key 会被忽略?/p>
SETNX foo.lock <current unix time>
DEL foo.lock
q得到foo.lock中老的旉戳T2
C1客户端获取锁Qƈ且处理完后，DEL掉锁Q在DEL锁之前。C2通过SETNX向foo.lock讄旉戳T0 发现有客L获取锁，q入GET操作?br />C2 向foo.lock发送GET命oQ获取返回值T1(nil)?br />C2 通过T0>T1+expireҎQ进入GETSET程?br />C2 调用GETSET向foo.lock发送T0旉戻Iq回foo.lock的原值T2
C2 如果T2=T1相等Q获得锁Q如果T2!=T1Q未获得锁?/p>
C1客户端获取锁Qƈ且处理完后，DEL掉锁Q在DEL锁之前。C2通过SETNX向foo.lock讄旉戳T0 发现有客L获取锁，q入GET操作?br />C2 向foo.lock发送GET命oQ获取返回值T1(nil)?br />C2 循环Q进入下一ơSETNX逻辑
W一U，锁重试时未做sleep。单ơ请求，加锁Q执行，解锁旉
可以看到加锁和解锁时间都很快Q当我们使用
AB q发100累计1000ơ请求，对这个方法进行压时?
我们会发玎ͼ获取锁的旉变成Q同时持有锁后，执行旉也变成，而delete锁的旉Q将q?0ms旉Qؓ什么会q样Q?br />1Q持有锁后，我们的执行逻辑中包含了再次调用Redis操作Q在大ƈ发情况下QRedis执行明显变慢?br />2Q锁的删除时间变长，从之前的0.2msQ变?.8msQ性能下降q?0倍?br />在这U情况下Q我们压的QPS?9Q最l发现QPS和压总量有关Q当我们q发100d100ơ请求时QQPS得到110多。当我们使用sleep?/p>
?图中l高部分是ؓ未采用sleep机制的时的压图Q矮胖部分ؓ采用sleep机制的压图Q通上囄到压力减?0%左右Q当Ӟsleepq种方式q?有个~点QPS下降明显Q在我们的压条件下Q仅?5Qƈ且有部分h出现时情况。不q综合各U情况后Q我们还是决定采用sleep机制Q主要是Z 防止在大q发情况下把Redis压垮Q很不行Q我们之前碰到过Q所以肯定会采用sleep机制?/p>
http://redis.io/commands/setnx
http://www.dentisthealthcenter.com/caojianhua/archive/2013/01/28/394847.html
引子
redis是一个很强大的数据结构存储的nosql数据库，很方侉K对业务模型进行效率的优化。最q我的工作是负责对现有Java服务器框架进行整理，q将|络层与逻辑层脱，以便于逻辑层和|络层的横向扩展?管我在逻辑层上使用了AKKA作ؓ核心框架Q尽可能lockfreeQ但是还是免不了需要跨jvm的锁。所以我需要实C个分布式锁?/p>
官方的实?/h2>
官方?a >SETNX q一늻了一个实现?/p>
- C4 sends SETNX lock.foo in order to acquire the lock
- The crashed client C3 still holds it, so Redis will reply with 0 to C4.
- C4 sends GET lock.foo to check if the lock expired. If it is not, it will sleep for some time and retry from the start.
- Instead, if the lock is expired because the Unix time at lock.foo is older than the current Unix time, C4 tries to perform: GETSET lock.foo (current Unix timestamp + lock timeout + 1)
- Because of the GETSET semantic, C4 can check if the old value stored at key is still an expired timestamp. If it is, the lock was acquired.
- If another client, for instance C5, was faster than C4 and acquired the lock with the GETSET operation, the C4 GETSET operation will return a non expired timestamp. C4 will simply restart from the first step. Note that even if C4 set the key a bit a few seconds in the future this is not a problem.
但是使用官方推荐的getset实现的话Q未竞争到锁的一方确实可以判断到自己未能竞争到锁Q但却将持有锁一方的旉修改了，q样的直接后果就是，持有锁的一Ҏ法解锁！Q！
Zlua的实?/h2>
其实官方实现出现的问题，是因Z用redis独立的命令不能将get-check-setq个q程q行原子化，所以我军_引入redis-luaQ将get-check-setq个q程使用lua脚本来实现?/p>
加锁Q?/p>
- script params: lock_key, current_timestamp, lock_timeout
- setnx lock_key (current_timestamp + lock_timeout). if not success, set lock_key (current_timestamp + lock_timeout) if current_timestamp > value
- client save current_timestamp(lock_create_timestamp)
解锁Q?/p>
- script params: lock_key, lock_create_timestamp, lock_timeout
- delete if lock_create_timestamp + lock_timeout == value
具体的实?
---locklocal now = tonumber(ARGV[1])local timeout = tonumber(ARGV[2])local to = now + timeoutlocal locked = redis.call('SETNX', KEYS[1], to)if (locked == 1) thenreturn 0endlocal kt = redis.call('type', KEYS[1]);if (kt['ok'] ~= 'string') thenreturn 2endlocal keyValue = tonumber(redis.call('get', KEYS[1]))if (now > keyValue) thenredis.call('set', KEYS[1], to)return 0endreturn 1---unlocklocal begin = tonumber(ARGV[1])local timeout = tonumber(ARGV[2])local kt = redis.call('type', KEYS[1]);if (kt['ok'] == 'string') thenlocal keyValue = tonumber(redis.call('get', KEYS[1]))if ((keyValue - begin) == timeout) thenredis.call('del', KEYS[1])return 0endendreturn 1
已知问题
redis的分布式锁会有单点的问题。当然我们的业务量也没有辑ֈ挂掉专门做锁的redis单点的水q?/p>
]]>
import org.beetl.sql.core.SQLManager;
import org.springframework.beans.BeansException;
import org.springframework.beans.factory.support.DefaultListableBeanFactory;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationContextAware;
import org.springframework.context.ConfigurableApplicationContext;
import org.springframework.core.io.Resource;
import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
import org.springframework.core.io.support.ResourcePatternResolver;
import java.io.IOException;
import java.util.Objects;
import java.util.stream.Stream;
public class DaoFactoryBean implements ApplicationContextAware {
@Override
public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
ConfigurableApplicationContext context = (ConfigurableApplicationContext) applicationContext;
DefaultListableBeanFactory beanFactory = (DefaultListableBeanFactory) context.getBeanFactory();
ResourcePatternResolver rpr = new PathMatchingResourcePatternResolver(applicationContext);
SQLManager sqlManager = applicationContext.getBean(SQLManager.class);
try {
Resource[] resources = rpr.getResources("classpath:com/duxiu/**/*.class");
Stream.of(resources).map(f -> {
try {
return f.getURI().getPath().split("(classes/)|(!/)")[1].replace("/", ".").replace(".class", "");
} catch (IOException e) {
e.printStackTrace();
return null;
}
}).filter(Objects::nonNull).forEach(f -> {
try {
Class<?> aClass = Class.forName(f);
boolean match = Stream.of(aClass.getAnnotations()).anyMatch(c -> c instanceof BeetlSQLDao);
if (match && !beanFactory.containsBean(aClass.getSimpleName())) {
System.out.println(sqlManager.getMapper(aClass));
//beanFactory.registerSingleton(aClass.getSimpleName(),sqlManager.getMapper(aClass));
}
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
});
} catch (IOException e) {
e.printStackTrace();
}
System.out.println(applicationContext.getBean(SQLManager.class));
/*if(!beanFactory.containsBean(beanName)){
BeanDefinitionBuilder beanDefinitionBuilder= BeanDefinitionBuilder.rootBeanDefinition(beanClass);
beanDefinitionBuilder.addPropertyValue("host", host);
beanDefinitionBuilder.addPropertyValue("port", port);
beanDefinitionBuilder.addPropertyValue("database", database);
beanDefinitionBuilder.setInitMethodName("init");
beanDefinitionBuilder.setDestroyMethodName("destroy");
beanFactory.registerBeanDefinition(beanName, beanDefinitionBuilder.getBeanDefinition());
logger.info("Add {} to bean container.", beanName);
}*/
}
}
]]>
?q本文，你可以对模块化开发和AMD规范有一个较直观的认识，q详l地学习RequireJSq个模块化开发工L常见用法。本文采取@序渐q的方式Q从 理论到实践，从RequireJS官方API文中，ȝ出在使用RequireJSq程中最常用的一些用法，q对文中不够清晰具体的内容Q加以例证和 分析Q希望本文的内容对你的能力提升有实质性的帮助?/p>
1. 模块?/h2>
怿每个前端开发h员在刚开始接触js~程Ӟ都写q类g面这样风格的代码Q?/p>
<script type="text/javascript"> var a = 1; var b = 2; var c = a * a + b * b; if(c> 1) { alert('c > 1'); } function add(a, b) { return a + b; } c = add(a,b); </script>
<a href="javascript:;" onclick="click(this);" title="">L?lt;/a>
q些代码的特ҎQ?/p>
- 到处可见的全局变量
- 大量的函?/strong>
- 内嵌在html元素上的各种js调用
当然q些代码本n在实现功能上q没有错误，但是从代码的可重用性，健壮性以及可l护性来_q种~程方式是有问题的，其是在面逻辑较ؓ复杂的应用中Q这些问题会暴露地特别明显：
- 全局变量极易造成命名冲突
- 函数式编E非怸利于代码的组l和理
- 内嵌的js调用很不利于代码的维护，因ؓhtml代码有的时候是十分臃肿和庞大的
所以当q些问题出现的时候，js大牛们就开始寻扑֎解决q些问题的究极办法，于是模块化开发就出现了。正如模块化q个概念的表面意思一P它要求在 ~写代码的时候，按层ơ，按功能，独立的逻辑Q封装成可重用的模块Q对外提供直接明了的调用接口Q内部实现细节完全私有，q且模块之间的内部实现在执行 期间互不q扰Q最l的l果是可以解决前面举例提到的问题。一个简单遵循模块化开发要求编写的例子Q?/p>
//module.js var student = function (name) { return name && { getName: function () { return name; } }; }, course = function (name) { return name && { getName: function () { return name; } } }, controller = function () { var data = {}; return { add: function (stu, cour) { var stuName = stu && stu.getName(), courName = cour && cour.getName(), current, _filter = function (e) { return e === courName; }; if (!stuName || !courName) return; current = data[stuName] = data[stuName] || []; if (current.filter(_filter).length === 0) { current.push(courName); } }, list: function (stu) { var stuName = stu && stu.getName(), current = data[stuName]; current && console.log(current.join(';')); } } }; //main.js var stu = new student('lyzg'), c = new controller(); c.add(stu,new course('javascript')); c.add(stu,new course('html')); c.add(stu,new course('css')); c.list(stu);
以上代码定义了三个模块分别表C学生，评和控制器Q然后在main.js中调用了controller提供的add和list接口Qؓlyzgq个学生d了三门课E，然后在控制台昄了出来。运行结果如下：
javascript;html;css
通过上例Q可以看出模块化的代码结构和逻辑十分清晰Q代码看h十分优雅Q另外由于逻辑都通过模块拆分Q所以达C解耦的目的Q代码的功能也会比较 健壮。不q上例用的q种模块化开发方式也q不是没有问题，q个问题是它还是把模块引用如studentq些直接dC全局I间下，虽然通过模块减少 了很多全局I间的变量和函数Q但是模块引用本w还是要依赖全局I间Q才能被调用Q当模块较多Q或者有引入W三Ҏ块库Ӟ仍然可能造成命名冲突的问题，所 以这U全局I间下的模块化开发的方式q不是最完美的方式。目前常见的模块化开发方式，全局I间方式是最基本的一U，另外常见的还有遵循AMD规范的开发方 式，遵@CMD规范的开发方式，和ECMAScript 6的开发方式。需要说明的是，CMD和ES6跟本文的核心没有关系Q所以不会在此介l，后面的内容主要介lAMD以及实现了AMD规范?RequireJS?/p>
2. AMD规范
正如上文提到Q实现模块化开发的方式Q另外常见的一U就是遵循AMD规范的实现方式，不过AMD规范q不是具体的实现方式Q而仅仅是模块化开发的一 U解x案，你可以把它理解成模块化开发的一些接口声明，如果你要实现一个遵循该规范的模块化开发工P必d现它预先定义的API。比如它要求在加?模块Ӟ必须使用如下的API调用方式Q?/p>
require([module], callback) 其中Q?[module]Q是一个数l，里面的成员就是要加蝲的模? callbackQ是模块加蝲完成之后的回调函?/span> 所有遵循AMD规范的模块化工具Q都必须按照它的要求d玎ͼ比如RequireJSq个库，是完全遵@AMD规范实现的，所以在利用 RequireJS加蝲或者调用模块时Q如果你事先知道AMD规范的话Q你q道该怎么用RequireJS了。规范的好处在于Q不同的实现却有相同的调 用方式，很容易切换不同的工具使用Q至于具体用哪一个实玎ͼq就跟各个工L各自的优点跟目的特Ҏ关系Q这些都是在目开始选型的时候需要确定的。目 前RequireJS不是唯一实现了AMD规范的库Q像Dojoq种更全面的js库也都有AMD的实现?/p>
最后对AMD全称做一个解释，译ؓQ异步模块定义。异步强调的是，在加载模块以及模块所依赖的其它模块时Q都采用异步加蝲的方式，避免模块加蝲d了网늚渲染q度。相比传l的异步加蝲QAMD工具的异步加载更加简便，而且q能实现按需加蝲Q具体解释在下一部分说明?/p>
3. JavaScript的异步加载和按需加蝲
html中的script标签在加载和执行q程中会d|页的渲染，所以一般要求尽量将script标签攄在body元素的底部，以便加快面昄的速度Q还有一U方式就是通过异步加蝲的方式来加蝲jsQ这样可以避免js文g对html渲染的阻塞?/p>
W?U异步加载的方式是直接利用脚本生成script标签的方式：
(function() { var s = document.createElement('script'); s.type = 'text/javascript'; s.async = true; s.src = 'http://yourdomain.com/script.js'; var x = document.getElementsByTagName('script')[0]; x.parentNode.insertBefore(s, x); })();
q段代码Q放|在script标记内部Q然后该script标记d到body元素的底部即可?/p>
W?U方式是借助script的属性：defer和asyncQdeferq个属性在IE览器和早v的火狐浏览器中支持，async在支?html5的浏览器上都支持Q只要有q两个属性，script׃以异步的方式来加载，所以script在html中的位置׃重要了：
<script defer async="true" type="text/javascript" src="app/foo.js"></script> <script defer async="true" type="text/javascript" src="app/bar.js"></script>
<script defer async="true" type="text/javascript" src="app/main.js"></script>
q种方式下，所有异步js在执行的时候还是按序执行的，不然׃存在依赖问题Q比如如果上例中的main.js依赖foo.js和bar.jsQ?但是main.js先执行的话就会出错了。虽然从来理Zq种方式也算不错了，但是不够好，因ؓ它用h很繁琐，而且q有个问题就是页面需要添加多?script标记以及没有办法完全做到按需加蝲?/p>
JS的按需加蝲分两个层ơ，W一个层ơ是只加载这个页面可能被用到的JSQ第二个层次是在只在用到某个JS的时 候才d载。传l地方式很容易做到第一个层ơ，但是不容易做到第二个层次Q虽然我们可以通过合ƈ和压~工P某个页面所有的JS都添加到一个文件中去， 最大程度减资源请求量Q但是这个JSh到客L以后Q其中有很多内容可能都用不上Q要是有个工兯够做到在需要的时候才d载相关js完解决问?了，比如RequireJS?/p>
4. RequireJS常用用法ȝ
前文多次提及RequireJSQ本部分对它的常用用法详细说明Q它的官方地址是：http://www.requirejs.cn/Q你可以到该地址M载最新版RequireJS文g。RequireJS作ؓ目前使用最q泛的AMD工具Q它的主要优ҎQ?/p>
- 完全支持模块化开?/li>
- 能将非AMD规范的模块引入到RequireJS中?/li>
- 异步加蝲JS
- 完全按需加蝲依赖模块Q模块文件只需要压~淆，不需要合q?/li>
- 错误调试
- 插g支持
4.01 如何使用RequireJS
使用方式很简单，只要一个script标记可以在|页中加载RequireJSQ?/p>
<script defer async="true" src="/bower_components/requirejs/require.js"></script>
׃q里用到了defer和asyncq两个异步加载的属性，所以require.js是异步加载的Q你把这个script标记攄在Q何地斚w没有问题?/p>
4.02 如何利用RequireJS加蝲q执行当前网늚逻辑JS
4.01解决的仅仅是RequireJS的用问题，但它仅仅是一个JS库，是一个被当前面的逻辑所利用的工P真正实现|页功能逻辑的是我们?利用RequireJS~写的主JSQ这个主JSQ假设这些代码都攄在main.js文g中）又该如何利用RJ来加载执行呢Q方式如下：
<script data-main="scripts/main.js" defer async="true" src="/bower_components/requirejs/require.js"></script>
Ҏ4.01Q你会发现script标记多了一个data-mainQRJ用这个配|当前页面的主JSQ你要把逻辑都写在这个main.js里面?当RJ自n加蝲执行后，׃再次异步加蝲main.js。这个main.js是当前网|有逻辑的入口，理想情况下，整个|页只需要这一个script?讎ͼ利用RJ加蝲依赖的其它文Ӟ如jquery{?/p>
4.03 main.js怎么?/h3>
假设目的目录结构ؓQ?/p>
main.js是跟当前面相关的主JSQapp文g夹存放本目自定义的模块Qlib存放W三方库?/p>
html中按4.02的方式配|RJ。main.js的代码如下：
require(['lib/foo', 'app/bar', 'app/app'], function(foo, bar, app) { //use foo bar app do sth });
在这DJS中，我们利用RJ提供的requireҎQ加载了三个模块Q然后在q个三个模块都加载成功之后执行页面逻辑。requireҎ??参数Q第一个参数是数组cd的，实际使用Ӟ数组的每个元素都是一个模块的module IDQ第二个参数是一个回调函敎ͼq个函数在第一个参数定义的所有模块都加蝲成功后回调，形参的个数和序分别与第一个参数定义的模块对应Q比如第一个模 块时lib/fooQ那么这个回调函数的W一个参数就是fooq个模块的引用，在回调函C我们使用q些形参来调用各个模块的ҎQ由于回调是在各模块?载之后才调用的，所以这些模块引用肯定都是有效的?/p>
从以上这个简短的代码Q你应该已经知道该如何用RJ了?/p>
4.04 RJ的baseUrl和module ID
在介lRJ如何去解析依赖的那些模块JS的\径时Q必d弄清楚baseUrl和module IDq两个概c?/p>
html中的base元素可以定义当前面内部Mhttph的url前缀部分QRJ的baseUrl跟这个base元素L作用是类似的Q由?RJL动态地h依赖的JS文gQ所以必然涉及到一个JS文g的\径解析问题，RJ默认采用一UbaseUrl + moduleID的解析方式，q个解析方式后箋会D例说明。这个baseUrl非常重要QRJ对它的处理遵循如下规则：
- 在没有用data-main和config的情况下QbaseUrl默认为当前页面的目录
- 在有data-main的情况下Qmain.js前面的部分就是baseUrlQ比如上面的scripts/
- 在有config的情况下QbaseUrl以config配置的ؓ?/li>
上述三种方式Q优先׃到高排列?/p>
data-main的用方式，你已l知道了Qconfig该如何配|，如下所C：
require.config({ baseUrl: 'scripts' });
q个配置必须攄在main.js的最前面。data-main与config配置同时存在的时候，以config为准Q由于RJ的其它配|也是在q个位置配置的，所?.03中的main.js可以Ҏ如下l构Q以便将来的扩展Q?/p>
require.config({ baseUrl: 'scripts' }); require(['lib/foo', 'app/bar', 'app/app'], function(foo, bar, app) { // use foo bar app do sth });
关于module IDQ就是在requireҎ以及后箋的defineҎ里，用在依赖数组q个参数里，用来标识一个模块的字符丌Ӏ上面代码中的['lib/foo', 'app/bar', 'app/app']是一个依赖数l，其中的每个元素都是一个module ID。值得注意的是Qmodule IDq不一定是该module 相关JS路径的一部分Q有的module ID很短Q但可能路径很长Q这跟RJ的解析规则有兟뀂下一节详l介l?/p>
4.05 RJ的文件解析规?/h3>
RJ默认按baseUrl + module ID的规则，解析文gQƈ且它默认要加载的文g都是jsQ所以你的module ID里面可以不包?js的后~Q这是为啥你看到的module ID都是lib/foo, app/barq种形式了。有三种module IDQ不适用q种规则Q?/p>
- ?开_?lib/jquey.js
- ?jsl尾Q如test.js
- 包含http或httpsQ如http://cdn.baidu.com/js/jquery.js
假如main.js如下使用Q?/p>
require.config({ baseUrl: 'scripts' }); require(['/lib/foo', 'test.js', 'http://cdn.baidu.com/js/jquery'], function(foo, bar, app) { // use foo bar app do sth });
q三个module 都不会根据baseUrl + module ID的规则来解析,而是直接用module ID来解析，{效于下面的代码Q?/p>
<script src="/lib/foo.js"></script> <script src="test.js"></script> <script src="http://cdn.baidu.com/js/jquery.js"></script>
各种module ID解析举例Q?/strong>
?Q项目结构如下：
main.js如下Q?/p>
require.config({ baseUrl: 'scripts' }); require(['lib/foo', 'app/bar', 'app/app'], function(foo, bar, app) { // use foo bar app do sth });
baseUrl为：scripts目录
moduleID为：lib/foo, app/bar, app/app
ҎbaseUrl + moduleIDQ以及自动补后缀.jsQ最l这三个module的js文g路径为：
scripts/lib/foo.js scripts/app/bar.js scripts/app/app.js ?Q项目结构同?Q?/p>
main.js改ؓQ?/p>
require.config({ baseUrl: 'scripts/lib', paths: { app: '../app' } }); require(['foo', 'app/bar', 'app/app'], function(foo, bar, app) { // use foo bar app do sth });
q里出现了一个新的配|pathsQ它的作用是针对module ID中特定的部分Q进行{义，如以上代码中对appq个部分Q{义ؓ../appQ这表示一个相对\径，相对位置是baseUrl所指定的目录，由项目结 构可知，../app其实对应的是scirpt/app目录。正因ؓ有这个{义的存在Q所以以上代码中的app/bar才能被正解析，否则q按 baseUrl + moduleID的规则，app/bar不是应该被解析成scripts/lib/app/bar.js吗，但实际ƈ非如此，app/bar被解析成 scripts/app/bar.jsQ其中v关键作用的就是paths的配|。通过q个举例Q可以看出module IDq不一定是js文g路径中的一部分Qpaths的配|对于\径过E的js特别有效Q因为可以简化它的module ID?/p>
另外W一个模块的ID为fooQ同时没有paths的{义，所以根据解析规则，它的文g路径Ӟscripts/lib/foo.js?/p>
paths的配|中只有当模块位于baseUrl所指定的文件夹的同层目录，或者更上层的目录时Q才会用?./q种相对路径?/p>
?Q项目结果同?Qmain.js同例2Q?/p>
q里要说明的问题E微ҎQ不以main.jsZQ而以app.jsZQ且app依赖barQ当然configq是需要在main.js中定义的Q由于这个问题在定义模块的时候更加常见，所以用define来D例，假设app.js模块如下定义Q?/p>
define(['./bar'], function(bar) { return { doSth: function() { bar.doSth(); } } }); 上面的代码通过define定义了一个模块，q个define函数后面介绍如何定义模块的时候再来介l，q里单了解。这里这U用法的W一个参数跟 require函数一P是一个依赖数l，W二个参数是一个回调，也是在所有依赖加载成功之后调用，q个回调的返回g成ؓq个模块的引用被其它模块所?用?/p>
q里要说的问题还是跟解析规则相关的，如果完全遵守RJ的解析规则，q里的依赖应该配|成app/bar才是正确的，但由于app.js?bar.js位于同一个目录，所以完全可利用./q个同目录的相对标识W来解析jsQ这L话只要app.js已经加蝲成功了，那么d目录下找 bar.jsp定能扑ֈ了。这U配|在定义模块的时候非常有意义Q这样你的模块就不依赖于攄q些模块的文件夹名称了?/p>
4.06 RJ的异步加?/h3>
RJ不管是requireҎq是defineҎ的依赖模块都是异步加载的Q所以下面的代码不一定能解析到正的JS文gQ?/p>
<script data-main="scripts/main" src="scripts/require.js"></script> <script src="scripts/other.js"></script>
//main.js
require.config({ paths: { foo: 'libs/foo-1.1.3' } });
//other.js
require( ['foo'], function( foo ) { //foo is undefined });
׃main.js是异步加载的Q所以other.js会比它先加蝲Q但是RJ的配|存在于main.js里面Q所以在加蝲other.jsM到RJ的配|，在other.js执行的时候解析出来的foo的\径就会变成scripts/foo.jsQ而正\径应该是scripts/libs/foo-1.1.3.js?/p>
管RJ的依赖是异步加蝲的，但是已加载的模块在多ơ依赖的时候，不会再重新加载：
define(['require', 'app/bar', 'app/app'], function(require) { var bar= require("app/bar"); var app= require("app/app"); //use bar and app do sth });
上面的代码，在callback定义的时候，只用了一个Ş参，q主要是Z减少形参的数量，避免整个回调的签名很ѝ依赖的模块在回调内部可以直接用require(moduleID)的参数得刎ͼ׃在回调执行前Q依赖的模块已经加蝲Q所以此处调用不会再重新加蝲。但是如果此处获取一个ƈ不在依赖数组中出现的module IDQrequire很有可能获取不到该模块引用，因ؓ它可能需要重新加载，如果它没有在其它模块中被加蝲q的话?/p>
4.07 RJ官方推荐的JS文gl织l构
RJQ文件组l尽量扁qI不要多层嵌套Q最理想的是跟项目相关的攑֜一个文件夹Q第三方库放在一个文件夹Q如下所C：
4.08 使用define定义模块
AMD规定的模块定义规范ؓQ?/p>
define(id?, dependencies?, factory); 其中Q?id: 模块标识Q可以省略?dependencies: 所依赖的模块，可以省略?factory: 模块的实玎ͼ或者一个JavaScript对象 关于W一个参敎ͼ本文不会涉及Q因为RJ所有模块都不要使用W一个参敎ͼ如果使用W一个参数定义的模块成ؓ命名模块Q不适用W一个参数的模块成ؓ匿名模块Q命名模块如果更名，所有依赖它的模块都得修改！W二个参数是依赖数组Q跟require一P如果没有q个参数Q那么定义的是一个无依赖的模块；最后一个参数是回调或者是一个简单对象，在模块加载完毕后调用Q当然没有第二个参数Q最后一个参C会调用?/p>
本部分所举例都采用如下项目结构：
1. 定义单对象模块：
app/bar.js
define({ bar:'I am bar.' });
利用main.js试Q?/p>
require.config({ baseUrl: 'scripts/lib', paths: { app: '../app' } }); require(['app/bar'], function(bar) { console.log(bar);// {bar: 'I am bar.'} });
2. 定义无依赖的模块Q?/p>
app/nodec.jsQ?/p>
define(function () { return { nodec: "yes, I don't need dependence." } });
利用main.js试Q?/p>
require.config({ baseUrl: 'scripts/lib', paths: { app: '../app' } }); require(['app/nodec'], function(nodec) { console.log(nodec);// {nodec: yes, I don't need dependence.'} });
3. 定义依赖其它模块的模块：
app/dec.jsQ?/p>
define(['jquery'], function($){ //use $ do sth ... return { useJq: true } });
利用main.js试Q?/p>
require.config({ baseUrl: 'scripts/lib', paths: { app: '../app' } }); require(['app/dec'], function(dec) { console.log(dec);//{useJq: true} });
4. 循环依赖Q?/p>
当一个模块foo的依赖数l中存在barQbar模块的依赖数l中存在fooQ就会Ş成@环依赖，E微修改下bar.js和foo.js如下?/p>
app/bar.jsQ?/p>
define(['foo'],function(foo){ return { name: 'bar', hi: function(){ console.log('Hi! ' + foo.name); } } });
lib/foo.jsQ?/p>
define(['app/bar'],function(bar){ return { name: 'foo', hi: function(){ console.log('Hi! ' + bar.name); } } });
利用main.js试Q?/p>
require.config({ baseUrl: 'scripts/lib', paths: { app: '../app' } }); require(['app/bar', 'foo'], function(bar, foo) { bar.hi(); foo.hi(); });
q行l果Q?/p>
如果改变main.js中require部分的依赖顺序，l果Q?/p>
循环依赖D两个依赖的module之间Q始l会有一个在获取另一个的时候，得到undefined。解x法是Q在定义module的时候，如果用到循环依赖的时候，在define内部通过require重新获取。main.js不变Qbar.jsҎQ?/p>
define(['require', 'foo'], function(require, foo) { return { name: 'bar', hi: function() { foo = require('foo'); console.log('Hi! ' + foo.name); } } });
foo.jsҎQ?/p>
define(['require', 'app/bar'], function(require, bar) { return { name: 'foo', hi: function() { bar = require('app/bar'); console.log('Hi! ' + bar.name); } } });
利用上述代码Q重新执行，l果是：
模块定义ȝQ?/strong>不管模块是用回调函数定义q是单对象定义，q个模块输出的是一个引用，所以这个引用必L有效的，你的回调不能q回undefinedQ但是不局限于对象cdQ还可以是数l，函数Q甚x基本cdQ只不过如果q回对象Q你能通过q个对象l织更多的接口?/p> 再看看这个代码： 依赖数组中的requireq个moduleID对应的是一个内|模块，利用它加载模块，怎么用你已经看到了，比如在main.js中，在define中。另外一个内|模块是moduleQ这个模块跟RJ的另外一个配|有养I具体用法请在W?大部分去了解?/p> 1. 生成相对于模块的URL地址 q个功能在你惌动态地加蝲一些文件的时候有用，注意要用相对\径?/p> 2. 控制台调?/p> 假如你想在控制台中查看某个模块都有哪些方法可以调用，如果q个模块已经在页面加载的时候通过依赖被加载过后，那么可以用以上代码在控制台中做各种试了?/p> 在RJ的配|中Q前面已l接触到了baseUrlQpathsQ另外几个常用的配置是： 为那些没有用define()来声明依赖关pR设|模块的"览器全局变量注入"型脚本做依赖和导出配|?/p> ?Q利用exports模块的全局变量引用与RequireJS兌 目l构如图Q?/p> main.js如下Q?/p> 如你所见，RJ在shim中添加了一个对underscoreq个模块的配|，q过exports属性指定该模块暴露的全局变量Q以便RJ能够对这些模块统一理?/p> ?Q利用deps配置js模块的依?/p> 目l构如图Q?/p> main.js如下Q?/p> ׃backboneq个lg依赖jquery和underscoreQ所以可以通过deps属性配|它的依赖，q样backbone会在另外两个模块加载完毕之后才会加载?/p> ?Qjquery{库插g配置Ҏ 代码举例如下Q?/p> 常常需要将配置信息传给一个模块。这些配|往往是applicationU别的信息，需要一个手D将它们向下传递给模块。在RequireJS中，Zrequirejs.config()的config配置Ҏ实现。要获取q些信息的模块可以加载特D的依赖“module”Qƈ调用module.config()?/p> ?Q在requirejs.config()中定义configQ以供其它模块?/p> 如你所见，config属性中的barq一节是在用于module ID为barq个模块的，bazq一节是用于module ID为bazq个模块的。具体用以bar.js举例Q?/p> 前面提到q，RJ的内|模块除了requireq有一个moduleQ用法就在此处，通过它可以来加蝲config的内宏V?/p> 如果讄为trueQ则当一个脚本不是通过define()定义且不具备可供查的shim导出字串值时Q就会抛出错误。这个属性可以强制要求所有RJ依赖或加载的模块都要通过define或者shim被RJ来管理，同时它还有一个好处就是用于错误检?/p> RequireJS获取资源旉加在URL后面的额外的query参数。作为浏览器或服务器未正配|时?#8220;cache bust”手段很有用。用cache bust配置的一个示例： IE中捕获加载错误不完美Q?/p> 所以ؓ了支持在IE中捕获加载错误，需要配|enforceDefine为trueQ这不得不要求你所有的模块都用define定义Q或者用shim配置RJ对它的引用?/p> 注意Q?/strong>如果你设|了enforceDefine: trueQ而且你用data-main=""来加载你的主JS模块Q则该主JS模块必须调用define()而不是require()来加载其所需的代码。主JS模块仍然可调用require/requirejs来设|config|但对于模块加载必M用define()。比如原来的q段׃报错Q?/p> 把最后三行改成： 才不会报错?/p> 上述代码先尝试加载CDN版本Q如果出错，则退回到本地的lib/jquery.js?/p> 注意: paths备错仅在模块ID_匚w时工作。这不同于常规的paths配置Q常规配|可匚w模块ID的Q意前~部分。备错主要用于非常的错误恢复Q而不是常规的path查找解析Q因为那在浏览器中是低效的?/p> Z捕获在局域的errback中未捕获的异常，你可以重载requirejs.onError()Q?/p> Q完Q?/p>4.09 内置的RJ模块
define(['require', 'app/bar'], function(require) { return { name: 'foo', hi: function() { var bar = require('app/bar'); console.log('Hi! ' + bar.name); } } });
4.10 其它RJ有用功能
define(["require"], function(require) { var cssUrl = require.toUrl("./style.css"); });
require("module/name").callSomeFunction() 5. RequireJS常用配置ȝ
5.01 shim
require.config({ baseUrl: 'scripts/lib', paths: { app: '../app' }, shim: { underscore: { exports: '_' } } }); require(['underscore'], function(_) { // 现在可以通过_调用underscore的api?/span> });
require.config({ baseUrl: 'scripts/lib', paths: { app: '../app' }, shim: { backbone: { deps: ['underscore', 'jquery'], exports: 'Backbone' } } }); require(['backbone'], function(Backbone) { //use Backbone's API });
requirejs.config({ shim: { 'jquery.colorize': { deps: ['jquery'], exports: 'jQuery.fn.colorize' }, 'jquery.scroll': { deps: ['jquery'], exports: 'jQuery.fn.scroll' }, 'backbone.layoutmanager': { deps: ['backbone'] exports: 'Backbone.LayoutManager' } } });
5.02 config
requirejs.config({ config: { 'bar': { size: 'large' }, 'baz': { color: 'blue' } } });
define(['module'], function(module) { //Will be the value 'large'var size = module.config().size; });
5.03 enforceDefine
5.04 urlArgs
urlArgs: "bust=" + (new Date()).getTime() 6. 错误处理
6.01 加蝲错误的捕?/strong>
require.config({ enforceDefine: true, baseUrl: 'scripts/lib', paths: { app: '../app' }, shim: { backbone: { deps: ['underscore', 'jquery'], exports: 'Backbone' } } }); require(['backbone'], function(Backbone) { console.log(Backbone); });
define(['backbone'], function(Backbone) { console.log(Backbone); });
6.02 paths备错
requirejs.config({ //To get timely, correct error triggers in IE, force a define/shim exports check. enforceDefine: true, paths: { jquery: [ 'http://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min', //If the CDN location fails, load from this location 'lib/jquery' ] } }); //Later require(['jquery'], function ($) { });
6.03 全局 requirejs.onError
requirejs.onError = function (err) { console.log(err.requireType); if (err.requireType === 'timeout') { console.log('modules: ' + err.requireModules); } throw err; };
]]>
Spark是基于内存的分布式计引擎，以处理的高效和稳定著U。然而在实际的应用开发过E中Q开发者还是会遇到U种问题Q其中一大类是和性能相关。在本文中，W者将l合自n实践Q谈谈如何尽可能地提高应用程序性能?/p>
分布式计引擎在调优斚w有四个主要关注方向，分别是CPU、内存、网l开销和I/OQ其具体调优目标如下Q?/p>
- 提高CPU利用率?
- 避免OOM?
- 降低|络开销?
- 减少I/O操作?/li>
W??数据倾斜
数据倾斜意味着某一个或某几个Partition中的数据量特别的大，q意味着完成针对q几个Partition的计需要耗费相当长的旉?/p>
?果大量数据集中到某一个PartitionQ那么这个Partition在计的时候就会成为瓶颈。图1是Spark应用E序执行q发的示意图Q在 Spark中，同一个应用程序的不同Stage是串行执行的Q而同一Stage中的不同Task可以q发执行QTask数目由Partition数来?定，如果某一个Partition的数据量特别大，则相应的task完成旉会特别长Q由此导致接下来的Stage无法开始，整个Job完成的时间就会非 帔R?/p>
要避免数据倾斜的出玎ͼ一U方法就是选择合适的keyQ或者是自己定义相关的partitioner。在Spark中Block使用 了ByteBuffer来存储数据，而ByteBuffer能够存储的最大数据量不超q?GB。如果某一个key有大量的数据Q那么在调用cache?persist函数时就会碰到spark-1476q个异常?/p>
下面列出的这些API会导致Shuffle操作Q是数据倾斜可能发生的关键点所?
1. groupByKey
2. reduceByKey
3. aggregateByKey
4. sortByKey
5. join
6. cogroup
7. cartesian
8. coalesce
9. repartition
10. repartitionAndSortWithinPartitions
def rdd: RDD[T] } // TODO View bounds are deprecated, should use context bounds // Might need to change ClassManifest for ClassTag in spark 1.0.0 case class DemoPairRDD[K <% Ordered[K] : ClassManifest, V: ClassManifest]( rdd: RDD[(K, V)]) extends RDDWrapper[(K, V)] { // Here we use a single Long to try to ensure the sort is balanced, // but for really large dataset, we may want to consider // using a tuple of many Longs or even a GUID def sortByKeyGrouped(numPartitions: Int): RDD[(K, V)] = rdd.map(kv => ((kv._1, Random.nextLong()), kv._2)).sortByKey() .grouped(numPartitions).map(t => (t._1._1, t._2)) } case class DemoRDD[T: ClassManifest](rdd: RDD[T]) extends RDDWrapper[T] { def grouped(size: Int): RDD[T] = { // TODO Version where withIndex is cached val withIndex = rdd.mapPartitions(_.zipWithIndex) val startValues = withIndex.mapPartitionsWithIndex((i, iter) => Iterator((i, iter.toIterable.last))).toArray().toList .sortBy(_._1).map(_._2._2.toLong).scan(-1L)(_ + _).map(_ + 1L) withIndex.mapPartitionsWithIndex((i, iter) => iter.map { case (value, index) => (startValues(i) + index.toLong, value) }) .partitionBy(new Partitioner { def numPartitions: Int = size def getPartition(key: Any): Int = (key.asInstanceOf[Long] * numPartitions.toLong / startValues.last).toInt }) .map(_._2) } }定义隐式的{?/p>
implicit def toDemoRDD[T: ClassManifest](rdd: RDD[T]): DemoRDD[T] = new DemoRDD[T](rdd) implicit def toDemoPairRDD[K <% Ordered[K] : ClassManifest, V: ClassManifest]( rdd: RDD[(K, V)]): DemoPairRDD[K, V] = DemoPairRDD(rdd) implicit def toRDD[T](rdd: RDDWrapper[T]): RDD[T] = rdd.rdd }在spark-shell中就可以使用?/p>
import RDDConversions._ yourRdd.grouped(5)W?? 减少|络通信开销
Spark 的Shuffleq程非常消耗资源，Shuffleq程意味着在相应的计算节点Q要先将计算l果存储到磁盘，后箋的Stage需要将上一个Stage的结 果再ơ读入。数据的写入和读取意味着Disk I/O操作Q与内存操作相比QDisk I/O操作是非怽效的?/p>
使用iostat来查看disk i/o的用情况，disk i/o操作频繁一般会伴随着cpu load很高?/p>
如果数据和计节炚w在同一台机器上Q那么可以避免网l开销Q否则还要加上相应的|络开销?使用iftop来查看网l带宽用情况，看哪几个节点之间有大量的|络传输?
?是Spark节点间数据传输的C意图，Spark Task的计函数是通过Akka通道由Driver发送到Executor上，而Shuffle的数据则是通过Netty|络接口来实现。由于Akka 通道中参数spark.akka.framesize军_了能够传输消息的最大|所以应该避免在Spark Task中引入超大的局部变量?/p>
W??选择合适的q发?/strong>
Z提高Spark应用E序的效率，可能的提升CPU的利用率。ƈ发数应该是可用CPU物理核数的两倍。在q里Qƈ发数q低QCPU得不到充分的利用Qƈ发数q大Q由于spark是每一个task都要分发到计结点，所以Q务启动的开销会上升?/p>
q发数的修改Q通过配置参数来改变spark.default.parallelismQ如果是sql的话Q可能通过修改spark.sql.shuffle.partitions来修攏V?/p>
W?? Repartition vs. Coalesce
repartition和coalesce都能实现数据分区的动态调_但需要注意的是repartition会导致shuffle操作Q而coalesce不会?/p>
W?? reduceByKey vs. groupBy
groupBy操作应该可能的避免Q第一是有可能造成大量的网l开销Q第二是可能DOOM。以WordCountZ来演CreduceByKey和groupBy的差?/p>
reduceByKey sc.textFile(“README.md”).map(l=>l.split(“,”)).map(w=>(w,1)).reduceByKey(_ + _)
Shuffleq程如图2所C?/p>
groupByKey sc.textFile(“README.md”).map(l=>l.split(“,”)).map(w=>(w,1)).groupByKey.map(r=>(r._1,r._2.sum))
: 可能用reduceByKey, aggregateByKey, foldByKey和combineByKey
假设有一RDD如下所C，求每个key的均?/p>
val data = sc.parallelize( List((0, 2.), (0, 4.), (1, 0.), (1, 10.), (1, 20.)) )
Ҏ一QreduceByKey
data.map(r=>(r._1, (r.2,1))).reduceByKey((a,b)=>(a._1 + b._1, a._2 + b._2)).map(r=>(r._1,(r._2._1/r._2._2)).foreach(println)Ҏ二：combineByKey
data.combineByKey(value=>(value,1), (x:(Double, Int), value:Double)=> (x._1+value, x._2 + 1), (x:(Double,Int), y:(Double, Int))=>(x._1 + y._1, x._2 + y._2))W?? BroadcastHashJoin vs. ShuffleHashJoin
在Joinq程中，l常会遇到大表和表的join. Z提高效率可以使用BroadcastHashJoin, 预先小表的内容q播到各个Executor, q样避免针对小表的Shuffleq程Q从而极大的提高q行效率?/p>
其实BroadCastHashJoin核心是利用了BroadCast函数Q如果理解清楚broadcast的优点，p比较好的明白BroadcastHashJoin的优势所在?/p>
以下是一个简单用broadcast的示例程序?/p>
val lst = 1 to 100 toList val exampleRDD = sc.makeRDD(1 to 20 toSeq, 2) val broadcastLst = sc.broadcast(lst) exampleRDD.filter(i=>broadcastLst.valuecontains(i)).collect.foreach(println)W?? map vs. mapPartitions
有时需要将计算l果存储到外部数据库Q势必会建立到外部数据库的连接。应该尽可能的让更多的元素共享同一个数据连接而不是每一个元素的处理旉d立数据库q接?
在这U情况下QmapPartitions和foreachPartitons比map操作高效的多?/p>
W?? 数据地d
Ud计算的开销q远低于Ud数据的开销?/p>
Spark中每个Task都需要相应的输入数据Q因此输入数据的位置对于Task的性能变得很重要。按照数据获取的速度来区分，由快到慢分别是：
1.PROCESS_LOCAL
2.NODE_LOCAL
3.RACK_LOCAL
Spark在Task执行的时候会优先考虑最快的数据获取方式Q如果想可能的在更多的机器上启动TaskQ那么可以通过调低spark.locality.wait的值来实现, 默认值是3s?/p>
?了HDFSQSpark能够支持的数据源来多Q如Cassandra, HBase,MongoDB{知名的NoSQL数据库，随着Elasticsearch的日渐兴Pspark和elasticsearchl合h提供 高速的查询解决Ҏ也成ZU有益的试?/p>
上述提到的外部数据源面的一个相同问题就是如何让spark快速读取其中的数据Q?可能的计结点和数据l点部v在一h辑ֈ该目标的基本ҎQ比如在部vHadoop集群的时候，可以HDFS的DataNode和Spark Worker׃n一台机器?/p>
以cassandraZ,如果Spark的部|和Cassandra的机器有部分重叠Q那么在dCassandra中数据的时候，通过调低spark.locality.wait可以在没有部vCassandra的机器上启动Spark Task?/p>
对于Cassandra, 可以在部|Cassandra的机器上部vSpark WorkerQ需要注意的是Cassandra的compaction操作会极大的消耗CPUQ因此在为Spark Worker配置CPU核数Ӟ需要将q些因素l合在一赯行考虑?/p>
q一部分的代码逻辑可以参考源码TaskSetManager::addPendingTask
private def addPendingTask(index: Int, readding: Boolean = false) { // Utility method that adds `index` to a list only if readding=false or it's not already there def addTo(list: ArrayBuffer[Int]) { if (!readding || !list.contains(index)) { list += index } } for (loc <- tasks(index).preferredLocations) { loc match { case e: ExecutorCacheTaskLocation => addTo(pendingTasksForExecutor.getOrElseUpdate(e.executorId, new ArrayBuffer)) case e: HDFSCacheTaskLocation => { val exe = sched.getExecutorsAliveOnHost(loc.host) exe match { case Some(set) => { for (e <- set) { addTo(pendingTasksForExecutor.getOrElseUpdate(e, new ArrayBuffer)) } logInfo(s"Pending task $index has a cached location at ${e.host} " + ", where there are executors " + set.mkString(",")) } case None => logDebug(s"Pending task $index has a cached location at ${e.host} " + ", but there are no executors alive there.") } } case _ => Unit } addTo(pendingTasksForHost.getOrElseUpdate(loc.host, new ArrayBuffer)) for (rack <- sched.getRackForHost(loc.host)) { addTo(pendingTasksForRack.getOrElseUpdate(rack, new ArrayBuffer)) } } if (tasks(index).preferredLocations == Nil) { addTo(pendingTasksWithNoPrefs) } if (!readding) { allPendingTasks += index // No point scanning this whole list to find the old task there } }如果准备让spark支持新的存储源，q而开发相应的RDDQ与位置相关的部分就是自定义getPreferredLocations函数Q以elasticsearch-hadoop中的EsRDDZQ其代码实现如下?/p>
override def getPreferredLocations(split: Partition): Seq[String] = { val esSplit = split.asInstanceOf[EsPartition] val ip = esSplit.esPartition.nodeIp if (ip != null) Seq(ip) else Nil }W?? 序列?/strong>
使用好的序列化算法能够提高运行速度Q同时能够减内存的使用?/p>
Spark在Shuffle的时候要数据先存储到磁盘中Q存储的内容是经q序列化的。序列化的过E牵涉到两大基本考虑的因素，一是序列化的速度Q二是序列化后内Ҏ占用的大?/p>
kryoSerializer与默认的javaSerializer相比Q在序列化速度和序列化l果的大方面都h极大的优ѝ所以徏议在应用E序配置中用KryoSerializer.
spark.serializer org.apache.spark.serializer.KryoSerializer默认的cache没有对缓存的对象q行序列化，使用的StorageLevel是MEMORY_ONLY,q意味着要占用比较大的内存。可以通过指定persist中的参数来对~存内容q行序列化?/p>
exampleRDD.persist(MEMORY_ONLY_SER)需要特别指出的是persist函数是等到job执行的时候才会将数据~存hQ属于gq执? 而unpersist函数则是立即执行Q缓存会被立x除?/p>
作者简介：讔RQ?《Apache Spark源码剖析》作者，x于大数据实时搜烦和实时流数据处理Q对elasticsearch, storm及drools多有研究Q现p于携E?/p>
]]>
Play uses sbt-native-packager, which supports the inclusion of arbitrary files by adding them to the mappings:
mappings in Universal ++= (baseDirectory.value / "scripts" * "*" get) map (x => x -> ("scripts/" + x.getName)) val jdk8 = new File("D:\\JDK\\JDK8\\jre1_8_0_40")
mappings in Universal ++= (jdk8 ** "*" get) map (x => x -> ("jre8/" + jdk8.relativize(x).getOrElse(x.getName)))
]]>
一直听说word2vec在处理词与词的相似度的问题上效果十分好，最q自׃上手跑了跑Google开源的代码Q?span style="color: #0000ff;">https://code.google.com/p/word2vec/Q?/span>
1、语?/span>
首先准备数据Q采用网上博客上推荐的全|新L?SogouCA)Q大ؓ2.1G?nbsp;
1 wget ftp://ftp.labs.sogou.com/Data/SogouCA/SogouCA.tar.gz --ftp-user=hebin_hit@foxmail.com --ftp-password=4FqLSYdNcrDXvNDi -r
解压数据包：
1 gzip -d SogouCA.tar.gz 2 tar -xvf SogouCA.tar
再将生成的txt文g归ƈ到SogouCA.txt中，取出其中包含content的行q{码，得到语料corpus.txtQ大ؓ2.7G?/span>
1 cat *.txt > SogouCA.txt 2 cat SogouCA.txt | iconv -f gbk -t utf-8 -c | grep "<content>" > corpus.txt
2、分?/span>
?/span>ANSJ对corpus.txtq行分词Q得到分词结果resultbig.txtQ大ؓ3.1G?/span>
1 nohup ./word2vec -train resultbig.txt -output vectors.bin -cbow 0 -size 200 -window 5 -negative 0 -hs 1 -sample 1e-3 -threads 12 -binary 1 &
vectors.bin是word2vec处理resultbig.txt后生成的词的向量文gQ在实验室的服务器上训练?个半时?/span>
1 ./distance vectors.bin ./distance可以看成计算词与词之间的距离Q把词看成向量空间上的一个点Qdistance看成向量I间上点与点的距R?/span>
下面是一些例子：
4.2 潜在的语a学规?/span>
4.3 聚类
经q分词后的语料resultbig.txt中的词聚cdƈ按照cd排序Q?/span>
1 nohup ./word2vec -train resultbig.txt -output classes.txt -cbow 0 -size 200 -window 5 -negative 0 -hs 1 -sample 1e-3 -threads 12 -classes 500 & 2 sort classes.txt -k 2 -n > classes_sorted_sogouca.txt
例如Q?/span>
4.4 短语分析
先利用经q分词的语料resultbig.txt中得出包含词和短语的文gsogouca_phrase.txtQ再训练该文件中词与短语的向量表C?/span>
1 ./word2phrase -train resultbig.txt -output sogouca_phrase.txt -threshold 500 -debug 2 2 ./word2vec -train sogouca_phrase.txt -output vectors_sogouca_phrase.bin -cbow 0 -size 300 -window 10 -negative 0 -hs 1 -sample 1e-3 -threads 12 -binary 1
下面是几个计相似度的例子：
5、参考链?/strong>Q?/span>
1. word2vecQTool for computing continuous distributed representations of wordsQ?a >https://code.google.com/p/word2vec/
2. 用中文把玩Google开源的Deep-Learning目word2vecQ?a >http://www.cnblogs.com/wowarsenal/p/3293586.html
3. 利用word2vec对关键词q行聚类Q?a >http://blog.csdn.net/zhaoxinfan/article/details/11069485
6、后l准备仔l阅ȝ文献Q?/span>
[1] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013.
[2] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013.
[3] Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of NAACL HLT, 2013.
[4] Collobert R, Weston J, Bottou L, et al. Natural language processing (almost) from scratch[J]. The Journal of Machine Learning Research, 2011, 12: 2493-2537.
]]>