具有子集搜索功能的快速记忆神经网络
rii的Python项目详细描述
可重构倒排索引（RII）：基于IVFPQ的快速记忆高效近似最近邻搜索方法 具有子集搜索功能。
参考：
功能摘要
| The search can be operated for a subset of a database. | Rii remains fast even after many new items are added. |
- 快速高效的神经网络。RII使您能够在10毫秒内运行十亿规模的搜索。
- 您可以在整个数据库的subset上运行搜索
- 即使新添加了许多向量，rii仍然保持快速（即，数据结构可以重新配置）
安装
您可以通过pip安装包。这个库在Linux上与Python3.5+一起工作。
pip install rii
Documentation
用法
基本ann
importriiimportnanopqimportnumpyasnpN,Nt,D=10000,1000,128X=np.random.random((N,D)).astype(np.float32)# 10,000 128-dim vectors to be searchedXt=np.random.random((Nt,D)).astype(np.float32)# 1,000 128-dim vectors for trainingq=np.random.random((D,)).astype(np.float32)# a 128-dim vector# Prepare a PQ/OPQ codec with M=32 sub spacescodec=nanopq.PQ(M=32).fit(vecs=Xt)# Trained using Xt# Instantiate a Rii class with the codece=rii.Rii(fine_quantizer=codec)# Add vectorse.add_configure(vecs=X)# Searchids,dists=e.query(q=q,topk=3)print(ids,dists)# e.g., [7484 8173 1556] [15.06257439 15.38533878 16.16935158]
注意，如果需要的话，可以构造一个pq编解码器并同时实例化rii类。
e=rii.Rii(fine_quantizer=nanopq.PQ(M=32).fit(vecs=Xt))e.add_configure(vecs=X)
此外，您甚至可以通过链接函数将它们写在一行中。
e=rii.Rii(fine_quantizer=nanopq.PQ(M=32).fit(vecs=Xt)).add_configure(vecs=X)
子集搜索
# The search can be conducted over a subset of the databasetarget_ids=np.array([85,132,236,551,694,728,992,1234])# Specified by IDsids,dists=e.query(q=q,topk=3,target_ids=target_ids)print(ids,dists)# e.g., [728 85 132] [14.80522156 15.92787838 16.28690338]
数据添加和重新配置
# Add new vectorsX2=np.random.random((1000,D)).astype(np.float32)e.add(vecs=X2)# Now N is 11000e.query(q=q)# Ok. (0.12 msec / query)# However, if you add quite a lot of vectors, the search might become slower# because the data structure has been optimized for the initial item size (N=10000)X3=np.random.random((1000000,D)).astype(np.float32)e.add(vecs=X3)# A lot. Now N is 1011000e.query(q=q)# Slower (0.96 msec/query)# In such case, run the reconfigure function. That updates the data structuree.reconfigure()e.query(q=q)# Ok. (0.21 msec / query)
通过酸洗输入/输出
importpicklewithopen('rii.pkl','wb')asf:pickle.dump(e,f)withopen('rii.pkl','rb')asf:e_dumped=pickle.load(f)# e_dumped is identical to e
效用函数
# Print the current parameterse.print_params()# Delete all PQ-codes and posting lists. fine_quantizer is kept.e.clear()# You can switch the verbose flage.verbose=False# You can merge two Rii instances if they have the same fine_quantizere1=rii.Rii(fine_quantizer=codec)e2=rii.Rii(fine_quantizer=codec)e1.add_reconfigure(vecs=X1)e2.add_reconfigure(vecs=X2)e1.merge(e2)# Now e1 contains both X1 and X2
Examples
作者
学分
- 标志由@richardbmx（#4）设计
欢迎加入QQ群-->： 979659372
