GitXplorerGitXplorer
d

sentencepiece

public
20 stars
6 forks
3 issues

Commits

List of commits on branch main.
Unverified
b0891a674dfa235b069756a58fb513599f59ce0b

Bump version to 0.11.2

ddanieldk committed a year ago
Unverified
8ea34a59b1b32dc63b8977b0e8e8d0ea658cd3e5

Disable +crt-static test for Ubuntu

SSystemcluster committed 2 years ago
Unverified
8d357c43309b2ad8f301eb4a4171b16022631964

Update to sentencepiece 0.1.99

SSystemcluster committed 2 years ago
Unverified
85492a8feaca9886df7cafb834c6b71746faa0cb

Add target argument to test invocations

SSystemcluster committed 2 years ago
Unverified
7d9d7ec03e57e88101de593b162da9c9aa584e05

Update test-all script with compatibility for macOS bash

SSystemcluster committed 2 years ago
Unverified
825dd6ea73dc6d933a0fa8eb0b05341c72c91f42

Add workflow test runs for multiple platforms

SSystemcluster committed 2 years ago

README

The README file for this repository.

sentencepiece

This Rust crate is a binding for the sentencepiece unsupervised text tokenizer. The crate documentation is available online.

libsentencepiece dependency

This crate depends on the sentencepiece C++ library. By default, this dependency is treated as follows:

  • If sentencepiece could be found with pkg-config, the crate will link against the library found through pkg-config. Warning: dynamic linking only works correctly with sentencepiece 0.1.95 or later, due to a bug in earlier versions.
  • Otherwise, the crate's build script will do a static build of the sentencepiece library. This requires that cmake is available.

If you wish to override this behavior, the sentencepiece-sys crate offers two features:

  • system: always attempt to link to the sentencepiece library found with pkg-config.
  • static: always do a static build of the sentencepiece library and link against that.