File tree Expand file tree Collapse file tree
Expand file tree Collapse file tree Original file line number Diff line number Diff line change 7575 - Python 3.9
7676 - Python 3.8
7777 - Python 3.7
78- - Python 3.6
7978 default : 1
8079 validations :
8180 required : true
8584 label : ToCount version
8685 description : Which version of ToCount are you using?
8786 options :
87+ - ToCount 0.2
8888 - ToCount 0.1
8989 default : 0
9090 validations :
Original file line number Diff line number Diff line change @@ -5,6 +5,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
55and this project adheres to [ Semantic Versioning] ( http://semver.org/spec/v2.0.0.html ) .
66
77## [ Unreleased]
8+ ## [ 0.2] - 2025-10-02
89### Added
910- ` TIKTOKEN_R50K.LINEAR_ALL ` model
1011- ` TIKTOKEN_R50K.LINEAR_ENGLISH ` model
@@ -17,5 +18,6 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
1718- ` RULE_BASED.GPT_3_5 ` model
1819
1920
20- [ Unreleased ] : https://github.com/openscilab/tocount/compare/v0.1...dev
21+ [ Unreleased ] : https://github.com/openscilab/tocount/compare/v0.2...dev
22+ [ 0.2 ] : https://github.com/openscilab/tocount/compare/v0.1...v0.2
2123[ 0.1 ] : https://github.com/openscilab/tocount/compare/8385d46...v0.1
Original file line number Diff line number Diff line change @@ -56,9 +56,9 @@ ToCount is a lightweight and extensible Python library for estimating token coun
5656
5757### PyPI
5858- Check [ Python Packaging User Guide] ( https://packaging.python.org/installing/ )
59- - Run ` pip install tocount==0.1 `
59+ - Run ` pip install tocount==0.2 `
6060### Source code
61- - Download [ Version 0.1 ] ( https://github.com/openscilab/tocount/archive/v0.1 .zip ) or [ Latest Source] ( https://github.com/openscilab/tocount/archive/dev.zip )
61+ - Download [ Version 0.2 ] ( https://github.com/openscilab/tocount/archive/v0.2 .zip ) or [ Latest Source] ( https://github.com/openscilab/tocount/archive/dev.zip )
6262- Run ` pip install . `
6363
6464## Models
Original file line number Diff line number Diff line change 44
55| Version | Supported |
66| ------------- | ------------------ |
7- | 0.1 | :white_check_mark : |
8- | < 0.1 | :x : |
7+ | 0.2 | :white_check_mark : |
8+ | < 0.2 | :x : |
99
1010## Reporting a vulnerability
1111
Original file line number Diff line number Diff line change 11{% set name = "tocount" %}
2- {% set version = "0.1 " %}
2+ {% set version = "0.2 " %}
33
44package :
55 name : {{ name|lower }}
Original file line number Diff line number Diff line change 44import sys
55import codecs
66Failed = 0
7- TOCOUNT_VERSION = "0.1 "
7+ TOCOUNT_VERSION = "0.2 "
88
99
1010SETUP_ITEMS = [
Original file line number Diff line number Diff line change @@ -32,14 +32,14 @@ def read_description() -> str:
3232 name = 'tocount' ,
3333 packages = [
3434 'tocount' , ],
35- version = '0.1 ' ,
35+ version = '0.2 ' ,
3636 description = 'ToCount: Lightweight Token Estimator' ,
3737 long_description = read_description (),
3838 long_description_content_type = 'text/markdown' ,
3939 author = 'ToCount Development Team' ,
4040 author_email = 'tocount@openscilab.com' ,
4141 url = 'https://github.com/openscilab/tocount' ,
42- download_url = 'https://github.com/openscilab/tocount/tarball/v0.1 ' ,
42+ download_url = 'https://github.com/openscilab/tocount/tarball/v0.2 ' ,
4343 keywords = "token tokenizer estimation llm ml nlp" ,
4444 project_urls = {
4545 'Source' : 'https://github.com/openscilab/tocount' ,
Original file line number Diff line number Diff line change 11# -*- coding: utf-8 -*-
22"""Tocount parameters and constants."""
33
4- TOCOUNT_VERSION = "0.1 "
4+ TOCOUNT_VERSION = "0.2 "
55
66INVALID_TEXT_ESTIMATOR_MESSAGE = "Invalid value. `estimator` must be an instance of TextEstimator enum."
77INVALID_TEXT_MESSAGE = "Invalid value. `text` must be a string."
You can’t perform that action at this time.
0 commit comments