Skip to content

Commit 92e5990

Browse files
youngdashuyoungdashu
committed
Migrate conda to pip (#21)
1 parent 12151cf commit 92e5990

15 files changed

Lines changed: 319 additions & 349 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 14 additions & 61 deletions
Original file line numberDiff line numberDiff line change
@@ -6,74 +6,27 @@ on:
66
jobs:
77
test:
88
runs-on: ubuntu-latest
9-
defaults:
10-
run:
11-
shell: bash -el {0}
9+
strategy:
10+
fail-fast: false
11+
matrix:
12+
python-version: ["3.10", "3.11", "3.12"]
1213

1314
steps:
1415
- name: Checkout repository
1516
uses: actions/checkout@v4
1617

17-
- name: Generate cache key
18-
id: cache-key
19-
run: |
20-
# Create a hash from the environment files and PyTorch CPU install string
21-
ENV_HASH=$(cat fridata_env_conda.yml requirements-fridata.txt | sha256sum | cut -d' ' -f1)
22-
PYTORCH_SPEC="pip-torch-torchvision-cpu-whl"
23-
CACHE_KEY="micromamba-${{ runner.os }}-${ENV_HASH}-${PYTORCH_SPEC}"
24-
echo "key=${CACHE_KEY}" >> $GITHUB_OUTPUT
25-
26-
- name: Restore micromamba environment cache
27-
id: cache-restore
28-
uses: actions/cache/restore@v4
29-
with:
30-
path: |
31-
~/micromamba/envs
32-
~/micromamba-bin
33-
key: ${{ steps.cache-key.outputs.key }}
34-
35-
- name: Setup micromamba (cache miss)
36-
if: steps.cache-restore.outputs.cache-hit != 'true'
37-
uses: mamba-org/setup-micromamba@v2
18+
- name: Set up Python ${{ matrix.python-version }}
19+
uses: actions/setup-python@v5
3820
with:
39-
environment-file: fridata_env_conda.yml
40-
environment-name: fridata_env
41-
cache-environment: false
42-
cache-downloads: false
43-
micromamba-binary-path: ~/micromamba-bin/micromamba
44-
micromamba-root-path: ~/micromamba
45-
init-shell: bash
21+
python-version: ${{ matrix.python-version }}
22+
cache: pip
4623

47-
- name: Initialize micromamba from cache (cache hit)
48-
if: steps.cache-restore.outputs.cache-hit == 'true'
49-
run: |
50-
# Make micromamba available to subsequent steps without shell hooks
51-
echo "$HOME/micromamba-bin" >> "$GITHUB_PATH"
52-
echo "MAMBA_ROOT_PREFIX=$HOME/micromamba" >> "$GITHUB_ENV"
24+
- name: Install CPU-only PyTorch
25+
# CPU wheels keep CI fast and avoid pulling multi-GB CUDA builds.
26+
run: pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
5327

54-
- name: Install pip requirements and CPU-only PyTorch
55-
if: steps.cache-restore.outputs.cache-hit != 'true'
56-
run: |
57-
# Pip packages are installed separately so read-only checkouts and CI caches stay reliable.
58-
~/micromamba-bin/micromamba run -n fridata_env python -m pip install --upgrade pip
59-
~/micromamba-bin/micromamba run -n fridata_env python -m pip install -r requirements-fridata.txt
60-
# Avoid a second micromamba/libmamba solve mixing pytorch + conda-forge on an
61-
# already-large env (can abort with libsolv solver_addrule assertion, exit 134).
62-
# CPU wheels from PyTorch are the usual CI approach. Install torchvision from the
63-
# same index as torch so C++/dispatch ops (e.g. torchvision::nms) match conda-forge
64-
# transformers alone pulls torchvision that can mismatch a pip-only torch build.
65-
~/micromamba-bin/micromamba run -n fridata_env python -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
66-
~/micromamba-bin/micromamba run -n fridata_env python -m pip install esm
67-
68-
- name: Save micromamba environment cache
69-
if: steps.cache-restore.outputs.cache-hit != 'true'
70-
uses: actions/cache/save@v4
71-
with:
72-
path: |
73-
~/micromamba/envs
74-
~/micromamba-bin
75-
key: ${{ steps.cache-key.outputs.key }}
28+
- name: Install FRIdata with embeddings + test extras
29+
run: pip install -e ".[embeddings,test]"
7630

7731
- name: Run tests
78-
run: |
79-
~/micromamba-bin/micromamba run -n fridata_env pytest ./tests
32+
run: pytest ./tests

‎.github/workflows/publish.yml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
name: Publish to PyPI
2+
3+
on:
4+
release:
5+
types: [published]
6+
7+
jobs:
8+
build:
9+
runs-on: ubuntu-latest
10+
steps:
11+
- uses: actions/checkout@v4
12+
13+
- uses: actions/setup-python@v5
14+
with:
15+
python-version: "3.11"
16+
17+
- name: Build sdist and wheel
18+
run: |
19+
python -m pip install --upgrade build
20+
python -m build
21+
22+
- name: Check distributions
23+
run: |
24+
python -m pip install --upgrade twine
25+
twine check dist/*
26+
27+
- uses: actions/upload-artifact@v4
28+
with:
29+
name: dist
30+
path: dist/
31+
32+
publish:
33+
needs: build
34+
runs-on: ubuntu-latest
35+
environment: pypi
36+
permissions:
37+
id-token: write
38+
steps:
39+
- uses: actions/download-artifact@v4
40+
with:
41+
name: dist
42+
path: dist/
43+
44+
- name: Publish to PyPI
45+
uses: pypa/gh-action-pypi-publish@release/v1

‎.gitignore‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,4 +36,5 @@ build
3636

3737
reports/
3838

39-
.cursor/**/*
39+
.cursor/**/*
40+
.DS_Store

‎MANIFEST.in‎

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,17 @@
1-
# include everything in the sdist (be careful — this will bundle all files)
2-
global-include *
1+
# Files to ship in the source distribution (sdist), beyond the packaged Python
2+
# modules. Keep this tight so releases stay small — never bundle test data.
3+
include LICENSE
4+
include README.md
5+
include example.config.json
6+
recursive-include scripts *.sh
37

4-
recursive-exclude tests *
5-
recursive-exclude .github *
6-
recursive-exclude .pytest_cache *
7-
recursive-exclude build *
8-
recursive-exclude *.egg-info *
9-
recursive-exclude dist *
8+
# Exclude everything that should not ship in a release.
9+
prune tests
10+
prune docs
11+
prune reports
12+
prune .github
13+
prune .gitnexus
14+
prune .cursor
15+
recursive-exclude * __pycache__
16+
recursive-exclude * *.py[cod]
17+
global-exclude *.h5 *.cif *.idx *.log .DS_Store

‎README.md‎

Lines changed: 63 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -22,75 +22,91 @@ Full documentation may be found [here](https://tomasz-lab.github.io/FRIdata/).
2222

2323
The repository is currently under active development. If you run into installation problems, find a bug, or would like to propose an improvement, please raise an issue or write directly to p.szczerbiak[at]sanoscience.org.
2424

25-
![FRIdata pipeline](diagram.png)
25+
![FRIdata pipeline](https://raw.githubusercontent.com/Tomasz-Lab/FRIdata/main/diagram.png)
2626

27-
## Installation and activation
27+
Generate sequences, coordinates, distograms, and embeddings from protein structures at scale. Supports PDB, AFDB, ESMatlas, and local/custom inputs. Full API reference: [docs/index.html](docs/index.html).
2828

29-
1. Download the repo
29+
## How FRIdata works
3030

31-
```
32-
git clone https://github.com/Tomasz-Lab/FRIdata.git
33-
cd FRIdata
34-
```
31+
Every dataset is defined by a **database type** (`-d`, `--db`) and a **collection type** (`-c`, `--collection`), plus optional `--proteome` and `--version`. Together they determine the dataset name (for example `AFDB-subset--test`) and how structures are resolved.
3532

36-
2. [Install miniconda](https://www.anaconda.com/docs/getting-started/miniconda/install)
33+
Use `-t` / `--type` to choose what to generate in a run: `sequences`, `coordinates`, `distograms`, `embeddings`, or `all`.
3734

38-
3. Install mamba
35+
### Database type (`-d`, `--db`)
3936

40-
```
41-
## prioritize 'conda-forge' channel
42-
conda config --add channels conda-forge
37+
Where structures come from.
4338

44-
## update existing packages to use 'conda-forge' channel
45-
conda update -n base --all
39+
| Value | Meaning |
40+
|-------|---------|
41+
| `PDB` | RCSB PDB structures (download by ID) |
42+
| `AFDB` | AlphaFold Database |
43+
| `ESMatlas` | ESM Atlas |
44+
| `other` | Local or custom files via `--input-path` or archives |
4645

47-
## install 'mamba'
48-
conda install -n base mamba
49-
```
46+
### Collection type (`-c`, `--collection`)
5047

51-
4. Create the environment (recommended)
48+
How much of that source to include.
5249

53-
Use the setup script to create the conda environment, install pip dependencies, and install a PyTorch build matched to your GPU driver:
50+
| Value | Meaning |
51+
|-------|---------|
52+
| `all` | Full database collection |
53+
| `part` | AFDB proteome partition (requires `--proteome`; foldcomp-based) |
54+
| `clust` | AFDB cluster partition (requires `--proteome`; foldcomp-based) |
55+
| `subset` | User-defined ID list via `-i` / `--ids` (optionally `--input-path` for local structures) |
5456

55-
```
56-
./scripts/setup_env.sh
57-
```
57+
## Installation
5858

59-
Use a custom environment name for a separate install:
59+
FRIdata is a pure-pip project (Python >= 3.10). No conda/mamba required.
6060

61-
```
62-
./scripts/setup_env.sh -n fridata_gpu_verify
63-
```
61+
The core install covers `sequences`, `coordinates` and `distograms`. Embedding
62+
generation needs the heavier `torch`/`esm`/`transformers` stack, which lives in
63+
an optional `embeddings` extra so the default install stays small.
6464

65-
For CPU-only systems:
65+
### Quick start (recommended)
66+
67+
The setup script creates a virtualenv, installs FRIdata with its dev extras, and
68+
installs a PyTorch build matched to your GPU driver:
6669

6770
```
68-
./scripts/setup_env.sh --cpu
71+
git clone https://github.com/Tomasz-Lab/FRIdata.git
72+
cd FRIdata
73+
./scripts/setup_env.sh # GPU (auto-detected); use --cpu for CPU-only
74+
source .venv/bin/activate
6975
```
7076

71-
5. Activate the environment
77+
Options: `--cpu` (CPU-only PyTorch), `--skip-pytorch` (core install, no
78+
embeddings), `-p/--path DIR` (virtualenv location, default `.venv`).
79+
80+
### Manual installation
7281

7382
```
74-
# Choose your shell type. Could be one of these: {bash,cmd.exe,dash,fish,nu,posix,powershell,tcsh,xonsh,zsh}
75-
eval "$(mamba shell hook --shell <replace with shell type>)"
76-
mamba activate fridata_env
77-
```
83+
python -m venv .venv
84+
source .venv/bin/activate
85+
python -m pip install --upgrade pip
7886
79-
### Manual installation
87+
# Core only (sequences / coordinates / distograms):
88+
pip install -e .
89+
90+
# ...or with embedding support (torch/esm/transformers from PyPI):
91+
pip install -e ".[embeddings]"
92+
```
8093

81-
If you prefer to run the steps yourself:
94+
On Linux the default PyPI `torch` wheel is CUDA-enabled. For a specific CUDA
95+
version or CPU-only wheels, use the helper after installing:
8296

8397
```
84-
mamba env create -f fridata_env_conda.yml -n fridata_env
85-
mamba run -n fridata_env python -m pip install -r requirements-fridata.txt
86-
./scripts/install_pytorch.sh
98+
./scripts/install_pytorch.sh # auto-detect CUDA from the driver
99+
./scripts/install_pytorch.sh --cpu # force CPU-only build
87100
```
88101

89-
Pip dependencies live in `requirements-fridata.txt` instead of `fridata_env_conda.yml` so environment creation works from read-only repository checkouts.
102+
### Install from PyPI
90103

91-
### Troubleshooting
104+
Once released, FRIdata can be installed without cloning the repo:
92105

93-
If you see `Error opening for writing ".../mambaf..."` while creating the environment, libmamba is trying to write a temporary pip requirements file next to the environment YAML. Use `./scripts/setup_env.sh` instead, or ensure the directory containing `fridata_env_conda.yml` is writable.
106+
```
107+
pip install fridata # core
108+
pip install "fridata[embeddings]" # with embedding support
109+
```
94110

95111
### Running tests
96112

@@ -102,7 +118,7 @@ pytest ./tests
102118

103119
### Running on AFDB structures locally
104120

105-
Requires having a directory with AFDB structures and a text file containing list of AFDB IDs with `\n` delimeter. Assuming all steps from `Installation and activation` succeded
121+
Requires having a directory with AFDB structures and a text file containing list of AFDB IDs with `\n` delimeter. Assuming all steps from [Installation](#installation) succeeded
106122

107123
```
108124
FRIDATA_PATH="<repository path>"
@@ -131,7 +147,7 @@ For subset runs with `--input-path`, new datasets store canonical keys as `{line
131147

132148
### Running as a CLI tool
133149

134-
Assuming all `Installation and activation` steps succeeded.
150+
Assuming all [Installation](#installation) steps succeeded.
135151

136152
0. Go into `FRIdata` directory
137153

@@ -169,8 +185,8 @@ Prerequisites:
169185
- `LAUNCH_WORKER_SLURM_PATH`: path to launch_worker_slurm.sh, defaults to `$DEEPFRI_PATH/FRIdata/scripts/hpc/launch_workers_slurm.sh`
170186
- `MEMORY_LIMIT`: memory limit per Dask worker, defaults to `288GiB`
171187
- `IP_INTERFACE`: network unix interface, where dask workers are connected. Defaults to `ens1f0`
172-
- `CONDA_ENV_PATH`: path to conda environment, defaults to `$DEEPFRI_PATH/conda_dev`
173-
- Have installed module miniconda3
188+
- `VENV_PATH`: path to the FRIdata virtualenv, defaults to `$DEEPFRI_PATH/.venv`
189+
- Have a Python module available (`module avail python` — the scripts try `python`/`python3`; adjust the candidate list in `scripts/hpc/*.sh` if your cluster names it differently)
174190
- Have installed module gcc
175191

176192
Steps:
@@ -188,10 +204,10 @@ cd FRIdata
188204
chmod u+x -R scripts/hpc/cpu
189205
```
190206

191-
3. Run `initialize_slurm.sh`. As an argument put the path into directory, where `.conda` directory should be installed and specify `--cpu` flag if the script is run on CPU cluster.
207+
3. Run `initialize_slurm.sh` to create the virtualenv (at `VENV_PATH`, default `$DEEPFRI_PATH/.venv`) and install dependencies. Add the `--cpu` flag on CPU clusters.
192208

193209
```
194-
./scripts/hpc/initialize_slurm.sh <path to .conda> [--cpu]
210+
./scripts/hpc/initialize_slurm.sh [--cpu]
195211
```
196212

197213
4. Schedule sbatch script into the HPC with all the args specified. Operations to be chosen are: `sequences`, `coordinates`, `embeddings`

0 commit comments

Comments
 (0)