Skip to content

Commit c6ba8c0

Browse files
mielvdsmielvds
committed
Add README in markdown
1 parent c3c6f4c commit c6ba8c0

1 file changed

Lines changed: 319 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 319 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,319 @@
1+
# PyLD
2+
3+
## Introduction
4+
5+
This library is an implementation of the JSON-LD specification in [Python](https://www.python.org/).
6+
7+
JSON, as specified in [RFC7159](http://tools.ietf.org/html/rfc7159), is a simple language for representing objects on the Web. Linked Data is a way of describing content across different documents or Web sites. Web resources are described using IRIs, and typically are dereferencable entities that may be used to find more information, creating a "Web of Knowledge". [JSON-LD](https://json-ld.org/) is intended to be a simple publishing method for expressing not only Linked Data in JSON, but for adding semantics to existing JSON.
8+
9+
JSON-LD is designed as a light-weight syntax that can be used to express Linked Data. It is primarily intended to be a way to express Linked Data in JavaScript and other Web-based programming environments. It is also useful when building interoperable Web Services and when storing Linked Data in JSON-based document storage engines. It is practical and designed to be as simple as possible, utilizing the large number of JSON parsers and existing code that is in use today. It is designed to be able to express key-value pairs, RDF data, [RDFa](http://www.w3.org/TR/rdfa-core/) data, [Microformats](http://microformats.org/) data, and [Microdata](http://www.w3.org/TR/microdata/). That is, it supports every major Web-based structured data model in use today.
10+
11+
The syntax does not require many applications to change their JSON, but easily add meaning by adding context in a way that is either in-band or out-of-band. The syntax is designed to not disturb already deployed systems running on JSON, but provide a smooth migration path from JSON to JSON with added semantics. Finally, the format is intended to be fast to parse, fast to generate, stream-based and document-based processing compatible, and require a very small memory footprint in order to operate.
12+
13+
## Conformance
14+
15+
This library aims to conform with the following W3C Recommendations:
16+
17+
| Standard | Status |
18+
| :--- | :--- |
19+
| [JSON-LD 1.1](https://www.w3.org/TR/json-ld11/) | W3C Recommendation |
20+
| [JSON-LD 1.1 Processing Algorithms and API](https://www.w3.org/TR/json-ld11-api/) | W3C Recommendation |
21+
| [JSON-LD 1.1 Framing](https://www.w3.org/TR/json-ld11-framing/) | W3C Recommendation |
22+
| [RDF Dataset Canonicalization](https://www.w3.org/TR/rdf-canon/) | W3C Recommendation |
23+
24+
25+
The [`test runner`](https://github.com/digitalbazaar/pyld/blob/master/tests/runtests.py) is often updated to note or skip newer tests that are not yet supported.
26+
27+
## Requirements
28+
29+
* Python (3.10 or later)
30+
* [Requests](http://docs.python-requests.org/) (optional)
31+
* [aiohttp](https://aiohttp.readthedocs.io/) (optional)
32+
33+
## Installation
34+
35+
PyLD can be installed with a [pip](http://www.pip-installer.org/) [package](https://pypi.org/project/PyLD/):
36+
37+
```bash
38+
pip install PyLD
39+
```
40+
41+
Defining a dependency on pyld will not pull in [Requests](http://docs.python-requests.org/) or [aiohttp](https://aiohttp.readthedocs.io/). If you need one of these for a [Document Loader]() then either depend on the desired external library directly or define the requirement as `PyLD[requests]` or `PyLD[aiohttp]`.
42+
43+
## Quick Examples
44+
45+
```python
46+
from pyld import jsonld
47+
import json
48+
49+
doc = {
50+
"http://schema.org/name": "Manu Sporny",
51+
"http://schema.org/url": {"@id": "http://manu.sporny.org/"},
52+
"http://schema.org/image": {"@id": "http://manu.sporny.org/images/manu.png"}
53+
}
54+
55+
context = {
56+
"name": "http://schema.org/name",
57+
"homepage": {"@id": "http://schema.org/url", "@type": "@id"},
58+
"image": {"@id": "http://schema.org/image", "@type": "@id"}
59+
}
60+
61+
# compact a document according to a particular context
62+
# see: https://json-ld.org/spec/latest/json-ld/#compacted-document-form
63+
compacted = jsonld.compact(doc, context)
64+
65+
print(json.dumps(compacted, indent=2))
66+
# Output:
67+
# {
68+
# "@context": {...},
69+
# "image": "http://manu.sporny.org/images/manu.png",
70+
# "homepage": "http://manu.sporny.org/",
71+
# "name": "Manu Sporny"
72+
# }
73+
74+
# compact using URLs
75+
jsonld.compact('http://example.org/doc', 'http://example.org/context')
76+
77+
# expand a document, removing its context
78+
# see: https://json-ld.org/spec/latest/json-ld/#expanded-document-form
79+
expanded = jsonld.expand(compacted)
80+
81+
print(json.dumps(expanded, indent=2))
82+
# Output:
83+
# [{
84+
# "http://schema.org/image": [{"@id": "http://manu.sporny.org/images/manu.png"}],
85+
# "http://schema.org/name": [{"@value": "Manu Sporny"}],
86+
# "http://schema.org/url": [{"@id": "http://manu.sporny.org/"}]
87+
# }]
88+
89+
# expand using URLs
90+
jsonld.expand('http://example.org/doc')
91+
92+
# flatten a document
93+
# see: https://json-ld.org/spec/latest/json-ld/#flattened-document-form
94+
flattened = jsonld.flatten(doc)
95+
# all deep-level trees flattened to the top-level
96+
97+
# frame a document
98+
# see: https://json-ld.org/spec/latest/json-ld-framing/#introduction
99+
framed = jsonld.frame(doc, frame)
100+
# document transformed into a particular tree structure per the given frame
101+
102+
# normalize a document using the RDF Dataset Normalization Algorithm
103+
# (URDNA2015), see: https://www.w3.org/TR/rdf-canon/
104+
normalized = jsonld.normalize(
105+
doc, {'algorithm': 'URDNA2015', 'format': 'application/n-quads'})
106+
# normalized is a string that is a canonical representation of the document
107+
# that can be used for hashing, comparison, etc.
108+
```
109+
110+
## Document Loader
111+
112+
The default document loader for PyLD uses [Requests](http://docs.python-requests.org/). In a production environment you may want to setup a custom loader that, at a minimum, sets a timeout value. You can also force requests to use https, set client certs, disable verification, or set other Requests parameters.
113+
114+
```python
115+
jsonld.set_document_loader(jsonld.requests_document_loader(timeout=...))
116+
```
117+
118+
The factory remains the compatibility API, and the concrete class is also available when class-based construction is preferred:
119+
120+
```python
121+
from pyld import RequestsDocumentLoader
122+
123+
jsonld.set_document_loader(RequestsDocumentLoader(timeout=...))
124+
```
125+
126+
An asynchronous document loader using aiohttp is also available. Please note that this document loader limits asynchronicity to fetching documents only. The processing loops remain synchronous.
127+
128+
```python
129+
jsonld.set_document_loader(jsonld.aiohttp_document_loader(timeout=...))
130+
```
131+
132+
The concrete aiohttp loader class is available from `pyld` as well:
133+
134+
```python
135+
from pyld import AioHttpDocumentLoader
136+
137+
jsonld.set_document_loader(AioHttpDocumentLoader(timeout=...))
138+
```
139+
140+
When no document loader is specified, the default loader is set to [Requests](http://docs.python-requests.org/). If Requests is not available, the loader is set to aiohttp. The fallback document loader is a dummy document loader that raises an exception on every invocation.
141+
142+
## Frozen Document Loader
143+
144+
For air-gapped runs, reproducible builds, and security-hardened deployments that must not perform any remote context fetches at all, PyLD ships `FrozenDocumentLoader`: a class-based loader that serves only the URLs in its `documents` allowlist and refuses everything else with `JsonLdError(code='loading document failed')`.
145+
146+
Instantiating with no arguments serves the curated `BUNDLED_CONTEXTS` set (ActivityStreams, DID v1, Verifiable Credentials v1 and v2, Linked Data Security v1/v2, Ed25519-2020, and JWS-2020). To extend the bundle with additional pre-vetted contexts, pass a merged mapping:
147+
148+
```python
149+
from pyld import jsonld, FrozenDocumentLoader, BUNDLED_CONTEXTS
150+
151+
loader = FrozenDocumentLoader(documents=dict(
152+
BUNDLED_CONTEXTS,
153+
**{'https://example.com/my-ctx': Path('contexts/my-ctx.jsonld')},
154+
))
155+
jsonld.expand(doc, options={'documentLoader': loader})
156+
```
157+
158+
This honors the W3C *JSON-LD Best Practices* recommendation that clients SHOULD attempt to use a locally cached version of contexts (see `§ Cache JSON-LD Contexts <https://w3c.github.io/json-ld-bp/#cache-json-ld-contexts>`_).
159+
Refresh the bundled copies with `make download-bundled-contexts`.
160+
161+
## Customizing the ContextLoader
162+
163+
You can customize the way contexts are loaded and cached by passing an instance of `ContextResolver`. The following example implements a loader with a prefilled custom document cache and uses a custom LRU cache for resolved contexts:
164+
165+
```python
166+
from pyld.jsonld import compact, expand, set_document_loader, ContextResolver
167+
import json
168+
from cachetools import LRUCache
169+
170+
# Load the Linked Art context from file-system
171+
fh = open('linked-art.json')
172+
js = json.load(fh)
173+
fh.close()
174+
175+
# Add to document cache
176+
docCache = {
177+
"https://linked.art/ns/v1/linked-art.json": {
178+
"contextUrl": None,
179+
"documentUrl": "https://linked.art/ns/v1/linked-art.json",
180+
"document": js
181+
}
182+
}
183+
184+
# Custom loader that uses the document cache
185+
def load_document_and_cache(url, options={}):
186+
if url in docCache:
187+
return docCache[url]
188+
doc = {"contextUrl": None, "documentUrl": url, "document": ""}
189+
resp = requests.get(url)
190+
doc["document"] = resp.json()
191+
docCache[url] = doc
192+
return doc
193+
194+
# Set the custom loader as global document loader
195+
set_document_loader(load_document_and_cache)
196+
# Create custom context resolver with custom LRU cache and custom loader
197+
resolved_context_cache = LRUCache(maxsize=1000)
198+
resolver = ContextResolver(resolved_context_cache, load_document_and_cache)
199+
200+
# Expand JSON-LD document using custom context resolver
201+
input = {"@context":"https://linked.art/ns/v1/linked-art.json", "id": "tag:foo", "type": "Person"}
202+
output = expand(input, options={'contextResolver': resolver})
203+
```
204+
205+
It is also possible to change the maximum number of times that the loader recursively fetches contexts, by passing the `max_context_urls` parameter:
206+
207+
```python
208+
resolver = ContextResolver(resolved_context_cache, load_document_and_cache, max_context_urls=20)
209+
# Or you can do...
210+
# resolver = ContextResolver(resolved_context_cache, load_document_and_cache)
211+
# resolver.max_context_urls = 20
212+
output = expand(input, options={'contextResolver': resolver})
213+
```
214+
215+
## Handling ignored properties during JSON-LD expansion
216+
217+
If a property in a JSON-LD document does not map to an absolute IRI then it is ignored. You can customize this behaviour by passing a customizable handler to `on_property_dropped` parameter of `jsonld.expand()`.
218+
219+
For example, you can introduce a strict mode by raising a ValueError on every dropped property:
220+
221+
```python
222+
def raise_this(value):
223+
raise ValueError(value)
224+
225+
jsonld.expand(doc, None, on_property_dropped=raise_this)
226+
```
227+
228+
## Commercial Support
229+
230+
Commercial support for this library is available upon request from [`Digital Bazaar`](mailto:support@digitalbazaar.com).
231+
232+
## Source
233+
234+
The source code for the Python implementation of the JSON-LD API is available at:
235+
236+
[https://github.com/digitalbazaar/pyld](https://github.com/digitalbazaar/pyld)
237+
238+
## Tests
239+
240+
This library includes a sample testing utility which may be used to verify that changes to the processor maintain the correct output.
241+
242+
To run the sample tests you will need to get the test suite files, which by default, are stored in the `specifications/` folder.
243+
The test suites can be obtained by either using git submodules or by cloning them manually.
244+
245+
### Using git submodules
246+
247+
The test suites are included as git submodules to ensure versions are in sync.
248+
When cloning the repository, use the `--recurse-submodules` flag to automatically clone the submodules.
249+
If you have cloned the repository without the submodules, you can initialize them with the following commands:
250+
251+
```bash
252+
git submodule init
253+
git submodule update
254+
```
255+
256+
### Cloning manually
257+
258+
You can also avoid using git submodules by manually cloning the `json-ld-api`, `json-ld-framing`, and `normalization` repositories hosted on GitHub using the following commands:
259+
260+
```bash
261+
git clone https://github.com/w3c/json-ld-api ./specifications/json-ld-api
262+
git clone https://github.com/w3c/json-ld-framing ./specifications/json-ld-framing
263+
git clone https://github.com/json-ld/normalization ./specifications/normalization
264+
```
265+
266+
Note that you can clone these repositories into any location you wish; however, if you do not clone them into the default `specifications/` folder, you will need to provide the paths to the test runner as arguments when running the tests, as explained below.
267+
268+
### Running the sample test suites and unit tests using pytest
269+
270+
If the suites repositories are available in the `specifications/` folder of the PyLD source directory, then all unittests, including the sample test suites, can be run with `pytest`:
271+
272+
```bash
273+
pytest
274+
```
275+
276+
If you wish to store the test suites in a different location than the default `specifications/` folder, or you want to test individual manifest `.jsonld` files or directories containing a `manifest.jsonld`, then you can supply these files or directories as arguments:
277+
278+
```bash
279+
# use: pytest --tests=TEST_PATH [--tests=TEST_PATH...]
280+
pytest --tests=./specifications/json-ld-api/tests
281+
```
282+
283+
The test runner supports different document loaders by setting `--loader requests` or `--loader aiohttp`. The default document loader is set to [Requests](http://docs.python-requests.org/).
284+
285+
```bash
286+
pytest --loader=requests --tests=./specifications/json-ld-api/tests
287+
```
288+
289+
An EARL report can be generated using the `--earl` option.
290+
291+
```bash
292+
pytest --earl=./earl-report.json
293+
```
294+
295+
### Running the sample test suites using the original test runner
296+
297+
You can also run the JSON-LD test suites using the original test runner script provided:
298+
299+
```bash
300+
python tests/runtests.py
301+
```
302+
303+
If you wish to store the test suites in a different location than the default `specifications/` folder, or you want to test individual manifest `.jsonld` files or directories containing a `manifest.jsonld`, then you can supply these files or directories as arguments:
304+
305+
```bash
306+
python tests/runtests.py TEST_PATH [TEST_PATH...]
307+
```
308+
309+
The test runner supports different document loaders by setting `-l requests` or `-l aiohttp`. The default document loader is set to [Requests](http://docs.python-requests.org/).
310+
311+
```bash
312+
python tests/runtests.py -l requests ./specifications/json-ld-api/tests
313+
```
314+
315+
An EARL report can be generated using the `-e` or `--earl` option.
316+
317+
```bash
318+
python tests/runtests.py -e ./earl-report.json
319+
```

0 commit comments

Comments
 (0)