Skip to content

Commit dbad94a

Browse files
mikelgg93mikelgg93
authored
Merge pull request #87 feat(docs): add support for bino + dual monocular gaze data
feat(docs): add support for bino + dual monocular gaze data
2 parents a5b6226 + 813d6da commit dbad94a

6 files changed

Lines changed: 243 additions & 4 deletions

File tree

‎docs/assets/scene_dual.webp‎

93.7 KB
Loading

‎docs/guides/under-the-hood.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -204,6 +204,14 @@ Gaze data is encoded in network byte order (big-endian) and consists of:
204204
3. `worn` - Boolean indicating whether the user is wearing the device. The value is
205205
encoded as an unsigned 8-bit integer as either `255` (device is being worn) or `0` (device is _not_ being worn).
206206

207+
**Dual Monocular Gaze Data in Neon**
208+
Starting from Neon Companion app version +2.9.31, both binocular and monocular gaze data are estimated simultaneously in Neon. In this case, the following additional parameters are included:
209+
210+
4. `mono_left_x` - Horizontal component of the left eye's monocular gaze location in pixels within the scene camera’s coordinate system. The value is encoded as a 32-bit float.
211+
5. `mono_left_y` - Vertical component of the left eye's monocular gaze location in pixels within the scene camera’s coordinate system. The value is encoded as a 32-bit float.
212+
6. `mono_right_x` - Horizontal component of the right eye's monocular gaze location in pixels within the scene camera’s coordinate system. The value is encoded as a 32-bit float.
213+
7. `mono_right_y` - Vertical component of the right eye's monocular gaze location in pixels within the scene camera’s coordinate system. The value is encoded as a 32-bit float.
214+
207215
**Eye State Data (Optional)**
208216

209217
If eye state computation is enabled (not available for Pupil Invisible), additional parameters are included(all encoded as a 32-bit float):
@@ -226,6 +234,9 @@ If eye state computation is enabled (not available for Pupil Invisible), additio
226234

227235
Each RTP packet contains one gaze datum. The payload length varies:
228236

237+
??? quote "Have a look at RTSPGazeStreamer.receive for payload unpacking details"
238+
::: pupil_labs.realtime_api.streaming.gaze.RTSPGazeStreamer.receive
239+
229240
- **21 bytes**: When only gaze data is included. To unpack - (!ffB)
230241
- **77 bytes**: When both gaze and eye state data are included. To unpack - (!ffBffffffffffffff)
231242

‎docs/methods/async/streaming/gaze.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ This function can return different types of gaze data ([GazeDataType][pupil_labs
66

77
- [GazeData][pupil_labs.realtime_api.streaming.gaze.GazeData] object for Pupil Invisible or Neon without "Compute eye state" enabled.
88
- [EyestateGazeData][pupil_labs.realtime_api.streaming.gaze.EyestateGazeData] or [EyestateEyelidGazeData][pupil_labs.realtime_api.streaming.gaze.EyestateEyelidGazeData] for Neon with "Compute eye state" enabled, depending on the version of the Neon Companion app.
9+
- Starting from Neon Companion app version +2.9.31, both binocular and monocular gaze data are exposed, it can return [BinoAndDualMonoGazeData][pupil_labs.realtime_api.streaming.gaze.BinoAndDualMonoGazeData] or [EyestateEyelidDualMonoGazeData][pupil_labs.realtime_api.streaming.gaze.EyestateEyelidDualMonoGazeData].
910

1011
See below samples for each type of gaze data.
1112

@@ -20,6 +21,30 @@ See below samples for each type of gaze data.
2021
)
2122
```
2223

24+
=== "Gaze data (Dual Mono)"
25+
26+
<!-- badge:product Neon -->
27+
<!-- badge:companion +2.9.31 -->
28+
<!-- badge:version +1.8.0 -->
29+
30+
```py linenums="0"
31+
BinoAndDualMonoGazeData(
32+
x=848.9487915039062,
33+
y=474.46356201171875,
34+
worn=True,
35+
mono_left_x=895.11083984375,
36+
mono_left_y=477.2065734863281,
37+
mono_right_x=800.8253173828125,
38+
mono_right_y=469.348388671875,
39+
timestamp_unix_seconds=1772109976.1031907
40+
)
41+
```
42+
43+
This method exposes gaze data for Neon in Dual Mono mode, which includes binocular and monocular gaze data.
44+
??? quote "BinoAndDualMonoGazeData"
45+
46+
::: pupil_labs.realtime_api.streaming.gaze.BinoAndDualMonoGazeData
47+
2348
=== "With Eye State"
2449

2550
<!-- badge:product Neon -->
@@ -86,6 +111,50 @@ See below samples for each type of gaze data.
86111
```
87112
This method also provides [eye openness](https://docs.pupil-labs.com/neon/data-collection/data-streams/#eye-openness) data.
88113

114+
=== "Dual Mono and Eye State & Eye Lid"
115+
116+
<!-- badge:product Neon -->
117+
<!-- badge:companion +2.9.31 -->
118+
<!-- badge:version +1.8.0 -->
119+
120+
```py linenums="0"
121+
EyestateEyelidDualMonoGazeData(
122+
x=852.3287353515625,
123+
y=517.3056030273438,
124+
worn=False,
125+
pupil_diameter_left=4.567640781402588,
126+
eyeball_center_left_x=-32.25,
127+
eyeball_center_left_y=18.6875,
128+
eyeball_center_left_z=-44.9375,
129+
optical_axis_left_x=0.20407231152057648,
130+
optical_axis_left_y=-0.020999005064368248,
131+
optical_axis_left_z=0.9787306189537048,
132+
pupil_diameter_right=3.035294771194458,
133+
eyeball_center_right_x=30.15625,
134+
eyeball_center_right_y=14.375,
135+
eyeball_center_right_z=-43.3125,
136+
optical_axis_right_x=-0.08533422648906708,
137+
optical_axis_right_y=-0.019046271219849586,
138+
optical_axis_right_z=0.9961703419685364,
139+
eyelid_angle_top_left=-0.63037109375,
140+
eyelid_angle_bottom_left=-1.138671875,
141+
eyelid_aperture_left=6.09653902053833,
142+
eyelid_angle_top_right=-0.06329345703125,
143+
eyelid_angle_bottom_right=0.1409912109375,
144+
eyelid_aperture_right=2.228158473968506,
145+
gaze_mono_left_x=864.9857177734375,
146+
gaze_mono_left_y=526.5974731445312,
147+
gaze_mono_right_x=834.6492309570312,
148+
gaze_mono_right_y=481.4454040527344,
149+
timestamp_unix_seconds=1772108289.6118817
150+
)
151+
```
152+
This method exposes binocular and monocular gaze data, [pupil diameter](https://docs.pupil-labs.com/neon/data-collection/data-streams/#pupil-diameters), and [eye poses](https://docs.pupil-labs.com/neon/data-collection/data-streams/#_3d-eye-poses).
153+
154+
??? quote "EyestateEyelidDualMonoGazeData"
155+
156+
::: pupil_labs.realtime_api.streaming.gaze.EyestateEyelidDualMonoGazeData
157+
89158
You can learn more about the payload in the [Under the Hood](../../../guides/under-the-hood.md) guide.
90159

91160
## Full Code Examples

‎docs/methods/simple/streaming/gaze.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ This function can return different types of gaze data ([GazeDataType][pupil_labs
1010

1111
- [GazeData][pupil_labs.realtime_api.streaming.gaze.GazeData] object for Pupil Invisible or Neon without "Compute eye state" enabled.
1212
- [EyestateGazeData][pupil_labs.realtime_api.streaming.gaze.EyestateGazeData] or [EyestateEyelidGazeData][pupil_labs.realtime_api.streaming.gaze.EyestateEyelidGazeData] for Neon with "Compute eye state" enabled, depending on the version of the Neon Companion app.
13+
- Starting from Neon Companion app version +2.9.31, both binocular and monocular gaze data are exposed, it can return [BinoAndDualMonoGazeData][pupil_labs.realtime_api.streaming.gaze.BinoAndDualMonoGazeData] or [EyestateEyelidDualMonoGazeData][pupil_labs.realtime_api.streaming.gaze.EyestateEyelidDualMonoGazeData].
1314

1415
See below samples for each type of gaze data.
1516

@@ -29,6 +30,30 @@ See below samples for each type of gaze data.
2930

3031
::: pupil_labs.realtime_api.streaming.gaze.GazeData
3132

33+
=== "Gaze data (Dual Mono)"
34+
35+
<!-- badge:product Neon -->
36+
<!-- badge:companion +2.9.31 -->
37+
<!-- badge:version +1.8.0 -->
38+
39+
```py linenums="0"
40+
BinoAndDualMonoGazeData(
41+
x=848.9487915039062,
42+
y=474.46356201171875,
43+
worn=True,
44+
mono_left_x=895.11083984375,
45+
mono_left_y=477.2065734863281,
46+
mono_right_x=800.8253173828125,
47+
mono_right_y=469.348388671875,
48+
timestamp_unix_seconds=1772109976.1031907
49+
)
50+
```
51+
52+
This method exposes gaze data for Neon in Dual Mono mode, which includes binocular and monocular gaze data.
53+
??? quote "BinoAndDualMonoGazeData"
54+
55+
::: pupil_labs.realtime_api.streaming.gaze.BinoAndDualMonoGazeData
56+
3257
=== "With Eye State"
3358

3459
<!-- badge:product Neon -->
@@ -103,6 +128,50 @@ See below samples for each type of gaze data.
103128

104129
::: pupil_labs.realtime_api.streaming.gaze.EyestateEyelidGazeData
105130

131+
=== "Dual Mono and Eye State & Eye Lid"
132+
133+
<!-- badge:product Neon -->
134+
<!-- badge:companion +2.9.31 -->
135+
<!-- badge:version +1.8.0 -->
136+
137+
```py linenums="0"
138+
EyestateEyelidDualMonoGazeData(
139+
x=852.3287353515625,
140+
y=517.3056030273438,
141+
worn=False,
142+
pupil_diameter_left=4.567640781402588,
143+
eyeball_center_left_x=-32.25,
144+
eyeball_center_left_y=18.6875,
145+
eyeball_center_left_z=-44.9375,
146+
optical_axis_left_x=0.20407231152057648,
147+
optical_axis_left_y=-0.020999005064368248,
148+
optical_axis_left_z=0.9787306189537048,
149+
pupil_diameter_right=3.035294771194458,
150+
eyeball_center_right_x=30.15625,
151+
eyeball_center_right_y=14.375,
152+
eyeball_center_right_z=-43.3125,
153+
optical_axis_right_x=-0.08533422648906708,
154+
optical_axis_right_y=-0.019046271219849586,
155+
optical_axis_right_z=0.9961703419685364,
156+
eyelid_angle_top_left=-0.63037109375,
157+
eyelid_angle_bottom_left=-1.138671875,
158+
eyelid_aperture_left=6.09653902053833,
159+
eyelid_angle_top_right=-0.06329345703125,
160+
eyelid_angle_bottom_right=0.1409912109375,
161+
eyelid_aperture_right=2.228158473968506,
162+
gaze_mono_left_x=864.9857177734375,
163+
gaze_mono_left_y=526.5974731445312,
164+
gaze_mono_right_x=834.6492309570312,
165+
gaze_mono_right_y=481.4454040527344,
166+
timestamp_unix_seconds=1772108289.6118817
167+
)
168+
```
169+
This method exposes binocular and monocular gaze data, [pupil diameter](https://docs.pupil-labs.com/neon/data-collection/data-streams/#pupil-diameters), and [eye poses](https://docs.pupil-labs.com/neon/data-collection/data-streams/#_3d-eye-poses).
170+
171+
??? quote "EyestateEyelidDualMonoGazeData"
172+
173+
::: pupil_labs.realtime_api.streaming.gaze.EyestateEyelidDualMonoGazeData
174+
106175
You can learn more about the payload in [Under the Hood](../../../guides/under-the-hood.md).
107176

108177
## Full Code Examples

‎examples/async/stream_video_with_overlayed_gaze.py‎

Lines changed: 47 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,16 @@
1515
receive_gaze_data,
1616
receive_video_frames,
1717
)
18+
from pupil_labs.realtime_api.streaming.gaze import ( # noqa: E402
19+
BinoAndDualMonoGazeData,
20+
EyestateEyelidDualMonoGazeData,
21+
)
22+
23+
LEGEND = {
24+
"mono_right": (255, 0, 0), # right eye monocular gaze
25+
"mono_left": (0, 255, 0), # left eye monocular gaze
26+
"bino": (0, 0, 255), # binocular gaze
27+
}
1828

1929

2030
async def main():
@@ -80,10 +90,45 @@ async def match_and_draw(queue_video, queue_gaze):
8090
cv2.circle(
8191
bgr_buffer,
8292
(int(gaze_datum.x), int(gaze_datum.y)),
83-
radius=80,
93+
radius=40,
8494
color=(0, 0, 255),
85-
thickness=15,
95+
thickness=5,
8696
)
97+
if isinstance(
98+
gaze_datum, (BinoAndDualMonoGazeData, EyestateEyelidDualMonoGazeData)
99+
):
100+
cv2.circle(
101+
bgr_buffer,
102+
(int(gaze_datum.mono_right_x), int(gaze_datum.mono_right_y)),
103+
radius=40,
104+
color=(255, 0, 0),
105+
thickness=5,
106+
)
107+
cv2.circle(
108+
bgr_buffer,
109+
(int(gaze_datum.mono_left_x), int(gaze_datum.mono_left_y)),
110+
radius=40,
111+
color=(0, 255, 0),
112+
thickness=5,
113+
)
114+
for eye, color in LEGEND.items():
115+
cv2.rectangle(
116+
bgr_buffer,
117+
(10, 10 + 30 * list(LEGEND.keys()).index(eye)),
118+
(30, 30 + 30 * list(LEGEND.keys()).index(eye)),
119+
color,
120+
-1,
121+
)
122+
cv2.putText(
123+
bgr_buffer,
124+
eye,
125+
(40, 30 + 30 * list(LEGEND.keys()).index(eye)),
126+
cv2.FONT_HERSHEY_SIMPLEX,
127+
0.5,
128+
color,
129+
1,
130+
cv2.LINE_AA,
131+
)
87132

88133
cv2.imshow("Scene camera with gaze overlay", bgr_buffer)
89134
cv2.waitKey(1)

‎examples/simple/stream_video_with_overlayed_gaze.py‎

Lines changed: 47 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,16 @@
66
cv2.destroyAllWindows()
77

88
from pupil_labs.realtime_api.simple import discover_one_device # noqa: E402
9+
from pupil_labs.realtime_api.streaming.gaze import ( # noqa: E402
10+
BinoAndDualMonoGazeData,
11+
EyestateEyelidDualMonoGazeData,
12+
)
13+
14+
LEGEND = {
15+
"mono_right": (255, 0, 0), # right eye monocular gaze
16+
"mono_left": (0, 255, 0), # left eye monocular gaze
17+
"bino": (0, 0, 255), # binocular gaze
18+
}
919

1020

1121
def main():
@@ -24,10 +34,45 @@ def main():
2434
cv2.circle(
2535
frame.bgr_pixels,
2636
(int(gaze.x), int(gaze.y)),
27-
radius=80,
37+
radius=40,
2838
color=(0, 0, 255),
29-
thickness=15,
39+
thickness=5,
3040
)
41+
if isinstance(
42+
gaze, (BinoAndDualMonoGazeData, EyestateEyelidDualMonoGazeData)
43+
):
44+
cv2.circle(
45+
frame.bgr_pixels,
46+
(int(gaze.mono_right_x), int(gaze.mono_right_y)),
47+
radius=40,
48+
color=(255, 0, 0),
49+
thickness=5,
50+
)
51+
cv2.circle(
52+
frame.bgr_pixels,
53+
(int(gaze.mono_left_x), int(gaze.mono_left_y)),
54+
radius=40,
55+
color=(0, 255, 0),
56+
thickness=5,
57+
)
58+
for eye, color in LEGEND.items():
59+
cv2.rectangle(
60+
frame.bgr_pixels,
61+
(10, 10 + 30 * list(LEGEND.keys()).index(eye)),
62+
(30, 30 + 30 * list(LEGEND.keys()).index(eye)),
63+
color,
64+
-1,
65+
)
66+
cv2.putText(
67+
frame.bgr_pixels,
68+
eye,
69+
(40, 30 + 30 * list(LEGEND.keys()).index(eye)),
70+
cv2.FONT_HERSHEY_SIMPLEX,
71+
0.5,
72+
color,
73+
1,
74+
cv2.LINE_AA,
75+
)
3176

3277
cv2.imshow("Scene camera with gaze overlay", frame.bgr_pixels)
3378
if cv2.waitKey(1) & 0xFF == 27:

0 commit comments

Comments
 (0)