Skip to content

README.md

Latest commit

 

History

History
270 lines (201 loc) · 16.7 KB

README.md

File metadata and controls

270 lines (201 loc) · 16.7 KB
Raw

Video Workflow

This document describes the components of the video workflow for OCW.

SECTIONS

  1. Overview
  2. Google Drive Sync and AWS Transcoding
  3. YouTube Submission
  4. Captioning and 3Play Transcript Request
  5. Completing the Workflow
  6. Management Commands
  7. YouTube API Quota Management
  8. Testing PRs with Transcoding
  9. Adding Captions and Transcript to Existing Videos

Overview

This assumes that Google Drive sync, YouTube integration, AWS MediaConvert, and 3Play submission are all enabled, which is required for the video workflow.

The high-level description of the process is below, and each subsequent section contains additional details, including links to the relevant code.

  • Browse to a course site in the Studio UI, go to the Resources page and click the icon to the right of the Sync w/ Google Drive button to open the site's Google Drive folder in the Google Drive UI.
  • Upload a video with the name <video_name>.<video_extension> to the videos_final folder on Google Drive, where <video_extension> is a valid video extension, such as mp4. If there are pre-existing captions that should be uploaded with the video (as opposed to requesting captions/transcript from 3Play), then these should be named exactly <video_name>_captions.vtt and <video_name>_transcript.pdf, and uploaded into the files_final folder on Google Drive.
  • Sync using the Studio UI. This uploads the video to S3.
  • As soon as the upload to S3 is complete, Studio initiates a celery task to submit the video to the AWS Media Convert service.
  • Once trancoding is complete, the video is uploaded to YouTube (set as unlisted prior to the course being published).
  • After the video has been successfully uploaded to YouTube, and if there are no pre-existing captions, Studio sends a transcript request to 3Play.
  • Once 3Play completes the transcript job, the captions (.vtt format) and transcript (.pdf format) are fetched and associated with the video.
  • On any publish action, the video metadata and YouTube metadata are updated, assuming the information has been received from the external services.
  • The YouTube video is set to public once the course has been published to live/production.

Google Drive Sync and AWS Transcoding

Users upload videos in a valid video format to the videos_final folder. Whether a file is located in this folder is used for defining the is_video property. The file is processed using the process_drive_file function, which triggers the stream_to_s3 function to upload the file to S3. After all resources are created via create_gdrive_resource_content_batch, videos are then transcoded via transcode_gdrive_videos_batch, which calls transcode_gdrive_video to submit the AWS MediaConvert transcoding job.

The parameters of the AWS transcode request are defined through the AWS interface, and the role is defined here. Some example JSONs used for triggering MediaConvert job are in this folder.

The TranscodeJobView endpoint listens for the webhook that is sent when the transcoding job is complete.

YouTube Submission

Videos are uploaded to YouTube via the resumable_upload function. The YouTube upload success notification is sent by email when the update_youtube_statuses task is complete; exceptions in this task trigger the YouTube upload failure notification. When the course is published to draft/staging, the video is set to unlisted. However, when it is published to live/production, the video is made public on YouTube, via the update_youtube_metadata function. When a video is made public on YouTube, all YouTube subscribers will be notified. There are nearly 5 million subscribers to the OCW YouTube channel, so be careful with this setting. Subsequently republishing the course to draft/staging will not change the visibility of the YouTube video. However, if the video resource is set to "Draft" and the course is republished, the video will again be set to unlisted.

Captioning and 3Play Transcript Request

If there are no pre-existing captions, a 3Play transcript request is generated. This is done via the threeplay_transcript_api_request function.

Completing the Workflow

Once the workflow is completed, the updates to the Video and WebsiteContent objects are nearly complete. The only remaining steps are triggered on course publish: updating the video metadata via update_transcripts_for_website and updating the YouTube metadata via update_youtube_metadata.

Management Commands

In cases where something may have gone wrong with the data, often due to legacy data issues, there are management commands that can be run to resolve them. The commands are defined here. These commands are:

  • backpopulate_video_downloads In the existing video workflow, the MediaConvert job creates a downloadable verion as well as the YouTube version. Initially, these downloadable versions were not in the same S3 path as the course site's other resource content, and running this command moves them to the appropriate location.
  • clear_webvtt_files Some captions were initially saved without an extension; this management command deletes them from S3 and clears the resource metadata, allowing them to be re-created.
  • sync_missing_captions This management command syncs captions and transcripts from 3Play to course videos missing them.
  • sync_transcripts. This management command syncs captions and transcripts for any videos missing them from one course (from_course) to another (to_course).
  • update_youtube_tags. This management command merges YouTube video tags with database tags and updates both. See YouTube API Quota Management for details on handling large-scale updates.

update_youtube_tags Usage

This command fetches current tags from YouTube, merges them with tags stored in the database, and pushes the merged set back to YouTube. It supports filtering, batching, quota tracking, and automated scheduling via Celery.

Basic usage

# Preview changes without modifying YouTube (dry run)
python manage.py update_youtube_tags --dry-run

# Update all videos
python manage.py update_youtube_tags

# Update only videos from specific courses
python manage.py update_youtube_tags --filter course-1,course-2

# Exclude specific courses
python manage.py update_youtube_tags --exclude course-to-skip

# Update specific YouTube IDs
python manage.py update_youtube_tags --youtube-id "yt_id_1,yt_id_2"

# Add course URL slug as a tag
python manage.py update_youtube_tags --add-course-tag

# Export results to CSV
python manage.py update_youtube_tags --out results.csv

# Set a custom quota limit (stop before exceeding)
python manage.py update_youtube_tags --quota-limit 5000

Automated scheduling via Celery

When the number of videos exceeds --threshold (default: 150), the --schedule flag dispatches Celery tasks spread across multiple days automatically:

# Auto-schedule: splits work across days based on quota
python manage.py update_youtube_tags --schedule --add-course-tag

# Combine with filters
python manage.py update_youtube_tags --schedule --filter course-1 --add-course-tag

# Custom threshold and daily quota
python manage.py update_youtube_tags --schedule --threshold 100 --daily-quota 5000

Weekend-only scheduling: Use --weekends-only to restrict scheduled tasks to Saturdays and Sundays, avoiding quota consumption during weekday operations.

python manage.py update_youtube_tags --schedule --weekends-only --add-course-tag
# Tasks will be delayed to the next Saturday/Sunday slots

If the video count is below the threshold, --schedule still runs updates immediately.

Command-line options

Option Default Description
--dry-run False Preview changes without modifying YouTube or DB
--youtube-id None Comma-separated YouTube IDs to update
--add-course-tag False Add course URL slug as a tag
--out None Export results to CSV file
--quota-limit 9000 Stop processing before exceeding this many quota units
--schedule False Dispatch Celery tasks when video count exceeds threshold
--threshold 150 Video count above which --schedule dispatches to Celery
--daily-quota 9000 Quota units available per day for scheduling calculations
--weekends-only False Restrict scheduled tasks to weekends (Sat/Sun) only

YouTube API Quota Management

The YouTube Data API v3 enforces a daily quota of 10,000 units (resets at midnight Pacific Time). Different operations have different costs:

Operation Quota Cost Notes
videos.list 1 unit Accepts up to 50 comma-separated IDs per call
videos.update 50 units One video per call (no batch endpoint)
videos.insert (upload) 1600 units Full video upload
captions.list 50 units
captions.insert 400 units
captions.update 450 units

Quota budget for tag updates

With a 10,000 unit daily quota and reserving 1,000 for other operations:

  • ~179 videos can have tags updated per day (9000 / 50 = 180, minus list overhead)
  • Reading tags for 1,000 videos costs only 20 units (batched 50 IDs per call)

Pattern: Spreading API-intensive work across days

When a management command or task needs to process more items than a single day's quota allows, follow this pattern:

  1. Check the count against a configurable threshold
  2. Below threshold → process synchronously in the management command with quota tracking
  3. Above threshold → split into chunks and schedule Celery tasks with countdown delays:
    • Day 0: countdown=0 (immediate)
    • Day 1: countdown=86400 (24h)
    • Day N: countdown=N*86400
  4. Each task should handle quota errors gracefully by retrying with self.retry(countdown=3600)
  5. Weekend-only mode can delay tasks to Saturday/Sunday slots to avoid consuming quota during peak weekday usage

This pattern is implemented in update_youtube_tags / update_youtube_tags_batch and can be reused for any YouTube API-intensive operation.

Testing PRs with Transcoding

Before working on, testing, or reviewing any PR that requires a video to be uploaded to YouTube, make sure that AWS buckets (instead of local Minio storage) are being used for testing. To do that, set OCW_STUDIO_ENVIRONMENT to any value other than dev.

Set the following variables to the same values as for RC:

AWS_ACCOUNT_ID
AWS_ACCESS_KEY_ID
AWS_REGION
AWS_ROLE_NAME
AWS_SECRET_ACCESS_KEY
AWS_STORAGE_BUCKET_NAME
DRIVE_SERVICE_ACCOUNT_CREDS
DRIVE_SHARED_ID
VIDEO_S3_TRANSCODE_ENDPOINT
VIDEO_S3_TRANSCODE_PREFIX

Upload the video to the course's Google Drive folder, as described in the Google Drive Sync and AWS Transcoding section above. Wait for the video transcoding job to complete, which requires an amount of time proportional to the length of the video; for a very short video, this should only take a few minutes.

Next, the response to the transcode request needs to be simulated. This is because the AWS MediaConvert service will not send a webhook notification to the local OCW Studio instance, but rather to the RC URL.

To simulate the response, use cURL, Postman, or an equivalent tool to POST a message to https://localhost:8043/api/transcode-jobs/, with the body as in the example below, updated to match the relevant environment variables, course name, and video name.

{
  "version": "0",
  "id": "c120fe11-87db-c292-b3e5-1cc90740f6e1",
  "detail-type": "MediaConvert Job State Change",
  "source": "aws.mediaconvert",
  "account": "<settings.AWS_ACCOUNT_ID>",
  "detail": {
    "timestamp": 1629911639065,
    "accountId": "<settings.AWS_ACCOUNT_ID>",
    "queue": "arn:aws:mediaconvert:us-east-1:919801701561:queues/Default",
    "jobId": "<VideoJob.job_id>",
    "status": "COMPLETE",
    "userMetadata": {},
    "outputGroupDetails": [
      {
        "outputDetails": [
          {
            "outputFilePaths": [
              "s3://<settings.AWS_STORAGE_BUCKET_NAME>/aws_mediaconvert_transcodes/<Website.short_id>/<DriveFile.file_id>/<original_video_filename_base>_youtube.mp4"
            ],
            "durationInMs": 45466,
            "videoDetails": {
              "widthInPx": 320,
              "heightInPx": 176
            }
          },
          {
            "outputFilePaths": [
              "s3://<settings.AWS_STORAGE_BUCKET_NAME>/aws_mediaconvert_transcodes/<Website.short_id>/<DriveFile.file_id>/<original_video_filename_base>_360p_16_9.mp4"
            ],
            "durationInMs": 45466,
            "videoDetails": {
              "widthInPx": 640,
              "heightInPx": 360
            }
          },
          {
            "outputFilePaths": [
              "s3://<settings.AWS_STORAGE_BUCKET_NAME>/aws_mediaconvert_transcodes/<Website.short_id>/<DriveFile.file_id>/<original_video_filename_base>_360p_4_3.mp4"
            ],
            "durationInMs": 45466,
            "videoDetails": {
              "widthInPx": 480,
              "heightInPx": 360
            }
          }
        ],
        "type": "FILE_GROUP"
      }
    ]
  }
}

making sure to set the values in <>. In particular, set

<settings.AWS_ACCOUNT_ID>
<VideoJob.job_id>
<settings.AWS_STORAGE_BUCKET_NAME>/aws_mediaconvert_transcodes/<Website.short_id>/<DriveFile.file_id>/<original_video_filename_base>

The DriveFile will be the one associated with the video: http://localhost:8043/admin/gdrive_sync/drivefile/.

If this completes successfully, the VideoJob status in Django admin should be COMPLETE, and there should now be three new VideoFile objects populated with status, destination, and s3_key fields.

Adding Captions and Transcript to Existing Videos

Existing caption (.vtt) and transcript (.pdf) resources can be associated with a video resource directly in OCW Studio without requiring a new upload or 3Play transcript request.

The Edit Resource form includes two fields: Video Captions Resource and Video Transcript Resource. These fields can be used to select resources that contain the corresponding caption and transcript files. The files associated with the resources are not required to follow the _captions.vtt and _transcript.pdf naming convention used by Google Drive sync.

When the video resource is saved, the serializer updates the Video Captions (WebVTT) URL and Video Transcript (PDF) URL fields based on the selected resources. Either field may be populated independently, and existing associations can be cleared by removing the selected resource (by clicking on the X) and saving the video resource again.

The site configuration metadata fields for these associations are defined by the environment variables YT_FIELD_CAPTIONS_RESOURCE and YT_FIELD_TRANSCRIPT_RESOURCE.

On course publish, the associated captions and transcript files are included in the site output and propagated to YouTube metadata as part of the standard publish process.